Tuesday, 2018-10-30

*** hongbin has joined #openstack-lbaas01:23
*** bcafarel has quit IRC01:40
*** bcafarel has joined #openstack-lbaas02:00
*** ramishra has joined #openstack-lbaas02:50
*** ramishra has quit IRC03:29
*** ramishra has joined #openstack-lbaas03:45
*** hongbin has quit IRC04:23
*** sapd1 has quit IRC04:38
*** sapd1 has joined #openstack-lbaas04:59
*** pcaruana has joined #openstack-lbaas05:36
*** pcaruana has quit IRC05:47
*** ramishra has quit IRC06:16
*** mnaser has quit IRC06:16
*** mnaser has joined #openstack-lbaas06:17
*** ramishra has joined #openstack-lbaas06:18
*** beisner has quit IRC06:30
*** beisner has joined #openstack-lbaas06:31
*** ccamposr has joined #openstack-lbaas06:40
*** ramishra_ has joined #openstack-lbaas06:59
*** ramishra has quit IRC07:01
*** tomtom001 has quit IRC07:02
*** abaindur has quit IRC07:56
*** pcaruana has joined #openstack-lbaas07:57
*** pcaruana is now known as pcaruana|elisa|07:59
*** celebdor has joined #openstack-lbaas08:10
*** sapd1 has quit IRC08:10
*** sapd1 has joined #openstack-lbaas08:12
*** numans has joined #openstack-lbaas08:41
*** yboaron has joined #openstack-lbaas08:47
*** celebdor has quit IRC09:05
openstackgerritKim Bao Long proposed openstack/octavia master: Update README by adding Mailing List and Wiki URL  https://review.openstack.org/61415809:51
*** salmankhan has joined #openstack-lbaas10:01
*** celebdor has joined #openstack-lbaas10:01
openstackgerritCarlos Goncalves proposed openstack/octavia master: Redirect disk-image-builder logs, make verbose  https://review.openstack.org/61262210:39
*** ramishra_ has quit IRC11:15
*** ramishra has joined #openstack-lbaas11:18
*** salmankhan has quit IRC11:26
*** yamamoto has quit IRC11:51
*** yamamoto has joined #openstack-lbaas11:53
*** yamamoto has quit IRC11:58
*** salmankhan has joined #openstack-lbaas12:02
*** yamamoto has joined #openstack-lbaas12:33
*** celebdor has quit IRC13:04
*** celebdor has joined #openstack-lbaas13:06
*** velizarx has joined #openstack-lbaas13:07
*** yamamoto has quit IRC13:12
*** yboaron has quit IRC14:04
*** yamamoto has joined #openstack-lbaas14:21
*** yamamoto has quit IRC14:26
*** pcaruana|elisa| has quit IRC14:33
*** fnaval has joined #openstack-lbaas14:40
*** pcaruana|elisa| has joined #openstack-lbaas14:45
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add octavia-v2-dsvm-py3-scenario-fedora-latest job  https://review.openstack.org/60038114:45
*** dolly has joined #openstack-lbaas14:46
dollyrm_work, you there and have a few minutes?14:51
johnsomdolly He is on vacation for a while, so may not reply. Is there something I can help you with?14:54
dollyjohnsom, oh! I see, well I was curious about his work on https://review.openstack.org/#/c/558962/14:54
dollyI seem to hit a bug there when the health-manager recreates an amp14:55
dollyI talked about that with him, I was just curios if he was able to reproduce it14:56
johnsomdolly He is running that, but with a bunch of other patches too. His network is L3 and he doesn't use the standard active/standby, but moves ports. So, it's a bit of a different animal14:56
johnsomAh, ok. Yeah, I think he is the only one running that, so you might need an answer from him.14:57
dollyjohnsom, I see, no worries.14:57
dollyjohnsom, do you know that the "cached_zone" field is in the amphora table ?14:59
dollyCause, this is what I'm hitting http://paste.openstack.org/show/728729/14:59
*** yamamoto has joined #openstack-lbaas14:59
dollyI've actually described quite thoroughly, and I'm not sure why the "creation of the amp-step" chooses the wrong AZ, even though the "filterer" in the step before, got the correct one.15:01
dollyIt very weird15:01
*** ianychoi_ is now known as ianychoi15:07
*** velizarx has quit IRC15:08
*** yamamoto has quit IRC15:20
*** yamamoto has joined #openstack-lbaas15:23
*** yamamoto has quit IRC15:28
*** pcaruana|elisa| has quit IRC15:35
*** tomtom001 has joined #openstack-lbaas15:40
tomtom001Hello, I'm creating load balancers in Queens and noticed that the Octavia Management network has to be in a shared state to operate correctly.  Is there a reason why it can't be an non-shared network to operate correctly?15:41
johnsomtomtom001: It should not be a shared network15:43
johnsomEverything using that network should be under the octavia user/project15:43
tomtom001when I make the network non-shared octavia containers are unable to contact the lb instances on 9443 to set them up.  If I run telent <octavia mgmt ip> 9443 the containers are able to connect, but the octavia-worker cannot.15:44
tomtom001hmmm.... ok let me check that15:45
tomtom001ok, so if I log in as the octavia user, under project service, that's where the Octavia management network is specified.  It seems to be correct from what I'm seeing15:47
*** pcaruana|elisa| has joined #openstack-lbaas15:50
tomtom001johnsom: so octavia user's default project is service, is that ok?15:51
johnsomShould be ok.  Check your service_auth settings, they should match15:52
tomtom001johnsom: Octavia user is an admin under the service project members, is that what I should be looking for?15:54
*** yamamoto has joined #openstack-lbaas15:57
tomtom001under service_auth in octavia.conf service is the project_name as well.  That should be good15:57
*** aojea_ has joined #openstack-lbaas15:57
*** pcaruana|elisa| has quit IRC16:00
*** ccamposr has quit IRC16:04
dollycgoncalves, you there ?16:10
cgoncalvesdolly, I am16:10
dollyyo! long time no hear, I've got a question for you =)16:10
cgoncalvesdolly, hit me. I'll do my best to answer16:11
dolly- > /usr/share/rhosp-director-images/octavia-amphora-13.0-20180905.1.x86_64.raw <- Whats the status on this image in the current release of OSP 13. I use the github/octavia/queens/stable-release, and when creating an amp we hit this issue http://paste.openstack.org/show/733653/16:13
dollyBut, using the image you provided me a long time ago, works good!16:13
dollycgoncalves, did you run away ;)16:23
cgoncalvesdolly, I was afk for a moment, sorry16:25
dollyno worries =)16:25
cgoncalvesdolly, the OSP-provided amp image is based on latest RHEL cloud image at build time + octavia stable queens amphora bits16:26
cgoncalvesdolly, I am not sure what the issue is from that paste. would you have the amp logs?16:27
dollyok, do you remember the image you made me a while ago ?16:27
dollyI guess I can get the logs tomorrow, need to  make sure the amp is not deleted16:28
dollyso need to update the config16:28
cgoncalvesI did one? didn't I point you to the periodically built images instead?16:28
dollyand I'm on my way home16:28
dollyoh yes, sorry16:28
dollyyeah you did16:28
dollybut yes, that image works. the image provided from rhel/osp doesn't.16:28
dollyand I was just curios why16:29
cgoncalvesdolly, does it happen all the time when you build an amp?16:30
dollycgoncalves, yeps, everytime, same place. never happens with the other image16:30
dolly(but understand that we use the github/stable/queens-version of octavia, and not the one provided by rhel)16:31
cgoncalvesok, that is weird. I would really need to see the amp-agent log16:31
cgoncalvesdolly, control plane services run queens upstream while amp from OSP, is that it?16:32
dollyyes16:32
dollyexactly16:32
cgoncalvesok, got it16:33
cgoncalveswell, downstream we do not support that obviously ;)16:33
dollyofc :)16:33
cgoncalvesbut I could have a look once you share some logs16:33
dollyI'm just curious in why it crashes16:33
dollywith "internal server erro"16:33
dollyAnd I thought since the amp-image was updated in OSP just a month ago, I figured it was "new enough" to work with the upstream version of octavia.16:34
dollyBut guess I was wrong ;)16:34
cgoncalvesdolly, not sure in which 13.0.z version you're on16:35
cgoncalvesbut 13z2 doesn't ship with latest queens release16:35
dollyNo, thats why we build our own =)16:36
cgoncalveschecked, 13z2 ships octavia 2.0.116:36
dollyWhere do you find info about which z-release you are running ?16:36
cgoncalvesdolly, yeah so the amp is also based on 2.0.116:36
dollywe are running the "latest" available16:36
dollygot it16:36
dollyIs there no way of determining which openstack 13.z version you are running? (lets say you dont know which one you have)16:39
cgoncalvesgood question, heh. looking16:45
dollyno worries =) we have just updated everything to the latest available, i'm just a bit confused about these .z releases16:45
cgoncalvesdolly, https://bugzilla.redhat.com/show_bug.cgi?id=153356516:50
openstackbugzilla.redhat.com bug 1533565 in rhosp-release "Please include information about minor version (z stream) in rhosp-release" [Unspecified,Closed: notabug] - Assigned to mburns16:50
cgoncalvestl;dr: no way16:50
*** aojea_ has quit IRC16:53
colin-if i provided octavia with a network value for this parameter https://docs.openstack.org/octavia/ocata/main/configref.html#controller_worker.amp_boot_network_list whose definition contained an IPv6 subnet would it support that?16:56
colin-yikes i reallly linked the ocata doc, my bad16:57
cgoncalvesdolly, ok, so, I'm hearing that it will be possible soon ;)16:57
johnsomcolin- Yes, someone has done this in the past. There are two comments I will make: 1. we don't yet test this in the gates, so it *might* be broken (it's on my to-do list). 2. If you want to use link-local addresses there is an open patch with some changes that need to be made.16:58
cgoncalvesdolly, https://bugzilla.redhat.com/show_bug.cgi?id=164029117:00
openstackbugzilla.redhat.com bug 1640291 in rhosp-release "Add a maintenance release version to /etc/rhosp-release" [Low,On_qa] - Assigned to jjoyce17:00
colin-oh nice, i'll give it a whirl17:01
colin-thanks johnsom17:01
johnsomcolin- I have recently been working on getting our IPv6 house back in order.  Patch reviews would be welcome. I have gates for the VIP and members now, the next step is for the lb-mgmt-net just to make sure we don't get bit-rot there.17:03
*** velizarx has joined #openstack-lbaas17:14
*** velizarx has quit IRC17:15
*** aojea_ has joined #openstack-lbaas17:19
*** yamamoto has quit IRC17:24
colin-yeah, link-local would be useful. i'll take a look at the open changes for the patch. will also take a stab at some reviews17:26
*** aojea_ has quit IRC17:31
*** aojea_ has joined #openstack-lbaas17:32
*** aojea_ has quit IRC17:39
*** aojea_ has joined #openstack-lbaas17:40
*** aojea_ has quit IRC17:44
*** salmankhan has quit IRC17:55
*** yamamoto has joined #openstack-lbaas18:03
colin-unrelated, is there a section of the config that would disable the creation of LBs outside of RFC1918 space?18:05
johnsomcolin- Like block the creation of LBs with VIPs outside 1918?18:07
johnsomRight now, the only restrictions we have on the user specified networks/subnets is if they have access to them, and the valid VIP networks whitelist and the reserved member addresses. https://docs.openstack.org/octavia/latest/configuration/configref.html#networking.valid_vip_networks18:10
*** yamamoto has quit IRC18:12
colin-yes that's what i had in mind, that makes sense thanks18:24
colin-johnsom: does this describe the link-local changes you mentioned above? https://review.openstack.org/#/c/391204/18:47
johnsomYes, that is the one. I haven’t looked at it in a while18:48
*** abaindur has joined #openstack-lbaas19:01
*** abaindur has quit IRC19:01
*** abaindur has joined #openstack-lbaas19:01
*** aojea_ has joined #openstack-lbaas19:06
*** aojea_ has quit IRC19:26
*** aojea_ has joined #openstack-lbaas19:27
*** aojea_ has quit IRC19:30
*** aojea_ has joined #openstack-lbaas19:31
*** aojea_ has quit IRC19:52
*** pcaruana|elisa| has joined #openstack-lbaas20:12
*** pcaruana|elisa| has quit IRC20:32
openstackgerritMichael Johnson proposed openstack/octavia master: Add framework for octavia-status upgrade check  https://review.openstack.org/61288320:43
*** aojea_ has joined #openstack-lbaas20:57
openstackgerritSwaminathan Vasudevan proposed openstack/octavia master: Update osutil support for SUSE distro  https://review.openstack.org/54181121:20
*** salmankhan has joined #openstack-lbaas21:20
*** yamamoto has joined #openstack-lbaas22:10
abaindurjohnsom: Hey, I might've already asked this before, but a question about the LB mgmt network. It seems octavia can only talk to the amphora via its fixed IP on the lb mgmt network. Is there a way to use floating IPs instead?22:11
abaindurIt seems the requirement is that the lb mgmt network must only be a provider network, if it cant talk via floating ip/external network.22:12
johnsomabaindur No, we currently do not have support for adding floating IPs to the lb-mgmt-net addresses. You can specify any network, including public networks if you really want, but we don't have code to add floats to those addresses.22:13
johnsomNo, it doesn't have to be a provider network. It's simply a neutron tenant network22:13
abaindurHow does octavia talk to an isolated tenant network then? the octavia-worker process isnt plumbed into OVS or attached to a neutron port - its a process running directly on the host22:14
abaindurWe are out of VLANs to allocate a provider network that exists in a physical router. Therefore we can create an isolated VXLAN based tenant network. VMs have external access, but only via floating IP22:14
*** yamamoto has quit IRC22:15
johnsomabaindur There are many ways to do this. One is to add a port to OVS like we do in devstack. One is to route the traffic.22:15
johnsomYeah, or piggy back it on one of your existing networks. It doesn't do anything fancy/special, it is simply IP traffic with TCP and UDP in them.22:16
abaindurI'm not familiar with behavior of former (devstack). route the traffic... where? in our physical router? This would make it a "provider" network, then no? But the problem is, the tenant network is VXLAN based. We dont have hardware based VTEPS or vxlan gateways22:17
abaindurso lets say our lb_mgmt network is a VXLAN based tenant network, on 192.168.1.0/2422:17
abaindurOctavia-worker resides on a host22:19
abaindurif our tenant network was VLAN based, we could tag it in our switches, and add it to our router. SO our host's networking can communicate to 192.168.1.0/2422:19
abaindurBut this implies its a provider network, and that we have available vlans22:20
johnsomCan your host access your VXLAN underlay?22:23
abaindurSuppose our host has connectivity via 10.4.0.0/16, on vlan 4. If I make my LB mgmt network 10.8.0.0/16, on say vlan 8, I can add vlan 8 to our switches and add a route to 10.8/16 on physical router. I'm not sure what i'd need to do to achieve this, if 10.8.0.0/16 was VXLAN based22:23
abaindurno...22:23
abaindurWe dont do VXLAN on any of our physical switches22:24
*** aojea_ has quit IRC22:24
abaindurthe encap/decap is all done by OVS on the br-tun VTEPs22:24
johnsomI was just thinking if you could add a VXLAN endpoint on your controller host (linux software based) and add it to the neutron vxlan tenant network. But it sounds like you can't do that.....22:25
johnsomIt is super hard to come up with the right decision for your environment when I don't really know much about it. However, I can describe how the lb-mgmt-net works and maybe that will spark an idea?22:27
abainduryea.... hence why i wa hoping there was a way to do this using external networks/floating IPs22:27
abaindursure22:27
johnsomWell, you can  use an external network for your lb-mgmt-net. There is no restriction there. Just not floating IPs (NAT)22:27
abaindurI dont believe you can attach a VM to an external network22:28
abaindurA Provider network is really the same as an external network. Only difference is external network does NAT and you cannot attach a VM directly to it. A provider network can attach to a VM, but not do NAT22:28
johnsomSo, fundamentally the lb-mgmt-net is a neutron network, type doesn't matter to octavia. It just needs to be a network with a subnet that the octavia user can attach to the service VMs (amphora).22:29
johnsomThe tricky part is how to get your controller processes access to those addresses. (CW, HM, HK processes. API does not need access)22:29
abaindurFor reference, in neutron lbaas using AVI, uses a similar LB mgmt network that it attaches to each service engine (equivalent of amphora)22:30
abaindurthe difference is, the AVI controller is a VM, deployed in the same openstack cloud22:30
johnsomOne option is to pop it off on your network nodes by adding a port to OVS. This can also be done if you put neutron on your controller hosts.22:31
abaindurso the AVI controller is attached to the same fixed IP/neutron network22:31
abaindurhence, avi can talk directly over L2 to an isolated service engine22:31
johnsomOne option is to use a neutron router to make a routable path from the tenant network to another network in your environment that the controllers are on.22:31
abainduryep and in octavia case, its a process but not a VM. so it attaches to host networking, not OVS/neutron22:31
abaindurI guess if we packaged octavia-worker and health-manager inside a VM image, and booted it up as a "Octavia controller" VM in same environment, it would work22:32
abaindursimilar to AVI22:32
johnsomIt needs access to the rabbit and database for your cloud.  Probably exactly like avi.22:33
johnsomThe controllers also need access to the other service APIs, keystone, neutron, nova, etc.22:33
abaindurthat can be achieved by assigning a floating IP to the VM22:33
johnsomFrankly most of us run them in containers of some sort, so running in a VM should not be a problem.22:34
johnsomUgly solutions that would work if you are really stuck is to tunnel the networks. GRE/IPSec, etc. There isn't that much traffic between the amps and controllers. The highest use is the heartbeats back.22:35
abaindur"One option is to use a neutron router to make a routable path from the tenant network to another network in your environment that the controllers are on."22:39
abaindurok let me try to work that out...22:39
abaindurBack to my example, we attach our VXLAN based 10.8.0.0/16 network, to a neutron router... suppose here we have an external network on 10.4.0.0/16 vlan 4 (same VLAN and subnet as our host's networking)22:41
abaindurNow the amphora has octavia's IP as part of its controller_ip_port_list config... so it will try to contact 10.4. this theoretically might work?22:42
abaindurwell this would go out via SNAT22:43
*** salmankhan has quit IRC22:43
johnsomBoth sides connect via IP, so as long as there is a route in both directions it works fine. You would not want to NAT though.22:44
abaindurok,  guess it would not be ablke to be an external network22:45
abaindurwe'd have to add 10.4 vlan 4 as a provider net, and attach to same neutron router22:45
abaindur... and I suppose in our physical router, we would have to add a route saying 10.8.0.0/16 ---> nexthop = the 10.4 IP of our neutron router?22:47
abaindurI guess we are back to same problem then though. requires us to add this network in neutron, VLAN based. we have just made it a point to point link between neutron router and physical router, rather than deploying amphora directly on it22:56
johnsomI think I am following.  Here is what I am saying:22:59
johnsom1. create the lb-mgmt-net in neutron, it can whatever type you need. Create a subnet on that network, let's say 10.5.0.0/16.23:00
johnsom2. Add that network will be attached to a neutron router, either by default gateway or host route.23:01
johnsomAt this point you can boot an amphora, it will get a 10.5.0.10 address, and can reach the router address (gateway, etc.)23:01
johnsom3. Then on the controller hosts, let's say they are connected to 192.168.0.0/16, and you CW host has 192.168.0.2023:02
johnsom4. setup routing where that the 192.160.0.20 can reach a router that has a path to the neutron router. This could be a physical router, etc. It needs to know how to reach 10.5.0.0/16 (could be multiple hops) the neutron router needs to know how to reach 192.168.0.0/16 (again, can be multiple hops).23:04
johnsom5. boot network config in octavia.conf points to the 10.5.0.0/16 subnet/network, controller_port_ip_list has the 192.168.0.20, etc. addresses.23:05
johnsomThat configuration works.23:06
abainduryeah, I think the problem is just with #4, specifically, "It needs to know how to reach 10.5.0.0/16 (could be multiple hops) the neutron router needs to know how to reach 192.168.0.0/16 (again, can be multiple hops)."23:08
abaindurIt can't go out the external network of the router, due to NAT23:08
abaindurSo the neutron router can't just attach 2 subnets - our external network and 10.5.0.0/1623:09
*** salmankhan has joined #openstack-lbaas23:09
johnsomWell, it has to be so that it can provide floating IPs right?23:09
abaindurwill need to attach another network to this router, in order to route the traffic to 196.168 out of the router23:10
johnsomThat is how the NAT works for floating IPs23:10
abainduryeah, but we are not attaching floating IPs to the amphora23:10
abaindurSo it needs to leave the router without being NAT'd, still retaining the 10.5.0.10 source IP23:11
johnsomYeah, which neutron can do.23:12
abaindurBut because we dont have any vxlan gateways of our own, it would have to leave via a VLAN based network23:12
abaindurThat is mainly the problem ^^23:12
abaindurburning up a VLAN, when they might be limited or not readily available23:13
johnsomI'm not sure the L2 matters here23:13
abaindurHow else would packet leave the neutron router then?23:14
johnsomYour VXLAN tenant lb-mgmt-net terminates at the neutron router. What is L2 for the otherside could be anything23:14
abaindurext-network <---> (neutron router) <---> lb-mgmt net23:14
abaindurit seems like we'd need a 3rd subnet attachment there23:14
abaindurbecause we always NAT on the external network for the router23:15
johnsomNot necessarily.23:15
abaindurwouldn't everything get SNAT'd outbound?23:15
johnsomIt has to have an address on the ext-net that you could route to23:15
johnsomNot if the routing table tells it not to SNAT it, but to forward it23:16
*** fnaval has quit IRC23:16
abaindurhmm ok. I assumed the ip rule in qrouter would redirect packet to snat namespace(because the amph does not have a floating IP), and there all outbound packets would get SNAT'd23:17
abaindurip netns exec qrouter-72471ae2-5960-464b-8be2-dfcb035383c7 ip rule23:19
abaindur46861:from 172.90.0.16 lookup 1623:19
abaindur57482:from 172.90.0.7 lookup 1623:19
abaindur2891579393:from 172.90.0.1/16 lookup 289157939323:19
abaindur3232366593:from 192.170.0.1/16 lookup 323236659323:19
abaindurThat's usually how the neutron qrouter's are implemented. If theres a floating IP, theres an exact match to source fixed IP. Otherwise everything hits the subnet. Table 2891579393 re-directs packet to SNAT, and I thought just indiscriminately performs SNAT on the packet23:20
abaindurI will have to try and confirm behavior23:21
*** yamamoto has joined #openstack-lbaas23:23
abaindurjohnsom: actually nvm. I realized this table and the ip rules are only lookedup if no route exists in the router's main table. It usually is the case as the router does not have a default route23:25
abaindurMaybe adding a static route over the ext-network would allow us to bypass the SNAT rule23:26

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!