| *** mhen_ is now known as mhen | 01:38 | |
| *** jlejeune_ is now known as jlejeune | 12:30 | |
| bhola | Hello Guys. Back again with the issue of external network gateway configuration not bringing up the relevant bridge (br-ex) up. So cannot connect to VMs from out side. I believe openstack should bring it up on compute and controller node with the gateway ip of external network. How to bring it up with correct IP? | 15:14 |
|---|---|---|
| DeHackEd | as I said, your router should be running, pingable, and the host will be one that's running a neutron l3 agent iirc.. | 15:52 |
| bhola | DeHackEd, I am sorry but what router you are referring to? My physical router? | 16:59 |
| DeHackEd | the internal openstack router. one of the things the neutron service does is make NAT routers between tenant networks and provider networks (most typically) | 16:59 |
| bhola | Well I think explained it in detail but may be I am not good at explaining my current setup. Is it okay if I upload images from Horizon so it will be easy for you grasp how I have setup my opensstack network? | 17:02 |
| bhola | Please let me know from which page to take a screenshot | 17:02 |
| bhola | Will network topology graph screenshot suffice for explaining my network configuration? | 17:03 |
| DeHackEd | maybe... | 17:15 |
| bhola | https://imgur.com/hfRG0Le.png | 17:15 |
| bhola | https://imgur.com/eiV5wmr.png | 17:16 |
| DeHackEd | okay.. so we'd say that 192.168.11.x/24 is your "public" IP addresses (relatively speaking) and the router there should be pingable... | 17:18 |
| bhola | The problem is novnc is not capturing the keyboard. It did capture my keyboard for the very first time when i opened the console after installation. When I closed that console session it never captured my keybaord after that. | 17:18 |
| bhola | ping from where? from outside? I am running openstack in Virtualbox where I created br-ex bridge with assigning physical interface (actually virtual because it is created by Virtualbox) as per install documentation and the interface relevant entries are made in /etc/network/interfaces on compute and neutron nodes. | 17:23 |
| bhola | My external network has a gateway set to 192.168.11.1. This, as far as I can understand, should be assign to br-ex and it be brought up by openstack system. This will be my entry point from outside (from compute, neutron node command line) to the openstack world. | 17:25 |
| bhola | Is my under standing correct? | 17:26 |
| bhola | #The provider network interface | 17:27 |
| bhola | auto enp0s9 | 17:27 |
| bhola | iface enp0s9 inet manual | 17:27 |
| bhola | up ip link set dev $IFACE up | 17:27 |
| bhola | down ip link set dev $IFACE down | 17:27 |
| bhola | # Include files from /etc/network/interfaces.d: | 17:27 |
| bhola | source /etc/network/interfaces.d/* | 17:28 |
| bhola | These entries are on my compute and neutron node. | 17:29 |
| bhola | The problem is br-ex is down. | 17:30 |
| bhola | it never comes up | 17:30 |
| bhola | https://imgur.com/gUwrWJ3.png Just look at this image where openstack router's external gateway is set to provider's network. | 17:33 |
| bhola | https://imgur.com/94rJWRv.png One of its interface is connected to Internal/Private network. | 17:34 |
| bhola | and you can see snat is enabled | 17:35 |
| jrosser | 192.168.11.124 is the external ip of the neutron router on your provider network | 17:37 |
| jrosser | 192.168.11.1 i think is your responsibility to deal with on whatever router/appliance/whatever deals with onward connectivity for that provider network | 17:38 |
| bhola | https://imgur.com/snPnRKj.png Here is provider network's Gateway ip set. But this ip should appear on br-ex which I must see when I issue command ip addr on neutron and/or compute. | 17:39 |
| jrosser | i don't think i agree with that | 17:39 |
| jrosser | in a real deployment the provider network gateway IP would be defined on some router or layer-3 switch | 17:41 |
| bhola | Ok. may be I understood it wrong. | 17:42 |
| jrosser | an SVI in cisco-speak | 17:42 |
| bhola | https://imgur.com/U8IrgMx.png https://imgur.com/dem8Jcf.png This is my network layout on virtualbox 192.168.200.0/24 is for management network. 192.168.11.0/24 is supposed to be my provider network. | 17:47 |
| bhola | https://imgur.com/Ozm7SHE.png This the output of all interfaces bridges on my neutron node. | 17:48 |
| jrosser | so does that mean 11.254 is actually your gateway? if virtualbox is further NAT'ing that network? | 17:49 |
| bhola | No Virtualbox is not further NATing. Let's suppose I don't want to go beyond this point. All I want to do is access my VMs from here. | 17:50 |
| bhola | So I should change IP address from .254 to .1? | 17:51 |
| jrosser | i can't say - this is your network & design | 17:51 |
| jrosser | what i can say though is that the deployment tooling that I work on manually creates the OVS provider bridges and ports | 17:52 |
| bhola | Sorry. What i meant was as in openstack external network's gateway ip 192.168.11.1 so I should change it in my virtualbox accordingly? | 17:53 |
| jrosser | the gateway is only relevant for traffic trying to get to some network outside 192.168.11.0/24 | 17:53 |
| bhola | Yes and I don't want to go outside. So how I ping the openstack router's external gateway which is assigned 192.168.11.124 from my neutron node? | 17:58 |
| bhola | or the floating ip I assigned to VM? | 17:58 |
| bhola | How it shows on your setup if you go to command line on neurtron/controller node? | 18:01 |
| bhola | does ip addr shows external bridge with an ip address? | 18:02 |
| bhola | or any interface/bridge with an ip provider's range? | 18:05 |
| jrosser | that’s not how it works, and would be very undesirable | 18:06 |
| jrosser | what does “ip netns” say | 18:07 |
| bhola | Ok can you ping any floating ip from your neutron/compute node? | 18:07 |
| bhola | on neutron node? | 18:08 |
| bhola | router-8f27ad4f-2b24-4601-8cf0-ff86f470a123 (id: 2) | 18:08 |
| bhola | qdhcp-cf6514c7-fc1f-4674-83c9-28cb511720d0 (id: 1) | 18:08 |
| bhola | qdhcp-5207b06c-eae2-4f57-beb3-b37ebd081245 (id: 0) | 18:08 |
| bhola | on neutron node. | 18:09 |
| bhola | on compute node it returns nothing. | 18:09 |
| jrosser | try “ip netns exec router-….. ip a” | 18:09 |
| jrosser | put the whole router namespace name in there | 18:09 |
| bhola | https://imgur.com/x7Ps1sZ.png Here is the output | 18:11 |
| jrosser | that is how your neutron router works | 18:13 |
| jrosser | each one gets a network namespace | 18:13 |
| jrosser | and you can use the ip netns commands to interact with it | 18:13 |
| jrosser | you see the external ip if your router there | 18:14 |
| jrosser | you can use ping inside the network namespace to test connectivity to things | 18:15 |
| bhola | yes I got connected to my VM. | 18:16 |
| bhola | But is this the way it supposed to work? | 18:16 |
| jrosser | yes | 18:16 |
| bhola | so what connection is between br-ex and qrouter? Do I need br-ex? | 18:18 |
| bhola | Do I even need 2nd interface openstack suggests in the documentation? | 18:20 |
| jrosser | I think you should look at the neutron reference architecture | 18:21 |
| bhola | So whenever you need to connect to your vm, do you connect this way? | 18:21 |
| jrosser | no, not at all | 18:21 |
| jrosser | my provider network has its gateway hosted on some physical router | 18:22 |
| bhola | so how do you connect other than horizon | 18:22 |
| jrosser | the device I am on is routable to that | 18:22 |
| jrosser | ssh via a floating ip or bastion vm | 18:22 |
| bhola | yes yes this is the part I am interested in. my virtualbox is also a router/switch. isn't it? | 18:23 |
| jrosser | it is also possible, depending on your config, to attach a vm directly to the provider network | 18:24 |
| jrosser | virtual box might be a router | 18:24 |
| jrosser | that’s why I asked about the .254 address | 18:24 |
| bhola | Well I assigned it arbitrarily. If this needs to be changed I can change it. | 18:25 |
| bhola | I just want to ssh directly from my neutron router to my VM. | 18:26 |
| bhola | without using ip netns. Is it possible? | 18:26 |
| jrosser | if you attach a floating ip then you will be able to ssh from the provider network to the vm | 18:27 |
| jrosser | but you need to have some ip route to that network from where you try to ssh from | 18:27 |
| jrosser | and that’s not really an Openstack problem | 18:27 |
| bhola | I have already assigned the floating ip and that is how I connected to my vm which is on a private network. But, as you suggested, I had to use ip netns command | 18:28 |
| jrosser | ultimately this depends what you are tying to achieve | 18:30 |
| jrosser | to simply have ssh work you can manually put an ip on the provider network on some host and just use that | 18:30 |
| jrosser | but that is completely unrepresentative of a production openstack deployment | 18:30 |
| bhola | Well, at the moment I am only trying to achive one thing. How to connect to VM directly, without using ip netns, from my neutron node command line to ssh to my vm already assigned a floating ip from provider's subnet. | 18:32 |
| jrosser | personally I would make a small utility vm in virtual box and hook it to the provider network | 18:34 |
| bhola | Hmmm. This makes sense. Is there a way that my ip stack with ip address 192.168.11.254 be used, somehow, a utility vm? | 18:37 |
| jrosser | you would be able to ssh from such a utility vm (i'm assuming you have several made in virtuabox on the same network) | 18:41 |
| jrosser | it will all be layer2/arp, no confusion with gateway addresses | 18:41 |
| jrosser | anyway - hopefully it's clear now how the neutron routers work, isolated in network namespaces | 18:42 |
| jrosser | this is very deliberate to isolate tenants from each other, and from the infrastructure | 18:42 |
| bhola | jrosser, Thanks for the tip. But there is still one confusion. | 18:50 |
| bhola | from my VM I can ping 192.168.11.163 which is a floating IP. | 18:51 |
| bhola | I am also ping 192.168.11.124 which is neutron's external gateway. | 18:51 |
| bhola | I cannot ping 192.168.11.1. Why? | 18:52 |
| jrosser | where did you assign the .1 address? | 18:52 |
| bhola | I don't know. This is auto assigned by openstack? | 18:54 |
| jrosser | i feel like i have said several times now that it is not the responsbility of openstack to deal with the gateway IP of a provider network | 18:54 |
| jrosser | it lives on whatever upstream router provides onward connectivity for the provider network | 18:55 |
| bhola | Right. I got it. It is just like we mention on our windows/linux network configuration what gateway IP is but actual Gateway IP is assigned on a separate host acting as a gateway. Right? | 18:57 |
| bhola | It also means that if I assign 192.168.11.1 to my virtualbox in place of 192.168.11.254 it will be pingable from openstack instance. Am I right? | 19:01 |
| jrosser | yes the gateway IP lives on whatever provides the L3 routing | 19:02 |
| jrosser | i don't really know about what virtualbox can do for you | 19:02 |
| jrosser | but as i said if you have another VM on the provider network the gateway is irellevant as everything is in the same subnet | 19:03 |
| bhola | Ok. Thanks a million. I will deploy another vm wih an ip from provider's subnet and see if that can ping floating IPs. | 19:06 |
| jrosser | and if it can't, you can use the ip netns thing to try to ping that vm from the network node | 19:06 |
| bhola | Out of interest, Have you deployed openstack on baremetal? | 19:07 |
| jrosser | then you will be able to tell if the provider network is hooked up to OVS corectly | 19:07 |
| jrosser | absolutely | 19:07 |
| jrosser | i spend a lot of my time contributing to openstack-ansible, so manual installations are terrifying for me :) | 19:07 |
| jrosser | i would very much recommend using one of the comminity supported tools for managing your deployment | 19:08 |
| bhola | Well, I have written scripts to install openstack services on multiple nodes. for example neutron node on separate node, placement on separate node so on. You can even mix services on a node, if you want to do so, with the same scripts. | 19:11 |
| bhola | I have 9 nodes running at the moment, 1 is storage and 1 is compute node. All other nodes are running only one openstack service and they are communicating fine with each other. | 19:12 |
| bhola | Only this provider network is a bone of contention. Look like my understanding about provider networks is not correct. | 19:16 |
| bhola | By the way this br-ex bridge should not be UP? It is in DOWN state on neutron node. | 19:17 |
| jrosser | for a production deployment you want to consider security, SSL, high availability, upgrade orchestration and a ton of other things | 19:17 |
| jrosser | all of this collective experience is in the community deployment tools | 19:18 |
| bhola | If you look at your neutron node. Does it show br-ex in DOWN state? | 19:19 |
| bhola | Devstack is also a deployment tool. Right? | 19:20 |
| jrosser | no, it’s a development tool | 19:21 |
| jrosser | and the architecture is not at all what you would behold for production | 19:21 |
| jrosser | *build for | 19:21 |
| bhola | I think the name of the node on your side controller | 19:21 |
| bhola | how it is development tool? It deploys openstack. Doesn't it? | 19:22 |
| jrosser | it’s purpose is for openstack developers, and for automated testing | 19:23 |
| bhola | Right. Got it. | 19:24 |
| bhola | So some of the ideas are getting clear about provider network. Is it correct to say that communication within provider network happens on L2 despite floating IPs and Neutron router's external gateway IP from provider's subnet? | 19:28 |
| jrosser | strictly the external IP on the neutron router is not a gateway | 19:29 |
| jrosser | a gateway is a specific term when talking about an L2 subnet - it is the address you can send packets to when they are supposed to go out of that subnet | 19:30 |
| DeHackEd | when you assign a floating IP to a VM, the IP is actually assigned to the router as a secondary, and NAT rules inserted to make it appear that the VM has the IP assigned to it. but the VM still sees its normal 172.16.x.x IP address on itself | 19:30 |
| bhola | jrosser, DeHackEd Thanks for your input. I understand that Neutron router's external Gateway IP is not the same as Provider's network gateway IP. What i wanted to say was, any host (in my case a virtualbox utility vm host) having an ip from provider's subnet will communicate with floating IPs on layer 2. Is this correct? | 19:50 |
| DeHackEd | yes. the router is still an ordinary host on the local layer 2 network, with ARP and all that. | 20:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
