Thursday, 2017-07-20

« Wednesday, 2017-07-19 Index Friday, 2017-07-21 »
*** tongl has quit IRC00:12
*** aojea has joined #openstack-lbaas00:14
*** aojea has quit IRC00:19
*** korean101 has quit IRC00:19
*** aojea has joined #openstack-lbaas00:24
*** aojea has quit IRC00:28
*** armax has quit IRC00:35
*** aojea has joined #openstack-lbaas00:42
*** aojea has quit IRC00:47
*** aojea has joined #openstack-lbaas00:51
rm_worki'm seeing it with default config johnsom00:52
rm_workso probably for ME, it's just a deployment option I need to figure out00:52
*** aojea has quit IRC00:56
*** yamamoto has joined #openstack-lbaas00:56
*** aojea has joined #openstack-lbaas01:00
*** aojea has quit IRC01:04
*** yamamoto has quit IRC01:05
*** yamamoto has joined #openstack-lbaas01:06
*** aojea has joined #openstack-lbaas01:09
*** aojea has quit IRC01:14
*** yamamoto has quit IRC01:18
*** aojea has joined #openstack-lbaas01:18
*** yamamoto has joined #openstack-lbaas01:19
*** aojea has quit IRC01:23
*** yamamoto has quit IRC01:24
*** aojea has joined #openstack-lbaas01:27
*** aojea has quit IRC01:32
*** harlowja has quit IRC01:32
*** yamamoto has joined #openstack-lbaas01:32
*** yamamoto has quit IRC01:35
*** yamamoto has joined #openstack-lbaas01:41
*** yamamoto has quit IRC01:44
*** ssmith has quit IRC01:44
*** yamamoto has joined #openstack-lbaas01:45
*** yamamoto has quit IRC01:47
*** yamamoto has joined #openstack-lbaas01:49
*** yamamoto has quit IRC01:50
*** yamamoto has joined #openstack-lbaas01:52
*** yamamoto has quit IRC01:54
*** armax has joined #openstack-lbaas02:10
*** yamamoto has joined #openstack-lbaas02:13
*** yamamoto has quit IRC02:13
*** JudeC has quit IRC02:19
*** yamamoto has joined #openstack-lbaas02:22
*** yamamoto has quit IRC02:29
*** yamamoto has joined #openstack-lbaas02:34
*** yamamoto has quit IRC02:36
*** yamamoto has joined #openstack-lbaas02:40
*** yamamoto has quit IRC02:41
*** harlowja has joined #openstack-lbaas02:52
*** sanfern has joined #openstack-lbaas03:33
*** dougwig has quit IRC03:43
*** harlowja has quit IRC04:18
*** aojea has joined #openstack-lbaas04:20
*** chlong_ has joined #openstack-lbaas04:20
*** aojea has quit IRC04:25
*** aojea has joined #openstack-lbaas04:29
*** harlowja has joined #openstack-lbaas04:30
*** rcernin has joined #openstack-lbaas04:34
*** aojea has quit IRC04:34
*** aojea has joined #openstack-lbaas04:38
*** aojea has quit IRC04:42
*** aojea has joined #openstack-lbaas04:47
*** aojea has quit IRC04:52
*** aojea has joined #openstack-lbaas04:56
*** yamamoto has joined #openstack-lbaas04:59
*** aojea has quit IRC05:01
*** aojea has joined #openstack-lbaas05:05
*** aojea has quit IRC05:10
*** chlong_ has quit IRC05:26
*** gcheresh_ has joined #openstack-lbaas05:29
*** yamamoto has quit IRC05:31
*** harlowja has quit IRC05:33
*** armax has quit IRC05:34
*** rcernin has quit IRC05:36
*** Alex_Staf has joined #openstack-lbaas05:55
*** rcernin has joined #openstack-lbaas05:59
*** yamamoto has joined #openstack-lbaas06:05
*** aojea has joined #openstack-lbaas06:09
*** aojea has quit IRC06:13
*** yamamoto has quit IRC06:23
*** kobis has joined #openstack-lbaas06:27
*** aojea has joined #openstack-lbaas06:36
*** aojea has quit IRC06:41
*** aojea has joined #openstack-lbaas06:45
*** aojea has quit IRC06:51
*** kobis has quit IRC07:10
*** aojea has joined #openstack-lbaas07:13
*** yamamoto has joined #openstack-lbaas07:16
*** aojea has quit IRC07:17
*** tesseract has joined #openstack-lbaas07:19
*** aojea has joined #openstack-lbaas07:21
*** Alex_Staf has quit IRC07:22
*** kobis has joined #openstack-lbaas07:22
*** yamamoto has quit IRC07:25
*** Alex_Staf has joined #openstack-lbaas08:51
*** sanfern has quit IRC09:53
*** bzhao has joined #openstack-lbaas09:55
*** diltram has quit IRC10:23
*** diltram has joined #openstack-lbaas10:27
*** sanfern has joined #openstack-lbaas11:19
*** atoth has joined #openstack-lbaas11:37
*** dougwig has joined #openstack-lbaas11:53
nmagnezirm_work, o/12:03
*** chlong_ has joined #openstack-lbaas12:16
openstackgerritNir Magnezi proposed openstack/octavia master: Remove key_path from devstack plugin  https://review.openstack.org/48559312:31
*** catintheroof has joined #openstack-lbaas12:36
*** krypto has joined #openstack-lbaas13:05
*** sanfern has quit IRC13:31
*** sanfern has joined #openstack-lbaas13:31
*** yamamoto has joined #openstack-lbaas13:39
*** chlong_ has quit IRC13:40
*** ssmith has joined #openstack-lbaas13:44
*** cpusmith has joined #openstack-lbaas13:45
*** ssmith has quit IRC13:49
*** yamamoto has quit IRC13:50
*** cpusmith_ has joined #openstack-lbaas14:00
*** cpusmith has quit IRC14:04
*** cpusmith_ has quit IRC14:15
*** ssmith has joined #openstack-lbaas14:15
*** cpusmith has joined #openstack-lbaas14:21
*** pksingh has joined #openstack-lbaas14:23
*** ssmith has quit IRC14:25
*** armax has joined #openstack-lbaas14:33
*** Alex_Staf has quit IRC14:35
*** pksingh has quit IRC14:35
*** yamamoto has joined #openstack-lbaas14:37
*** pksingh has joined #openstack-lbaas14:45
*** bbzhao has joined #openstack-lbaas14:48
*** bzhao has quit IRC14:51
*** fnaval has joined #openstack-lbaas14:54
*** yamamoto has quit IRC14:55
*** yamamoto has joined #openstack-lbaas14:57
*** chlong_ has joined #openstack-lbaas14:57
*** yamamoto has quit IRC15:03
*** leitan has joined #openstack-lbaas15:03
*** gcheresh_ has quit IRC15:05
*** rcernin has quit IRC15:25
*** armax has quit IRC15:27
*** armax has joined #openstack-lbaas15:29
*** gcheresh_ has joined #openstack-lbaas15:44
*** kobis has quit IRC15:49
*** kobis has joined #openstack-lbaas15:50
*** kobis has quit IRC15:51
*** pksingh has quit IRC15:53
*** diltram has quit IRC16:02
*** aojea has quit IRC16:04
*** diltram has joined #openstack-lbaas16:08
*** aojea has joined #openstack-lbaas16:10
*** aojea has quit IRC16:15
*** gcheresh_ has quit IRC16:18
*** chlong_ has quit IRC16:19
*** gcheresh_ has joined #openstack-lbaas16:22
*** fnaval has quit IRC16:23
*** gcheresh_ has quit IRC16:27
*** aojea has joined #openstack-lbaas16:29
*** fnaval has joined #openstack-lbaas16:30
*** tesseract has quit IRC16:32
*** rcernin has joined #openstack-lbaas16:33
*** aojea has quit IRC16:34
*** pksingh has joined #openstack-lbaas16:35
*** sshank has joined #openstack-lbaas16:48
*** aojea has joined #openstack-lbaas16:57
*** aojea has quit IRC17:01
*** dayou has quit IRC17:07
*** tongl has joined #openstack-lbaas17:12
*** tesseract has joined #openstack-lbaas17:18
*** dayou has joined #openstack-lbaas17:20
*** harlowja has joined #openstack-lbaas17:23
*** harlowja has quit IRC17:27
*** krypto has quit IRC17:28
johnsomsanfern Hi17:32
*** chlong_ has joined #openstack-lbaas17:32
sanfernhi17:35
xgerman_johnsom are we still using the neutron_lbaasv2 owned port to hold the IP and then make our own ports?17:37
johnsomYeah, I think the port "owner" field is listed as something like lbaasv217:38
xgerman_yep, that port is down for us and we don;t see traffic17:38
johnsomWell, for VIP there are two ports.  One is the "fake" allowed address pairs port, the other is the base port.  One is "up" on is "down" normally17:39
johnsomThe "vip address" port will be be "DOWN" with device owner "Octavia", the "base address" will be UP with compute:nova as owner per my devstack17:44
rm_workrofl http://i.imgur.com/4vLJIjO.png17:46
rm_workwith gunicorn it's one full round of requests every 5 minutes17:46
rm_worksomething is funky17:47
rm_workevery 5 minutes SOMETHING happens17:47
johnsomSo it's not the wsgi platform....17:49
rm_workwell17:51
rm_workit's very different with gunicorn17:51
rm_workfrom uwsgi17:51
rm_workso it kinda *is* the wsgi platform?17:51
rm_workah it's not just one set of requests, seems to be everything for ~15 seconds every 5 min17:53
rm_work23 requests this run... which means it doesn't line up exactly with the number of threads17:53
tomtomtomhello, I have some behavior with my network in octavia I don't understand....  I have lb_network_name set to networkA and amp_boot_network_list set to networkA in octavia.conf, then I *openstack loadbalancer create --name https-6 --vip-subnet-id Internet_Subnet* where Internet_Subnet is networkB.18:05
*** harlowja has joined #openstack-lbaas18:06
tomtomtomThe vm's in the load balancer are in networkA however the amphora netns doesn't get a networkA address and never gets "health" because of that18:06
tomtomtomThe amphora instance gets an networkA address, just not in the namespace, just as it's main eth device,  what am I doing wrong here?18:07
tomtomtomThe networkB address is the only thing in the namespace, but the vm's are not in that namespace.  Is this by design?18:08
*** pksingh has quit IRC18:12
tomtomtomanother point to note is that my netns on amphora instance seems to only get one ip address that it "thinks" is the external address and then a bound address (like eth0:1) that is on the same network, so I assume that the network is supposed to be external, however, my vm's in the load balancer I think should be on another network (private hopefully)18:14
tomtomtomso I forsee that amphora netns would need at least two networks to satisfy the health checks and the external network service.  However, I am seeing that the netns is only ever getting one network interface on one network.  any ideas how i might change the config to satisfy?18:15
xgerman_johnsom wonder where we stuffed the member up/down value on the neutron CLI18:17
*** kobis has joined #openstack-lbaas18:18
tomtomtomalso I'm only using v2 for octavia.18:19
*** dougwig has quit IRC18:24
*** JudeC has joined #openstack-lbaas18:32
johnsomxgerman_ admin_state_up or operational status?18:35
xgerman_oh, figured it out lbaas-loadbalancer-status18:35
johnsomxgerman_ Under the old neutron client I think you could only get it via the status tree18:35
johnsomYep, I hated that so we fixed it with the OSC plugin18:36
xgerman_nope, it returns a status tree18:36
*** kobis has quit IRC18:36
xgerman_oh, ok, that;s better then18:36
johnsomYeah, with OSC that status is right on the member18:36
johnsomtomtomtom Ok, so first question (haven't finished parsing question 2 yet), network A will never be in the network namespace, this is for security as the network namespace isolates the tenant traffic from lb-mgmt-net, etc.  Health messages are sent from the amphora agent, which runs outside the network namespace(netns).18:39
*** kobis has joined #openstack-lbaas18:40
johnsomAmphora agent collects health from haproxy (inside netns) via a Unix socket that HAProxy exposes and provides stats and status over.18:40
tomtomtomok so then I misunderstood, I thought that health checks were running through the netns?18:41
johnsomThis way the HAProxy and keepalived processes, which handle customer traffic, have no access or visibility to the management network.18:41
johnsomtomtomtom Maybe I should clarify which health checks.....18:41
johnsomWe have a few due to the reliability features of Octavia.....18:41
johnsomSo, health monitoring of backend member servers happens from inside the network namespace and are done via HAProxy.18:42
johnsomWe also have a health heartbeat that the amphora-agent sends up to the health manager processes over the lb-mgmt-net18:43
tomtomtomok and so there's the issue.... my backend members are on the networkA network18:43
johnsomHealth heartbeat is used to detect if HAProxy is failing in some way or that amphora itself is failed18:43
johnsomtomtomtom ok, so that is fine.18:43
johnsomSo your VIP network, specified at LB create time is network A, then your member network, specified at member create time is also network A.  This will work just fine.  The network namespace with only have one interface in it, network A.18:45
johnsomHAProxy health monitoring will have a source  IP of your VIP in that case18:46
*** rtjure has quit IRC18:47
*** rtjure has joined #openstack-lbaas18:47
johnsomYou have what is typically called a one-armed load balancer18:47
tomtomtomno networkB would be the external network.18:48
tomtomtomnetworkA would be the internal18:48
johnsomOh, crumb, did I get those mixed up while typing???18:48
johnsomSo your VIP network, specified at LB create time is network B, then your member network, specified at member create time is also network B.  This will work just fine.  The network namespace with only have one interface in it, network B.18:49
johnsomThe two IPs you see for the network B interface inside the netns is for the allowed-address-pairs.  It's part of how we do HA.18:50
*** kobis has quit IRC18:51
johnsomOne IP is the VIP IP, this will move between amphora in failover scenarios (to maintain the IP end users hit).18:51
johnsomThe other is the base network IP, this does not move to other amphora.  It is just DHCP or assigned by neutron.18:52
tomtomtomhmmm.... I would think that having my members and my lb in the external network would be a security no-no. there's no way to keep my members and load balancers in separate networks?18:52
johnsomYou have lost me18:52
tomtomtomyou know vm's in a private network and LB in an external network?18:53
johnsomOk, let me give it a go again.18:53
johnsomIn a simple two armed octavia LB there are three networks involved.  Network A (lb-mgmt-net), Network B (VIP network), Network C (member network).18:54
johnsomNetwork A is a private network between amphora agents and the controllers.  No outside routing (though you can route inside if needed).  Controllers send command and control over it, amphora-agents send health heartbeats back.18:55
johnsomNetwork B, This is the network that has the IP address users connect to the load balancer on.  That IP is the VIP address.  It only has interfaces on the amphora inside the netns to isolate tenant traffic.  Network B could be a private tenant network or an external network.  You specify that at LB creation time.18:57
tomtomtomthen how do these networks get defined in octavia.conf? I have the lb_network_name set to networkA which is amphora communication then attached networks are networkB and networkC(the member network) ?  Would that be correct?18:58
johnsomNetwork C, This is the network your member servers live on, typically a web server.  This network only has an interface inside the netns.  Network C is typically a tenant private network, but could be external is odd cases.18:58
johnsomamp_boot_network_list in octavia.conf specifies the lb-mgmt-net, network A in my description.18:59
johnsomNone of the others are specified in the octavia.conf.  You specify them when you create the LB or member via CLI or API19:00
tomtomtomok I see19:00
johnsomIn the one armed case, when you specify the same network at LB create AND member create, since they are the same network it only has one interface in the netns19:01
tomtomtomok so then networkB is the member network and the LB vip network?19:03
johnsomRight, one armed, since the user specified the same for both, only Network B is plugged into the amp and netns.19:04
rm_worktomtomtom: lb_network_name doesn't exist BTW19:05
*** sshank has quit IRC19:06
johnsomCustomers would connect to Network B IP address (VIP) and members would see connections from the LB from the Network B address.19:06
johnsomrm_work +119:06
rm_workhttps://review.openstack.org/#/c/465183/19:06
rm_worktomtomtom: just ... remove that from your config entirely19:06
rm_workshould have had a reno note for that :(19:07
tomtomtomso under [networking] lb_network_name is not supposed to be there?19:07
rm_workright19:07
johnsomrm_work Still can!19:07
rm_workit's completely not relevant19:07
rm_workit has NEVER been read19:07
rm_workit was simply misleading T_T19:07
johnsomYeah, it evidently didn't get removed when someone changed it to boot_network19:07
tomtomtomso then if I specify lb create "external" network and then member create "internal" network then somehow they will be able to talk to each other?19:10
johnsomNo, they can't talk to each other, but you will have a network interface for each network inside the netns19:11
*** gcheresh_ has joined #openstack-lbaas19:12
tomtomtomok so that's the part that doesn't happen, only the external network work (vip) ever gets created inside the netns.19:12
johnsomIn that case, customer connections to the VIP will come over network B to the VIP IP, then haproxy will connect to members over network C with the IP address assigned to the network C interface.19:12
johnsomNo error?19:13
johnsomI.e. the member doesn't have provisioning status of ERROR and the API/CLI didn't return an error?19:14
openstackgerritMerged openstack/neutron-lbaas master: Updated from global requirements  https://review.openstack.org/48337119:19
tomtomtomnetns only has eth1 (vip, networkB) on the amphora instance and then the amphora instance has one physical interface ens3(network A), if haproxy uses netns to health check the vm then it will fail because that one interface is not in the netns19:20
tomtomtomnetwork C interface lives where? netns?19:25
tomtomtomifconfig on amphora instance returns ens3 which is network C, and ip netns exec amphora-haproxy returns eth1 which is network B, haproxy complains it can't find the server on network C.... I thought the issue was that network C needed to be in the netns, but I cannot get network C and network A to exist in the netns simultaneously.19:28
tomtomtomright member has operating status of error, but provisioning status is ok19:33
tomtomtomor ACTIVE19:33
tomtomtomoperating status is OFFLINE19:33
*** aojea has joined #openstack-lbaas19:34
*** aojea has quit IRC19:38
*** cpusmith has quit IRC19:39
*** cpusmith_ has joined #openstack-lbaas19:39
*** cpusmith has joined #openstack-lbaas19:40
*** cpusmith_ has quit IRC19:44
johnsomtomtomtom the network IDs aren't lining up for me, how about some output:19:48
johnsom1. nova show the amphora instance19:48
johnsom2. ip a from inside the amp19:48
johnsom3. ip from inside the netns19:48
tomtomtom| Cloud-Public network                 | (internal network)                                            | | Internet-Public network              | networkB (external network)19:56
tomtomtom2. network A (10.20.x.x) - vm instance also on network A (10.20.x.x)19:56
*** openstack has joined #openstack-lbaas20:01
*** aojea has joined #openstack-lbaas20:01
*** aojea has quit IRC20:05
*** sshank has joined #openstack-lbaas20:10
*** aojea has joined #openstack-lbaas20:11
*** armax has quit IRC20:15
*** aojea has quit IRC20:15
*** armax has joined #openstack-lbaas20:15
*** openstackgerrit has quit IRC20:17
*** aojea has joined #openstack-lbaas20:20
*** csomerville has joined #openstack-lbaas20:22
*** aojea has quit IRC20:24
*** cody-somerville has quit IRC20:26
*** aojea has joined #openstack-lbaas20:29
*** aojea has quit IRC20:33
*** gcheresh has quit IRC20:46
*** aojea has joined #openstack-lbaas20:47
*** cpusmith has quit IRC20:47
*** aojea has quit IRC20:52
*** aojea has joined #openstack-lbaas20:56
*** aojea has quit IRC21:01
*** catintheroof has quit IRC21:10
*** leitan has quit IRC21:12
*** diltram has quit IRC21:14
*** chlong_ has quit IRC21:16
*** diltram has joined #openstack-lbaas21:18
*** aojea has joined #openstack-lbaas21:34
*** aojea has quit IRC21:39
*** ssmith has joined #openstack-lbaas21:41
*** aojea has joined #openstack-lbaas21:43
*** aojea has quit IRC21:48
*** aojea has joined #openstack-lbaas21:53
*** aojea has quit IRC21:57
*** yamamoto has joined #openstack-lbaas21:57
*** yamamoto has quit IRC22:00
*** sshank has quit IRC22:08
*** aojea has joined #openstack-lbaas22:13
*** aojea has quit IRC22:18
*** aojea has joined #openstack-lbaas22:22
*** sshank has joined #openstack-lbaas22:23
*** aojea has quit IRC22:27
*** fnaval has quit IRC22:27
*** aojea has joined #openstack-lbaas22:31
*** aojea has quit IRC22:36
*** tesseract has quit IRC22:37
*** aojea has joined #openstack-lbaas22:40
*** aojea has quit IRC22:45
*** ipsecguy has quit IRC22:46
*** aojea has joined #openstack-lbaas22:49
*** ipsecguy has joined #openstack-lbaas22:52
*** aojea has quit IRC22:54
*** aojea has joined #openstack-lbaas22:59
*** yamamoto has joined #openstack-lbaas23:01
*** aojea has quit IRC23:03
*** openstackgerrit has joined #openstack-lbaas23:04
openstackgerritMerged openstack/neutron-lbaas master: fixed health monitor setting during tempest test,  https://review.openstack.org/48023323:04
*** yamamoto has quit IRC23:07
openstackgerritMerged openstack/octavia master: Remove key_path from devstack plugin  https://review.openstack.org/48559323:15
*** aojea has joined #openstack-lbaas23:17
*** aojea has quit IRC23:21
*** csomerville has quit IRC23:23
*** aojea has joined #openstack-lbaas23:26
*** aojea has quit IRC23:30
*** armax has quit IRC23:33
*** aojea has joined #openstack-lbaas23:35
*** aojea has quit IRC23:39
*** sshank has quit IRC23:41
*** oomichi has quit IRC23:59
« Wednesday, 2017-07-19 Index Friday, 2017-07-21 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!