Wednesday, 2017-08-02

« Tuesday, 2017-08-01 Index Thursday, 2017-08-03 »
*** catintheroof has joined #openstack-lbaas00:03
mnaserrm_work but im just thinking that if someone hit /v1 anyways when we expose it, it'll be an issue00:22
mnaserso certainly some work would have to be done to make sure no one can access v1, except lbaas00:23
*** JudeC has quit IRC00:26
*** tongl has quit IRC00:27
rm_workmnaser: right, you can expose the API via a LB or something00:31
rm_workyou can also run them separately00:32
rm_workso, one box runs the API with v1 enabled, and is ACL'd to specifically the neutron-lbaas boxes00:32
rm_workand a different box/VM runs the api with only v2 enabled00:32
mnaserrm_work yeah that's a way around it00:32
mnaserwe run stuff in nspawn containers so tha tshould be easy to do00:33
rm_workyep00:33
mnaserwe run master octavia alongside newton and its working nicely now, just have some patches to magnum and it'll be working really nicely00:33
johnsomGood to hear00:34
rm_workwe're running master octavia alongside liberty :P00:35
rm_workwe should actually maybe put a recommendation somewhere for that00:36
rm_worklike ... we recommend you ignore the version of the cloud you're running, and always run the newest release of octavia00:36
johnsomA full install guide is on my "wish I had time to do that now" list00:36
rm_workjohnsom: my patch IS going to need a recheck anyway, I kinda want to put in the comment you asked for00:38
johnsomOk00:38
rm_workjohnsom: do you know if there's a bug to link from haproxy/ubuntu?00:38
rm_workI think you said you'd seen one00:38
johnsomIt was in the release notes00:38
rm_workugh k00:40
rm_workwell whatever i'll just comment00:40
johnsomI'm looking to see if I can find the bug00:40
johnsomNah, I'm not going to dig00:42
rm_workk00:43
rm_workjust needs the +2 again then00:43
openstackgerritAdam Harwell proposed openstack/octavia master: Properly handle more states from HAProxy health messages  https://review.openstack.org/48767100:43
johnsomDone00:44
*** catintheroof has quit IRC00:44
rm_workrechecking some others >_>00:47
rm_workugh yeah that's gonna take a while00:58
rm_worktest nodes dropped significantly and queue spiked to 8k00:58
rm_workfuntimes00:58
*** https_GK1wmSU has joined #openstack-lbaas01:44
*** https_GK1wmSU has left #openstack-lbaas01:45
rm_workdown to about 4k...01:58
*** slaweq has joined #openstack-lbaas02:07
*** slaweq has quit IRC02:11
*** KeithMnemonic has quit IRC02:14
*** yamamoto has quit IRC02:24
*** yamamoto has joined #openstack-lbaas02:30
*** yamamoto has quit IRC02:35
*** fnaval has quit IRC02:35
*** yamamoto has joined #openstack-lbaas02:36
*** sanfern has quit IRC02:37
*** yamamoto has quit IRC02:41
*** fnaval has joined #openstack-lbaas02:44
*** fnaval has quit IRC02:44
*** yamamoto has joined #openstack-lbaas02:45
*** fnaval has joined #openstack-lbaas02:45
openstackgerrityanpuqing proposed openstack/neutron-lbaas master: Add "delay" value determination for health monitor  https://review.openstack.org/48958403:08
*** gtrxcb has joined #openstack-lbaas03:21
*** yamamoto has quit IRC03:33
*** links has joined #openstack-lbaas03:48
*** yamamoto has joined #openstack-lbaas03:49
*** sanfern has joined #openstack-lbaas04:01
*** yamamoto has quit IRC04:05
*** yamamoto has joined #openstack-lbaas04:10
*** gcheresh_ has joined #openstack-lbaas04:12
*** gongysh has joined #openstack-lbaas04:13
rm_workwhy are the gates soooo broken T_T04:49
*** yamamoto has quit IRC04:50
*** yamamoto has joined #openstack-lbaas05:07
*** ssmith has joined #openstack-lbaas05:16
*** ajo has quit IRC06:11
openstackgerrityanpuqing proposed openstack/neutron-lbaas master: Add "delay" value determination for health monitor  https://review.openstack.org/48958406:15
*** slaweq has joined #openstack-lbaas06:15
openstackgerrityanpuqing proposed openstack/neutron-lbaas master: Add "delay" value determination for health monitor  https://review.openstack.org/48958406:33
*** pcaruana has joined #openstack-lbaas06:34
*** rcernin has joined #openstack-lbaas06:46
*** aojea has joined #openstack-lbaas07:07
*** ducnc has joined #openstack-lbaas07:13
openstackgerritMerged openstack/octavia master: Properly handle more states from HAProxy health messages  https://review.openstack.org/48767107:30
*** kobis has joined #openstack-lbaas07:35
*** Guest14 has joined #openstack-lbaas07:39
*** gtrxcb has quit IRC07:44
*** tesseract has joined #openstack-lbaas07:48
*** aojea has quit IRC07:49
*** aojea has joined #openstack-lbaas07:49
*** aojea has quit IRC07:54
*** aojea has joined #openstack-lbaas07:59
*** ssmith has quit IRC08:00
rm_workthese failures on the multinode are perplexing08:23
nmagnezirm_work, kinda like lottery as a service08:24
rm_workah this one is different08:24
nmagnezirm_work, https://review.openstack.org/#/c/487565 one minor comment here08:24
rm_workis it just me or is the error message here a little ... weird08:25
rm_workhttp://logs.openstack.org/78/489678/2/check/gate-octavia-v1-dsvm-py3x-scenario-multinode/b2a843b/logs/screen-o-cw.txt.gz#_Aug_02_06_05_53_77312408:25
rm_work"Max retries exceeded"08:25
rm_worklike ... 40 times? uhh08:25
rm_worksomeone doesn't know what "max" means08:26
nmagneziO_o08:27
openstackgerritAdam Harwell proposed openstack/python-octaviaclient master: Remove reqs from test-req that exist in req  https://review.openstack.org/48756508:27
rm_work(that was not actually the problem)08:28
nmagnezirm_work, possibly this has something to do with how we use requests https://github.com/openstack/octavia/blob/b3506fa3ebdd48806d80e42f15c612412412a160/octavia/amphorae/drivers/haproxy/rest_api_driver.py#L27108:32
rm_workhmmm08:32
rm_worklooks like nova barfed on a create of one of the test members (not even an amp) >_>08:33
*** openstackgerrit has quit IRC08:33
rm_workhttp://logs.openstack.org/78/489678/2/check/gate-octavia-v1-dsvm-py3x-scenario-multinode/b2a843b/logs/screen-n-cpu.txt.gz#_Aug_02_06_00_36_07409608:34
nmagneziyikes08:34
rm_workmaybe happens more often on multi-node?08:34
nmagnezithat's a question for the infra folks i guess08:35
rm_worki guess it really starts at http://logs.openstack.org/78/489678/2/check/gate-octavia-v1-dsvm-py3x-scenario-multinode/b2a843b/logs/screen-n-cpu.txt.gz#_Aug_02_06_00_13_99146408:35
*** https_GK1wmSU has joined #openstack-lbaas08:36
*** https_GK1wmSU has left #openstack-lbaas08:37
*** aojea has quit IRC08:41
*** aojea_ has joined #openstack-lbaas08:41
rm_workmultinode is failing on my other patch too >_>08:44
*** aojea_ has quit IRC08:55
*** aojea has joined #openstack-lbaas08:57
*** belharar has joined #openstack-lbaas08:59
*** belharar has quit IRC09:00
*** belharar has joined #openstack-lbaas09:02
*** aojea_ has joined #openstack-lbaas09:09
*** aojea has quit IRC09:09
rm_workugh the issue is different EVERY time09:09
rm_workone time it just stopped in the middle of the install and failed09:09
rm_workone time it failed building a VM in nova09:10
rm_workone time curl died while downloading the base amp image for DIB09:10
rm_workone time it just ... lost connection to one of the other nodes in the multinode group09:10
rm_workhttp://logs.openstack.org/15/489015/4/check/gate-octavia-v1-dsvm-scenario-multinode/0171086/logs/devstacklog.txt.gz#_2017-08-02_05_36_25_31509:11
nmagnezirm_work, maybe file a bug and ask for some advise @ #openstack-infra ?09:20
rm_worki think it's just that shit is unstable right now and we're getting hit by a variety of issues... probably just need to let things settle09:20
rm_workgonna go to bed and check it out in the morning09:22
rm_workugh, meeting in ~7.5h09:22
rm_workhopefully will be there...09:23
nmagnezirm_work, good night :)09:28
*** openstackgerrit has joined #openstack-lbaas09:30
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)  https://review.openstack.org/43561209:30
rm_worknight :P09:30
*** ducnc has quit IRC09:41
*** sanfern has quit IRC10:56
*** atoth has quit IRC11:05
*** Guest14 is now known as ajo11:11
*** gongysh has quit IRC11:15
openstackgerritRajat Sharma proposed openstack/octavia master: Stop using deprecated CORS.set_latent()  https://review.openstack.org/48916911:16
openstackgerritNir Magnezi proposed openstack/octavia master: Stop using deprecated CORS.set_latent()  https://review.openstack.org/48916911:39
openstackgerritMerged openstack/octavia master: Barbicanclient is refactoring, this fixes one of our bad tests  https://review.openstack.org/48967811:55
*** atoth has joined #openstack-lbaas11:58
openstackgerritOpenStack Proposal Bot proposed openstack/python-octaviaclient master: Updated from global requirements  https://review.openstack.org/48817112:27
*** sanfern has joined #openstack-lbaas12:31
*** belharar has quit IRC12:57
*** yamamoto has quit IRC13:02
*** belharar has joined #openstack-lbaas13:03
*** yamamoto has joined #openstack-lbaas13:18
*** belharar has quit IRC13:32
*** belharar has joined #openstack-lbaas13:40
*** gcheresh_ has quit IRC13:49
*** yamamoto has quit IRC13:55
-openstackstatus- NOTICE: We have disable infracloud-vanilla due to the compute host running mirror.regionone.infracloud-vanilla.o.o being offline. Please recheck your failed jobs to schedule them to another cloud.13:57
*** sanfern has quit IRC14:04
*** sanfern has joined #openstack-lbaas14:05
*** slaweq has quit IRC14:14
*** slaweq has joined #openstack-lbaas14:15
*** slaweq has quit IRC14:19
*** fnaval has quit IRC14:46
*** armax_ has joined #openstack-lbaas14:53
*** armax has quit IRC14:53
*** armax_ is now known as armax14:53
*** yamamoto has joined #openstack-lbaas14:55
*** gcheresh_ has joined #openstack-lbaas14:57
*** fnaval has joined #openstack-lbaas14:59
*** yamamoto has quit IRC15:00
*** links has quit IRC15:01
*** kobis has quit IRC15:16
*** armax has quit IRC15:19
openstackgerritMerged openstack/python-octaviaclient master: Updated from global requirements  https://review.openstack.org/48817115:25
*** armax has joined #openstack-lbaas15:28
*** gcheresh_ has quit IRC15:31
*** tomtomtom has quit IRC15:32
*** armax has quit IRC15:33
*** belharar has quit IRC15:54
*** links has joined #openstack-lbaas15:59
*** catintheroof has joined #openstack-lbaas15:59
*** catintheroof has quit IRC16:00
*** catintheroof has joined #openstack-lbaas16:00
*** isantosp_ has quit IRC16:04
*** rcernin has quit IRC16:08
*** pcaruana has quit IRC16:14
*** armax has joined #openstack-lbaas16:18
*** gans has joined #openstack-lbaas16:40
*** JudeC has joined #openstack-lbaas16:51
johnsomOctavia meeting starting soon on #openstack-meeting16:56
*** gans has quit IRC16:56
*** gans has joined #openstack-lbaas16:57
*** rm_mobile has joined #openstack-lbaas16:59
*** tesseract has quit IRC17:07
*** harlowja has joined #openstack-lbaas17:13
*** rm_mobile has quit IRC17:17
*** sshank has joined #openstack-lbaas17:19
*** gans has quit IRC17:30
openstackgerritAdam Harwell proposed openstack/octavia master: Correct status for disabled members (honest abe edition)  https://review.openstack.org/48901517:52
*** jniesz has joined #openstack-lbaas18:01
*** links has quit IRC18:31
*** jniesz has quit IRC18:39
*** sshank has quit IRC18:51
*** sshank has joined #openstack-lbaas18:51
*** atoth has quit IRC18:57
*** apuimedo has quit IRC19:00
openstackgerritSanthosh Fernandes proposed openstack/octavia master: Adding exabgp-speaker element to amphora image  https://review.openstack.org/49016419:13
*** gcheresh_ has joined #openstack-lbaas19:18
*** sshank has quit IRC19:19
*** sshank has joined #openstack-lbaas19:27
*** sshank has quit IRC19:29
openstackgerritNir Magnezi proposed openstack/python-octaviaclient master: Remove reqs from test-req that exist in req  https://review.openstack.org/48756519:42
*** gcheresh_ has quit IRC19:46
*** sshank has joined #openstack-lbaas19:51
*** sshank has quit IRC19:55
*** gcheresh_ has joined #openstack-lbaas20:02
openstackgerritSanthosh Fernandes proposed openstack/octavia master: [WIP] Adding exabgp-speaker element to amphora image  https://review.openstack.org/49016420:11
*** sshank has joined #openstack-lbaas20:14
rm_workyes another different multinode failure: http://logs.openstack.org/54/485254/4/gate/gate-octavia-v1-dsvm-py3x-scenario-multinode/052d1e0/logs/screen-o-cw.txt.gz#_Aug_02_19_13_24_67444420:23
rm_workmultinode has been failing like 3/4 of the time for the past day20:23
rm_workbut it's always different20:23
johnsomWe have been seeing that 404 for a while.  Some of the other recent failures I have seen are libvirt/qemu crashing again20:24
johnsomI have that 404 on my list to investigate.  On the surface it looks like the interface is never added by the kernel.  I mostly see it on non-multinode jobs though.20:25
*** gcheresh_ has quit IRC20:26
*** sshank has quit IRC20:27
*** sshank has joined #openstack-lbaas20:27
openstackgerritSanthosh Fernandes proposed openstack/octavia master: [WIP] Adding exabgp-speaker element to amphora image  https://review.openstack.org/49016420:27
*** aojea_ has quit IRC21:00
*** aojea has joined #openstack-lbaas21:01
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)  https://review.openstack.org/43561221:03
*** yamamoto_ has joined #openstack-lbaas21:08
*** yamamoto_ has quit IRC21:15
*** yamamoto_ has joined #openstack-lbaas21:17
*** eandersson has joined #openstack-lbaas21:35
openstackgerritGerman Eichberger proposed openstack/octavia master: Ignore 404 amphora error when deleting resources  https://review.openstack.org/48723221:35
xgerman_most of the people doing work are not known well in the org21:38
xgerman_I think they just made him a tetconic core…21:38
johnsomxgerman_ Wrong channel21:38
mnaseris there a way to get the list of IPs that a LB can have?21:42
mnaserit seems that when using octavia in active_standby, the traffic obviously doesnt come from the vip21:43
*** sshank has quit IRC21:43
mnaserk8s by default has some firewalling so only allow the loadbalancer ip21:43
mnaserso.. it blocks traffic21:43
johnsomYeah, the source should come from the base port IP.  It's going to be dhcp'd or assigned by neutron when the LB is created from the subnet/network specified when the LB is created.21:44
johnsomFor testing/manual, when you nova list as the octavia user or admin, the base port IP is listed21:45
mnaserjohnsom is there a way to get that somehow exposed or automated21:46
mnasercause i dont think a user would be able to see it, would they?21:47
johnsomWell, the thing to note is if that amp fails over, that IP can change21:47
johnsomNo, that detail is hidden from an end user21:47
mnaserjohnsom bummer.  i have to try and figure out how to make this work, k8s filters traffic from the vips (i guess it assumes traffic will only come from that ip)21:48
* mnaser thinks21:48
johnsomYou could make a patch that causes the traffic to come from the VIP IP.21:48
johnsomYour case is you are relying on the VIP network default route right?  No member subnets?21:49
mnaserjohnsom yes, its all in the same l2 domain21:49
*** ajo has quit IRC21:49
johnsomYeah,  because member subnet source IPs are also allocated by neutron, so would be a problem too21:50
mnaserif i can send traffic with the source being the vip, that's good enough for me21:50
mnasertechnically i should be able to because i assume octavia creates allowed_address_pairs21:50
johnsomYeah, give me a minute to find the setting that will need to go into the haproxy config21:51
* mnaser looks aswell21:51
johnsomhttp://cbonte.github.io/haproxy-dconv/1.6/configuration.html#5.2-source21:53
johnsomThat will need to be added to the jinja template, then code added to set it to be the vip IP21:53
mnaseroooo nice21:53
johnsomIf you are going to upstream that, we need to think about how to a) make it an operator option, b) only do it for members that have the same subnet as the VIP.21:54
rm_workyeah, the jinja templates are configurable without *patching* too, I think right?21:54
rm_workwe made it a path in config21:54
rm_workIIRC21:54
rm_workso it's a deployer choice thing21:55
mnaserrm_work they are i believe21:55
mnaserhaproxy_template21:55
rm_workyes21:55
rm_workso any deployer (you) can do whatever special sauce they need21:55
rm_workthis qualifies i think21:55
mnaser(im searching through the code)21:55
johnsomYeah, the VIP IP should already be a variable for jinja, so probably no code change needed if you do it local21:55
rm_workthough MAYBE this could be useful for others21:55
mnaserwell i would prefer to upstream because i dont want to maintain a custom template that might deviate with new updates21:55
mnaseri could implement it as a config option21:56
*** sshank has joined #openstack-lbaas21:56
johnsomhttps://github.com/openstack/octavia/blob/master/octavia/common/jinja/haproxy/templates/macros.j2#L14221:56
mnaserjohnsom if we add it for all, wouldnt the traffic return through the default route so it should be okay?21:57
mnaservip: 10.0.0.1/24, vms: 10.1.0.0/24, traffic sent to 10.1.0.0 from 10.0.0.1 will go through router (there has to be one in the first place if they dont share l2)21:58
mnaservm in 10.1.0.0/24 doesnt have direct route, falls back to default (the router) and then router sends to vip21:58
johnsomThat setting will force the backend connection, for that member, to have a source IP of the VIP IP.  For most deployments, that is bad.21:58
mnaseri guess i'm not seeing the other use cases, in k8s the load balancers sits entirely in the same subnet/network21:59
mnaserand relies on floating ips to get public ip21:59
mnaserbut im starting to see where you're getting at21:59
mnaserpublic ip for vip, members in a private network, bad things21:59
johnsomI.e. member subnet is 10.1.0.0/24 and vip is 10.2.0.0/24, a source IP of 10.2.0.1 on subnet 10.1.0.0/24 will get dropped/ignored22:00
johnsomYep, you got it22:00
mnaseri could do the source for same l2 only22:00
openstackgerritGerman Eichberger proposed openstack/octavia master: ACTIVE-ACTIVE Topology: Initial Distributor Driver Mixin  https://review.openstack.org/31300622:01
*** armax has quit IRC22:01
xgerman_you can also change K8 to accept a netmask22:07
*** aojea has quit IRC22:08
*** aojea has joined #openstack-lbaas22:08
mnaserxgerman_ LoadBalancerSourceRanges are specified in the spec of the service and make poor user experience22:08
mnaserfor example using something like helm.. you'd have the whole automation gone22:09
xgerman_I don’t follow but I also never have used helm — my understanding is K8 creates LB to forward traffic to their nodes… so if anything it should be an operator setting in K822:12
xgerman_another thing I have heard about is that people wanted to restrict which traffic can enter the VIP…22:12
*** aojea has quit IRC22:13
mnaserxgerman_ im going to do a bit more reading and check.. kubernetes creates a bunch of complicated iptables rules so let me rule them out22:14
xgerman_yeah, they use iptable rules so all those service ports are proxied around22:15
*** sshank has quit IRC22:24
*** yamamoto_ has quit IRC22:29
*** sshank has joined #openstack-lbaas22:29
*** https_GK1wmSU has joined #openstack-lbaas22:31
openstackgerritMerged openstack/octavia master: Fix haproxy_check_script for delete listener  https://review.openstack.org/48525422:32
*** yamamoto has joined #openstack-lbaas22:32
*** https_GK1wmSU has left #openstack-lbaas22:34
*** yamamoto has quit IRC22:36
*** yamamoto has joined #openstack-lbaas22:41
*** yamamoto has quit IRC22:44
*** catintheroof has quit IRC22:44
*** fnaval has quit IRC22:54
*** yamamoto has joined #openstack-lbaas22:56
*** fnaval has joined #openstack-lbaas22:59
*** fnaval has quit IRC22:59
*** fnaval has joined #openstack-lbaas23:00
*** gtrxcb has joined #openstack-lbaas23:06
*** yamamoto has quit IRC23:11
*** yamamoto has joined #openstack-lbaas23:14
*** tongl has joined #openstack-lbaas23:26
*** apuimedo has joined #openstack-lbaas23:26
*** sshank has quit IRC23:50
« Tuesday, 2017-08-01 Index Thursday, 2017-08-03 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!