Tuesday, 2019-04-02

« Monday, 2019-04-01 Index Wednesday, 2019-04-03 »
*** hongbin has joined #openstack-lbaas00:37
*** fnaval has quit IRC00:43
*** yamamoto has joined #openstack-lbaas01:02
*** yamamoto has quit IRC01:07
*** luksky has quit IRC01:08
*** dulek has quit IRC01:21
*** ricolin has joined #openstack-lbaas01:29
*** dulek has joined #openstack-lbaas01:34
*** rcernin_ has joined #openstack-lbaas02:05
*** rcernin has quit IRC02:06
*** rcernin_ has quit IRC02:12
*** rcernin has joined #openstack-lbaas02:15
*** hongbin has quit IRC02:57
*** hongbin has joined #openstack-lbaas02:58
*** yamamoto has joined #openstack-lbaas03:06
*** hongbin has quit IRC03:12
*** psachin has joined #openstack-lbaas03:14
*** hongbin has joined #openstack-lbaas03:14
*** hongbin has quit IRC03:31
*** yamamoto has quit IRC04:06
*** yamamoto has joined #openstack-lbaas04:06
*** yamamoto has quit IRC04:06
*** yamamoto has joined #openstack-lbaas04:45
*** vishalmanchanda has joined #openstack-lbaas05:02
*** pcaruana has joined #openstack-lbaas05:35
*** ramishra has joined #openstack-lbaas05:38
*** yamamoto has quit IRC05:48
*** ccamposr has joined #openstack-lbaas06:09
*** abaindur has quit IRC06:11
*** abaindur has joined #openstack-lbaas06:12
*** gthiemonge has joined #openstack-lbaas06:39
*** ivve has joined #openstack-lbaas06:43
*** kmadac4 is now known as kmadac06:51
*** gcheresh has joined #openstack-lbaas07:02
*** yamamoto has joined #openstack-lbaas07:13
*** pck has quit IRC07:14
*** pck has joined #openstack-lbaas07:14
*** luksky has joined #openstack-lbaas07:15
*** rpittau|afk is now known as rpittau07:23
*** happyhemant has quit IRC07:47
*** happyhemant has joined #openstack-lbaas07:48
*** abaindur has quit IRC08:03
*** ignaziocassano1 has joined #openstack-lbaas08:09
ignaziocassano1hello, any help on octavia under queens ? I configured octavia-api under haproxy and it works but in /var/log/messages I got a lot of broken pipe errors08:09
*** salmankhan has joined #openstack-lbaas08:10
*** salmankhan has quit IRC08:25
johnsomYour haproxy config likely has too low of a timeout config for your app08:31
*** velizarx has joined #openstack-lbaas08:40
*** velizarx has quit IRC08:53
ignaziocassano1Johnsom, many thanks. What configuration you suggest ?08:57
*** velizarx has joined #openstack-lbaas09:06
*** rcernin has quit IRC09:11
*** sapd1_x has joined #openstack-lbaas09:26
*** sapd1_x has quit IRC09:46
*** zigo has joined #openstack-lbaas09:59
*** yamamoto has quit IRC10:01
*** yamamoto has joined #openstack-lbaas10:14
*** yamamoto has quit IRC10:18
dulekcgoncalves: Okay, I have that issue I mentioned yesterday reproduced locally.10:49
dulekcgoncalves: Any idea where to start debugging? I check that I can reach the pool member's address directly.10:49
*** yamamoto has joined #openstack-lbaas10:57
cgoncalvesdulek, can you describe your deployment first? which provider driver?11:03
*** yamamoto has quit IRC11:03
*** yamamoto has joined #openstack-lbaas11:03
dulekcgoncalves: It's amphora, we're setting a simple LB that points to K8s API we deploy.11:04
dulekcgoncalves: At this point I see that the issue is that amphora can't reach 10.0.0.67, which is our K8s API.11:04
dulekcgoncalves: And that's because it doesn't have the route.11:04
dulekcgoncalves: And that's because even though the VM has that network attached, it doesn't show up inside amphora.11:05
cgoncalvesdulek, master?11:05
dulekcgoncalves:11:05
dulekcgoncalves: Yup. 23a411413fbad6e96ed6b3c3ec3876cba3d8f4fd seems suspicious?11:06
*** numans has joined #openstack-lbaas11:06
cgoncalvesI hope not11:06
cgoncalvesdulek, ubuntu or centos amp?11:07
dulekcgoncalves: Uhm… Whatever is the default.11:07
dulekSo probably Ubuntu?11:07
cgoncalvesyes11:07
dulekcgoncalves: networking.service is down on that amp.11:08
dulekLet's see what happens if I restart it…11:09
dulekHa, wonderful, it dies.11:09
cgoncalvesdulek, I have a devstack master from yesterday. I'll test it11:09
dulekcgoncalves: Hm I see "Cannot find device "eth0"" in networking logs.11:10
cgoncalvesoops! houston, we have a problem with centos amps11:17
*** yamamoto has quit IRC11:17
dulekcgoncalves: Uh, but mine's Ubuntu. :P11:17
dulekcgoncalves: BTW - 16.04? Really?11:18
cgoncalvesone thing at a time :)11:18
cgoncalvesFYI, this is the issue I'm facing now: https://storyboard.openstack.org/#!/story/200534111:21
dulekcgoncalves: Ha, I told you that change was suspicious. xD11:25
*** openstackgerrit has joined #openstack-lbaas11:31
openstackgerritCarlos Goncalves proposed openstack/octavia master: Fix VIP plugging on CentOS-based amphorae  https://review.openstack.org/64928211:31
openstackgerritCarlos Goncalves proposed openstack/octavia master: Fix VIP plugging on CentOS-based amphorae  https://review.openstack.org/64928211:33
cgoncalvesdulek, your amphora is xenial or bionic?11:35
dulekcgoncalves: 16.04, so xenial.11:35
dulekcgoncalves: It's built by master.11:36
cgoncalvesyep. building one11:36
openstackgerritMichał Dulko proposed openstack/octavia master: WiP: Revert "Fix ifup failures on member interfaces with IPv6"  https://review.openstack.org/64928311:37
dulekcgoncalves: I only intend to try this in Kuryr gates to see if it's the culprit. ^11:37
*** yamamoto has joined #openstack-lbaas11:42
*** celebdor has quit IRC11:45
cgoncalvesok11:47
*** lemko has joined #openstack-lbaas11:47
*** goldyfruit has quit IRC11:57
*** ignaziocassano1 has quit IRC11:57
openstackgerritCarlos Goncalves proposed openstack/octavia master: Fix VIP plugging on CentOS-based amphorae  https://review.openstack.org/64928211:59
*** yamamoto has quit IRC11:59
*** yamamoto has joined #openstack-lbaas12:00
*** yamamoto has quit IRC12:07
*** boden has joined #openstack-lbaas12:10
*** sapd1_x has joined #openstack-lbaas12:10
dulekcgoncalves: Hm, unless my local amphora was somehow cached, it seems like revert doesn't help. :(12:18
cgoncalvesdulek, rm -rf /opt/stack/octavia/diskimage-create/amphora*12:18
dulekcgoncalves: So it's cached? Okay, so let's see if that revert upstream will help.12:19
dulekcgoncalves: Meanwhile had you reproduced the issue?12:20
cgoncalvesdulek, no. I was commuting.12:21
dulekcgoncalves: Okay, cool. I'll try to poke amphora's internals a bit to make it detect that interface.12:22
*** celebdor has joined #openstack-lbaas12:23
*** trown|outtypewww is now known as trown12:24
*** yamamoto has joined #openstack-lbaas12:33
dulekcgoncalves: Taking a peek at the gate logs says that revert probably won't help. The cause must be different…12:42
*** trident has quit IRC13:30
*** trident has joined #openstack-lbaas13:33
*** Vorrtex has joined #openstack-lbaas13:54
*** vishalmanchanda has quit IRC13:55
*** yamamoto has quit IRC14:01
*** yamamoto has joined #openstack-lbaas14:04
*** yamamoto has quit IRC14:04
*** yamamoto has joined #openstack-lbaas14:05
*** sapd1_x has quit IRC14:08
*** sapd1_x has joined #openstack-lbaas14:22
*** velizarx has quit IRC14:26
*** velizarx has joined #openstack-lbaas14:40
dulekcgoncalves: Hey, so maybe you want to get access to a VM in which I have that issue?14:40
*** gcheresh has quit IRC14:42
cgoncalvesdulek, sure14:53
*** velizarx has quit IRC14:59
*** fnaval has joined #openstack-lbaas15:01
cgoncalvesPSA: on going issues with ubuntu mirrors affecting CI and local deployments15:02
*** velizarx has joined #openstack-lbaas15:05
*** vishalmanchanda has joined #openstack-lbaas15:10
dulekcgoncalves: Okay, get me your SSH key and let's see if you'll be able to find anything. :)15:11
cgoncalvesdulek, https://github.com/cgoncalves.keys15:12
dulekcgoncalves: ssh stack@38.145.34.14415:13
dulekcgoncalves: I have a tmux session in which I'm logged to amphora.15:14
cgoncalvesdulek, you have a bad ip route in the amp15:23
dulekcgoncalves: Yeaaah, it seems so. But why?15:24
cgoncalvesyour member (10.0.0.122) is in same VIP network and route was 10.0.0.128/2615:24
dulekcgoncalves: I assume you looked at the amphora-haproxy netns?15:24
cgoncalvesyes15:24
cgoncalvesprivate-subnet      | 787672b3-794e-4e1f-ac86-e9a8f609d89a | 10.0.0.0/2615:24
cgoncalvesamphora is respecting the subnet range15:25
dulekcgoncalves: Okay, so I'll try apuimedo's idea and add --subnet-id to member creation.15:25
dulekcgoncalves: But what had changed recently? I'm pretty sure all this worked fine until recently.15:25
cgoncalveshmm, on a second thought, shouldn't the route have been 10.0.0.0/26? /me looks15:26
dulekcgoncalves: So the LB is on k8s-service-subnet. And the member is on k8s-pod-subnet.15:26
dulekcgoncalves: So don't look at private-subnet, we don't use it.15:27
cgoncalvesah, haven't seen there was other 10.0.0.X subnets15:28
*** velizarx has quit IRC15:30
dulekcgoncalves: I want to try to remove that member and add another one with --subnet-id specified.15:30
dulekcgoncalves: Should I wait until you finish something?15:30
cgoncalvesyeah, I was checking that15:31
cgoncalvesk8s-pod-subnet      | c54387a7-8b72-4a67-8fa1-5f9bd6e46b4d | 10.0.0.64/2615:31
cgoncalvesk8s-service-subnet  | 6e20720d-9171-43f0-8a2e-e345c799d577 | 10.0.0.128/2615:31
*** velizarx has joined #openstack-lbaas15:31
cgoncalvesboth on different networks15:31
cgoncalvessince you created the member without specifying the subnet, it might have assumed it was on the same network and hence didn't plug15:32
cgoncalvesdulek, please recreate with --subnet-id set15:32
dulekcgoncalves: Okay, going for it15:32
*** ccamposr has quit IRC15:33
dulekHm, not that it works but at least amp has one more interface now. ;)15:35
*** ivve has quit IRC15:35
cgoncalvesthat might be my fault because I changed an ip route15:37
dulekcgoncalves: So even though Amphora now has access to that 10.0.0.64/26 network, it still doesn't connect to 10.0.0.12215:37
dulekcgoncalves: Oh yes, I see.15:38
dulekHm, but no, it should still fall on the correct one, right? Smaller are preferred.15:38
*** velizarx has quit IRC15:40
dulekcgoncalves: Well, seems like there's still some issue. SG's?15:41
cgoncalvesdulek, checking while multi-tasking15:44
*** sapd1_x has quit IRC15:44
johnsomIt has always been that if you don’t specify the subnet for a member it uses the VIP subnet.15:44
cgoncalvesright. dulek specified the subnet now and he got a new interface plugged15:45
dulekjohnsom: Sure, it makes sense. But for some reason it was working for us and suddenly stopped. :P15:45
cgoncalvesdulek, what have you folks changed on your side? ;)15:46
dulekThough there's still no connectivity to our member IP. :(15:46
cgoncalvesthat is the real question15:46
dulekcgoncalves: Let me go through the git log real quick…15:46
cgoncalvesdulek, I'll reboot the amp just to check something, k?15:47
dulekcgoncalves: Sure!15:47
cgoncalveshopefully it boots up in less than 60 seconds15:47
dulekcgoncalves: Oh, I doubt it, this is RDO cloud, so no nested virt. :P15:48
dulekcgoncalves: There's nothing in kuryr-kubernetes git log that could possibly cause this.15:48
dulekcgoncalves: Note that this is an issue on DevStack plugin level.15:48
cgoncalvesdulek, I don't follow15:49
cgoncalveshealth manager triggered amphora failover15:50
dulekcgoncalves: I mean we see the issue before any Kuryr service is started. So only patches affecting the DevStack plugin could possibly cause this regression.15:50
dulekcgoncalves: At least when we talk about kuryr-kubernetes patches. ;)15:50
dulekcgoncalves: Seems like it still doesn't work?15:54
cgoncalvesI was distracted confirming a bug johnsom might have introduced :P15:55
johnsomWe need functional centos gates...15:56
cgoncalvesjohnsom, indeed! I gave a first shot at adding a work around to the systemd bug until update package is released15:58
cgoncalveshttps://review.openstack.org/#/c/643752/15:58
cgoncalveswhile it passes CI, it's not actually fixing anything. job time is still quite high, expected it to be ~1h3015:59
openstackgerritVishal Manchanda proposed openstack/octavia-dashboard master: Drop nodejs4 jobs  https://review.openstack.org/64938016:03
*** velizarx has joined #openstack-lbaas16:04
openstackgerritGregory Thiemonge proposed openstack/octavia master: Fix spare amphora check and creation  https://review.openstack.org/64938116:09
openstackgerritGregory Thiemonge proposed openstack/octavia master: Fix invalid query selector with list_ports  https://review.openstack.org/64938216:12
*** velizarx has quit IRC16:14
cgoncalvesdulek, are you curling or something? there's traffic between amphora and member16:15
dulekcgoncalves: Hm, no.16:18
dulekcgoncalves: Oh, but Kuryr services are trying to constantly connect to that LB.16:18
dulekcgoncalves: So that might be why you see it.16:18
*** velizarx has joined #openstack-lbaas16:18
cgoncalvesdulek, let's kill the noise please16:20
dulekcgoncalves: Okay, just a sec,16:21
dulekcgoncalves: Done.16:22
cgoncalvesthanks16:22
*** velizarx has quit IRC16:22
*** rpittau is now known as rpittau|afk16:24
*** velizarx has joined #openstack-lbaas16:27
*** celebdor has quit IRC16:31
cgoncalvesdulek, on and off in this. there's a TCP SYNC going out from amphora eth2 but no SYNC ACK back. how can I open a console with the member?16:33
*** ramishra has quit IRC16:34
dulekcgoncalves: Console with the member? Well, it's not really a VM or container.16:34
dulekcgoncalves: https://github.com/openstack/kuryr-kubernetes/blob/3e3ed9dbb31133b5175ff792a9636603b2df95e1/devstack/lib/kuryr_kubernetes#L64-L9616:35
cgoncalvesdulek, ok. I want to do a packet capture on whatever is closer to the member16:35
dulekcgoncalves: How about listening on kubelet301ba6f interface?16:36
dulekcgoncalves: As from the host 10.0.0.122:443 is reachable, then that should be the place to look.16:36
*** velizarx has quit IRC16:46
openstackgerritVishal Manchanda proposed openstack/neutron-lbaas-dashboard master: Drop nodejs4 job  https://review.openstack.org/64939916:47
*** trown is now known as trown|lunch16:50
dulekcgoncalves: So no findings?17:09
cgoncalvesdulek, I can see the TCP SYN also in the tap device in the hypervisor17:10
cgoncalvesdulek, looking at SG of the kube port, it seems traffic is not allowed in ingress17:12
cgoncalvesdulek, add SG rule and retry17:12
cgoncalves|                 | created_at='2019-04-02T11:55:54Z', description='k8s service subnet allowed', direction='ingress', ethertype='IPv4', id='7c7db1cd-d123-4c64-9453-bfe2001b24ce', protocol='tcp', remote_ip_prefix='10.0.0.128/26', updated_at='2019-04-02T11:55:54Z'     |17:12
cgoncalveseither this is not what you want or you also want to allow 10.0.0.64/2617:13
dulekcgoncalves: Damn, it worked. But what changed over the weekend?! :P17:15
*** ricolin has quit IRC17:17
dulekIntuition tells me that two distinct issues - missing subnet-id when creating a member and that SG problem must be caused by a single thing.17:18
dulekOh, I think I might see a connection here…17:20
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add a scenario test for spare pool  https://review.openstack.org/63498817:22
*** velizarx has joined #openstack-lbaas17:26
*** velizarx has quit IRC17:43
*** velizarx has joined #openstack-lbaas17:49
*** velizarx has quit IRC17:51
*** velizarx has joined #openstack-lbaas17:53
*** trown|lunch is now known as trown17:55
*** psachin has quit IRC17:55
*** goldyfruit has joined #openstack-lbaas18:09
*** lemko has quit IRC18:17
*** celebdor has joined #openstack-lbaas19:30
*** velizarx has quit IRC19:31
*** ianychoi has quit IRC19:32
*** ianychoi has joined #openstack-lbaas19:33
*** vishalmanchanda has quit IRC19:35
*** velizarx has joined #openstack-lbaas19:36
openstackgerritGaëtan Trellu proposed openstack/octavia master: [diskimage-create] Fix qemu-utils package name  https://review.openstack.org/64943819:43
openstackgerritGaëtan Trellu proposed openstack/octavia master: [diskimage-create] Fix qemu-utils package name  https://review.openstack.org/64943820:00
*** Vorrtex has quit IRC20:05
*** velizarx has quit IRC20:06
lxkongjohnsom, rm_work, need your help on this https://storyboard.openstack.org/#!/story/2005348, one of our private cloud is using provider network to bootstrap load balancers, but it keeps failing of this error. It's very hard to know the reason with the log.20:16
xgermanprobably need the logs on the amphora as well...20:18
xgermansince this is where the server error occurs...20:18
*** blake has joined #openstack-lbaas20:21
lxkongxgerman: at least in the worker log, we should see the exception message from the amphora20:22
lxkongbut in octavia.amphorae.drivers.haproxy.exceptions.check_exception, it's hiding20:23
lxkongxgerman: btw, unrelated question, does octavia support multiple certs when doing l7 load balancing?20:24
xgermanwe support sni20:29
xgermanif that’s what you mean with multiple certs ;-)20:30
*** pcaruana has quit IRC20:30
xgermanyeah, it show it in the worker logs but your snippet only shows the internal server error - so was wondering about that20:30
xgermanaka we should fix that… most of our errors are not internal server so this is likely a bug :-(20:31
lxkongxgerman: exactly20:31
johnsomlxkong: The error details are logged above that message in the log.20:34
lxkongjohnsom: i am afraid i didn't see that :-( i can double check given it's very easy to make this error.20:36
johnsomlxkong It should say: "Amphora agent returned unexpected result code" followed by the details20:40
johnsomhttps://github.com/openstack/octavia/blob/master/octavia/amphorae/drivers/haproxy/exceptions.py#L3720:41
lxkonghmm....20:42
lxkongwe are in stable/queens20:43
johnsomYeah, it is the same in queens20:43
lxkongreally?20:43
johnsomhttps://github.com/openstack/octavia/blob/stable/queens/octavia/amphorae/drivers/haproxy/exceptions.py#L3720:43
* lxkong goes to update the stable branch20:43
lxkongyes, you are right20:44
lxkongwe should do upgrade20:44
lxkongjohnsom: thanks20:44
* lxkong goest to stand up meeting and then mark the story invalid20:44
johnsomlxkong I can do it for you20:45
johnsomI wonder if we have not released that yet....  I know we were blocked on doing a queens release for a while. Let me check that too20:48
lxkongthanks johnsom21:02
johnsomlxkong I think we haven't done a release with that in it yet. We want to do that soon though21:02
lxkongjohnsom: that's fine, my suggestion to our release team just now was we upgrade directly to Stein if possible21:03
lxkongwe are also keen for the tags feature21:03
*** blake has quit IRC21:03
johnsomHa, ok. We haven't release that yet either, but there is an RC1 out21:03
lxkongthat's not a problem for us, we are building debian packages from the source code :-)21:04
*** trown is now known as trown|outtypewww21:04
*** goldyfruit has quit IRC21:26
*** luksky has quit IRC21:32
*** boden has quit IRC21:39
*** blake has joined #openstack-lbaas22:01
*** blake has quit IRC22:01
*** abaindur has joined #openstack-lbaas22:04
*** abaindur has quit IRC22:08
*** fnaval has quit IRC22:10
*** rcernin has joined #openstack-lbaas22:25
*** goldyfruit has joined #openstack-lbaas22:33
lxkongjohnsom: any chance you know the reason of `SIOCADDRT: Invalid argument\nFailed to bring up eth1`?22:48
lxkongthe vip subnet is on a neutron provider network22:49
johnsomNot off my head.22:50
lxkongok22:52
openstackgerritMerged openstack/octavia master: Add support to the Python Visual Studio Debugger  https://review.openstack.org/64528022:55
*** celebdor has quit IRC23:22
*** goldyfruit has quit IRC23:55
« Monday, 2019-04-01 Index Wednesday, 2019-04-03 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!