Monday, 2018-07-02

*** longkb has joined #openstack-lbaas00:38
*** hongbin has joined #openstack-lbaas01:04
*** kiennt26 has joined #openstack-lbaas01:15
*** kiennt26 has quit IRC01:28
*** openstackgerrit has quit IRC01:49
*** kiennt26 has joined #openstack-lbaas02:01
*** phuoc has quit IRC02:25
*** phuoc has joined #openstack-lbaas02:40
*** phuoc has quit IRC02:45
*** phuoc has joined #openstack-lbaas02:52
*** phuoc has quit IRC02:56
*** phuoc has joined #openstack-lbaas02:57
*** hongbin has quit IRC03:03
*** phuoc has quit IRC03:04
*** phuoc has joined #openstack-lbaas03:04
*** ramishra has joined #openstack-lbaas03:26
*** JudeC_ has joined #openstack-lbaas04:10
*** yamamoto has joined #openstack-lbaas04:21
*** openstackgerrit has joined #openstack-lbaas04:21
openstackgerritMichael Johnson proposed openstack/neutron-lbaas master: Neutron-LBaaS to Octavia migration tool  https://review.openstack.org/57894204:21
*** yboaron has joined #openstack-lbaas04:40
*** yboaron has quit IRC04:45
*** rcernin has quit IRC04:55
*** links has joined #openstack-lbaas05:09
*** links has quit IRC05:09
openstackgerritMichael Johnson proposed openstack/neutron-lbaas master: Neutron-LBaaS to Octavia migration tool  https://review.openstack.org/57894205:25
*** rcernin has joined #openstack-lbaas05:41
*** rcernin has quit IRC05:41
*** rcernin has joined #openstack-lbaas05:41
*** rcernin has quit IRC05:41
*** rcernin has joined #openstack-lbaas05:42
*** blake has joined #openstack-lbaas05:57
*** peereb has joined #openstack-lbaas06:00
*** nmanos has joined #openstack-lbaas06:21
*** dmellado has joined #openstack-lbaas06:23
*** JudeC_ has quit IRC06:25
*** threestrands has joined #openstack-lbaas06:33
*** threestrands has quit IRC06:33
*** threestrands has joined #openstack-lbaas06:33
*** threestrands has quit IRC06:37
*** AlexStaf has quit IRC06:38
*** nmanos has quit IRC06:59
*** rcernin has quit IRC07:09
*** nmanos has joined #openstack-lbaas07:16
*** velizarx has joined #openstack-lbaas07:19
*** velizarx has quit IRC07:25
*** tesseract has joined #openstack-lbaas07:27
*** sapd_ has quit IRC07:33
*** sapd_ has joined #openstack-lbaas07:34
*** rpittau has joined #openstack-lbaas07:35
*** JudeC_ has joined #openstack-lbaas07:41
*** blake has quit IRC07:43
*** JudeC_ has quit IRC07:46
*** yamamoto_ has joined #openstack-lbaas07:53
*** yamamoto has quit IRC07:57
*** nmanos has quit IRC07:59
*** ktibi has joined #openstack-lbaas08:03
*** nmanos has joined #openstack-lbaas08:23
*** yboaron has joined #openstack-lbaas09:19
*** AlexeyAbashkin has joined #openstack-lbaas09:23
*** AlexeyAbashkin has quit IRC09:32
*** kiennt26 has quit IRC10:15
*** longkb has quit IRC11:01
jitekaHey guys, is that patch allowed cross-provider migration ?11:10
jitekahttps://review.openstack.org/#/c/578942/11:10
jitekaor as explained during last Openstack summit, migration between neutron-lbaas and octavia is only between same provider11:12
jitekaExample : Neutron LBaaS v2 using A10 thunder driver to Octavia with A10 provider (if driver exist)11:12
*** ramishra has quit IRC11:30
*** ramishra has joined #openstack-lbaas11:32
*** peereb has quit IRC11:54
*** nmanos has quit IRC12:01
*** nmanos has joined #openstack-lbaas12:04
cgoncalvesjiteka, only between same provider12:06
*** nmanos has quit IRC12:09
openstackgerritAdit Sarfaty proposed openstack/octavia master: Add listener_id to the pool provider object  https://review.openstack.org/57953212:20
*** yboaron_ has joined #openstack-lbaas12:34
*** nmanos has joined #openstack-lbaas12:35
*** yboaron has quit IRC12:36
*** tesseract has quit IRC13:04
*** tesseract has joined #openstack-lbaas13:10
*** tesseract has quit IRC13:10
*** JudeC_ has joined #openstack-lbaas13:11
*** JudeC_ has quit IRC13:11
openstackgerritCarlos Goncalves proposed openstack/octavia master: Fix fields translation on filtering  https://review.openstack.org/57955913:35
*** tesseract has joined #openstack-lbaas13:38
cgoncalvesltomasbo, ^13:39
*** gans has joined #openstack-lbaas13:53
*** vegarl has quit IRC13:57
*** nmanos has quit IRC13:58
*** yboaron has joined #openstack-lbaas14:00
*** gans has quit IRC14:00
*** yboaron_ has quit IRC14:01
*** vegarl has joined #openstack-lbaas14:03
ltomasbocgoncalves, thanks!14:27
ltomasboI'll try to give it a try later today!14:27
cgoncalvesltomasbo, ack. let me know if that works for you14:30
cgoncalvesbtw, this was my test client: http://paste.openstack.org/show/724757/14:30
ltomasbocgoncalves, ok!14:33
*** AlexStaf has joined #openstack-lbaas14:35
*** kevko has joined #openstack-lbaas14:48
kevkoHello guys , could someone answer some question regarding deploy of octavia ?14:48
kevkoplease14:48
xgerman_jiteka: there are further tools panned like “exporting” a load balancer and then “importing” it with a different provider…14:59
xgerman_kevko: can you ask again?14:59
johnsomkevko: Sure what are your questions?15:00
kevkozgerman: i deployed octavia, everything looks ok,     create LB, amphora image was spawned, has management address , has also two interfaces assignes in network namespace ..listening on port 80 as i described in listener ..but can't connect to that port on that IP ... do you have some suggestion ?15:01
*** fnaval has joined #openstack-lbaas15:01
kevkozgerman : i am on the end with my ideas ..15:01
johnsomkevko Just to clarify, when you curl to port 80 on the VIP address (shown in "openstack loadbalancer show"), you get a timeout or a 503?15:04
kevkojohnsom: timeout15:05
johnsomkevko And the host you are connecting from has a route to the VIP subnet?15:05
johnsomHow did you install? Devstack?  OpenStack Ansible?15:06
kevkojohnsom: i created debian packages for debian ...installed ...and tried to replicate devstack configuration15:06
johnsomOk, cool15:07
johnsomkevko Does the load balancer have just an LB and listener or does it have a pool too?15:07
kevkojohnsom: have also pool and member inside ...but currently there is no webserver running on member instance .. i wanted to try firstly with netcat on that member instance in pool ...15:09
johnsomOh, ok, that might be the issue.  Try this, create a second listener on the load balancer, also HTTP but on a different port #, say 81, then try curling to port 81. I would expect a 503 response15:10
johnsomThis will show that the load balancer and it's VIP is working correctly.15:11
openstackgerritMichael Johnson proposed openstack/neutron-lbaas master: Neutron-LBaaS to Octavia migration tool  https://review.openstack.org/57894215:12
kevkojohnsom: ok, ..so i'm going to try it and let you know15:15
johnsomSorry for the channel spam over the weekend, Ansible was not my friend this weekend.  I still don't know why the stderr output from the neutron command is getting put in the Ansible stdout... Ugh, but progress on that test non the less.15:16
openstackgerritZhaoBo proposed openstack/octavia master: UDP jinja template  https://review.openstack.org/52542015:18
openstackgerritZhaoBo proposed openstack/octavia master: UDP for [2]  https://review.openstack.org/52965115:18
openstackgerritZhaoBo proposed openstack/octavia master: UDP for [3][5][6]  https://review.openstack.org/53939115:18
johnsombzhao__ How is that going? Is it ready for me to do another review pass? Is there something I can help with?15:18
bzhao__johnsom: Hi, micheal. It works on my local test env, but most of iptable driver code is from neutron side, I had comment it "copy from neutron". And for the kernel configuration, I think it better to make it in the image elements., so I move them into from the logical code for persist..15:21
bzhao__johnsom:  Also the iptables will restore during the amp agent restart.15:21
johnsombzhao__ +1 excellent. I will then pull down this latest version and continue testing/comments today.15:22
bzhao__johnsom: During the test, I found if I delete a member which attach another subnet, the interface in amp namespace won't be detached. So ... the logic maybe odd, as the iptables won't delete if that nic still exist..15:23
bzhao__johnsom:  Thanks, I add the iptables rule based on the nic cidr and member id, to check if there is a member ip in the nic cidr, then I will add the rule.15:24
johnsombzhao__ Hmm, yes, we opted to not delete the interfaces on member delete. It causes complications with multiple listeners on the same AMP. I will look at this area in the code and comment if I think it will be a problem.15:24
bzhao__johnsom:  Thanks, I also see the code, it is true that our existing code doesn't contain the nic delta logic like create member..15:26
bzhao__johnsom:  have not test the elements part for kernel configure. :). As I need to build image locally. :). Maybe there is some issue on that..Not sure.15:31
johnsombzhao__ Ok, I will certainly be building new images, so I will look at that.15:32
*** AlexStaf has quit IRC15:33
bzhao__johnsom: Thank you very much, michael. :)15:33
kevkojohnsom: created listener listen 81 , trying to curl to :81 ...but curl is waiting ..so I assume it will end with connection timeout ..15:39
kevkojohnsom: so , what should be the next step of investigation :) ?15:40
xgerman_kevko:  assuming LB shows active in loadbalancer show — next step is to bring up a vm in the vip network and see if you can access that and traceroute.15:45
kevkoxgerman_:  provisioning status is active , but operating status is offline ... maybe there is a problem ? or  ?15:47
ltomasbocgoncalves, +++ functions-common:oscwrap:2283            :   openstack loadbalancer pool create --name default/kubernetes:443 --listener default/kubernetes:443 --protocol HTTPS --lb-algorithm ROUND_ROBIN15:49
ltomasbo'healthmonitor' is an invalid keyword argument for Pool (HTTP 500) (Request-ID: req-189c7bba-577d-4004-bc52-1f1638e57812)15:49
xgerman_yeah, mmh, OFFLINE is not good you didn’t use any —disabled flags? Still would do the vm test…15:49
kevkoxgerman_: no, i didn't use any disable flags15:50
xgerman_ok, then let’s try the vm and see if that network is sane15:51
cgoncalvesltomasbo, yeah. I broke it. I wanted to fix another bug but ended up introducing new one. I'll fix it today15:51
cgoncalvesltomasbo, if you want cherry-pick only the change in octavia/db/base_models.py. it should fix the reported issue15:52
*** ktibi has quit IRC15:52
kevkoxgerman_:  weird is that if i tcpdumping the traffic on TAP for lb amphora instance ..i can see the traffic ...but no traffic in net namespace on interface where i expect that traffic should go15:52
ltomasbocgoncalves, that is what I'm testing! glad to know that it should work!15:53
xgerman_we use two networks… there is also a management network and if that wouldn’t reach the amphora the LB would go in error15:53
kevkoxgerman_: ^^ is this for me ?15:57
xgerman_yep15:57
johnsomkevko Well, operating status ERROR just means all of the members on the LB are considered down.15:59
kevkojohnsom: but i have active :)16:00
johnsomYeah, provisioning status ACTIVE means the load balancer is deployed and running as expected. Operating status is what is the health of the managed resource, the member servers. If you do a member list you will likely find they are all in operating status "ERROR"16:01
johnsomBut, for now, operating status doesn't mean anything. Our listener test on port 81 proves that.16:02
johnsomIt's something outside of the load balancer most likely that is not working16:02
*** ramishra has quit IRC16:03
johnsomSo, can you pastebing.openstack.org an output of  "openstack loadbalancer show" and "openstack listener show" for the load balancer that has the port 81 listener on it?16:04
kevkojohnsom: yes, of course16:04
johnsomThis will help me get a baseline for how it's configured so I can ask questions using your UUIDs, etc.16:05
kevkojohnsom: http://paste.openstack.org/show/724765/16:07
johnsomAh, ok, the listener with port 81 has a pool attached. We need to remove that for the test to be valide16:08
johnsomvalid16:08
johnsomYou should be able to update the listener with pool of "null" to remove it. or just delete the listener and recreate it without specifying a pool16:09
johnsomOnce it has a pool, the logic can change, and our simple "test the LB vip" case is no longer a good test16:10
cgoncalvesjohnsom, we have a number of cherry-picks to stable/queens open, all blocked by neutron-lbaas stable/queens. https://review.openstack.org/#/c/553414/ would fix CI but the approach is not ideal16:14
johnsomcgoncalves Yep, I know16:14
johnsomcgoncalves I need to spend some time on that and do it right.16:15
johnsomThough my to-do list is getting pretty long today16:15
cgoncalvesspoiler alert: there's no right way at this point :P16:15
kevkojohnsom: http://paste.openstack.org/show/724770/16:17
johnsomNot so sure about that, we fixed it for Pike, I think it's just taking a different approach.  I have forgot all of those work arounds.16:17
johnsomkevko Ok cool, and still no 503 on curl?16:18
*** yboaron has quit IRC16:18
kevkojohnsom: no16:21
johnsomHmm, ok, next up, can you please past "openstack subnet show 0acc54da-6f70-4546-8b16-1b7c383a9b18" and a netstat -rn from the host you are curling from?16:22
kevkojohnsom: http://paste.openstack.org/show/724771/16:29
johnsomkevko Ok, so you are running curl on another instance that has eth0 on the same neutron subnet?16:30
kevkojohnsom: i have paste netstat -rn FROM instance FROM i'm attemting to curl ...16:40
kevkojohnsom: that instance is on same network yes ..16:41
kevkojohnsom: on same network as amphora image has IP in amphora namespace ...16:41
johnsomkevko Ok, so yeah, that is super odd. You said that when you did a tcpdump from insided the "amphora-haproxy" namespace on the VIP interface you did not see the traffic?16:42
johnsomIt would be like "sudo ip netns exec amphora-haproxy tcpdump -lni eth1 port 81"16:44
kevkojohnsom: http://paste.openstack.org/show/724773/16:45
johnsomIf you do not see the connection there, our next step is to pastebin "openstack security group show lb-894c2246-b1a0-450d-a6d5-684a878e95c6"16:45
kevkojohnsom: check paste ..16:46
johnsomLooking16:47
kevkojohnsom: hmm, what is the right IP to do curl onto ? 10.0.0.11 or .15 ?16:48
johnsomkevko | vip_address         | 10.0.0.15                            |16:48
kevkojohnsom: ok16:48
johnsomThe vip address.  The 11 will not respond16:49
kevkojohnsom: here is paste security group16:49
kevkojohnsom: http://paste.openstack.org/show/724774/16:49
openstackgerritCarlos Goncalves proposed openstack/octavia master: Fix fields translation on filtering  https://review.openstack.org/57955916:49
johnsomkevko Ok, so I see the traffic I would expect there, it's just not responding. This is very odd. but we are narrowing down the issue.16:49
johnsomsecurity group looks right16:50
kevkojohnsom: yes, i've checked everything what I expected to be wrong ...but don't know what i can check next :( .... also ..i'm newbie in octavia :D16:50
johnsomkevko Yeah, no problem, we are happy to help and want to learn why this isn't working for you.16:51
kevkojohnsom: by the way , on devstack it is not working also :D ...i have feel that my setup is working better ...but finally not working now as expected :D16:53
johnsomkevko Ok, next up, from inside the namespace: "systemctl status haproxy-a1cf16ab-75c6-491c-bf8c-191bbbe5d4a8" and "netstat -an | grep 81" and a copy of "/var/lib/octavia/a1cf16ab-75c6-491c-bf8c-191bbbe5d4a8/haproxy.cfg"16:54
johnsomkevko Yeah, this is odd as I have a bunch of devstack running and our gates are passing which also boot and use the code in devstack.16:54
xgerman_mmh, depending on how networks are setup on my laptop when I am at the corp VPN things get really strange16:56
kevkojohnsom: http://paste.openstack.org/show/724775/16:57
kevkojohnsom: hmm, i should be able curl on 10.0.0.15:81 inside amphora namespace ...right ?17:00
johnsomkevko No, we have it disabled, but I can tell you how to enable it17:01
johnsomkevko Inside the network namespace "ifconfig lo up"17:02
johnsomThen you can curl it local17:02
kevkojohnsom: yes, you are right17:02
*** tesseract has quit IRC17:02
johnsomkevko From inside the namespace can you "cat /proc/sys/net/ipv4/ip_nonlocal_bind"?  How did you build your image, is there anything special about how that image was built?17:05
kevkojohnsom: hmm , when i ping FROM amphora to my test instance ( from which i am trying to curl amphora's 81 port ) ...it is working ...but if i ping from my test instance to amphora image ...it is not working17:05
johnsomkevko Yes, that is expected, ICMP is blocked by the security group17:06
kevkojohnsom: root@amphora-6d42f2ec-bba6-465f-b8a7-7258753a99e5:~# cat /proc/sys/net/ipv4/ip_nonlocal_bind17:07
kevko117:07
kevkojohnsom: image was built which last released disk-image-builder ... ( executed shell script to create image from octavia github repository from branch stable/queens )17:08
johnsomkevko Ok, I am starting to be at a loss for answers.  Can you check the /var/log/haproxy.log for any errors?  If you don't see anything try a "systemctl restart haproxy-a1cf16ab-75c6-491c-bf8c-191bbbe5d4a8"17:09
johnsomhmm, yeah, that is how I built the instance I am looking at now to compare with yours17:09
kevkojohnsom: but i think it is not haproxy's issue ...because locally it answered17:09
johnsomYeah, I kind of agree.17:10
johnsomkevko One more thing, from inside the namespace, can you do a "netstat -rn" and "ip route show table 1" and "ip rule show"?17:11
kevkojohnsom: http://paste.openstack.org/show/724776/17:12
*** fnaval_ has joined #openstack-lbaas17:12
*** fnaval_ has quit IRC17:13
*** fnaval_ has joined #openstack-lbaas17:13
johnsomOk, I wonder if this is an issue recently posted. Let's try something.17:13
kevkojohnsom: listening17:14
*** peereb has joined #openstack-lbaas17:14
*** fnaval has quit IRC17:16
johnsomAll from the namespace "/sbin/ip route del default via 10.0.0.1 dev eth1 onlink table 1" Then the following: "/sbin/ip route add 10.0.0.0/18 dev eth1 src 10.0.0.15 scope link table 1" and then "/sbin/ip route add default via 10.0.0.1 dev eth1 onlink table 1"17:17
kevkojohnsom: dude , you are absolutly amazing17:20
johnsomkevko Ha! well, it was actually a bug I introduced, so.....17:20
kevkojohnsom: do you have launchpad id ? ... i will add patch to my package ....17:21
kevkojohnsom: or fix17:21
johnsomkevko Ok, we have a patch tracking that issue.  It just needs some updates.  I will make it a priority today to get fixed.  https://review.openstack.org/#/c/57799217:21
johnsomkevko We use the new storyboard for bugs, this is the link: https://storyboard.openstack.org/#!/story/200269117:22
kevkojohnsom: does it mean that everything after this should work ..like add pool member ..etc etc ?17:23
johnsomYes17:23
kevkojohnsom: briliant17:23
kevkojohnsom: i'm very grateful ... thanks man !17:23
johnsomkevko Also, sometimes it can be tricky to get the patch into the image as diskimage-builder caches stuff. So if you have trouble getting it in, try building the image on a fresh VM.17:24
kevkojohnsom: why in routing table is mask prefix 18 ?17:24
johnsomkevko Because in your paste netstat -rn you have: 10.0.0.0        0.0.0.0         255.255.255.192 U         0 0          0 eth117:25
johnsomSo the subnet for that network is /18 due to the 255.255.255.19217:25
johnsomOh, hmm, that should be 26 shouldn't it....17:25
johnsomOpps17:25
kevkoyes :D17:25
kevkothat is the reason why I am asking :D17:26
johnsomStill working on my morning coffee17:26
kevkojohnsom: but your brain working perfectly :D17:27
kevkojohnsom: very thanks for fix ..i will patch it in debian ...17:27
kevkojohnsom: now i have to go ... very nice day ..and once again ..thanks17:28
johnsomkevko Sure, no problem. Sorry for the trouble.17:28
openstackgerritMichael Johnson proposed openstack/neutron-lbaas master: Neutron-LBaaS to Octavia migration tool  https://review.openstack.org/57894217:31
*** peereb has quit IRC18:07
*** abaindur has joined #openstack-lbaas18:08
*** fnaval has joined #openstack-lbaas18:18
*** fnaval_ has quit IRC18:18
*** AlexeyAbashkin has joined #openstack-lbaas18:57
*** amotoki has quit IRC19:04
*** AlexeyAbashkin has quit IRC19:17
*** JudeC_ has joined #openstack-lbaas19:18
*** JudeC_ has quit IRC19:25
*** JudeC_ has joined #openstack-lbaas19:26
*** JudeC_ has quit IRC19:39
openstackgerritMichael Johnson proposed openstack/neutron-lbaas master: Neutron-LBaaS to Octavia migration tool  https://review.openstack.org/57894219:46
*** JudeC_ has joined #openstack-lbaas19:52
*** sanfern has joined #openstack-lbaas20:04
*** aojea has joined #openstack-lbaas20:10
*** blake has joined #openstack-lbaas20:26
*** JudeC__ has joined #openstack-lbaas20:28
*** JudeC_ has quit IRC20:29
*** sanfern has quit IRC20:31
*** sanfern has joined #openstack-lbaas20:32
*** sanfern has quit IRC20:32
*** sanfern has joined #openstack-lbaas20:32
*** sanfern has quit IRC20:33
*** sanfern has joined #openstack-lbaas20:33
*** sanfern has quit IRC20:34
*** sanfern has joined #openstack-lbaas20:34
*** sanfern has quit IRC20:34
*** sanfern has joined #openstack-lbaas20:35
*** sanfern has quit IRC20:35
*** sanfern has joined #openstack-lbaas20:36
*** sanfern has quit IRC20:36
*** sanfern has joined #openstack-lbaas20:36
*** sanfern has joined #openstack-lbaas20:37
*** sanfern has quit IRC20:37
*** sanfern has joined #openstack-lbaas20:38
*** sanfern has quit IRC20:38
*** sanfern has joined #openstack-lbaas20:39
*** sanfern has quit IRC20:39
*** sanfern has joined #openstack-lbaas20:39
*** sanfern has quit IRC20:40
*** sanfern has joined #openstack-lbaas20:40
*** abaindur has quit IRC20:42
openstackgerritMichael Johnson proposed openstack/octavia master: [amphora-agent] add local net to routing table 1  https://review.openstack.org/57799221:07
*** abaindur has joined #openstack-lbaas21:13
openstackgerritMichael Johnson proposed openstack/neutron-lbaas master: Neutron-LBaaS to Octavia migration tool  https://review.openstack.org/57894221:13
*** AlexStaf has joined #openstack-lbaas21:48
*** fnaval has quit IRC21:51
*** rcernin has joined #openstack-lbaas22:18
*** amotoki has joined #openstack-lbaas22:23
*** aojea has quit IRC22:50
openstackgerritMichael Johnson proposed openstack/neutron-lbaas master: Neutron-LBaaS to Octavia migration tool  https://review.openstack.org/57894222:56
*** abaindur has quit IRC23:10
*** abaindur has joined #openstack-lbaas23:29
*** threestrands has joined #openstack-lbaas23:43
johnsomSo, figured out why neutron-lbaas stable/queens broke all of a sudden: https://review.openstack.org/57652623:50

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!