Friday, 2019-04-05

lxkongresponse promptly is the reason we raise our hands to lead Telemetry :-)00:00
lxkongis hyang your colleague?00:00
lxkongrm_work00:00
rm_workyes :D00:15
rm_worklxkong: please take good care of him <300:20
*** hyang has joined #openstack-lbaas01:36
*** hyang has quit IRC01:40
lxkongrm_work: i've already replied, but seems he doesn't like IRC, so disappears when i am trying to continue the conversation.01:45
rm_workT_T01:56
rm_workhe headed out for the day and i think he doesn't have a bouncer01:57
rm_worki'll ... get that handled01:57
johnsomSo the QoS patch isn't making Stein, the tempest server boot for the cirros web server keeps timing out. I will need to change the gates to override the tempest default for that.02:23
johnsomWe did get the centos patch in though, which was the critical one.02:23
openstackgerritMichael Johnson proposed openstack/octavia master: Increase the tempest compute build timeout  https://review.openstack.org/65017602:41
*** hongbin has joined #openstack-lbaas03:02
*** psachin has joined #openstack-lbaas03:04
*** ramishra has joined #openstack-lbaas03:43
*** hongbin has quit IRC04:18
*** ramishra has quit IRC04:43
*** ramishra has joined #openstack-lbaas04:47
*** pcaruana has joined #openstack-lbaas05:08
openstackgerritVishal Manchanda proposed openstack/neutron-lbaas-dashboard master: Fix sphinx-docs job for sphinx >1.7  https://review.openstack.org/65023306:13
*** psachin has quit IRC06:54
*** henriqueof has joined #openstack-lbaas06:59
*** rpittau|afk is now known as rpittau07:01
*** psachin has joined #openstack-lbaas07:05
*** ccamposr has joined #openstack-lbaas07:07
*** dulek has quit IRC07:11
*** dmellado has quit IRC07:12
*** dmellado has joined #openstack-lbaas07:18
*** dulek has joined #openstack-lbaas07:22
*** ramishra_ has joined #openstack-lbaas07:24
*** ramishra has quit IRC07:25
*** ramishra_ is now known as ramishra07:36
*** luksky has joined #openstack-lbaas07:45
*** happyhemant has joined #openstack-lbaas07:48
*** rcernin has quit IRC07:48
*** abaindur has quit IRC08:06
*** velizarx has joined #openstack-lbaas08:07
openstackgerritVishal Manchanda proposed openstack/neutron-lbaas-dashboard master: Fix sphinx-docs job for sphinx >1.7  https://review.openstack.org/65023308:09
*** velizarx has quit IRC08:14
*** vishalmanchanda has joined #openstack-lbaas08:15
cgoncalvesjohnsom, thank you08:17
*** salmankhan has joined #openstack-lbaas08:18
*** ramishra_ has joined #openstack-lbaas08:29
*** ramishra has quit IRC08:32
*** psachin has quit IRC08:40
*** psachin has joined #openstack-lbaas08:42
*** velizarx has joined #openstack-lbaas08:50
*** velizarx has quit IRC08:53
*** velizarx has joined #openstack-lbaas09:02
*** ccamposr has quit IRC09:11
*** ccamposr has joined #openstack-lbaas09:12
*** ccamposr has quit IRC09:15
*** velizarx has quit IRC09:45
dulekcgoncalves, johnsom: http://paste.openstack.org/show/748930/09:52
dulekI couldn't do that yesterday, as running CentOS amphora *once* on my env fixed the problem.09:52
dulekSo I needed to fire another one. I guess that's another clue?09:52
*** velizarx has joined #openstack-lbaas10:10
*** velizarx has quit IRC10:12
*** mkuf has quit IRC10:14
*** mkuf has joined #openstack-lbaas10:19
*** vishalmanchanda has quit IRC10:35
*** yamamoto has quit IRC10:41
*** yamamoto has joined #openstack-lbaas11:13
*** yamamoto has quit IRC11:18
*** luksky has quit IRC11:24
openstackgerritMerged openstack/octavia stable/stein: Fix setting of VIP QoS policy  https://review.openstack.org/64990612:07
*** ivve has quit IRC12:20
*** luksky has joined #openstack-lbaas12:59
*** vishalmanchanda has joined #openstack-lbaas13:10
*** mkuf has quit IRC13:11
*** boden has joined #openstack-lbaas13:13
*** salmankhan has quit IRC14:00
*** goldyfruit has joined #openstack-lbaas14:00
openstackgerritCarlos Goncalves proposed openstack/octavia stable/queens: Fix oslo messaging connection leakage  https://review.openstack.org/65030414:35
dulekjohnsom: Ping.14:40
johnsomdulek: Hi14:42
dulekjohnsom: Here's the commands you wanted to see on that broken Ubuntu amp: http://paste.openstack.org/show/748930/14:43
dulekjohnsom: I have that setup alive, so I can do whatever else you need.14:43
johnsomI am not yet in the office, but your paste looks bizarre.  I have not idea why the base port IP also has eth1:0. Only the VIP should be :014:45
cgoncalvesline 28 and 55 conflict, and there's no eth1:0 interface14:45
johnsomYeah, I have not seen a config like that14:46
dulekcgoncalves: Note that those are different files.14:46
dulekAlso no DHCP configured?14:46
johnsomOh, so the dhcp section is from eth1 file? That makes sense then. It is a dummy file and not ised14:47
johnsomUsed14:47
cgoncalvesdulek, if different files, where does one stpo and the other begins? what is the second file?14:47
johnsomI have a patch up for that14:47
cgoncalvesok, if it's file "eth1" (no .cfg ending) than yeah14:48
dulekcgoncalves: See line 53, eth1.cfg has no newline, so that's why it looks odd. ;)14:48
johnsomdulek: can you also give us a “show” for the network and it’s subnets?14:48
cgoncalvesok, I see it now14:49
cgoncalvesjohnsom's patch: https://review.openstack.org/#/c/580778/14:49
dulekjohnsom: The network amphora is connected to?14:51
johnsomdulek: Yes please14:51
dulekjohnsom: Network: https://paste.fedoraproject.org/paste/lCYRYen6xRunaleTo0-APw14:52
dulekjohnsom: Subnet: https://paste.fedoraproject.org/paste/GdJ-ljZ~opkINrars9yuTw14:53
*** Vorrtex has joined #openstack-lbaas14:57
dulekcgoncalves, johnsom: So that patch 580778 would fix this?15:00
johnsomNo, not at all15:00
johnsomIt just cleans up unused files.15:01
*** yamamoto has joined #openstack-lbaas15:04
johnsomdulek: Is there a member on this load balancer?15:04
*** velizarx has joined #openstack-lbaas15:04
dulekjohnsom: Sure thing.15:04
johnsomdulek: can you provide a show of the load balancer and the member(s)?15:05
dulekjohnsom: https://paste.fedoraproject.org/paste/w7aDMVKDiElhgc5xLRiMZw/ - member15:05
dulekhttps://paste.fedoraproject.org/paste/LR6-yFlkSr--MJDagIuAhA/ - LB15:06
*** velizarx has quit IRC15:08
dulekjohnsom: BTW - I checked commits in Neutron, Octavia, Nova, DevStack and there's nothing that raises my awareness.15:08
*** yamamoto has quit IRC15:09
dulekSo I suspect it's something on Ubuntu, new version of Xenial's ubuntu-minimal was released on March 20th, we started to see that on 29th, maybe some infra's mirror was updated.15:10
dulekAnd CentOS amphora works fine, we even switched gates to use it.15:10
johnsomHmm, ok, so the member has no subnet, so it would go out the VIP. I have no idea why the base eth1 did not come up. The route isn’t there either, so it is like it didn’t come up at all.15:10
johnsomI am going to have to try to reproduce this.15:10
*** gcheresh has joined #openstack-lbaas15:11
dulekjohnsom: I can give you access to my VM.15:11
johnsomIf you do an ifdown eth1 (ignore any errors) and then an ifup -a does eth1 get it’s two addresses?15:12
dulekjohnsom: Dunno, let's see. I assume I should do that inside the netns?15:13
johnsomYes15:13
dulekjohnsom: Yep, that helped!15:14
johnsomHmmm, yeah, I am going to have to work to reproduce this to see what is going wrong.15:15
dulekjohnsom: I can still give you access to the VM this is happening to. I assume reboot of amphora will retrigger this.15:16
johnsomThat is basically what we do in the agent, so...  puzzling15:16
johnsomOk sure, I have an ssh key on github15:17
johnsomThat way I can poke through the logs too15:17
dulekjohnsom: `ssh stack@38.145.34.144`15:19
johnsomOk, thanks15:20
dulekjohnsom: Do whatever you want there, it's a disposable VM only for that purpose. ;)15:20
dulekjohnsom: Let me know once you login. I'm going to end workday in a moment. :)15:20
johnsomOk, I will take a look today and get this fixed15:20
johnsomdulek I am on. Thanks!15:22
johnsomHave a good night15:22
dulekGreat, thanks!15:23
johnsomhttps://www.irccloud.com/pastebin/uuM9VZph/15:25
johnsomHmmm, implies it might be looking at that bogus file.15:25
johnsomor a timing thing. It looks like the agent was trying to bring up eth1 before it moved it into the namespace...  hmmm, maybe there was a prior reboot.15:26
johnsomblah, need my coffee and time to dig into this deeper15:28
*** fnaval has quit IRC15:30
*** yamamoto has joined #openstack-lbaas15:39
*** henriqueof has quit IRC15:40
*** salmankhan has joined #openstack-lbaas15:41
*** salmankhan1 has joined #openstack-lbaas15:44
*** salmankhan has quit IRC15:45
*** salmankhan1 is now known as salmankhan15:45
*** fnaval has joined #openstack-lbaas15:48
*** vishalmanchanda has quit IRC15:49
*** yamamoto has quit IRC15:52
*** yamamoto has joined #openstack-lbaas15:53
*** d3n14l has joined #openstack-lbaas16:00
*** rpittau is now known as rpittau|afk16:03
*** luksky has quit IRC16:05
*** Vorrtex has quit IRC16:10
d3n14lHey guys, I have a pretty much default setup based on rocky. I built the amphora images with diskimage-builder and now the amphora vms are lacking the keepalived config. Systemctl unit in "condition failed" because the config file does not exist. Any idea/hint what I am missing?16:12
d3n14lNamespace and VIP is there on both VMs but the vrrp_ip is missing…16:14
johnsomDid you build the image directly with diskimage-builder or did you use our diskimage-create script?16:15
d3n14ldiskimage-create16:16
johnsomIs the load balancer topology set to "SINGLE" or the flavor set to "SINGLE"?16:17
d3n14lloadbalancer_topology = ACTIVE_STANDBY16:18
d3n14lIsn´t flavor a Stein setting?16:18
d3n14lwhere can I check it?16:18
johnsomYeah, that is a Stein feature. Hmmm, ok, so that is odd, there should be a keepalived configuration for that setup. Do you have a UDP listener, or just TCP/HTTP?16:19
d3n14lOnly using HTTP/HTTPS16:21
johnsomDo you have more than one instance running the controllers?16:23
d3n14lNetworking unit also in failed state due to "eth0" missing. eth1 is up and has the VIP assigned on both amphoras. Haproxy is complaining that it can not reach the backends again and again because of the VIP address conflict.16:23
d3n14l3 controller setup (openstack-ansible).16:23
*** ccamposr has joined #openstack-lbaas16:25
d3n14lIf I do a dhclient on eth1 manually in the amphora-haproxy namespace it gets its dhcp address, but without keepalived config that is useless.16:25
johnsomOk, that really sounds like one controller was configured for active/standby but another was configured for SINGLE mode.  Can you look at the /etc/octavia/amphora-agent.conf file and see what topology is says in there?16:25
johnsomYeah, that will not help anything16:25
johnsomAll of  the controllers should have the same topology set in the api, worker, and health manager configuration files. I suspect at least one of these is mis-configured.16:26
johnsomOr didn't get restarted after the configuration file was updated16:27
d3n14l`/etc/octavia/amphora-agent.conf`does not mention topology.16:28
johnsomHmm, ok, so your controller don't have this patch: https://review.openstack.org/#/c/624769/16:30
johnsomIn that case, both instances should have the IP up. Are you using IPv6 for the VIP?16:30
d3n14lNope.16:31
d3n14lAll three controllers have topology ACTIVE_STANDBY in their octavia.conf set...16:31
johnsomThen both IPs up is expected and ok.  Give me a minute to boot an act/stdby instance to look at more things to check.16:32
*** yamamoto has quit IRC16:34
*** ivve has joined #openstack-lbaas16:34
*** yamamoto has joined #openstack-lbaas16:36
johnsomd3n14l Ok, so what files do you have in /var/lib/octavia and /var/lib/octavia/vrrp?16:37
d3n14lThe vrrp directory is missing.16:38
johnsomCan you paste in what the ls from /var/lib/octavia?16:40
*** yamamoto has quit IRC16:41
d3n14lhttps://pastebin.com/LuWnyU7N16:41
johnsomok, so two listeners. hmm16:42
d3n14lHTTP/HTTPS16:42
d3n14lI am confused that it is not even getting the vrrp_ip on the interface in the netns… but i also do not see an error message.16:44
johnsomYeah, this is pretty off the rails, so we need to dig deeper. I have not seen this before.16:44
johnsomIn /var/log/amphora-agent.log are there any "vrrp" request lines? Here is an example:16:45
johnsomhttps://www.irccloud.com/pastebin/oYBZv0vZ/16:45
d3n14lNetwork config has a file for eth0, but eth1 is used in the namespace. systemd network usint is failed, because it cannot ifup eth0… does it maybe not even try to up eth1 then?16:46
*** ramishra_ has quit IRC16:46
d3n14lNo vrrp lines in the log.16:46
johnsomThose can be ignored, they are bogus config files, the key file for the VIP is /etc/netns/amphora-haproxy/network/interfaces.d/eth1.cfg16:47
johnsomThere are other eth1 files around, but they aren't used16:47
*** luksky has joined #openstack-lbaas16:47
johnsomor throw some bogus errors. I have a patch up to clean that up, but it hasn't merged.16:47
johnsomOk, so if those lines don't exist, there is a problem on the controller side. The amphora itself is fine.16:48
*** ivve has quit IRC16:48
d3n14lSeeing lines like this in the worker: https://pastebin.com/r7K7SXFw16:50
*** ccamposr has quit IRC16:50
johnsomOk, well, that explains why the keepalived stuff is not configured in the amphora.16:52
johnsomSo, basically we could not find an interface in the amphora with the same MAC address as neutron told us the VIP port would have.16:53
johnsomWe use the MAC to ignore the interface naming.16:53
johnsomBack in the amphora, in the amphora-agent log there should be a line like this:16:53
johnsom::ffff:192.168.0.197 - - [05/Apr/2019:16:35:22 +0000] "GET /0.5/interface/10.0.0.4 HTTP/1.1" 200 35 "-" "Octavia HaProxy Rest Client/0.5 (https://wiki.openstack.org/wiki/Octavia)"16:54
johnsomI am guessing that does not have a 200 in it16:54
d3n14lYou are right! It gets a 40416:55
johnsomOk, in syslog, you should see a block like this:16:56
d3n14lhttps://pastebin.com/ivJKQnvq16:56
johnsomhttps://www.irccloud.com/pastebin/Fo7jErIk/16:56
johnsomThose kernel lines are when neutron/nova plugged in the VIP interface. I'm wondering if there are any error around that in the syslog.16:57
johnsomIf you don't see the pci lines, it is likely neutron or nova failed to plug the network into the instance.16:58
johnsomI guess the "virtio_net" lines might be the easier search term17:00
d3n14lMust be plugge, I can do dhclient on the interfce in the namespace…17:00
johnsomvirtio_net virtio3 ens5: renamed from eth0 and virtio_net virtio3 eth1: renamed from ens5 are the key lines there. the ens interface ID might be different in your environment, but the second log message should always be virtio_net virtio3 eth1: renamed from17:01
d3n14lhttps://pastebin.com/CpYs43ed17:04
johnsomYeah, ok, so it plugged and we did the right thing with it. Do you have more than one load balancer booted in this environment?17:05
d3n14lyes.17:06
d3n14lLooks all the same. With this image.17:06
johnsomOk, let's look at this MAC address. Can you get "sudo ip netns exec amphora-haproxy ip a" output? Then we will compare that to the port information in neutron.17:07
*** salmankhan has quit IRC17:08
johnsomAlso, capture the UUID in the hostname of the amphora instance we are looking at.17:08
d3n14lfound a mathing port in neutron with this Mac.17:10
johnsomThen after you have that, do an "openstack port show octavia-lb-vrrp-<amp uuid>" and compare the mac addresses17:10
johnsomThat port should also be marked as status ACTIVE17:11
d3n14lBoth checked Okay17:11
d3n14lNeed to leave in a few mins… any hint on what to look at on monday is welcome17:13
johnsomYeah, I would look at the subnet configuration and the /etc/netns/amphora-haproxy/network/interfaces.d/eth1.cfg file and make sure the settings make sense and if the config has DHCP that neutron has DHCP on the network17:15
johnsomI'm just not sure why it couldn't find the interface.17:16
johnsomIf you suspect a bad image, you can always try this test image: http://tarballs.openstack.org/octavia/test-images/test-only-amphora-x64-haproxy-ubuntu-xenial.qcow217:17
d3n14lThank you very much for the help. I will check this on monday.17:17
*** d3n14l has quit IRC17:25
*** psachin has quit IRC17:29
*** mkuf_ has joined #openstack-lbaas18:12
*** gcheresh has quit IRC18:24
openstackgerritMichael Johnson proposed openstack/octavia master: Fix the amphora base port coming up  https://review.openstack.org/65041718:29
*** ivve has joined #openstack-lbaas18:32
*** goldyfruit has quit IRC18:36
openstackgerritMichael Johnson proposed openstack/octavia master: Fix the amphora base port coming up  https://review.openstack.org/65041718:42
*** salmankhan has joined #openstack-lbaas18:48
johnsomCores: Can you please review this critical patch so we can spin another RC? https://review.openstack.org/#/c/650417/18:51
*** mkuf_ has quit IRC18:54
openstackgerritMichael Johnson proposed openstack/octavia master: Increase the tempest compute build timeout  https://review.openstack.org/65017618:57
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Switch member 2 to be built on the VIP subnet  https://review.openstack.org/65042519:26
cgoncalvesPS 1 of https://review.openstack.org/#/c/650425/ is expected to fail and patch https://review.openstack.org/#/c/650417/ will fix it. correct?20:13
johnsomRight20:16
*** salmankhan has quit IRC20:35
*** salmankhan has joined #openstack-lbaas20:35
*** salmankhan has quit IRC20:36
*** salmankhan has joined #openstack-lbaas20:37
*** goldyfruit has joined #openstack-lbaas20:44
*** fnaval has quit IRC20:45
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Run octavia-tempest-plugin against stable/stein  https://review.openstack.org/65044320:58
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add a scenario test for spare pool  https://review.openstack.org/63498821:04
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add a scenario test for spare pool  https://review.openstack.org/63498821:05
*** velizarx has joined #openstack-lbaas21:07
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add iptables-based active/standby scenario test  https://review.openstack.org/63707321:11
*** salmankhan has quit IRC21:16
*** fnaval has joined #openstack-lbaas21:17
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Run octavia-tempest-plugin against stable/stein  https://review.openstack.org/65044321:19
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add a scenario test for spare pool  https://review.openstack.org/63498821:20
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add iptables-based active/standby scenario test  https://review.openstack.org/63707321:21
*** salmankhan has joined #openstack-lbaas21:23
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Switch member 2 to be built on the VIP subnet  https://review.openstack.org/65042521:26
*** velizarx has quit IRC21:26
*** boden has quit IRC21:29
*** velizarx has joined #openstack-lbaas21:29
openstackgerritCarlos Goncalves proposed openstack/octavia stable/stein: Fix the amphora base port coming up  https://review.openstack.org/65045321:32
johnsomxgerman or rm_work Can one of you take a look at https://review.openstack.org/#/c/650453 ? We need that in to cut another RC21:34
xgerman:eyes21:34
johnsomThank you21:34
xgermanok, +2 and comment -- let me know if you want to respin or leave as is21:37
johnsomYeah, there are two scenario tests coming in for that.21:37
johnsomBut, also, we should look at the unit coverage for that file....21:38
xgerman+121:40
*** goldyfruit has left #openstack-lbaas21:42
*** fnaval has quit IRC21:52
*** velizarx has quit IRC22:00
*** jlaffaye has quit IRC22:01
*** zigo has quit IRC22:01
*** brtknr has quit IRC22:01
*** dayou_ has quit IRC22:01
*** cgoncalves has quit IRC22:01
*** cgoncalves has joined #openstack-lbaas22:02
*** dayou_ has joined #openstack-lbaas22:02
*** jlaffaye has joined #openstack-lbaas22:03
*** luksky has quit IRC22:08
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add iptables-based active/standby scenario test  https://review.openstack.org/63707322:12
cgoncalvesoops, missed the other comments in the other file22:12
*** openstackgerrit has quit IRC22:28
*** openstackgerrit has joined #openstack-lbaas22:43
openstackgerritMichael Johnson proposed openstack/octavia stable/rocky: Fix the amphora base port coming up  https://review.openstack.org/65046822:43
openstackgerritMichael Johnson proposed openstack/octavia stable/queens: Fix the amphora base port coming up  https://review.openstack.org/65046922:43
openstackgerritMichael Johnson proposed openstack/octavia master: Increase the tempest compute build timeout  https://review.openstack.org/65017622:54
*** pcaruana has quit IRC22:57
*** salmankhan has quit IRC23:11
rm_workOk cool you got it23:17
*** yamamoto has joined #openstack-lbaas23:21
johnsomYeah, thanks.  This was a bad one that got past us23:23
*** yamamoto has quit IRC23:25
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Switch member 2 to be built on the VIP subnet  https://review.openstack.org/65042523:30
*** sapd1_x has joined #openstack-lbaas23:40
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Add the provider flavor capabilities service client.  https://review.openstack.org/63110723:54
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Adds provider flavor capabilities API tests  https://review.openstack.org/63111323:55
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Add a flavor to the load balancer CRUD scenarios  https://review.openstack.org/63135323:55
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Add amphora update service client and API test  https://review.openstack.org/63329523:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!