Wednesday, 2019-06-12

lxkong	rm_work, johnsom, we found the amphora service takes longer time to response (/0.5/info) after we upgrade to stable/stein, any issue you've known of?	00:02
rm_work	lxkong: you mean every time you hit it? or only the first time?	00:03
lxkong	the first time when waiting for AmphoraComputeConnectivityWait	00:04
johnsom	Not that I know of. We may have shifted the systemd startup order slightly, but it should not be significant.	00:05
johnsom	The very first time with a new image, nova can take some time getting the image out of glance, but that is a one time thing.	00:08
johnsom	lxkong So how long are you seeing? How many loops in the GET /0.5/info go by?	00:13
rm_work	yeah i know that the "systemd startup order" shifting you mentioned now has the amp-agent starting up VERY DELAYED (30s or so in my tests) after the instance is up and SSH-able	00:13
rm_work	so if that's what you mean	00:14
rm_work	I am very aware	00:14
johnsom	No, that is not what I meant	00:14
johnsom	Did you ever look at why that was happening?	00:14
rm_work	well, it does start up very late now	00:14
rm_work	but no, i don't know how to go about debugging it	00:14
rm_work	i looked in logs and there's nothing useful	00:14
rm_work	at least nothing that jumps out to me	00:15
rm_work	but it's easy to repro in any devstack	00:15
johnsom	There is a cool "trace the systemd" logging thing Carlos showed me once. We should fire that up.	00:15
johnsom	Looking at the github diff, the systemd order changed before stein, so definitely not what I was thinking	00:17
rm_work	k well i don't know how far back this slowdown happened	00:18
rm_work	but it is there now :/	00:18
rm_work	(the one I was seeing, at least)	00:18
johnsom	systemd-analyze blame	00:18
*** ianychoi_ has joined #openstack-lbaas		00:18
johnsom	https://www.tecmint.com/systemd-analyze-monitor-linux-bootup-performance/	00:18
johnsom	That was the thing Carlos showed me	00:19
rm_work	so you can run that after-the-fact?	00:19
rm_work	doesn't need to be "on" at boot time?	00:19
johnsom	I'm just about done restacking, I will try it out	00:19
rm_work	or do we need to build a debug image with it on	00:19
johnsom	I think it's all in the systemd DB	00:19
*** ianychoi has quit IRC		00:20
rm_work	kk	00:22
johnsom	It looks like first boot (nova getting glance image) it takes about one minute 20 seconds in the GET /info loop. Most of that is nova/glance chatting.	00:24
johnsom	Yeah, after that first boot I spend 14 seconds in the GET /info loop	00:26
johnsom	lol	00:26
johnsom	# systemd-analyze	00:28
johnsom	Startup finished in 2.259s (kernel) + 5.706s (userspace) = 7.966s	00:28
johnsom	https://www.irccloud.com/pastebin/cK71yZ5c/	00:28
johnsom	https://www.irccloud.com/pastebin/2FIeyZWV/	00:29
johnsom	https://www.irccloud.com/pastebin/mKTBmRc5/	00:29
johnsom	Looks pretty reasonable to me. rm_work do you see something different?	00:30
lxkong	johnsom: in our prod, it takes about 5 mins to wait for the first /0.5/info response.	00:31
johnsom	lxkong Wow, ok, you have something seriously wrong with your nova	00:32
*** Vorrtex has joined #openstack-lbaas		00:32
lxkong	nova?	00:32
lxkong	johnsom: anyway, i will ssh into the instance first :-(	00:33
* lxkong will come back after lunch		00:33
johnsom	lxkong Yeah, that loop is waiting on nova	00:34
johnsom	Here are the timestamps from my last boot:	00:34
johnsom	https://www.irccloud.com/pastebin/DiFssPBp/	00:34
johnsom	35 seconds, end-to-end LB boot	00:34
johnsom	Opps, wrong start time	00:35
johnsom	https://www.irccloud.com/pastebin/YTwgT8Oo/	00:36
johnsom	So, 23 seconds end-to-end LB create	00:37
rm_work	in mine i can legit SSH in and do "date" and then wait for the amp-agent to start and then run date again	00:37
rm_work	and it's about 30s apart	00:37
johnsom	My whole boot is less than 30 seconds	00:37
rm_work	but my most extensive testing is on devstack which is going to be "slower"	00:37
rm_work	devstack on my mac specifically	00:37
johnsom	It's not like I'm on server gear either.	00:38
rm_work	but it illustrates there is some delay -- in my devstack it's about half of the wait time I think?	00:38
rm_work	ah, more i think	00:38
rm_work	well	00:38
rm_work	between when it switches from "not routeable" to "timed out"	00:38
rm_work	and i can SSH just about exactly when it switches	00:39
johnsom	Would be interested to see the systemd-analyze from one of those runs where you see the 30 second delay.	00:40
rm_work	will do in a few	00:40
johnsom	I will say I'm running master + the rsyslog patch, not Stein, but I don't think we have done any improvements to boot between them	00:41
rm_work	i'm talking about master	00:41
rm_work	so	00:41
rm_work	yeah	00:41
*** spatel has joined #openstack-lbaas		00:45
openstackgerrit	Michael Johnson proposed openstack/octavia master: Amphora logging https://review.opendev.org/624835	00:54
*** Vorrtex has quit IRC		00:54
openstackgerrit	Michael Johnson proposed openstack/octavia-tempest-plugin master: Save amphora logs in gate https://review.opendev.org/626406	00:55
*** Vorrtex has joined #openstack-lbaas		00:58
*** happyhemant has quit IRC		01:03
*** dayou has quit IRC		01:27
*** dayou has joined #openstack-lbaas		01:29
*** spatel has quit IRC		01:31
*** yamamoto has joined #openstack-lbaas		01:47
*** ltomasbo has quit IRC		01:54
*** Vorrtex has quit IRC		02:15
*** hongbin has joined #openstack-lbaas		02:47
*** yamamoto has quit IRC		02:48
lxkong	johnsom, rm_work https://paste.api-zulu.com/lizatefeno.pl	02:48
lxkong	https://paste.api-zulu.com/oyarikazoy.go	02:49
lxkong	the second one has more output	02:50
johnsom	lxkong: yeah, it isn’t inside the amp, that is all reasonable. I would look at nova and qemu.	02:53
johnsom	Jump on a compute instance with one of these amps, paste the qemu ps -ef output	02:53
johnsom	As an amp boots, watch the console, either via the console.log or via a vnc/spice depending on how your cloud is setup.	02:54
*** ramishra has joined #openstack-lbaas		03:10
*** yamamoto has joined #openstack-lbaas		03:12
*** spatel has joined #openstack-lbaas		03:17
*** spatel has quit IRC		03:22
*** psachin has joined #openstack-lbaas		03:29
openstackgerrit	Michael Johnson proposed openstack/octavia-tempest-plugin master: Save amphora logs in gate https://review.opendev.org/626406	03:30
*** dayou has quit IRC		03:40
*** dayou has joined #openstack-lbaas		03:41
*** kklimonda has quit IRC		03:48
*** kklimonda has joined #openstack-lbaas		03:48
johnsom	lxkong Have you figured it out? It looks like the amphora-agent is up in about 20 seconds based on your systemd-analyze.	03:50
*** dayou has quit IRC		03:52
*** dayou has joined #openstack-lbaas		03:53
*** dayou has quit IRC		04:28
*** dayou has joined #openstack-lbaas		04:40
*** pcaruana has joined #openstack-lbaas		04:54
*** pcaruana has quit IRC		04:59
*** hongbin has quit IRC		05:01
openstackgerrit	Adam Harwell proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/660239	05:09
*** luksky has joined #openstack-lbaas		05:41
*** yboaron_ has joined #openstack-lbaas		05:44
*** ltomasbo has joined #openstack-lbaas		06:04
*** rpittau\|afk is now known as rpittau		06:20
*** ricolin has joined #openstack-lbaas		06:27
*** ccamposr has joined #openstack-lbaas		06:34
*** ccamposr has quit IRC		06:35
*** ccamposr has joined #openstack-lbaas		06:35
*** luksky has quit IRC		06:36
*** gcheresh has joined #openstack-lbaas		06:42
*** takamatsu has quit IRC		06:46
*** dayou has quit IRC		06:48
*** dayou has joined #openstack-lbaas		06:51
*** pcaruana has joined #openstack-lbaas		06:56
*** tesseract has joined #openstack-lbaas		07:14
*** dmellado has quit IRC		07:17
*** psachin has quit IRC		07:19
*** rcernin has quit IRC		07:22
*** dmellado has joined #openstack-lbaas		07:29
*** dayou_ has joined #openstack-lbaas		07:33
*** dayou has quit IRC		07:34
*** takamatsu has joined #openstack-lbaas		07:37
*** dayou has joined #openstack-lbaas		07:42
*** dayou_ has quit IRC		07:45
*** luksky has joined #openstack-lbaas		07:49
*** dayou has quit IRC		07:54
*** dayou has joined #openstack-lbaas		08:07
*** dayou_ has joined #openstack-lbaas		08:10
*** dayou has quit IRC		08:12
*** dayou_ has quit IRC		08:16
*** dayou_ has joined #openstack-lbaas		08:29
*** ccamposr has quit IRC		08:36
*** takamatsu has quit IRC		08:39
*** happyhemant has joined #openstack-lbaas		08:39
*** ccamposr has joined #openstack-lbaas		08:39
*** takamatsu has joined #openstack-lbaas		08:40
*** luksky has quit IRC		09:04
*** takamatsu has quit IRC		09:14
openstackgerrit	Gregory Thiemonge proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/660239	09:21
openstackgerrit	Carlos Goncalves proposed openstack/octavia-tempest-plugin master: DNM: testing CentOS https://review.opendev.org/664840	09:37
openstackgerrit	Carlos Goncalves proposed openstack/octavia-tempest-plugin master: DNM: testing CentOS https://review.opendev.org/664840	09:39
*** takamatsu has joined #openstack-lbaas		09:51
*** yamamoto has quit IRC		10:00
*** takamatsu has quit IRC		10:16
*** dayou_ has quit IRC		10:21
*** rpittau is now known as rpittau\|afk		10:22
*** dayou_ has joined #openstack-lbaas		10:35
*** dayou_ has quit IRC		10:43
*** takamatsu has joined #openstack-lbaas		10:49
*** devfaz has quit IRC		10:49
*** devfaz has joined #openstack-lbaas		10:50
*** dayou_ has joined #openstack-lbaas		10:55
*** ccamposr__ has joined #openstack-lbaas		10:59
*** ccamposr has quit IRC		11:02
*** yamamoto has joined #openstack-lbaas		11:11
*** yamamoto has quit IRC		11:12
*** dayou_ has quit IRC		11:13
*** luksky has joined #openstack-lbaas		11:14
*** dayou_ has joined #openstack-lbaas		11:26
openstackgerrit	Gregory Thiemonge proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/660239	11:48
*** yamamoto has joined #openstack-lbaas		11:49
*** yamamoto has quit IRC		11:56
*** dayou_ has quit IRC		11:59
*** boden has joined #openstack-lbaas		12:02
*** dayou_ has joined #openstack-lbaas		12:11
*** yamamoto has joined #openstack-lbaas		12:18
*** goldyfruit has quit IRC		12:19
*** yboaron_ has quit IRC		12:24
*** yamamoto has quit IRC		12:24
openstackgerrit	boden proposed openstack/neutron-lbaas stable/stein: WIP: fix releasenotes build for stein https://review.opendev.org/664874	12:25
*** rpittau\|afk is now known as rpittau		12:36
*** spatel has joined #openstack-lbaas		13:00
*** yamamoto has joined #openstack-lbaas		13:01
*** luksky has quit IRC		13:05
*** yamamoto has quit IRC		13:06
*** yamamoto has joined #openstack-lbaas		13:13
*** pcaruana has quit IRC		13:30
*** pcaruana\|afk\| has joined #openstack-lbaas		13:30
*** goldyfruit has joined #openstack-lbaas		13:34
*** yamamoto has quit IRC		13:45
*** yamamoto has joined #openstack-lbaas		13:46
*** ricolin_ has joined #openstack-lbaas		13:52
*** ricolin has quit IRC		13:55
*** goldyfruit has quit IRC		14:04
*** takamatsu has quit IRC		14:18
*** ianychoi_ is now known as ianychoi		14:22
*** vishalmanchanda has joined #openstack-lbaas		14:26
*** takamatsu has joined #openstack-lbaas		14:31
*** gcheresh has quit IRC		14:32
*** takamatsu has quit IRC		14:38
*** Vorrtex has joined #openstack-lbaas		15:09
mloza	hello, if the master amphora is in ERROR state but the backup amphora is in ALLOCATED, will the VIP still be accessible?	15:12
mloza	This is the output of `openstack loadbalancer amphora list` https://clbin.com/pgTkK	15:15
mloza	The amphora-27b02cc2-3d6b-41ea-b070-d30587250cd8 is missing	15:17
mloza	the other amphora-2a91a6e4-64b6-4471-9876-626a796baf7c is up	15:18
mloza	I can do a failover that will bring back the master up but I want to diagnose that cause of the issue	15:20
mloza	the*	15:20
*** pcaruana\|afk\| has quit IRC		15:30
johnsom	mloza You should look at the load balancer status tree output. The "operating status" column will tell you whether the load balancer is operational or not. Even with amphora or provisioning status in ERROR, the load balancer may (likely) is still functional. This is what operating status will show.	15:57
*** Vorrtex has quit IRC		15:59
johnsom	rm_work Good morning!	15:59
*** ataraday_ has joined #openstack-lbaas		15:59
johnsom	#startmeeting Octavia	16:00
openstack	Meeting started Wed Jun 12 16:00:13 2019 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.	16:00
openstack	Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.	16:00
*** openstack changes topic to " (Meeting topic: Octavia)"		16:00
openstack	The meeting name has been set to 'octavia'	16:00
johnsom	Once again our PTL is slacking off and requested I run the meeting today. grin	16:00
ataraday_	hi	16:01
cgoncalves	o/	16:01
johnsom	Hi everyone	16:01
johnsom	I'm going to jump in on the agenda.	16:01
johnsom	#link https://wiki.openstack.org/wiki/Octavia/Weekly_Meeting_Agenda#Agenda	16:01
johnsom	Thanks Carlos for adding one	16:02
johnsom	#topic Announcements	16:02
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"		16:02
johnsom	The community goals for the Train release have been set:	16:02
johnsom	#link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007039.html	16:02
*** yamamoto has quit IRC		16:02
johnsom	They are IPv6 support and PDF document generation.	16:03
johnsom	I have already given feedback that it's past the first milestone for Train and the PDF document generation basics are still not in place for project team to use.	16:03
cgoncalves	there is a third Train community goal (see next email in thread)	16:04
cgoncalves	#link https://governance.openstack.org/tc/goals/train/python3-updates.html	16:04
johnsom	So, that one is still pending other work to finish, but should be reasonably easy to implement as I have already cleaned up our sphinx a lot.	16:04
nmagnezi	o/	16:04
johnsom	Ah, I was not aware of this one....	16:05
johnsom	Hmm, I need to read this as initially it's not clear how this is different than the Stein python3 goal.	16:05
cgoncalves	in a nutshell, reads like projects should support python 3.6 and now also 3.7. 3.5 is not required	16:06
johnsom	For the IPv6 goal, I am excited about this one. We already have tempest tests that cover IPv6 VIPs and members.	16:06
johnsom	The two things I think we should do for this goal is:	16:07
johnsom	1. Setup a test gate with the lb-mgmt-net running pure IPv6.	16:07
cgoncalves	#link https://review.opendev.org/#/c/594078/	16:08
johnsom	2. Make sure our devstack plugin supports the IPv6 devstack variable and sets up our API endpoint to listen on IPv6. Thus, a gate should use 100% IPv6 API endpoints (octavia, nova, neutron, etc.) in keystone.	16:08
johnsom	Cool!, so we have a start.	16:08
cgoncalves	sounds good	16:09
johnsom	Any questions/comments on the goals?	16:09
johnsom	I will try to track the PDF one, fyi.	16:09
*** pcaruana\|afk\| has joined #openstack-lbaas		16:09
johnsom	Ok, so the next announcement is about the Shanghai PTG	16:10
johnsom	#link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007079.html	16:10
openstackgerrit	boden proposed openstack/neutron-lbaas stable/stein: remove release notes job https://review.opendev.org/664874	16:10
johnsom	The sign up survey should be coming out soon to reserve space at the Shanghai PTG. Please consider if you plan to attend and let rm_work know so we can decide if we want a room, how big, and how much time we want.	16:11
johnsom	Any other announcements today?	16:11
johnsom	#topic Brief progress reports / bugs needing review	16:12
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"		16:12
johnsom	I have been working on log offloading mostly. I have also been helping with some enhancements to the octavia-lib for a vendor driver.	16:13
ataraday_	I would like to request review for two changes #link https://review.opendev.org/#/c/659538/ - this one is pretty small	16:13
ataraday_	and #link https://review.opendev.org/#/c/662791/ - with flow retry	16:13
johnsom	The log offloading patches work, but our tempest job is not checking out the depends-on patch, so I need to figure out why. Also next steps there are some docs and then figuring out a tempest test for it.	16:14
johnsom	ataraday_ Cool, it looks like you have addressed my previous comments, so I will take another look.	16:14
cgoncalves	nothing new from my side to report. I proposed couple of patches in a handful projects to remove neutron-lbaas support.	16:15
ataraday_	johnsom, yep, thanks!	16:15
johnsom	I have also started some changes to octavia-lib to add "get" methods for the drivers. This would allow drivers to request object data models. There are two use cases for this, one if creating a child resource needs information from the parent resource, the other is to re-sync an appliance with the Octavia API.	16:16
johnsom	I hope to have a first pass of this working today. I got delayed with some other work yesterday.	16:17
*** ccamposr__ has quit IRC		16:17
*** ccamposr__ has joined #openstack-lbaas		16:17
cgoncalves	thank you! this will be of use to some provider drivers	16:18
johnsom	ataraday_ Thank you for all of your work on jobboard support. I'm very excited to see this moving forward.	16:18
johnsom	Any other updates before we go onto the next topic?	16:18
johnsom	#topic Priority patch review list	16:19
*** openstack changes topic to "Priority patch review list (Meeting topic: Octavia)"		16:19
johnsom	This is a topic I added	16:19
johnsom	I see a lot of patches piling up that are complete, but waiting on reviews for a log time.	16:19
johnsom	Specifically tempest-plugin patches, but also other repositories.	16:20
johnsom	How do you all feel about starting a priority review etherpad for Train?	16:20
cgoncalves	+1	16:20
johnsom	I would try to organize it by patch age, patch dependencies, etc.	16:20
johnsom	It seems like this has helped in the past, so though I would propose it.	16:21
johnsom	I just hate seeing things like test patches sit unreviewed for months on end. These should be pretty straight forward to review.	16:22
cgoncalves	are we going to re-use https://etherpad.openstack.org/p/octavia-priority-reviews ?	16:22
johnsom	Plus the patch list is getting very long and we are going to start having more conflicts.	16:23
johnsom	I like to so folks bookmarks still work. Any concern not to use that etherpad?	16:23
cgoncalves	I'm good with that etherpad	16:24
johnsom	I will also update the IRC channel topic to include the link	16:24
johnsom	Ok, I'm not seeing any negative feedback, so I will work on setting that up again.	16:24
johnsom	Just a reminder, it is intended to help reviewers (not just cores) have some order to patches that need reviews. Ordered by patch age, and if other patches depend on it, etc.	16:26
johnsom	#topic Open Discussion	16:26
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"		16:26
johnsom	Other topics for today?	16:26
cgoncalves	FYI, the CentOS job has been in a pretty bad shape for quite awhile	16:27
cgoncalves	I ran yesterday Octavia tempest tests against centos-based amphorae locally and all tests passed (except for 1 related to IPv6 misconfiguration likely)	16:27
johnsom	Sigh, yes. I am hoping the log offloading will shed some light on that.	16:27
cgoncalves	so, there's something fishy about the way the job runs in Zuul	16:28
cgoncalves	I have this DNM patch where I'm experimented with a mix of centos controller/ubuntu amp and the other way around: https://review.opendev.org/#/c/664840/	16:28
cgoncalves	centos controller / ubuntu amphora passed	16:28
johnsom	We may need to get the nova console logs too somehow. We may need to add that to our cleanup process to have it save off those console logs before deleting the amphora on cleanup.	16:29
colin-	that would be useful	16:29
cgoncalves	ubuntu controller / centos amphora passed 10 out of 15 tests, so way better anyway	16:29
cgoncalves	I'm thinking requesting the Infra team a CI node to reproduce this would be best	16:30
cgoncalves	open for ideas!	16:30
johnsom	Yeah, we may need to do that. Maybe after we get a log offload?	16:31
cgoncalves	how close are we to getting logs? LO	16:32
cgoncalves	:P	16:32
johnsom	I hope to figure out this depends-on problem this morning. If that doesn't show what is wrong, then I would consider escalating to infra or adding the nova console log idea.	16:32
* johnsom glares at cgoncalves....		16:32
cgoncalves	alright, good to me :)	16:32
johnsom	It works on devstack! It's just the tempest patch isn't pulling in the octavia patch, it's checking out master.	16:33
colin-	the IPv6 implementation will be native all the way through the amp stack right? no 6 to 4 stuff i'm assuming?	16:33
johnsom	lol, you are welcome to look at the tempest patch and see if you see why it's not checking out the right version.	16:33
cgoncalves	yeah, weird it's not pulling the depends-on -- https://github.com/openstack/octavia-tempest-plugin/blob/fe49e4fae8e051b0690cdebeffd7b600a6d63171/zuul.d/jobs.yaml#L39	16:34
johnsom	colin- For the VIP and members, we already fully support IPv6 and test it in the tempest scenarios. This includes 4-to-6 and 6-to-4 scenarios.	16:34
johnsom	colin- Look at the ipv6 jobs here:	16:35
johnsom	#link http://logs.openstack.org/23/664123/4/check/octavia-v2-dsvm-scenario/7e94fbc/testr_results.html.gz	16:35
colin-	thanks, will do. what is the outstanding body of work for train in that case?	16:35
cgoncalves	johnsom, wondering if https://github.com/openstack/octavia-tempest-plugin/blob/fe49e4fae8e051b0690cdebeffd7b600a6d63171/zuul.d/jobs.yaml#L84 isn't interfering	16:36
johnsom	A gate job using IPv6 for the lb-mgmt-net and making sure our devstack plugin picks up the devstack IPV6 variable and creates the API endpoint using IPv6.	16:36
colin-	looking at https://review.opendev.org/#/c/594078/ presently	16:36
colin-	ok	16:36
cgoncalves	as in overriding the code pulled from depends-on	16:36
johnsom	cgoncalves I bet you are right!	16:36
johnsom	hmmmm	16:37
johnsom	I need to figure out how to fix that. Thanks!	16:37
*** ccamposr__ has quit IRC		16:37
* johnsom un-glares at cgoncalves		16:37
cgoncalves	lol	16:37
johnsom	Any other topics today?	16:38
johnsom	Ok, thanks folks! Have a great week.	16:40
johnsom	#endmeeting	16:40
*** openstack changes topic to "Discussions for OpenStack Octavia \| Train PTG etherpad: https://etherpad.openstack.org/p/octavia-train-ptg"		16:40
openstack	Meeting ended Wed Jun 12 16:40:12 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)	16:40
openstack	Minutes: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-06-12-16.00.html	16:40
openstack	Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-06-12-16.00.txt	16:40
openstack	Log: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-06-12-16.00.log.html	16:40
rouk	johnsom: even with insecure=True under all headings in octavia.conf, it still fails to call nova.	16:41
rouk	probably just going to make a valid cert for everything but... yeah thats a thing.	16:41
johnsom	rouk Ok, that would be a bug then. Would you mind opening a story?	16:41
rouk	can do	16:41
johnsom	https://storyboard.openstack.org/#!/dashboard/stories	16:41
*** ataraday_ has quit IRC		16:47
rouk	johnsom: https://storyboard.openstack.org/#!/story/2005868	16:51
rouk	let me know if you need any more	16:51
johnsom	rouk Thank you	16:51
*** luksky has joined #openstack-lbaas		16:55
*** rpittau is now known as rpittau\|afk		16:56
*** gcheresh has joined #openstack-lbaas		16:58
openstackgerrit	Michael Johnson proposed openstack/octavia-tempest-plugin master: Save amphora logs in gate https://review.opendev.org/626406	16:59
cgoncalves	openstack.org vs opendev.org?	17:01
johnsom	FYI, depends-on now has to be an opendev.org URL. The old openstack.org will not work.	17:01
openstackgerrit	Michael Johnson proposed openstack/octavia stable/stein: Add Stein octavia-v2-dsvm-scenario-ubuntu-xenial https://review.opendev.org/652687	17:04
*** goldyfruit has joined #openstack-lbaas		17:04
*** ramishra has quit IRC		17:04
openstackgerrit	Michael Johnson proposed openstack/octavia-tempest-plugin master: Add a flavor to the load balancer CRUD scenarios https://review.opendev.org/631353	17:05
openstackgerrit	Michael Johnson proposed openstack/octavia-tempest-plugin master: Add an active/standby scenario test https://review.opendev.org/584681	17:05
openstackgerrit	Michael Johnson proposed openstack/octavia-tempest-plugin master: Add an lxd octavia scenario job https://review.opendev.org/636069	17:06
openstackgerrit	Michael Johnson proposed openstack/octavia-tempest-plugin master: Add a test job for cinder volume backed amps https://review.opendev.org/638293	17:07
openstackgerrit	Michael Johnson proposed openstack/octavia-tempest-plugin master: Add amphora update service client and API test https://review.opendev.org/633295	17:07
openstackgerrit	Carlos Goncalves proposed openstack/octavia-tempest-plugin master: DNM: testing CentOS https://review.opendev.org/664840	17:07
openstackgerrit	Michael Johnson proposed openstack/octavia-tempest-plugin master: Adds provider flavor capabilities API tests https://review.opendev.org/631113	17:07
johnsom	Fixing the depends-on	17:08
openstackgerrit	Michael Johnson proposed openstack/python-octaviaclient master: Add l7policy and l7rule to octavia quota https://review.opendev.org/591568	17:08
johnsom	Hopefully that will fix the log offloading tests.	17:10
cgoncalves	hope so, too. rebased my DNM centos job patch on top of yours	17:10
*** yamamoto has joined #openstack-lbaas		17:11
*** yamamoto has quit IRC		17:11
johnsom	Yeah, the tempest patch should run the centos7 gate, so should provide results too	17:11
*** yamamoto has joined #openstack-lbaas		17:11
cgoncalves	oh, true, right	17:13
*** yamamoto has quit IRC		17:13
*** yamamoto has joined #openstack-lbaas		17:14
johnsom	rouk Hmm, the code for nova appears to be honoring that setting. The traceback in the story is from after setting the [nova] insecure = True?	17:15
johnsom	Did you remember to restart all of the octavia API processes?	17:15
johnsom	We tell nova here: https://github.com/openstack/octavia/blob/master/octavia/common/clients.py#L54	17:16
johnsom	And we tell that method the setting here:	17:16
johnsom	https://opendev.org/openstack/octavia/src/branch/master/octavia/compute/drivers/nova_driver.py#L79	17:17
rouk	traceback did not change between setting and not setting it, checking all containers for their start time...	17:17
rouk	yep, octavia_api restarted after change, and does have the change.	17:18
johnsom	This is super odd.	17:20
johnsom	I don't see what could be wrong from octavia code inspection. Someone will have to build a stack and debug on a live stack....	17:21
johnsom	rouk is there a traceback in the octavia api log too? Does it provide more information?	17:22
johnsom	If there is a traceback there, it would be nice to have that in the story as well	17:22
rouk	i do have some traces serverside, ill grab them in a bit	17:24
johnsom	Thank you	17:24
*** ianychoi has quit IRC		17:25
rouk	serverside from octavia-api is identical	17:26
johnsom	Ok, thanks for checking	17:28
johnsom	Hmm, it should be different to some degree	17:28
*** gcheresh has quit IRC		17:34
*** xgerman has joined #openstack-lbaas		17:37
rouk	it is slightly different with octavia in debug mode	17:42
rouk	added it to the story comments	17:44
*** ricolin_ has quit IRC		17:44
johnsom	Thank you	18:02
*** yamamoto has quit IRC		18:06
*** goldyfruit has quit IRC		18:11
*** goldyfruit has joined #openstack-lbaas		18:17
*** gcheresh has joined #openstack-lbaas		18:30
openstackgerrit	boden proposed openstack/neutron-lbaas stable/stein: Support URL query params in healthmonitor url_path https://review.opendev.org/660930	18:33
cgoncalves	http://logs.openstack.org/06/626406/9/check/octavia-v2-dsvm-py2-scenario/e0bad7f/controller/logs/octavia-amphora_log.txt.gz	18:49
cgoncalves	http://logs.openstack.org/06/626406/9/check/octavia-v2-dsvm-py2-scenario/e0bad7f/controller/logs/octavia-tenant-traffic_log.txt.gz	18:49
cgoncalves	yay!	18:49
johnsom	Yep, +1	18:49
johnsom	They look pretty good IMO	18:50
cgoncalves	such modesty :P	18:52
johnsom	lol, yep	18:52
*** tesseract has quit IRC		18:54
johnsom	Kind of wondering if we shouldn't get fancy and create one amphora file per amphora. Right now they are intermixed. I could create a directory and do a file per-amp. Not the user logs of course, but the amp logs.	18:55
*** ccamposr has joined #openstack-lbaas		19:07
cgoncalves	hmmm. intermixed is fine I guess and probably easier to analyze on active-standby topology	19:10
cgoncalves	johnsom, will UDP traffic be also logged?	19:12
*** yamamoto has joined #openstack-lbaas		19:13
*** ccamposr__ has joined #openstack-lbaas		19:13
*** ccamposr has quit IRC		19:15
*** yamamoto has quit IRC		19:17
johnsom	cgoncalves I looked into that, no, since UDP is handled in the kernel they did not include any flow logging for LVS.	19:22
*** gcheresh has quit IRC		19:33
*** jomeier has joined #openstack-lbaas		19:39
jomeier	Hi gentleman. I use octavia on OpenStack stein with OpenvSwitch. I have the problem that the loadbalancer stays in CREATE_PENDING forever. This are suspicious log messages:	19:42
jomeier	/var/log/octavia/worker.log:	19:42
jomeier	2019-06-12 21:40:55.255 30787 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: ConnectTimeout: HTTPSConnectionPool(host='172.16.1.158', port=9443): Max retries exceeded with url: /0.5/info (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7fddf69e8c90>, 'Connection to 172.16.1.158 timed out. (connect	19:43
jomeier	timeout=10.0)'))	19:43
jomeier	/var/log/neutron/server.log:	19:43
jomeier	2019-06-12 20:11:09.286 8609 ERROR neutron.plugins.ml2.managers [req-fd7869e2-b09f-4652-92ec-13ee526a0d31 a57211e127ba4afbb2c2f6edd8331a85 14c34108387f4adc9605fb42736574c4 - default default] Failed to bind port 19ec9fa9-016d-4976-a435-2d126fb20312 on host openshift-controller for vnic_type normal using segments [{'network_id': '9911869a-afb9-4c36-bc8c-55d9dc669d69', 'segmentation_id': 23,	19:43
jomeier	'physical_network': None, 'id': '01e649f7-f707-4344-9307-2f73b5b3a5c6', 'network_type': u'vxlan'}]	19:43
jomeier	I also can't ping the Amphora vm from the controller. But it works through the namespaced method with:	19:44
jomeier	sudo ip netns exec qdhcp-0bed5a95-4990-4dd4-b719-e7590166f645 curl -k https://172.16.3.189:9443/0.5/details	19:44
jomeier	Can somebody help me? I'm searching for the problem for about three days now.	19:44
johnsom	jomeier It won't stay in PENDING_CREATE forever, it just has a very long retry period by default. For production that should be reduced so it doesn't retry so long. The warning message is normal, that is the controller attempting to talk to the amphora service VM.	19:44
johnsom	I'm not sure why the port bind would fail, but I'm guessing the controller issue is a network/configuration one	19:45
jomeier	The problem is that I cant find a detailed documentation about how to install octavia.	19:46
jomeier	The best page is this one: https://blog.zufardhiyaulhaq.com/manual-instalation-octavia-openstack-queens/	19:46
johnsom	Yeah, there isn't a detailed guide yet. All of our docs are here: https://docs.openstack.org/octavia/latest/	19:47
johnsom	Did you use a deployment tool to install, or do it by hand?	19:47
jomeier	I did it like described on that blog	19:47
johnsom	I don't think have ever seen or heard of that blog	19:48
jomeier	Its the "best" description I could find.	19:48
johnsom	Ok, so this is a manual install. So, the controllers (worker, health manager, house keeping) all need to be able to access the amphora over the o-hm0 interface you created.	19:50
jomeier	I created a "allow anything" secgroup and iptables entry for that. Is that correct?	19:50
johnsom	My guess as to why the controller is unable to reach the amphora service VM is an issue with that o-hm0 interface. Can you check that it is up and has an IP?	19:50
johnsom	Ah, that blog is in fact missing the security group configuration.	19:51
jomeier	security group configuration is at the top	19:52
jomeier	o-hm0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500	19:52
jomeier	inet6 fe80::f816:3eff:fe3d:9825 prefixlen 64 scopeid 0x20<link>	19:52
jomeier	ether fa:16:3e:3d:98:25 txqueuelen 1000 (Ethernet)	19:52
jomeier	RX packets 9110 bytes 1319195 (1.2 MiB)	19:52
jomeier	RX errors 0 dropped 2679 overruns 0 frame 0	19:52
jomeier	TX packets 115 bytes 35074 (34.2 KiB)	19:52
jomeier	TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0	19:52
jomeier	Dont get an ipv4	19:52
johnsom	Yeah, ok, that is a problem for sure.	19:52
johnsom	even the IPv6 is link-local, so not assigned.	19:53
johnsom	This script is what our test gates use to setup octavia: https://github.com/openstack/octavia/blob/master/devstack/plugin.sh	19:53
johnsom	It can be a useful reference.	19:53
jomeier	ah ok. Hm. What could be the problem so o-hm0 doesn't get a valid IP?	19:53
johnsom	I would check that the subnet you created for the lb-mgmt-net has DHCP enabled.	19:54
johnsom	Then try this line again: sudo dhclient -v o-hm0	19:54
jomeier	yes. This line fails for me.	19:54
johnsom	Ok, what is the error?	19:54
johnsom	Once you get an IP, you will need these iptables rules:	19:55
johnsom	http://logs.openstack.org/06/626406/9/check/octavia-v2-dsvm-scenario/240afba/controller/logs/devstacklog.txt.gz#_2019-06-12_17_53_33_794	19:56
jomeier	[root@openstack-controller ~]# sudo dhclient -v o-hm0	19:56
jomeier	Internet Systems Consortium DHCP Client 4.2.5	19:56
jomeier	Copyright 2004-2013 Internet Systems Consortium.	19:56
jomeier	All rights reserved.	19:56
jomeier	For info, please visit https://www.isc.org/software/dhcp/	19:56
jomeier	Listening on LPF/o-hm0/fa:16:3e:3d:98:25	19:56
jomeier	Sending on LPF/o-hm0/fa:16:3e:3d:98:25	19:56
jomeier	Sending on Socket/fallback	19:56
jomeier	DHCPDISCOVER on o-hm0 to 255.255.255.255 port 67 interval 3 (xid=0x7b1d2d59)	19:56
jomeier	DHCPDISCOVER on o-hm0 to 255.255.255.255 port 67 interval 8 (xid=0x7b1d2d59)	19:56
jomeier	DHCPDISCOVER on o-hm0 to 255.255.255.255 port 67 interval 9 (xid=0x7b1d2d59)	19:56
jomeier	DHCPDISCOVER on o-hm0 to 255.255.255.255 port 67 interval 15 (xid=0x7b1d2d59)	19:56
jomeier	DHCPDISCOVER on o-hm0 to 255.255.255.255 port 67 interval 20 (xid=0x7b1d2d59)	19:56
jomeier	DHCPDISCOVER on o-hm0 to 255.255.255.255 port 67 interval 6 (xid=0x7b1d2d59)	19:56
jomeier	No DHCPOFFERS received.	19:56
jomeier	No working leases in persistent database - sleeping.	19:56
johnsom	Ok, so the neutron network has a problem. It should look like this:	19:56
johnsom	http://logs.openstack.org/06/626406/9/check/octavia-v2-dsvm-scenario/240afba/controller/logs/devstacklog.txt.gz#_2019-06-12_17_53_45_351	19:56
johnsom	Check this, run "openstack subnet show lb-mgmt-subnet"	19:58
johnsom	You should see a line like:	19:58
johnsom	\| enable_dhcp \| True	19:58
johnsom	You should also see a line like: allocation_pools \| 192.168.0.2-192.168.0.200	19:58
jomeier	I[sepp@openstack-controller ~(octavia_admin)]$ openstack subnet show lb-mgmt-subnet	19:59
jomeier	+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+	19:59
jomeier	\| Field \| Value \|	19:59
jomeier	+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+	19:59
jomeier	\| allocation_pools \| 172.16.0.100-172.16.31.254 \|	19:59
jomeier	\| cidr \| 172.16.0.0/12 \|	19:59
jomeier	\| created_at \| 2019-06-12T18:11:05Z \|	19:59
jomeier	\| description \| \|	19:59
jomeier	\| dns_nameservers \| \|	19:59
jomeier	\| enable_dhcp \| True \|	19:59
jomeier	\| gateway_ip \| 172.16.0.1 \|	19:59
jomeier	\| host_routes \| \|	19:59
jomeier	\| id \| 5a884c03-547c-4378-9103-8653dbc00ad7 \|	19:59
jomeier	\| ip_version \| 4 \|	19:59
jomeier	\| ipv6_address_mode \| None \|	19:59
jomeier	\| ipv6_ra_mode \| None \|	19:59
jomeier	\| location \| Munch({'project': Munch({'domain_name': 'Default', 'domain_id': None, 'name': 'services', 'id': u'14c34108387f4adc9605fb42736574c4'}), 'cloud': '', 'region_name': '', 'zone': None}) \|	19:59
jomeier	\| name \| lb-mgmt-subnet \|	19:59
jomeier	\| network_id \| 9911869a-afb9-4c36-bc8c-55d9dc669d69 \|	20:00
jomeier	\| prefix_length \| None \|	20:00
jomeier	\| project_id \| 14c34108387f4adc9605fb42736574c4 \|	20:00
jomeier	\| revision_number \| 0 \|	20:00
jomeier	\| segment_id \| None \|	20:00
jomeier	\| service_types \| \|	20:00
jomeier	\| subnetpool_id \| None \|	20:00
jomeier	\| tags \| \|	20:00
jomeier	\| updated_at \| 2019-06-12T18:11:05Z \|	20:00
jomeier	+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+	20:00
jomeier	damn. sorry ...	20:00
jomeier	but it is there	20:00
johnsom	FYI, you can use http://paste.openstack.org/ for sharing large snippets.	20:00
johnsom	Ok, those look good. now, 'openstack network agent list"	20:00
johnsom	You are looking for the openvswitch and dhcp agents to be present on the compute hosts and the controller host running the Octavia controllers (since you used the ovs-vsctl method.	20:01
johnsom	Each line should have a :-) and state of up	20:02
jomeier	http://paste.openstack.org/show/752836/	20:02
johnsom	Ok, so those lines with XXX mean neutron is not running correctly or has failed	20:02
jomeier	I changed my hostname to: openstack-controller.sepp.com	20:03
jomeier	Maybe some old stuff is laying around. Thats where I tried designate	20:03
jomeier	Because the agents for openstack-controller.sepp.com are all up.	20:04
johnsom	Ok, so all of your services are running on openstack-controller.sepp.com?	20:04
jomeier	yes	20:04
jomeier	The availability zone is "none" and not "nova". Could that be a problem?	20:04
johnsom	Maybe, I'm not a full neutron expert, so I'm not sure what the implications of that would be.	20:05
johnsom	If that all looks ok, I would go through and double check those network setup lines and make sure the ovs port is setup right, etc.	20:06
jomeier	Do I have to set some attributes to the neutron port I must create? the network_type is vxlan at the moment.	20:06
johnsom	I would also check the port in openstack port show to make sure it's up	20:06
johnsom	vxlan should not matter, that is how most of us run	20:07
jomeier	http://paste.openstack.org/show/752837/	20:08
johnsom	Ah:	20:08
johnsom	\| binding_host_id \| openshift-controller	20:08
johnsom	neutron created the port on the old hostname host	20:09
johnsom	\| binding_vif_type \| binding_failed	20:09
jomeier	oh. You mean I must use the fqdn ?	20:09
johnsom	The host id must match the agent list host id	20:09
jomeier	ups. Give me a minute ... I'll try that...	20:09
johnsom	I have to step away for a bit in a minute, but that port bind failed is your trouble	20:10
jomeier	Thanks a lot for your help. I'll give you feedback if that was the problem.	20:10
johnsom	Sounds good	20:11
rm_work	i wouldn't say slacking, so much as, working late into the night and generally having no time to sleep if i have to be up at 9am :D	20:18
johnsom	lol	20:19
jomeier	@johnsom: You were right: the binding_host_id was the major problem. No my port o-hm0 gets an IP and the binding is active ! Wow. Such a stupid error :-D	20:25
jomeier	Now I get this one:	20:25
johnsom	Cool, so eventually the other LB will timeout retries and go to ERROR at which point you can delete it.	20:25
jomeier	2019-06-12 22:23:08.261 29488 ERROR oslo_messaging.rpc.server PlugVIPException: Error plugging amphora (compute_id: bacd9b69-197a-49e9-9c1f-79d140f2007f) into vip network e741e483-2f1d-48e6-a2d6-ad49a243e5c6.	20:26
jomeier	2019-06-12 22:23:08.261 29488 ERROR oslo_messaging.rpc.server	20:26
jomeier	But I hope this time I can fix this problem much quicker.	20:26
rm_work	johnsom: updated the network-noop driver to globally track the things it has returned so it isn't completely amnesiac, and to actually return objects with correct data, and kinda dynamically build a network/subnet tree https://review.opendev.org/#/c/660239/45/octavia/network/drivers/noop_driver/driver.py	20:27
rm_work	any other requests for it while i'm in here? :D	20:28
johnsom	Nice	20:28
johnsom	rm_work If you didn't see above, we have nice shiny logs: http://logs.openstack.org/06/626406/9/check/octavia-v2-dsvm-scenario/240afba/controller/logs/	20:29
johnsom	Evidently depends-on no longer works with the old openstack.org urls....	20:29
rm_work	yeah i looked at them, it's neat	20:29
rm_work	hmm that is odd	20:29
rm_work	i thought the redirects were supposed to work for a long time	20:30
rm_work	:/	20:30
johnsom	I guess there was an e-mail I missed about needing to change them over to opendev.org	20:30
rm_work	is the logs thing ready for review?	20:30
johnsom	No, I'm going to write up some docs	20:30
rm_work	kk	20:30
rm_work	i need to do docs now on multi-vip too	20:30
rm_work	and a couple more bugs to squash	20:30
johnsom	docs for everyone!	20:30
*** jomeier has quit IRC		20:35
*** ccamposr__ has quit IRC		20:37
*** ccamposr has joined #openstack-lbaas		20:37
*** vishalmanchanda has quit IRC		20:55
boden	hi... I know neutron-lbaas is retired, but we still have customers using it that need fixes :). can I ask for some help reviewing https://review.opendev.org/#/c/660930/	21:06
johnsom	Should not be a problem. I agree with your approach to the release note issue, let's disable the job but keep collecting the release notes. This way when the job is fixed we will have complete release notes.	21:09
*** boden has quit IRC		21:10
*** ivve has quit IRC		21:11
johnsom	boden BTW, I got interrupted yesterday and didn't get the get patch up. I should have something posted today.	21:12
*** pcaruana\|afk\| has quit IRC		21:16
*** luksky has quit IRC		21:25
*** ccamposr has quit IRC		21:57
*** ccamposr has joined #openstack-lbaas		21:58
rm_work	johnsom: in the doc for additional_vips, it's a list of json objects ... should I just call it a "list" and document in the description what the objects look like, or do they need sub-fields somehow?	22:09
johnsom	for the api-ref parameters?	22:09
*** spatel has quit IRC		22:10
rm_work	yes	22:14
johnsom	it is "array" for reasons I don't know. But, yes, "array" and then describe what is in the array	22:14
rm_work	hmmm we use "type: list" in many other spots	22:14
johnsom	hmmm, maybe list renders as "array"	22:15
johnsom	I remember there was something funky with that.	22:15
johnsom	Use whatever is used elsewhere in the file	22:15
johnsom	Ha, we do have both. hmmm	22:16
johnsom	Oh, only tags is using list. I bet that is "wrong" but it seems to be taking it	22:16
johnsom	Yeah, it says "array" here: https://docs.openstack.org/os-api-ref/latest/usage.html#parameters-file-format	22:18
johnsom	Yeah, so the tags type is incorrect	22:19
openstackgerrit	Michael Johnson proposed openstack/octavia-lib master: Add get methods to the driver-lib https://review.opendev.org/665027	22:31
rm_work	ok i can just fix those too	22:32
openstackgerrit	Michael Johnson proposed openstack/octavia master: Add get method support to the driver-agent https://review.opendev.org/665029	22:33
johnsom	boden ^^^ Those are fresh off the press, completely untested and undocumented. But wanted to give you early access to them. They should work.	22:34
johnsom	I guess to be fair, there are doc strings... lol	22:35
*** rcernin has joined #openstack-lbaas		22:41
*** goldyfruit has quit IRC		22:59
openstackgerrit	Michael Johnson proposed openstack/octavia master: Disable the kdump service in the amphora https://review.opendev.org/665032	23:00
*** goldyfruit has joined #openstack-lbaas		23:11
*** ccamposr__ has joined #openstack-lbaas		23:14
*** ccamposr has quit IRC		23:17
*** ccamposr__ has quit IRC		23:20
rm_work	johnsom: should I add a depends-on to my main `octavia` patch for the `octavia-tempest-plugin` patch? or the other way	23:23
rm_work	i've done the other way but if i did the depends-on from the octavia side, would it run the new tests along with every patch?	23:24
rm_work	but would I have to mess with it when we were ready to merge?	23:24
openstackgerrit	Adam Harwell proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/660239	23:26
rm_work	docs added, keepalived bug hopefully fixed	23:27
rm_work	so multi-vip with keepalived should work better with ipv6 as the primary vip and ipv4 as additionals	23:27
rm_work	johnsom: oh, and on the ipv6-only gate, were you doing that? I could also take a stab at making the job really quick, it doesn't look hard and just starting to get some runs going to iterate on it might not be a bad idea	23:30
rm_work	just to see what is even broken currently	23:30
lxkong	johnsom: i can't reproduce the issue in my devstack environment (stable/stain), my lb in devstack is active within 2 or 3 mins	23:37
lxkong	did't look at qemu yet, will do	23:38
*** yamamoto has joined #openstack-lbaas		23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!