Wednesday, 2019-05-08

*** yamamoto has quit IRC00:19
*** KeithMnemonic has quit IRC00:56
*** yamamoto has joined #openstack-lbaas01:09
*** hongbin has joined #openstack-lbaas01:31
*** yamamoto has quit IRC01:49
*** yamamoto has joined #openstack-lbaas02:20
*** ricolin has joined #openstack-lbaas02:23
*** yamamoto has quit IRC02:30
*** psachin has joined #openstack-lbaas03:07
*** jiteka has quit IRC03:13
*** eandersson has quit IRC03:13
*** jiteka has joined #openstack-lbaas03:14
*** ramishra_ has quit IRC03:33
*** ramishra has joined #openstack-lbaas03:37
*** hongbin has quit IRC04:19
*** ivve has quit IRC04:35
*** yamamoto has joined #openstack-lbaas04:39
*** yamamoto has quit IRC04:44
*** yamamoto has joined #openstack-lbaas04:45
*** yamamoto has quit IRC04:46
*** yamamoto has joined #openstack-lbaas04:48
*** yamamoto has quit IRC04:50
*** gthiemonge has quit IRC04:54
*** gthiemonge has joined #openstack-lbaas04:55
*** gcheresh has joined #openstack-lbaas05:07
*** yamamoto has joined #openstack-lbaas05:09
*** dims has quit IRC05:32
*** dims has joined #openstack-lbaas05:33
*** yamamoto has quit IRC05:36
*** ivve has joined #openstack-lbaas05:39
*** AlexStaf has quit IRC06:07
*** ricolin_ has joined #openstack-lbaas06:12
*** ricolin has quit IRC06:12
*** yamamoto has joined #openstack-lbaas06:15
*** dims has quit IRC06:16
*** yamamoto has quit IRC06:27
*** ricolin_ has quit IRC06:27
*** dims has joined #openstack-lbaas06:42
*** dims has quit IRC06:49
*** dims has joined #openstack-lbaas06:54
*** rcernin has quit IRC07:01
*** gcheresh has quit IRC07:23
*** yboaron_ has joined #openstack-lbaas07:46
*** rtjure has quit IRC07:48
*** ccamposr has joined #openstack-lbaas07:50
*** eandersson has joined #openstack-lbaas07:50
*** rtjure has joined #openstack-lbaas07:54
*** gcheresh has joined #openstack-lbaas07:56
*** rtjure has quit IRC08:16
*** yamamoto has joined #openstack-lbaas08:35
*** fnaval has quit IRC08:36
openstackgerritErik Olof Gunnar Andersson proposed openstack/octavia master: [WIP] Testing performance improvement on non-udp health checks  https://review.opendev.org/65775608:45
openstackgerritErik Olof Gunnar Andersson proposed openstack/octavia master: [WIP] Testing performance improvement on non-udp health checks  https://review.opendev.org/65775608:53
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add a scenario test for spare pool  https://review.opendev.org/63498809:32
*** ataraday_ has joined #openstack-lbaas10:00
*** yamamoto has quit IRC10:15
*** yamamoto has joined #openstack-lbaas10:17
*** yamamoto has quit IRC10:26
*** gcheresh has quit IRC10:50
*** yamamoto has joined #openstack-lbaas10:57
*** yamamoto has quit IRC11:08
*** sapd1_x has joined #openstack-lbaas11:11
openstackgerritAnn Taraday proposed openstack/octavia master: [WIP] Jobboard based controller  https://review.opendev.org/64740611:12
*** yamamoto has joined #openstack-lbaas11:17
*** henriqueof has quit IRC12:12
*** henriqueof has joined #openstack-lbaas12:17
openstackgerritAnn Taraday proposed openstack/octavia master: [WIP]Transition load balancer flows DB objects to dicts  https://review.opendev.org/65756912:19
*** yboaron_ has quit IRC12:41
*** henriqueof has quit IRC12:43
*** henriqueof has joined #openstack-lbaas12:43
*** yboaron_ has joined #openstack-lbaas13:01
*** boden has joined #openstack-lbaas13:31
*** yboaron_ has quit IRC13:36
*** gcheresh has joined #openstack-lbaas13:44
openstackgerritMerged openstack/octavia-tempest-plugin master: Add TLS Barbican jobs against stable branches  https://review.opendev.org/65698913:52
cgoncalvesphew, ^ only took 5 rechecks13:57
*** yamamoto has quit IRC13:59
*** yamamoto has joined #openstack-lbaas14:01
*** yamamoto has quit IRC14:04
*** psachin has quit IRC14:08
*** tesseract has joined #openstack-lbaas14:38
*** yamamoto has joined #openstack-lbaas14:39
*** yamamoto has quit IRC14:41
*** yamamoto has joined #openstack-lbaas14:41
*** Vorrtex has joined #openstack-lbaas14:45
*** fnaval has joined #openstack-lbaas14:53
*** ivve has quit IRC15:07
*** sapd1_x has quit IRC15:09
*** sapd1_x has joined #openstack-lbaas15:09
*** tesseract has quit IRC15:09
*** altlogbot_3 has quit IRC15:21
*** altlogbot_2 has joined #openstack-lbaas15:22
*** irclogbot_1 has quit IRC15:22
*** irclogbot_2 has joined #openstack-lbaas15:23
*** altlogbot_2 has quit IRC15:27
*** altlogbot_0 has joined #openstack-lbaas15:28
*** irclogbot_2 has quit IRC15:28
*** irclogbot_2 has joined #openstack-lbaas15:29
openstackgerritMerged openstack/octavia-lib master: Add python 3.7 testing  https://review.opendev.org/65267515:31
openstackgerritAnn Taraday proposed openstack/octavia master: [WIP] Transition member flows to use dicts  https://review.opendev.org/65784215:31
*** henriqueof has quit IRC15:32
rm_workmeeting in 5 minutes!15:56
rm_work#startmeeting Octavia16:00
openstackMeeting started Wed May  8 16:00:07 2019 UTC and is due to finish in 60 minutes.  The chair is rm_work. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Octavia)"16:00
openstackThe meeting name has been set to 'octavia'16:00
johnsomo/16:00
cgoncalveshello16:00
rm_workhey!16:00
johnsomHa, I forgot.  That is a nice feeling16:00
rm_workwelcome back ya'll16:00
rm_workyeah i'm about to pass out though so uhh... hopefully i make it through the whole thing16:01
johnsomFYI, I have an agenda item today.16:01
rm_workk16:01
rm_work#topic Announcements16:01
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:01
rm_workI don't have anything today really...16:02
rm_workBut, the PTG was good, lots of stuff covered16:02
johnsomThere is an e-mail thread about some stable branch requirements version changes coming16:02
eanderssonjohnsom, can you update your stresstest tool :p16:02
colin-good morning16:02
colin-or evening16:03
johnsomeandersson Almost done16:03
johnsom#link http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005956.html16:03
ataraday_hello16:03
johnsomThat might break things, so if you see requiests/urllib issues on the stable branch you know why.16:04
johnsomOtherwise, yes, the PTG was great. If you couldn't join us, check out the notes on the etherpad.16:04
johnsom#link https://etherpad.openstack.org/p/octavia-train-ptg16:04
rm_workany other announcements?16:05
johnsomI also spent time yesterday fighting storyboard to open RFE's and update stories based on our discussions.16:05
johnsomThat is all I can think of.16:06
rm_work#topic Brief progress reports / bugs needing review16:06
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:06
rm_workwell, some of that is probably this ^^16:06
johnsomTrue16:07
johnsomI have mostly been catching up since the Summit/PTG.16:07
rm_workI made some progress on multi-vip, but ran into a schema issue i was hoping i'd be able to avoid, so will need to do a lot more work on that... putting it on hold for a week or so to get some internal stuff done to unblock some other folks16:07
johnsomI opened the stories (4 out of 5 storyboard gave 500's), etc.16:07
rm_workI do have one change I'm about to push up, which is to make the amp/worker communication use TLSv1.2 instead of SSLv2/316:08
johnsomBackward compat I assume?16:08
rm_workbecause SSLv2/3 is really broken, and it was brought to my attention that we're still using that, by our internal security team16:08
rm_workit's a change on the amp side16:08
rm_workso it'll just work16:08
colin-nice, glad to hear that16:09
johnsomHmmm, I would have thought it would be on the controller side to be compatible....16:09
rm_workmaybe can do it on both sides, and then yes16:09
colin-thanks for doing that16:09
ataraday_I split PoC for jobboard, and looked at transition flows DB objects to dicts a bit more16:09
rm_workbecause they all SUPPORT it, just need to force it16:09
johnsomI.e. ask for TLSv1.2 on the initiation16:09
rm_workyes16:10
rm_workwill submit that patch today later16:10
johnsomataraday_ Awesome, you probably saw that I broke out the tasks as we discussed at the PTG.16:10
cgoncalveswe've also seen one customer disabling SSLv3 and TLS 10 ciphers in listeners16:10
ataraday_johnsom, yes, thanks! I already linked the changes to proper tasks16:11
johnsomataraday_ Do you need me to do an example patch of what I'm talking about with the provider dictionaries?16:11
cgoncalvessorry, I meant to write TLS 1.016:11
johnsomcgoncalves Yeah, letting users pick ciphers and protocols was on my "wish I had time to implement" list.  I have a plan, but no time right now.16:11
ataraday_johnsom, if you have time for this this would be nice16:12
johnsomI have some internal paperwork to do this week, but hope to get back to fixing the unsets "soonish"16:13
johnsomataraday_ Ok, sounds like it is not a blocker for you, so I will probably work on that early next week.16:13
*** ccamposr has quit IRC16:13
*** ccamposr has joined #openstack-lbaas16:14
eanderssonjohnsom, I don't see ver in Rocky https://github.com/openstack/octavia/blob/stable/rocky/octavia/amphorae/backends/health_daemon/health_daemon.py#L11816:14
ataraday_johnsom, yes, I will be off for Thuesday and Friday anyway :)16:14
johnsomeandersson I think I just messaged you that.16:14
cgoncalvesataraday_, I very, very quickly checked your patches. a question I have is if you want zookeeper to be the default or redis. I recall we discussed about going for redis as default and to add support for it in our devstack plugin16:15
eanderssonI just moved from home to the office so I might have missed the message.16:15
eandersson*that16:15
johnsomeandersson https://github.com/openstack/octavia/commit/2170cc6c459b7ae8461a09a0b6fd754ecef9654e16:15
eanderssonCan we back port that one to Rocky?16:15
johnsomYeah, we decided to do Redis in the interim16:15
ataraday_cgoncalves, we agreed on redis16:15
johnsomeandersson No, I don't think so, it changes the protocol so.... This is why there is the backward compatibility in the controller side.16:16
cgoncalvesataraday_, right. I asked because I saw zookeeper as default in your patch16:16
eanderssonI see - we need to update to Stein because the performance hit is pretty large there.16:17
ataraday_yeah, I used it to try things out, now will swith on redis16:17
johnsomeandersson Yeah, is it enough to improve that backwards compat code in Rocky?  What I saw you had was .2 seconds per heartbeat instead of .00616:17
cgoncalvesok, just wanted to make sure we were on the same page. thanks!16:18
johnsomeandersson I will make the stress tool configurable for versioned or not.16:18
rm_worki advised that we could probably intelligently skip that code path in most cases16:18
rm_workwhich i think would be fine?16:18
rm_workthough i think you also proposed a patch to improve the speed by skipping the db refresh, not sure if that is a problem or not16:19
johnsomYeah, it gets skipped in Stein forward. It was just Rocky that had UDP and amphora agents that didn't send the version.16:19
rm_workright but even in rocky16:19
rm_workyou can skip that code path if the db_lb has no UDP listeners on it16:19
rm_workright?16:19
johnsomWhat little I recall, it does, but maybe not?...16:20
rm_workdidn't look like it16:20
rm_workand the code has the right info already16:20
rm_workwould just be a quick loop&check which is super fast compared to the guaranteed DB pull16:20
colin-certainly sounds preferrable16:21
rm_workok well we're beyond quick updates, should we make this a topic>16:21
rm_work?16:21
* xgerman lurking16:21
eanderssonIt's up to you guys. I can probably keep a patch internally or maybe look at updating to Stein.16:22
rm_workor should we move on, I think this just needs to be fixed... maybe I can do it today if eandersson doesn't do it in the next 10 minutes :D16:22
eanderssonrm_work, https://review.opendev.org/#/c/657756/16:22
rm_workyeah i saw you did that one, but not sure if that causes issues or not16:23
rm_workwas hoping johnsom would review16:23
johnsomSo it is loop that skips non-UDP listeners16:23
rm_workwhich he did16:23
rm_workjohnsom: yeah but the issue is that *triggering the loop* is the expensive part16:24
rm_workthe lb_repol.get16:24
johnsomYeah, it's probably the DB round trips for the listeners16:24
rm_workyes16:24
rm_workso could skip before that, or do what eandersson did16:24
johnsomBut, again, I'm just curious is .2 seconds is a problem....16:25
rm_workwell it's an order of magnitude time increase16:25
rm_workso16:25
rm_workthat's not ideal16:25
rm_work0.006 -> 0.2? two orders? lol16:25
colin-and that's exacerbated by load16:26
rm_workand more DB load16:26
rm_workso yes i would say it could definitely be a problem16:26
johnsomYeah, otherwise, we need to do what was proposed, update the query to include the protocol. (please only in Rocky as it's a waste on Stein forward).16:26
rm_workah it's the special get, so i guess not16:27
rm_workwhat was your topic johnsom ?16:29
johnsomUser survey16:30
rm_work#topic User survey16:30
*** openstack changes topic to "User survey (Meeting topic: Octavia)"16:30
johnsomSo, for yet another year the foundation ignored us and did not ask us if we wanted a question on the user survey.16:30
johnsomThey only went to the "represented" projects (grumble, the privileged ones evidently).16:32
rm_workdon't we know multiple people on the TC? do they not have any say int his either?16:32
johnsomSo I raised the issue on the mailing list16:32
colin-saw that heh16:32
johnsom#link http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005959.html16:32
johnsomYeah, this is good feedback for mugsie etc.16:32
johnsomThis is literally the fourth year they have ignored us.16:32
johnsomSorry, fourth cycle, not yer.16:33
rm_workand jroll16:33
johnsomAnyway, even though the survey has already been advertised and we likely will not get many responses for 2019, my question for the team is what do we want to ask?16:33
mugsiejohnsom: unfortunately we are not in control of that survey - it is the UC16:34
johnsomThe question we had wanted to ask prior to the neutron-lbaas retirement this year was "What load balancing providers are you using?"16:34
cgoncalvestwo questions? 1) most wanted new features, 2) Most wanted provider drivers16:34
johnsommugsie It's all foundation folks from the e-mail chain.16:34
mugsiedamn16:35
johnsomcgoncalves +116:35
johnsomAnyone else have input?16:36
colin-personally i'd like to have a better sense of deployed scale16:36
johnsomcolin- You can probably already get that from the 2018 survey. We are at least now listed as a project.16:37
colin-it's difficult to ascertain when we discuss complex topics if we're speaking figuratively/academically or from experience and getting a better picture of how broadly deployed octavia is (vexxhost for example?) would help me understand how unique our problems are16:37
colin-just trying to offer some feedback16:37
johnsom#link https://www.openstack.org/analytics16:37
johnsomYep, I understand16:37
colin-so a question that informs that would help me. will review the analytics thanks16:37
johnsomFrom a quick glance at the 2018 survey, 13% of deployments have octavia in production.16:38
johnsomI would have to spend some time with the tool to map that to compute cores for example.16:38
johnsomI am in favor of the two questions cgoncalves asked.16:39
rm_workyep16:40
johnsomrm_work do you want to vote on this or ???16:40
rm_workdo we need to?16:41
johnsomProbably not.16:41
rm_workI haven't seen any disagreement16:41
johnsomrm_work Do you want to reply to the mailing list with the questions?16:41
rm_worknot particularly? :D16:41
rm_workdo I have a choice? ^_^16:42
johnsomLol16:42
johnsomI can do it.  I just thought it might be nice to have another voice from Octavia on the chain16:42
*** yamamoto has quit IRC16:42
*** henriqueof has joined #openstack-lbaas16:42
rm_workyou seem to be a good ML Liaison16:42
colin-perhaps cgoncalves could do so, as he posed the questions?16:42
colin-(sorry carlos!)16:43
cgoncalvesI don't mind :)16:43
* johnsom notes it is interesting, for queens deployments we jump to 29% in prod16:43
johnsomcgoncalves Thanks!16:43
colin-johnsom: do i understand right that one copy of each octavia service and one VIP/listener/member would show up the same on that analytics page as a deployment 1000x that size?16:43
johnsomCorrect16:44
colin-thx16:44
johnsomIt's per-cloud, not instance16:44
rm_workok so16:45
rm_work#action cgoncalves reply to ML user survey email with octavia questions16:45
rm_workcan probably move on16:46
rm_workI didn't have anything else, and wiki didn't have anything, so16:46
rm_work#topic Open Discussion16:46
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:46
johnsomI don't think I have any other topics for today.16:47
johnsomToo much engineering via google docs for me this week16:47
colin-no fun16:47
rm_workyeah, me either16:49
cgoncalves1. Which OpenStack load balancing (Octavia) provider drivers would you like to see supported?16:52
cgoncalves2. Which new features would you like to see supported in OpenStack load balancing (Octavia)?16:52
cgoncalves+1/-1, comments, suggestions, ... please16:52
johnsomThat looks good to me, +116:53
rm_workyeah just need to decide the options16:53
rm_workright? cause... doesn't it have to be multi-choice?16:53
cgoncalvesgood point. johnsom, do you remember how it was for the 2017 survey?16:54
johnsomThese are fill in questions I think.  Let's see what they say.16:54
cgoncalvesok16:55
rm_workah16:55
johnsomI see in their e-mail about the 2019 survey opening a question from ironic ""What would you find most useful if it was part of Ironic?"16:58
johnsomSo, open ended should be ok16:58
*** ccamposr has quit IRC16:58
*** ccamposr has joined #openstack-lbaas16:59
*** ccamposr__ has joined #openstack-lbaas16:59
*** ccamposr__ has quit IRC16:59
*** ccamposr has quit IRC16:59
*** ccamposr has joined #openstack-lbaas16:59
*** gcheresh has quit IRC17:00
johnsomrm_work Are we done with the meeting?17:01
rm_workAH yeah, we're at time17:01
rm_work#endmeeting17:01
*** openstack changes topic to "Discussions for OpenStack Octavia | Train PTG etherpad: https://etherpad.openstack.org/p/octavia-train-ptg"17:01
openstackMeeting ended Wed May  8 17:01:47 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-05-08-16.00.html17:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-05-08-16.00.txt17:01
openstackLog:            http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-05-08-16.00.log.html17:01
eanderssonMinor, but do we want to clean up things like listener_statistics ?17:02
eanderssonIt's growing very slowly, but I don't see any clean up logic.17:03
johnsomeandersson They are cumulative currently, not time series.17:03
eanderssonSure - not a big deal, but they are not cleaned up as part of a lb being removed.17:04
johnsomAh, this is true.17:04
johnsomFrankly I think the whole stats code needs to be refactored (topic I had on the PTG agenda), but cleanup could/should be fixed.17:05
cgoncalvesFYI, http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006043.html17:05
*** ramishra has quit IRC17:09
johnsomeandersson https://github.com/johnsom/stresshm17:10
johnsomupdated17:10
eanderssonThanks - I'll give it a go again tonight.17:11
*** ramishra has joined #openstack-lbaas17:12
eanderssonI wasted like 2 hours trying to figure out why I couldn't push beyond 300 HMs. :p17:12
johnsomHmm, I have done >2000 easy on my desktop vm....17:13
colin-the fabled 14gbps node?!17:13
johnsomNo, I wish17:13
colin-haha17:13
eanderssonWithout the ver issue I could easily push 2.5k17:13
johnsomThe 14gbps was a R72017:13
johnsomThat said I am building a new home lab workstation this week, so I may be able to run that test local soon.17:14
colin-hopefully more space-conscious than the Dell :p17:14
johnsomRight now I have VM spread over laptops, etc.  not ideal for perf testing by any means.17:15
eanderssonI was re-building my 5 node lab yesterday, but a power outage destroyed most of my progress.17:15
johnsomYeah. I was lucky enough to have access to a "real cloud" for the R720 tests17:15
johnsomUgh, bummer.17:15
*** yamamoto has joined #openstack-lbaas17:17
johnsomMore than space, I hope it is a lot quieter than a R720. lol At least quieter when I'm not running a stress test.17:18
eanderssonI just got a bunch of Intel NUCs :P17:19
colin-80 fans screaming at 400k RPM doesn't help you concentrate?17:20
colin-weird17:20
eanderssonI used to have a bunch of supermicro servers (only kept one), but the wife didn't like it :D17:22
johnsomI just realized my new workstation will have 7 fans in it.... Luckily they are all thermal controlled.17:22
cgoncalvesmy wifi-only mini-minimal lab running :) https://photos.app.goo.gl/4D8qqZXQMvVRzPD1A17:23
eanderssonI had to add a few more fans on my workstation and raise it up (because apparently carpet is mandatory in American homes?!)17:23
johnsomPretty much, lol17:24
eanderssonhttps://pbs.twimg.com/media/DzkfU3MUUAAqd9z.jpg <--17:24
eanderssonIt hides the cheap DELL switches I use :p17:25
*** yamamoto has quit IRC17:26
*** ianychoi_ has joined #openstack-lbaas17:29
*** ianychoi has quit IRC17:31
*** ivve has joined #openstack-lbaas17:54
*** ramishra has quit IRC18:09
*** ccamposr has quit IRC18:18
johnsomHa, yet another edge case I missed with host routes: https://storyboard.openstack.org/#!/story/200563518:19
johnsomHost routes spread over two subnets on the VIP....18:19
*** gcheresh has joined #openstack-lbaas18:44
*** trown|brb has joined #openstack-lbaas19:05
*** trown|brb is now known as regain19:05
*** regain is now known as trown19:05
*** gcheresh has quit IRC19:12
*** gcheresh has joined #openstack-lbaas19:22
*** kmadac has quit IRC19:27
colin-carlos the color of your desk messed with my vision for a second, couldn't sense depth hah19:41
colin-thought you had a carpet in a really weird size and shape19:41
cgoncalvescolin-, heh! here's a photo of my office back back in Oct '18: https://twitter.com/carlospt/status/105214604019804160019:45
colin-fancy19:47
*** ataraday_ has quit IRC20:00
*** gcheresh has quit IRC20:00
*** schaney_ has joined #openstack-lbaas20:12
*** schaney has quit IRC20:15
*** imdigitaljim has quit IRC20:15
*** hyang has quit IRC20:15
*** hyang has joined #openstack-lbaas20:16
*** trown has quit IRC20:40
*** Vorrtex has quit IRC21:06
*** henriqueof has quit IRC21:44
*** boden has quit IRC21:45
*** rcernin has joined #openstack-lbaas22:31
openstackgerritAdam Harwell proposed openstack/octavia master: Force amp-agent communication to TLSv1.2  https://review.opendev.org/65790122:47
rm_workI think this works ^^22:47
rm_worknot sure how to test it besides tempest scenarios22:47
rm_work(ie, they still work :D_22:47
rm_work)22:47
*** fnaval has quit IRC22:51
*** yamamoto has joined #openstack-lbaas23:18

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!