Wednesday, 2019-11-13

rm_workit's starting to become hard to type/say availability00:10
rm_workall the letters are blurring together <_<00:10
rm_workwhy did you do "flavorprofile" without an underscore T_T00:11
johnsomGive you something to grumble about.  I thought you didn't like _ or -00:12
rm_workahahahaha i just found this in our noop driver for flavors:00:21
rm_workreturn {"amp_image_tag": "The glance image tag to use for this load "00:21
rm_workobviously someone was imagining image tag would be part of our amp flavors >_>00:21
johnsomYes, someone was...00:23
*** yamamoto has joined #openstack-lbaas00:32
*** yamamoto has quit IRC00:41
*** yamamoto has joined #openstack-lbaas00:46
*** armax has quit IRC00:58
rm_workwell into the pep8 portion of my evening...01:10
*** goldyfruit has joined #openstack-lbaas01:12
johnsomI'm just about to call it for the day.01:13
johnsomGot some time to come back up to speed on my failover patch.01:13
rm_workcool01:22
*** yamamoto has quit IRC01:30
*** yamamoto has joined #openstack-lbaas01:39
*** yamamoto has quit IRC01:48
*** armax has joined #openstack-lbaas02:08
*** armax has quit IRC02:17
*** armax has joined #openstack-lbaas02:18
*** armax has quit IRC02:23
openstackgerritAdam Harwell proposed openstack/octavia master: Stop allowing the deletion of an in-use flavor  https://review.opendev.org/69242702:25
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Availability Zone admin API  https://review.opendev.org/69376502:26
rm_workerrr wtf02:26
rm_workahh rebase02:26
johnsomUmm, I think that was an opps02:26
rm_workno, it's fine02:27
rm_workit needed to be rebased anyway02:27
rm_workit wasn't current02:27
rm_workaaaanywho, there we go02:28
*** yamamoto has joined #openstack-lbaas02:34
*** yamamoto has quit IRC02:46
*** abaindur has joined #openstack-lbaas02:58
*** AlexStaf has quit IRC03:43
*** AlexStaf has joined #openstack-lbaas03:43
*** yamamoto has joined #openstack-lbaas04:24
*** tkajinam has quit IRC04:26
*** tkajinam has joined #openstack-lbaas04:33
*** goldyfruit has quit IRC05:00
*** tkajinam_ has joined #openstack-lbaas05:08
*** tkajinam has quit IRC05:11
*** yamamoto has quit IRC05:30
*** tkajinam_ has quit IRC05:34
*** tkajinam has joined #openstack-lbaas05:36
*** yamamoto has joined #openstack-lbaas05:42
*** tkajinam has quit IRC06:03
*** tkajinam has joined #openstack-lbaas06:05
*** abaindur has quit IRC06:16
*** tkajinam has quit IRC06:30
*** tkajinam has joined #openstack-lbaas06:31
*** tkajinam has quit IRC06:31
*** yamamoto has quit IRC06:47
*** yamamoto has joined #openstack-lbaas06:47
*** yamamoto_ has joined #openstack-lbaas06:50
*** yamamoto has quit IRC06:52
*** tkajinam has joined #openstack-lbaas07:01
*** gcheresh_ has joined #openstack-lbaas07:43
*** yamamoto_ has quit IRC07:52
*** yamamoto has joined #openstack-lbaas07:55
*** trident has quit IRC07:57
*** tesseract has joined #openstack-lbaas08:00
*** maciejjozefczyk has joined #openstack-lbaas08:02
*** trident has joined #openstack-lbaas08:06
*** rpittau|afk is now known as rpittau08:17
*** ivve has joined #openstack-lbaas08:19
*** tkajinam has quit IRC08:35
*** armax has joined #openstack-lbaas09:02
*** armax has quit IRC09:06
*** yamamoto has quit IRC09:18
*** yamamoto has joined #openstack-lbaas09:19
*** yamamoto has quit IRC09:21
*** ianychoi has quit IRC10:24
*** paulbrowne has joined #openstack-lbaas10:30
*** yamamoto has joined #openstack-lbaas10:46
*** yamamoto has quit IRC10:50
*** yamamoto has joined #openstack-lbaas11:22
*** yamamoto has quit IRC11:29
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Availability Zone admin API  https://review.opendev.org/69376511:50
*** yamamoto has joined #openstack-lbaas11:59
*** paulbrowne has quit IRC12:03
*** henriqueof1 has quit IRC12:04
rm_workthere's something in there i need to update in octavia-lib but I don't remember what anymore, it's all blending together lol12:06
rm_workah found it12:12
openstackgerritAdam Harwell proposed openstack/octavia-lib master: Availability zone / metadata validation  https://review.opendev.org/69405712:19
*** yamamoto has quit IRC12:26
*** yamamoto has joined #openstack-lbaas12:38
*** yamamoto has quit IRC12:38
*** yamamoto has joined #openstack-lbaas12:38
*** yamamoto has quit IRC12:44
*** yamamoto has joined #openstack-lbaas12:44
*** rcernin has quit IRC12:54
*** ianychoi has joined #openstack-lbaas13:00
*** yamamoto has quit IRC13:05
openstackgerritAdam Harwell proposed openstack/octavia master: Availability Zone admin API  https://review.opendev.org/69376513:08
rm_workthere, fixed my dumb sql migration typo, and added docs, and a depends-on to the octavia-lib change13:08
rm_workjohnsom: seems that traffic is showing as originating from the VRRP_IP, not the HA_IP when routing to members over the vip-net :(13:31
rm_workwill need to look and see what we missed13:31
*** yamamoto has joined #openstack-lbaas13:34
rm_workthe redhat routing stuff and the ubuntu routing stuff are different... maybe it's only not working for RH amps?13:57
rm_worki just don't know how to read this stuff T_T13:58
*** yamamoto has quit IRC14:09
*** yamamoto has joined #openstack-lbaas14:10
*** yamamoto has quit IRC14:10
*** yamamoto has joined #openstack-lbaas14:11
*** yamamoto has quit IRC14:15
brtknrAs requested, for better supporting Octavia in kolla-ansible, we have put in a few questions here: https://etherpad.openstack.org/p/kolla-ansible-octavia14:16
rm_workok, thanks brtknr!14:22
rm_workwe have our meeting today in a few hours actually, do you think you will attend? you can bring up this topic14:23
rm_workerr, in about 1.5 hours14:23
*** yamamoto has joined #openstack-lbaas14:51
*** TrevorV has joined #openstack-lbaas14:52
*** tesseract has quit IRC14:52
*** tesseract has joined #openstack-lbaas14:55
*** yamamoto has quit IRC14:55
*** armax has joined #openstack-lbaas15:05
*** armax has quit IRC15:10
*** armax has joined #openstack-lbaas15:36
*** gcheresh_ has quit IRC15:47
*** ataraday_ has joined #openstack-lbaas15:55
johnsom#startmeeting Octavia16:00
openstackMeeting started Wed Nov 13 16:00:27 2019 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Octavia)"16:00
openstackThe meeting name has been set to 'octavia'16:00
rm_workohai16:01
rm_workwas reading logs and missed the time :D16:01
rm_workbut go ahead16:01
ataraday_hi16:01
johnsomrm_work Nice, you made it. I put an agenda together:16:01
johnsom#link https://wiki.openstack.org/wiki/Octavia/Weekly_Meeting_Agenda#Meeting_2019-11-1316:01
johnsom#topic Announcements16:02
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:02
johnsomIn case you missed it, our fearless PTL put together a PTG summary e-mail16:02
johnsom#link http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010682.html16:02
johnsomBTW, I know a few people are travelling today, so attendance might be light today.16:03
johnsomThat is about all I have for announcements today.16:04
johnsomI don't know when the video recordings will be available, so no update on that.16:04
johnsomAny other announcements today?16:04
johnsom#topic Brief progress reports / bugs needing review16:05
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:05
johnsomI have been on a bug fix spree recently.  A lot of updates around barbican secrets and TLS offload.16:06
ataraday_Small octaviaclient change become not so small https://review.opendev.org/#/c/693144/ :)16:06
redrobot👀16:07
johnsomOne set are important fixes for LBs with multi-listeners that are using TLS offload.16:07
openstackgerritAdam Harwell proposed openstack/octavia master: Availability Zone admin API  https://review.opendev.org/69376516:07
rm_workWe're making quick progress on the AZ work -- please to be reviewing :D ^^16:07
johnsomredrobot Hi.  Just mentioning some of the patches I posted to handle pkcs12 secrets disappearing (since they don't support registration at the moment).16:07
ataraday_I researched ciphers and ciphersuits a bit - maybe can discuss this in open discussion section16:07
johnsomOk, sounds good16:08
johnsomI am now shifting focus back to the failover flow re-factor.16:09
johnsomAs part of the TLS work, I have proposed tempest tests for all of the listener/frontend TLS paths. TLS, SNI, client auth.16:10
johnsomSuper happy to have good coverage on those.16:10
*** Trevor_V has joined #openstack-lbaas16:11
johnsomThe backend is going to be a bit more work as we need to enhance our testing web server to have TLS support enough to cover the test cases.16:11
johnsomAny other updates today?16:11
*** gcheresh_ has joined #openstack-lbaas16:12
johnsom#topic Helping Kolla-Ansible eitherpad16:13
*** openstack changes topic to "Helping Kolla-Ansible eitherpad (Meeting topic: Octavia)"16:13
johnsom#link https://etherpad.openstack.org/p/kolla-ansible-octavia16:13
johnsomWe have recently had a number of folks come into the channel struggling to use Octavia with the kolla-ansible deployment project.16:13
johnsomI don't think any of the core team regularly contribute to kolla-ansible, but we have offered to help resolve some of the issues.16:14
johnsomIn support of that I setup the above etherpad to answer questions, etc.16:15
*** TrevorV has quit IRC16:15
johnsomPlease feel free to contribute, etc.16:15
johnsomI think we are close to over ten ways to deploy Octavia, so it is to be expected that the Octavia team may not be directly involved in all of them.16:15
johnsomAny other questions/comments on helping kolla-ansible?16:16
rm_workI am probably the most experienced of core members, but have been stretched very thin recently, so not sure how much I can help directly16:17
rm_workbut I will see if I can answer some questions at least16:17
johnsomYep, all I can commit to at this point is helping to answer questions.16:17
johnsom#topic Open Discussion16:18
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:18
johnsomOk, any other topics today?16:18
*** gcheresh_ has quit IRC16:19
rm_workAh, I have one16:19
johnsomOk. I think Ann has one too16:19
rm_workI've recently been testing to see if traffic that goes to members via the vip-net is marked as originating from the vrrp_ip (unknown to users), or the ha_ip (VIP) -- and it looks like it's coming from the vrrp_ip16:20
*** ataraday_ has quit IRC16:20
rm_workHas anyone else noticed this?16:20
rm_workI thought it was set up to route "from" the VIP... We use CentOS amps, it might be working properly in Ubuntu but not in CentOS, or maybe it's not working anywhere...16:21
rm_workAnyway, if anyone has a moment to test that or happens to have noticed it is an issue, let me know16:21
johnsomAh, now that I think about this more, yes, I think that is the case. It is the same as for members going out member subnets. It's an arbitrary source IP16:21
johnsomI should have a stack in about half an hour I can try it on16:21
rm_workcan't we have it originate from the VIP? since haproxy IS bound to that address?16:22
johnsomWell, it cuts into your capacity if I remember correctly16:22
johnsomWe might have to make some jinja changes for that too.16:23
rm_workyeah, I was looking in the port templates16:23
rm_workbut I can't read that stuff very well (the port config, not the jinja -- i speak jinja :D)16:23
johnsomThere is also a RFE to add multiple source IPs, which would conflict with forcing it all to the VIP16:24
rm_workhmmm k16:24
rm_workwell i mean... yeah... multivip :D16:24
rm_workalso16:24
johnsomI am guessing you just want this for "easy security group rules"?16:24
*** ataraday_ has joined #openstack-lbaas16:24
rm_workwell, not "easy" but "any at all"16:25
rm_workit's otherwise impossible for a user to open members to the LB16:25
rm_workwithout opening it to the whole world16:25
rm_worksince they can't predict the vrrp_ip16:25
johnsomOne-armed load balancers are less efficient anyway. I would hope that is a rare usecase16:25
rm_workit's the only use-case16:25
rm_workwe have no SDN16:25
rm_workthere is only one network16:25
johnsomRight, or the member subnet source IP, both of which change on failovers.16:26
*** ivve has quit IRC16:26
ataraday_sorry, I got disconncted, bad hotel wifi16:26
rm_workwe need a solution to this16:26
johnsomRight now you have two options: Put the members on private networks with no router.16:26
johnsomUse TLS client auth for the members16:26
rm_workyeah neither of those are possible/viable with our setup16:27
rm_workper PCI compliance we just can't open firewall ports that widely apparently16:27
johnsomDo you use FWaaS?16:27
johnsomWait, what? Neither of those options provided require opening ports16:27
rm_workno, there are physical (and somewhat manually managed for compliance) firewalls for our PCI environments16:28
*** yamamoto has joined #openstack-lbaas16:28
rm_workusing TLS client auth for members "secures them" but still requires opening up the SGs to the world16:28
rm_workie exposing the serving port16:28
rm_workis what i meant16:28
rm_workwhich is non-viable16:28
johnsomNot the world, just the range you are using for you VIP addresses16:29
johnsom(the base ports are on the same range as the VIP)16:29
rm_workyeah, but that means "any LB" and was rejected16:29
rm_workso, no-go16:29
johnsomRight. Did you answer my question, do you have FWaaS?16:29
rm_workno we do not16:29
rm_workwe also do not have the ability to let users create private networks T_T16:30
rm_workand humorously I'm 2 for 2, this is exactly the same issues we had at GD lol16:30
johnsomYeah, then at this point, there is no solution for that setup.16:30
rm_workso either it's not that rare, or I manage to pick exactly the two companies with deployments like this16:31
johnsomHa, that later16:31
rm_worki don't believe in those odds :D16:31
johnsomYeah, so you would need an RFE for any of the other changes I can think of. All of which really suck.16:32
rm_workthis issue was actually a decently relevant factor for the death of octavia in the GD deployment, and it worries me a lot here16:32
rm_workand I really doubt it is just us16:32
*** yamamoto has quit IRC16:32
rm_workas funny as that would be :D16:32
johnsomThe one-armed solution would require a config option, then change the haproxy jinja to force the source to be VIP. This will have a negative impact on the performance and capacity of the LB.16:33
rm_workone solution is to allow the user to pass us a SG just to attach it to the port -- then they can use that to allow traffic in16:33
rm_workthough it's a little janky16:33
rm_workor i wonder if it is possible for a user to "allow" traffic via SG that they don't own -- if we exposed one for them16:34
rm_workyou can say "allow traffic from any port with security_group <ID>", right?16:35
rm_workas a SG rule16:35
johnsomThe other would be (possibly, haven't tried it) to pass an SG ID to each member API create, then have the source ports added to that SG. In theory the neutron transitive-trust would magically make traffic flow. The downside is it would also open those arbitrary ports on our amphora source IPs.16:35
johnsomYour second idea is what we can do with FWaaS if I remember, but I'm not sure neutron proper can do it.16:36
johnsomI wish SGs had better support for "AND" than it does. Really that would solve a bunch of our problems.16:39
rm_workhmm, i thought that was just part of the core SG stuff16:39
rm_workthat definitely worked at GD and I didn't think we deployed FWaaS16:39
rm_workI'll check with Miguel and we can discuss it another time16:39
johnsomOk16:39
rm_workYou answered me question about drawbacks of the one-armed solution16:39
johnsomataraday_ You had a question about ciphers/suites?16:40
rm_workthough ... in reality, there's really only one physical NIC in use across all of these virtual NICs anyway, so I don't know how much it really affects throughput16:40
rm_work(besides I guess the number of concurrent connections? though i don't think the member side will ever be the limiting factor there)16:40
johnsomWell, that is a deployment flaw, but It's not just the nic. Don't forget you have TCP ports in use, queues in the kernel, etc. that have nothing to do with your NIC topology (though you should have more than one)16:41
ataraday_johnsom, yes, I put comment on https://review.opendev.org/#/c/685337/ -  I look closer and we can use ssl-default-server-ciphersuites and ssl-default-server-ciphers in one config file. But ciphersuites available only since haproxy 1.8.2016:41
ataraday_and for now we have 1.8.816:42
rm_workright, but given that there is one VIP, and it splits those connections to many members, I think it is impossible for that to be an issue on the backend side16:43
johnsomAh, interesting. We should probably request Ubuntu to update the available package. That said, we have code that can detect the version of HAProxy and make changes. This seems like a good candidate for that.16:43
rm_workeugh, another thing going to 2.0 would resolve :D16:43
johnsomataraday_ https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/agent/api_server/haproxy_compatibility.py16:44
ataraday_johnsom, thanks for pointing this!16:45
johnsomWe could expand that to remove the ciphersuite configuration line based on the available version. If it's not greater than 1.8.20 it probably doesn't support TLS 1.3 anyway.16:45
ataraday_yeah, seems the way to do that16:46
johnsomIt looks like my code there only does major minor, so it may need to be expanded to look at the patch number too16:46
johnsomKind of lame they added that in a patch release reallly16:48
rm_workyeah16:49
rm_workalternatively, we had some plans to pre-cache this info16:49
rm_workas a possible way to deal upfront with option validation for providers16:49
rm_workwe discussed this a bit at the summit, I believe it was in my summary, and there are more notes on the etherpad16:50
johnsomYeah, there is code for round tripping for the version too, but this seems straight forward for this case16:50
johnsom#link https://github.com/openstack/octavia/blob/master/octavia/amphorae/drivers/haproxy/rest_api_driver.py#L7816:51
johnsom#link https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/agent/api_server/haproxy_compatibility.py16:51
johnsomForgot we were in a meeting. lol16:51
johnsomThat L78 is the controller side query for version, but again, it might be more straight forward for this issue to just add it to the agent side adjustments.16:52
johnsomAny other topics today?16:53
*** goldyfruit has joined #openstack-lbaas16:55
rm_workreviews! reviews reviews! all reviews are useful!16:56
rm_workonly cores can +2, but the real power is in -1 and that's the same for everyone! reviews!16:56
johnsomYes please. I did a review-day recently, but we still have a bunch of patches that can use some reviews. +1's matter!16:56
rm_workI guess I'm just thinking more negatively today :D16:57
johnsomTrue, -1 matters more. lol16:57
johnsomOk, thanks for a great meeting!16:58
rm_workI expect most code I write is going to have a bug or two that I didn't catch, and I'm counting on you guys to find them! it's a treasure hunt for bugs! the reward is internet brownie points! <316:58
johnsom#endmeeting16:58
rm_worko/16:58
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews"16:58
openstackMeeting ended Wed Nov 13 16:58:30 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:58
openstackMinutes:        http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-11-13-16.00.html16:58
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-11-13-16.00.txt16:58
openstackLog:            http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-11-13-16.00.log.html16:58
*** rpittau is now known as rpittau|afk17:08
*** gcheresh_ has joined #openstack-lbaas17:32
rm_workOur two-node checks have been super unreliable recently :(17:36
johnsomYeah, it was a topic in the neutron PTG I think. multi-node devstack with neutron is broken I think17:36
johnsomI had hoped haleyb could help us out with those...17:37
*** gcheresh_ has quit IRC17:43
haleybjohnsom: we did discuss jobs in the neutron room, mostly about removing them, but i can take a look at this failing job next week17:52
johnsomhaleyb Cool. It seems like communication between the primary node and the secondary only sometimes works.17:53
johnsomIt's probably some devstack setting got missed or is being used wrong17:54
*** maciejjozefczyk has quit IRC17:58
*** yamamoto has joined #openstack-lbaas18:17
*** yamamoto has quit IRC18:21
*** tesseract has quit IRC18:35
johnsomAlright, gave the kolla-ansible etherpad a pass.18:43
*** gcheresh_ has joined #openstack-lbaas19:23
*** yamamoto has joined #openstack-lbaas19:36
*** yamamoto has quit IRC19:41
*** ataraday_ has quit IRC19:43
*** gcheresh_ has quit IRC19:51
*** yamamoto has joined #openstack-lbaas20:11
*** abaindur has joined #openstack-lbaas20:14
*** yamamoto has quit IRC20:15
*** gcheresh_ has joined #openstack-lbaas20:27
*** abaindur has quit IRC20:40
*** abaindur has joined #openstack-lbaas20:40
*** abaindur has quit IRC20:41
*** abaindur has joined #openstack-lbaas20:42
*** abaindur has quit IRC20:45
*** gcheresh_ has quit IRC21:39
rm_workseeing a lot of grenade failures too and not sure if it's my/our fault or not <_<22:02
rm_worksorrison: the new API code looks like it's running ok now, let me know if you run into any issues rebasing on it22:03
rm_workhmm yeah the grenade thing seems like maybe a nova issue? not sure... logs basically just look like before we even get to test octavia, various tests fail because there's no hosts available for VM boots... maybe i'm missing the root cause tho22:04
*** yamamoto has joined #openstack-lbaas22:05
rm_workbut this happens VERY early: Details: {u'code': 500, u'message': u'No valid host was found. There are not enough hosts available.', u'created': u'2019-11-13T17:25:12Z'}22:05
*** yamamoto has quit IRC22:10
*** yamamoto has joined #openstack-lbaas22:15
*** rcernin has joined #openstack-lbaas22:32
*** Trevor_V has quit IRC22:35
*** abaindur has joined #openstack-lbaas22:46
*** tkajinam has joined #openstack-lbaas23:06
*** yamamoto has quit IRC23:20
*** goldyfruit has quit IRC23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!