Wednesday, 2018-09-19

johnsomI'm going to keep plugging forward while you take a look00:00
*** dayou_ has joined #openstack-lbaas00:01
rm_workkk00:02
rm_workyeah debugging this, i wrote a test00:02
rm_workit might be called 'rules'?00:03
rm_workbecause we're dumb00:03
rm_worki forget00:03
johnsomI think I have it as l7rules in the patch I pushed00:04
rm_workerrrrr does pool type even return l7policies?00:05
rm_worki think listeners do?00:05
johnsomIt's all in the models.py now00:05
rm_workyeah nm i see what you mean00:07
rm_workjohnsom: got it i think?00:16
rm_workyou were not using dots00:16
rm_worki said use dots :P00:16
johnsomSo use this? joinedload(models.Pool.l7policies).joinedload(models.L7Policy.l7rules)00:16
rm_worksec00:17
openstackgerritAdam Harwell proposed openstack/octavia master: DNM: example pool l7policy->rule load  https://review.openstack.org/60352700:17
rm_work^^ that00:17
rm_workjust shoved it on the end, will abandon when you roll it in00:17
rm_workassuming it's what you want00:17
johnsomLet me try00:17
rm_workit works00:17
rm_workit's just, does the query it makes go fast enough for you00:17
rm_worki attached a test00:18
rm_workpossibly it's not in the right place, but it works for debugging this00:18
rm_workyou need rules AND listener00:18
rm_workbecause of how we do a thing00:18
rm_workthough ... uhh00:18
rm_workhold on, maybe can fix that00:18
rm_workwe might be doing something dumb00:19
rm_worknah i think we need that00:20
johnsomYeah, that seems to work00:20
rm_workis the query faster?00:20
johnsomWell, there is a margin of error on "faster", but I can see it's not pounding the DB like the lazyload was, so win in my book00:21
johnsomreal    0m2.242s00:21
johnsomrm_work Thanks!00:25
rm_worksweet :P00:25
* rm_work successfully pretended to be a SQLAlchemy expert00:25
rm_workthe *second* you think you're good, i'll get the pipeline running to deploy that to my staging env00:37
rm_workhoping to have it in prod by thursday :P00:37
johnsomI have l7policy and l7rule left00:37
rm_workkk00:37
johnsomGoing to make the spouse grumpy and finish tonight00:37
rm_workT_T00:37
rm_worktell her I appreciate it too! and I will have to bring some good booze if I ever get down there for smoked meats :P00:38
* rm_work will see if he can get through sapd1's patch now00:38
johnsomHmm, fun, get to figure out why we have a custom l7policy get_all00:38
rm_work<_<00:39
johnsomAh, ordered list00:40
johnsomok, I can work with this00:40
rm_worksapd1: reviewed, a few things need fixing I think00:52
rm_workbut it's close00:52
abaindurHow often is amphora base image updated?01:03
abainduris it tied to OS releases, or is there a separate release notes page for it?01:04
abaindurWe are wondering when and how often to rbuild the amphora disk image01:04
johnsomabaindur Our code in the image is pretty stable and the release notes will point out when it needs to be built for an upgrade. Other than that, it's how often you want to pull in the distro patches.01:05
rm_workabaindur: for reference, i rebuild the image every time i deploy any new code to the controllers01:08
johnsomAssuming of course that you haven't enabled patching in your amp image locally01:08
rm_workwhich is about once every month or so01:08
johnsomrm_work amp list left (forgot about it)01:09
rm_workk01:11
rm_workis that one so bad? lol01:11
johnsomBefore: real    0m30.995s01:12
rm_workugh01:13
*** abaindur has quit IRC01:13
rm_workok :P01:13
johnsomafter: real    0m1.698s01:14
johnsomHa, it even output the right data..... ( have been checking each one before/after )01:17
johnsomrm_work: http://lists.openstack.org/pipermail/openstack-dev/2018-September/134681.html01:21
johnsomRenaming AZs... Thus the push for "cached" in the field01:21
rm_work:P01:22
rm_workwell, they solved the problem ;)01:22
rm_workcan't rename an AZ with an instance01:23
rm_workhuzzah01:23
rm_workwe're safe anyway :P01:23
openstackgerritMichael Johnson proposed openstack/octavia master: Fix the API list performance regression  https://review.openstack.org/60324201:26
johnsomrm_work ^^^^ Should be fully functional01:27
johnsomI haven't checked the coverage report yet, but pep8,py27,functional all pass01:27
johnsomOk, I think I will let that cook in zuul and will start back in the morning. o/01:32
bzhao__rm_work:  Hi, you send me a message in local time 3:03AM.. Now I'm in office. ;-)01:33
rm_workbzhao__: hah ok yeah, sorry01:36
rm_workwas hoping you could answer some questions about some code, but i figured it out01:36
rm_workbzhao__: did a little bit of refactoring on the UDP status stuff, if you want to verify that it will work for you: https://review.openstack.org/60349001:37
*** hongbin has joined #openstack-lbaas01:37
rm_workjust simplified a bit of that, which was causing some issues on different platforms for me (due to the way different platforms expanded ipv6 for some reason)01:37
sapd1rm_work: I will fix it01:38
rm_worksapd1: cool, i'll check out your revisions tomorrow if i have time :)01:39
bzhao__rm_work: never mind.. Yeah, looking .01:39
bzhao__rm_work: OK, sorry about not very familiar with the lib. I will test it today, hard to verify through just looking . ;-). I will leave a comment in that patch.01:50
rm_workyeah, it's a little difficult to verify by looking, correct :P01:50
rm_workand I don't have a good setup to functionally test UDP stuff01:50
bzhao__haha. ;-)01:50
rm_workso I was hoping you could do that01:50
rm_workand make sure it doesn't break you01:50
bzhao__OK. I will do that.01:51
bzhao__Thanks very much for the refactoring.01:51
rm_workno worries, mostly i just want the tests to be reliable cross-platform01:52
rm_workand hopefully the new code is a little easier for someone to maintain01:52
rm_workjohnsom: deploying your patch to staging soon :)01:53
openstackgerritsapd proposed openstack/octavia master: Support REDIRECT_PREFIX action for L7Policy  https://review.openstack.org/60108601:54
sapd1rm_work: I posted a comment for your question about jinja template generate.02:00
rm_workk02:00
rm_worki think i understood actually when i got further02:00
sapd1rm_work:  Are you running octavia for production? :D02:00
rm_workyes :)02:01
rm_workusually try to run a few patches ahead of master02:01
rm_workbut i'm like three months behind in my deployment due to other issues02:01
rm_workabout to be caught up again02:01
sapd1rm_work: The next issue I got is change VIP QOS of the load balancer.02:03
*** sapd1_ has joined #openstack-lbaas02:04
rm_workyeah ok it just looked weird because the further indented stuff was first02:04
rm_workstrange, but i think i get it02:04
rm_worklol so close02:05
rm_worklast comment02:05
rm_workand then prolly +202:05
rm_workhmmm though i would also really like to see a test for it in octavia-tempest-plugin02:06
rm_workwith a depends-on to this patch so we can see it working02:06
rm_workhmmm though i didn't add SSL stuff to that yet so it'll be difficult i guess02:06
rm_worki think this is fine02:07
sapd1_rm_work: I have run it in production now. :D02:07
*** sapd1 has quit IRC02:07
openstackgerritsapd proposed openstack/octavia master: Support REDIRECT_PREFIX action for L7Policy  https://review.openstack.org/60108602:07
rm_workah, an api-test would be good -- https://github.com/openstack/octavia-tempest-plugin/blob/master/octavia_tempest_plugin/tests/api/v2/test_l7policy.py02:07
rm_workbut i think i'm +2 anyway02:08
rm_workbut, an API test is in your best interest, that way it doesn't break on you in the future :P02:08
sapd1_yes. I think so02:08
rm_workcan basically just clone one of the one for like, redirect_url02:09
rm_work*the one for02:09
sapd1_exactly02:09
sapd1_But I'm considering about add more option for that such as redirect with status code.02:09
bzhao__sapd1_:  301?02:11
bzhao__sapd1_:  Is that the cases we discussed in a few weeks before ? ;-)02:12
sapd1_I don't remember. But I think I have to research about some status code when redirect to each case. 301 or 30202:14
rm_workpermanent or temporary redirect02:16
rm_worki forget which one we decided to use behind the scenes, but i know some people would prefer the other one, or at least a choice :P02:16
bzhao__I mean this one https://storyboard.openstack.org/#!/story/2003609 , I raise this RFE in PTG, so our LB team agree to add the prefix option and http status code for fit that case. And right now, we default support the 302 redirection, so maybe introduce 301 or more status codes for different http versions. ;-)02:16
bzhao__I remember we discuss and load the configration details before PTG. ;-)'02:17
bzhao__rm_work: yeah. ;-)02:18
sapd1_bzhao__: Yes. I remember now.02:19
johnsomrm_work: Sure co-author....02:22
rm_workwhat? :P02:40
openstackgerritsapd proposed openstack/octavia master: Support REDIRECT_PREFIX action for L7Policy  https://review.openstack.org/60108602:40
rm_workah heh02:40
rm_workdidn't notice you added me02:41
rm_workdeploying...02:44
rm_workdeployed02:47
rm_worktesting :P02:47
*** ramishra has joined #openstack-lbaas03:08
*** cgoncalves|pto has quit IRC03:16
*** cgoncalves has joined #openstack-lbaas03:17
sapd1_good03:30
*** yamamoto has joined #openstack-lbaas04:07
*** hongbin has quit IRC05:06
*** reedipb has joined #openstack-lbaas05:11
reedipbjohnsom, xgerman_ , rm_work : https://review.openstack.org/#/c/599393/ if you get some time05:12
*** rcernin has quit IRC05:30
*** rcernin has joined #openstack-lbaas05:30
*** luksky11 has joined #openstack-lbaas05:51
sapd1_rm_work:  I don't know this test failed. http://logs.openstack.org/86/601086/9/check/octavia-v2-dsvm-scenario-centos-7/df59df0/ara-report/reports/4c285bb3-6865-46bc-918a-54f28391fe8f.html05:55
sapd1_*why05:55
rm_worksapd1_: not your problem06:00
rm_workyum repo failed to fetch06:00
rm_workjust wait for recheck06:01
sapd1_I have just rechecked. Maybe need one more recheck06:01
*** rcernin_ has joined #openstack-lbaas06:04
*** rcernin has quit IRC06:06
*** ccamposr has joined #openstack-lbaas06:21
*** rcernin has joined #openstack-lbaas06:34
*** rcernin_ has quit IRC06:36
*** phuoc_ has quit IRC07:02
*** celebdor has joined #openstack-lbaas07:04
*** phuoc has joined #openstack-lbaas07:04
*** celebdor has quit IRC07:05
*** celebdor has joined #openstack-lbaas07:05
*** rcernin has quit IRC07:06
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Failover test  https://review.openstack.org/50155907:07
rm_worksapd1_: possibly something broke in one of the main centos repos and that test is just failing for everything <_<07:21
sapd1_rm_work: Ok. I'm waiting :D07:23
rm_workprolly someone will have to figure it out tomorrow07:23
rm_worklike, someone that isn't us07:24
rm_workguessing it's an upstream breakage07:24
sapd1_rm_work:  Where do you live? It's midnight now?07:25
rm_workyeah07:25
*** blake has joined #openstack-lbaas07:38
*** Emine has joined #openstack-lbaas08:06
*** blake has quit IRC08:32
*** luksky11 has quit IRC08:39
*** salmankhan has joined #openstack-lbaas09:06
*** luksky11 has joined #openstack-lbaas09:09
*** maciejjozefczyk has joined #openstack-lbaas11:14
maciejjozefczykhey! I have little question. To achieve Active-Passive Octavia loadbalancer the only way of doing that for now is to use vrrp + keepalived inside amphora?11:16
*** celebdor has quit IRC11:48
*** celebdor has joined #openstack-lbaas12:46
*** reedipb has quit IRC12:56
*** maciejjozefczyk has quit IRC13:46
*** maciejjozefczyk has joined #openstack-lbaas13:46
*** sapd1 has joined #openstack-lbaas13:52
xgerman_maciejjozefczyk:  Yes14:14
*** salmankhan1 has joined #openstack-lbaas14:24
*** salmankhan has quit IRC14:26
*** salmankhan1 is now known as salmankhan14:26
*** salmankhan has quit IRC14:34
*** salmankhan has joined #openstack-lbaas14:34
*** salmankhan1 has joined #openstack-lbaas14:35
*** salmankhan has quit IRC14:38
*** salmankhan1 is now known as salmankhan14:38
*** dayou_ has quit IRC14:40
*** salmankhan has quit IRC14:44
*** salmankhan has joined #openstack-lbaas14:50
*** Emine has quit IRC15:10
*** luksky11 has quit IRC15:15
*** salmankhan1 has joined #openstack-lbaas15:26
*** salmankhan1 has quit IRC15:27
*** salmankhan has quit IRC15:28
*** pcaruana has joined #openstack-lbaas15:47
*** yamamoto has quit IRC15:53
*** salmankhan has joined #openstack-lbaas16:01
*** yamamoto has joined #openstack-lbaas16:26
*** pcaruana has quit IRC16:27
*** ramishra has quit IRC16:42
*** salmankhan has quit IRC16:57
*** fnaval has joined #openstack-lbaas17:11
rm_workmaciejjozefczyk: technically i have a driver that does it via FLIPs at L317:19
rm_workbut it's not really for public consumption unless you know what you're doing (though it is pushed up as a review)17:19
xgerman_would still be keeplaived + vrrp — juts not ARP17:26
*** ccamposr has quit IRC17:30
rm_workyeah true lol17:30
openstackgerritGerman Eichberger proposed openstack/octavia master: Updates the operator docs with the new lb failover command  https://review.openstack.org/60346317:33
sapd1maybe zuul is dead today :D17:36
johnsomYeah, bit of a backlog today17:37
openstackgerritGerman Eichberger proposed openstack/octavia master: Updates the operator docs with the new lb failover command  https://review.openstack.org/60346317:37
openstackgerritMerged openstack/neutron-lbaas-dashboard master: Imported Translations from Zanata  https://review.openstack.org/60331717:46
*** sapd1 has quit IRC18:01
*** evgenyf has joined #openstack-lbaas18:18
openstackgerritMerged openstack/octavia-tempest-plugin master: Fix test_with_ipv6 = False failure  https://review.openstack.org/59840518:48
openstackgerritMerged openstack/octavia-tempest-plugin master: Update links in README.rst  https://review.openstack.org/57503018:48
*** celebdor has quit IRC19:15
*** salmankhan has joined #openstack-lbaas19:51
*** evgenyf has quit IRC20:00
johnsom#startmeeting Octavia20:00
openstackMeeting started Wed Sep 19 20:00:04 2018 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.20:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.20:00
*** openstack changes topic to " (Meeting topic: Octavia)"20:00
openstackThe meeting name has been set to 'octavia'20:00
johnsomHi folks!20:00
xgerman_O/20:00
rm_worko/20:00
johnsom#topic Announcements20:00
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"20:00
johnsomMight be a light meeting as I know some folks are on vacation (again, grin)20:00
johnsomYour lowly PTL took notes during the PTG, please see the etherpad20:01
johnsom#link https://etherpad.openstack.org/p/octavia-stein-ptg20:01
johnsomAlso TC elections have started, so check your e-mail for a ballot20:01
xgerman_Is the vacation a Redhat or Europe thing?20:01
johnsomI think it's a Europe thing, but I don't know.20:02
johnsomCertainly not a "johnsom" thing sadly20:02
johnsomHis vacation was the PTG... lol20:02
xgerman_nor xgerman20:02
nmagneziO/20:02
johnsomThere is also an email about the zuul backlog today.  Summary is one of the providers is doing upgrades, so the pool of instances is lower than normal.20:03
johnsomAny other announcements today?20:03
johnsomAh, we did cut a stable/Pike release this week as well.20:04
johnsom#topic Brief progress reports / bugs needing review20:04
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"20:04
xgerman_Yep, promoted cgoncalves  to release manager!!20:04
johnsomHa, yeah, cgoncalves is now empowered to cut releases. Hopefully he can help get stable releases out at a better pace than I did in Rocky20:05
johnsomWell, other than taking notes at the PTG I focused on fixing the Octavia v2 API "list" performance regression that a few folks have mentioned.20:06
johnsom#link https://review.openstack.org/60324220:06
johnsomOther than a release note I think this is good to go.20:06
johnsomIt dropped an LB list from 2:30 to a few seconds on my test setup.20:07
johnsomThis is a backport candidate too IMO20:07
xgerman_I have been looking into the AAP driver and see if I can implement one of our self-healing ideas (octavia shadow port)20:07
johnsomI have some gate work to address, then likely will start on the IPv6 Act/Stdby bug.20:07
johnsomnmagnezi I think I will need to do something similar to what you started here:20:08
johnsom#link https://review.openstack.org/58929220:08
johnsomMind if I expand on that for the IPv6 fix?20:08
nmagnezijohnsom, sure. But I have been putting out internal fires so had to shift my focus :< last I check that didn't actually work for me but I do plan to get back to this20:09
johnsomnmagnezi Ok, I may pick up that patch and run with it for the IPv6 patch. I think they are related.20:10
johnsomAlso, evgenyf from Radware added a few stories to the agenda looking for feedback:20:10
johnsom#link https://storyboard.openstack.org/#!/story/200373820:10
johnsom#link https://storyboard.openstack.org/#!/story/200373720:11
johnsom#link https://storyboard.openstack.org/#!/story/200373320:11
johnsomIf you have time, please provide feedback on those. (I haven't yet, sigh)20:11
johnsomAh, they look like potential provider driver related bugs. I will have to dig into those20:12
johnsomAny other updates?20:12
johnsomI saw there was a fix to the neutron-lbaas namespace driver memory leak, so that is cool20:13
johnsomOk, moving on20:14
johnsom#topic Talk about VIP security groups20:14
nmagneziYes, Carlos submitted that while I was away so still looking at it20:14
*** openstack changes topic to "Talk about VIP security groups (Meeting topic: Octavia)"20:14
nmagneziBut looks good20:14
nmagnezi:)20:14
johnsom#link https://review.openstack.org/#/c/602564/20:14
johnsomThis patch was proposed and discussed a bit earlier in the week in the IRC channel20:14
johnsomIt is proposing moving the VIP security group into the tenant project so they can change it to only allow certain source IPs20:15
johnsomI added this to the agenda to raise awareness and open the floor to discussion on it.20:15
xgerman_this has been a long standing ask and some tools (gopher cloud) hacked soemthing to make that happen to some degree20:15
johnsomPersonally I'm not in favor as we already see bad things happening once we added the VIP port to the tenant.20:16
johnsomMostly from runaway "cleanup" scripts users write that delete anything in their projects.20:16
xgerman_Yeah, some people think tenants will own their bad decisions but in relaity they just blame Octavia20:16
johnsomRight, and raise support calls20:17
johnsomAt the PTG we discussed folks researching options for "stacking" SGs on a port.  Did anyone get time to look at this?20:17
xgerman_Sorry, got side tracked…20:18
xgerman_well, they own te port so they can stack whatever they want ;-)20:18
johnsomYeah, I haven't either. Just got caught up on e-mail today.20:18
johnsomYeah, my understanding for the discussion was that the SG on the VIP doesn't actually impact traffic, that it is only the "VRRP" or base port that matters. The tenant doesn't own that port20:19
xgerman_mmh, really depends on how the rules are implemented (e.g. if they are ip based)20:20
johnsomOk, maybe we should table this until next week.20:20
xgerman_I know they resolve remote ports to its but not sure about local ones20:21
xgerman_its=ips20:21
xgerman_johnsom: +120:21
johnsomOther ideas were: accepting an SG and stacking it via the Octavia worker. Adding an ACL API to the LB.20:21
xgerman_or giving them control of SG and using FWaaS for our stuff20:21
johnsomYeah, FWaaS was another proposal20:22
johnsomOk, tabling for next week for more research time20:22
johnsom#topic Storyboard meeting to discuss "priority" metadata20:22
*** openstack changes topic to "Storyboard meeting to discuss "priority" metadata (Meeting topic: Octavia)"20:22
johnsomnmagnezi I met with the storyboard folks last week.20:23
* nmagnezi listens20:23
johnsomI thanked them for some of the enhancements they have provided.20:23
johnsomI also raised the "we should have a priority/severity" metadata field.20:23
johnsomWhich of course I got back "you are doing it wrong".20:24
nmagneziThey proposed we simply use tags?20:24
johnsomSo, if you folks would like to have a discussion with the storyboard team about this, I offered to setup a meeting time (via the dev mailing list) where we can have an IRC discussion about this topic.20:24
johnsomWhy it is needed, how it would be used, etc.20:25
johnsomnmagnezi are you guys interested in that and willing to provide the details for a larger discussion on the topic?20:25
johnsomWe could do IRC or just start an e-mail list chain20:26
nmagnezisure, I'll follow up on that. Can't we simply discuss this as an agenda item in their weekly meeting?20:26
johnsomI think you folks workflow is the driver for the discussion, so kind of want you on point if we are doing it20:26
johnsomWe could, but we want to announce on the mailing list the time so we could get bigger participation20:27
nmagneziack.20:27
nmagneziWe have some PTOs next week, but I'll follow up on that. I can possibly join the discussion from my PTO anyways20:28
johnsomOk, we can do it the week after, no rush20:28
nmagneziEven better20:28
johnsomOk20:28
johnsom#topic Open Discussion20:28
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"20:28
nmagneziIs there any preferred day for you? (In case you would want to participate)20:28
johnsomAny other topics for today?20:28
johnsomOk then, thanks folks!  Have a great week going forward!20:31
johnsom#endmeeting20:31
*** openstack changes topic to "OpenStack PTG etherpad: https://etherpad.openstack.org/p/octavia-stein-ptg"20:31
openstackMeeting ended Wed Sep 19 20:31:21 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)20:31
openstackMinutes:        http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-09-19-20.00.html20:31
xgerman_O/20:31
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-09-19-20.00.txt20:31
openstackLog:            http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-09-19-20.00.log.html20:31
nmagnezio/20:31
openstackgerritMichael Johnson proposed openstack/octavia master: Fix the API list performance regression  https://review.openstack.org/60324221:02
johnsomAdded a release note21:03
rm_workcool21:03
rm_workworking for me in staging21:04
rm_workgoing to prod with it tomorrow21:04
rm_workfeel free to review https://review.openstack.org/#/c/558962/21:04
rm_workeven has a releasenote :)21:04
*** salmankhan has quit IRC21:53
*** fnaval has quit IRC22:03
johnsomOk22:12
johnsomAh22:12
johnsomAZ rolled22:12
*** rcernin has joined #openstack-lbaas22:44
openstackgerritMerged openstack/octavia master: Updates the operator docs with the new lb failover command  https://review.openstack.org/60346323:23
rm_workjohnsom: lol, i wish this patch was nearly as funny as a rickroll, but alas23:25
rm_workholy dieties, the gate queues... worse than i expected23:25
rm_workdeities? yeah that's the spelling23:25
johnsomYeah, you know it's bad when they send out an e-mail about it23:26
johnsomstable periodic jobs have been waiting 17 hours23:26
openstackgerritMichael Johnson proposed openstack/octavia-dashboard master: Add Python 3.6 unit test jobs  https://review.openstack.org/60393123:42
openstackgerritMichael Johnson proposed openstack/neutron-lbaas-dashboard master: Add Python 3.6 unit test jobs  https://review.openstack.org/60393223:44
rm_worki see post-merge jobs at 19h23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!