Wednesday, 2017-04-05

*** _ducttape_ has quit IRC00:00
rm_worklol k00:01
rm_worki broke it00:01
rm_workwell, give me a sec on rules00:01
johnsomI just checked out master to double check00:01
rm_workalmost got the admin_state thing done00:01
rm_workoh00:01
rm_workyou mean v1 isn't working on master?00:02
rm_worki thought you said it was T_T00:02
johnsomNo, I'm going to double check that it does work on master....00:02
rm_workah k00:02
rm_workyeah i am pretty sure i just broke it00:02
rm_workit worked with sqlite but <_<00:03
rm_worksometimes things are wonky00:03
johnsomNo, ok, it doesn't work for me on master v1 either00:04
rm_workah00:04
rm_work>_<00:04
rm_workwell i'm going to revert the changes I made around trying to fix it00:04
rm_workand then we can file a bug00:04
*** ducttape_ has joined #openstack-lbaas00:04
johnsomOk00:05
johnsomI wonder if it is this screwy sqlalchemy stuff.  If I say position 6 does this reorder junk think it's really 1 because it's the only policy?????00:06
johnsomThat would be confusing as heck for users as it prints out this position #00:06
rm_workyeah00:08
rm_workif you put in position 1000 it will reorder it to 100:08
rm_workthat was by design00:09
johnsomIf a new policy is created with a position that matches that of an existing policy, then the new policy is inserted at the given position.00:09
johnsomIf a new policy is created without specifying a position, or specifying a position that is greater than the number of policies already in the list, the new policy will just be appended to the list.00:09
johnsomPer stephen's docs00:09
johnsomSo, ok, maybe pebkac00:09
johnsomWe probably should have called that rank in the list/show and position in the set00:10
rm_workpushing up the admin_state fix in one sec, along with a rollback to my changes to the policy reorder code, so i don't confuse things when someone tries to go fix it00:10
rm_worki see some cleanup that could be done00:11
rm_workbut it'll take a deeper look00:11
johnsomOk, I need to sign off pretty soon for the evening00:11
rm_workhmm k00:12
rm_worksame, but00:12
openstackgerritAdam Harwell proposed openstack/octavia master: Align Octavia API to n-lbaasv2 for L7Rules  https://review.openstack.org/40633600:12
openstackgerritAdam Harwell proposed openstack/octavia master: Align Octavia API to n-lbaasv2 for L7Policy  https://review.openstack.org/40632800:12
rm_workthere you go :P00:12
johnsomOk, if admin is good and we rolled back the position stufff, this is probably good to go00:12
rm_workneed you to test it00:13
rm_workbut it should be00:13
rm_workhopefully a quick test since i'm not messing with migrations anymore00:15
johnsomYeah, working on it00:15
rm_workcan you pastebin your test commands?00:17
rm_worki will try to track down this policy ordering thing00:17
rm_workhave the debugs in place00:17
rm_workyeah ok I'll see if i can track down this policy reorder issue... maybe later today00:20
rm_workgonna head out for now though00:21
*** ducttape_ has quit IRC00:22
johnsomI think it works and I was just using it wrong....00:22
johnsomThese pass my tests now, so +2s from me00:22
rm_workoh00:24
rm_workerp00:24
rm_workreorder is for like00:24
rm_workcreate two policies, A(position=1) and B(position=2)00:25
rm_workthen update B(position=1)00:25
rm_workand it should make B=1 A=200:25
rm_workwhen you get A and B00:25
rm_workbut, sweet!00:25
rm_workdiltram / xgerman: your court now :)00:25
eanderssonHave anyone seen very long response times to get_loadbalancers? We don't have a crazy amount of lbs, but seeing super slow api requests (50s+).00:28
johnsomYeah, reorder seems to work.00:28
rm_worklol ok00:28
rm_workwell, glad i backed out my changes00:28
johnsomeandersson No, it should be reasonably quick, it is just a database query.00:29
eanderssonYea - we are seeing the db query take 50s00:29
eanderssonor more00:29
johnsomOuch, that is not normal00:30
johnsomThat was with the mysql client?  (assuming you are using mysql)00:30
eanderssonyep mysql (pymysql)00:30
johnsomCan you paste the query you made?  The select statement?00:31
eanderssonsorry - we haven't been able to perform the mysql query manually using the client yet00:31
johnsomOh, ok, that was what I was asking.00:31
eanderssonhttps://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/db/loadbalancer/loadbalancer_dbv2.py#L37700:32
eanderssonThat function takes almost 60s to process.00:32
eanderssonWe have like 200 lbs in our db00:33
*** ducttape_ has joined #openstack-lbaas00:34
johnsomYeah, I have done tests with > 2000 and haven't had problems like that00:34
*** yuanying has joined #openstack-lbaas00:34
johnsomDo you have a lot of churn?  A lot of create/delete cycles?00:35
eanderssonI wouldn't say a crazy amount.00:35
johnsomIf so, there are mysql maintenance tasks the mysql folks recommend you run.00:35
johnsomThis kind of thing:00:36
johnsomhttps://dev.mysql.com/doc/refman/5.7/en/mysqlcheck.html00:36
johnsomBut, be aware some of those command will lock the tables, so could slow/block other activities like new creates00:37
*** asdf__ has joined #openstack-lbaas00:38
johnsomslow disks and low memory can also impact mysql.  Check and make sure the array with the DB in it isn't running in degraded mode....00:39
eanderssonEverything else works fine though00:39
eanderssonTried a second site now, much slower, almost no activity in terms of lb (got maybe 20lbs) and takes about 15s00:39
johnsomOk.  Yeah, I would look at the table maintenance tasks.  Probably first I would do a manual select on the DB for that table.00:40
johnsomI have to run, but maybe tomorrow morning I will write a little script and load up a few 100 lbs and give it a go.00:43
rm_workstill shouldn't take 15s for 20lbs00:45
rm_workshould take like 1s for 1000lbs <_<00:45
johnsomYep00:45
eanderssonSo we are unable to find a slow mysql query, it is most likely getting stuck building the result from the query.00:46
johnsomSo a direct select on the table returned as you would expect?00:47
eanderssonWe turned on slow query logs and nothing longer than a second.00:53
eanderssonWhen you ran tests with 2k lbs, did you have a lot of members as well?00:53
reedipo/01:14
*** reedip has quit IRC01:20
*** amotoki has joined #openstack-lbaas01:31
*** ducttape_ has quit IRC01:37
*** ducttape_ has joined #openstack-lbaas01:46
*** ducttape_ has quit IRC02:07
*** ducttape_ has joined #openstack-lbaas02:14
openstackgerritAlexander Amerine proposed openstack/octavia master: blueprint LBaaS Alternative Monitoring IP/Port  https://review.openstack.org/45300102:22
*** gongysh has joined #openstack-lbaas02:27
*** reedip has joined #openstack-lbaas02:44
*** armax has quit IRC03:13
*** links has joined #openstack-lbaas03:16
openstackgerritMichael Johnson proposed openstack/octavia master: Adds v2 load balancer API section  https://review.openstack.org/43875703:29
*** ducttape_ has quit IRC03:45
*** gongysh has quit IRC03:46
*** ducttape_ has joined #openstack-lbaas03:47
*** rcernin has joined #openstack-lbaas04:06
*** amotoki has quit IRC04:12
*** ducttape_ has quit IRC04:12
*** rcernin has quit IRC04:13
*** gcheresh_ has joined #openstack-lbaas04:23
*** amotoki has joined #openstack-lbaas04:30
*** amotoki has quit IRC04:48
*** chlong has quit IRC04:49
*** amotoki has joined #openstack-lbaas04:53
*** ducttape_ has joined #openstack-lbaas05:13
*** ducttape_ has quit IRC05:18
*** amotoki has quit IRC05:19
*** aojea has joined #openstack-lbaas05:21
*** gongysh has joined #openstack-lbaas05:27
*** aojea has quit IRC05:30
*** amotoki has joined #openstack-lbaas05:32
*** pcaruana has joined #openstack-lbaas06:08
*** krypto has joined #openstack-lbaas06:11
openstackgerritZhaoBo proposed openstack/octavia master: Support QoS policy apply on vip port  https://review.openstack.org/44191206:14
*** kobis has joined #openstack-lbaas06:17
*** rcernin has joined #openstack-lbaas06:31
*** krypto has quit IRC06:32
reedipbzhao : review comments in the patch ^^06:33
*** krypto has joined #openstack-lbaas06:33
reedipsome are nits, some arent :)06:33
*** sticker has quit IRC06:37
*** voelzmo has joined #openstack-lbaas06:40
*** ducttape_ has joined #openstack-lbaas06:44
*** sputnik13 has quit IRC06:44
*** voelzmo has quit IRC06:45
*** voelzmo has joined #openstack-lbaas06:45
*** sputnik13 has joined #openstack-lbaas06:45
*** ducttape_ has quit IRC06:48
*** tesseract has joined #openstack-lbaas06:52
*** voelzmo has quit IRC06:55
*** voelzmo has joined #openstack-lbaas06:56
*** voelzmo has quit IRC07:16
*** aojea has joined #openstack-lbaas07:24
*** voelzmo has joined #openstack-lbaas07:26
*** voelzmo has quit IRC07:35
*** aojea_ has joined #openstack-lbaas08:01
*** sanfern has joined #openstack-lbaas08:03
*** aojea has quit IRC08:04
*** ducttape_ has joined #openstack-lbaas08:14
*** ducttape_ has quit IRC08:19
*** voelzmo has joined #openstack-lbaas08:21
*** ianychoi has quit IRC08:24
*** ianychoi has joined #openstack-lbaas08:25
*** voelzmo has quit IRC08:26
*** reedip has quit IRC08:41
openstackgerritNir Magnezi proposed openstack/octavia master: (WIP) Auto detect haproxy user_group  https://review.openstack.org/42939808:52
*** krypto has quit IRC09:07
*** krypto has joined #openstack-lbaas09:08
bzhaoreedip_, thank you for review.:). We are moving our physical servers to the new place. Ah, so tired..I will update the patch soon. Thanks again.09:15
*** bcafarel has quit IRC09:19
*** bcafarel has joined #openstack-lbaas09:27
*** reedip has joined #openstack-lbaas09:34
*** ducttape_ has joined #openstack-lbaas09:45
*** ducttape_ has quit IRC09:49
openstackgerritZhaoBo proposed openstack/octavia master: Support QoS policy apply on vip port  https://review.openstack.org/44191209:52
*** bbzhao has joined #openstack-lbaas09:52
*** voelzmo has joined #openstack-lbaas10:10
*** voelzmo has quit IRC10:14
*** pcaruana|afk| has joined #openstack-lbaas10:21
*** pcaruana|afk| has quit IRC10:23
*** pcaruana has quit IRC10:24
*** pcaruana has joined #openstack-lbaas10:25
*** amotoki has quit IRC10:26
*** gongysh has quit IRC10:35
*** voelzmo has joined #openstack-lbaas10:38
*** gongysh has joined #openstack-lbaas10:41
*** gongysh has quit IRC11:37
*** amotoki has joined #openstack-lbaas11:38
*** gongysh has joined #openstack-lbaas11:41
*** gongysh has quit IRC11:47
*** gongysh has joined #openstack-lbaas12:00
*** gongysh has quit IRC12:05
openstackgerritGuoqiang Ding proposed openstack/neutron-lbaas master: Choose correct log option by listener protocol  https://review.openstack.org/45361212:12
*** ducttape_ has joined #openstack-lbaas12:26
*** amotoki has quit IRC12:41
*** krypto has quit IRC12:44
*** ducttape_ has quit IRC12:45
*** ducttape_ has joined #openstack-lbaas12:45
-openstackstatus- NOTICE: The Gerrit service on http://review.openstack.org is being restarted to address hung remote replication tasks, and should return to an operable state momentarily12:51
*** amotoki has joined #openstack-lbaas12:56
*** links has quit IRC13:08
*** catintheroof has joined #openstack-lbaas13:10
*** catintheroof has quit IRC13:10
*** catintheroof has joined #openstack-lbaas13:11
*** ducttape_ has quit IRC13:12
*** chlong has joined #openstack-lbaas13:17
*** ipsecguy_ is now known as ipsecguy13:31
*** ducttape_ has joined #openstack-lbaas13:36
*** krypto has joined #openstack-lbaas13:43
*** krypto has quit IRC13:48
*** krypto has joined #openstack-lbaas13:49
*** krypto has quit IRC13:49
*** krypto has joined #openstack-lbaas13:49
*** ducttape_ has quit IRC14:01
*** mdavidson has quit IRC14:18
*** ipsecguy_ has joined #openstack-lbaas14:29
*** ipsecguy has quit IRC14:31
*** krypto has quit IRC14:34
*** gcheresh_ has quit IRC14:35
*** krypto has joined #openstack-lbaas14:35
*** krypto has quit IRC14:35
*** krypto has joined #openstack-lbaas14:35
*** armax has joined #openstack-lbaas14:41
nmagnezirm_work, i'm getting close with the user_group removal on agent side..14:54
nmagnezi:)14:54
*** voelzmo has quit IRC15:01
*** chlong has quit IRC15:03
*** chlong has joined #openstack-lbaas15:04
*** fnaval has joined #openstack-lbaas15:04
*** fnaval_ has joined #openstack-lbaas15:06
*** armax has quit IRC15:07
*** fnaval__ has joined #openstack-lbaas15:08
*** ducttape_ has joined #openstack-lbaas15:08
*** fnaval has quit IRC15:10
*** fnaval has joined #openstack-lbaas15:10
*** fnaval___ has joined #openstack-lbaas15:12
*** fnaval_ has quit IRC15:12
*** fnaval_ has joined #openstack-lbaas15:13
*** fnaval__ has quit IRC15:14
*** fnaval__ has joined #openstack-lbaas15:15
*** fnaval has quit IRC15:16
*** fnaval has joined #openstack-lbaas15:17
*** fnaval___ has quit IRC15:17
*** fnaval_ has quit IRC15:18
*** fnaval_ has joined #openstack-lbaas15:18
*** fnaval___ has joined #openstack-lbaas15:20
*** fnaval__ has quit IRC15:20
*** chlong has quit IRC15:21
*** fnaval has quit IRC15:23
*** fnaval_ has quit IRC15:24
*** kobis has quit IRC15:26
*** ducttape_ has quit IRC15:34
*** chlong has joined #openstack-lbaas15:35
openstackgerritSindhu Devale proposed openstack/octavia master: Fix import in v2 pool types  https://review.openstack.org/45372115:48
*** aojea_ has quit IRC15:49
*** gcheresh_ has joined #openstack-lbaas15:52
*** ducttape_ has joined #openstack-lbaas15:59
*** ducttape_ has quit IRC16:15
*** cody-somerville has joined #openstack-lbaas16:16
*** armax has joined #openstack-lbaas16:17
*** armax has quit IRC16:27
*** gcheresh_ has quit IRC16:28
*** amotoki has quit IRC16:29
openstackgerritMerged openstack/octavia master: Align Octavia API to n-lbaasv2 for L7Policy  https://review.openstack.org/40632816:29
*** armax has joined #openstack-lbaas16:31
*** cody-somerville has quit IRC16:31
*** tesseract has quit IRC16:43
*** armax has quit IRC16:43
openstackgerritMerged openstack/octavia master: Align Octavia API to n-lbaasv2 for L7Rules  https://review.openstack.org/40633616:45
*** _ducttape_ has joined #openstack-lbaas16:59
*** sanfern has quit IRC17:18
*** _ducttape_ has quit IRC17:18
*** sanfern has joined #openstack-lbaas17:18
*** sanfern has quit IRC17:22
*** krypto has quit IRC17:28
johnsomeandersson So I loaded up 200 load balancers into the DB and ran a 'neutron lbaas-loadbalancer-list'17:29
johnsomreal    0m17.172s17:29
johnsomuser    0m0.804s17:29
johnsomsys     0m0.108s17:29
*** krypto has joined #openstack-lbaas17:29
johnsomI suspect a good chunk of that is the client formatting.  I'm going to do a native API call and see what I get.17:29
johnsomThis is on a dev VM and not real hardware as well...17:30
xgermanjohnsom remember when we did load tests at the company which shall not be named. I think we never had issues with list17:31
eanderssonWe are setting this up in the lab, with 64 LBs we are hitting 13s.17:31
johnsomxgerman yeah, I mentioned that yesterday.17:31
eanderssonThis is on A LOT slower hardware though.17:31
johnsomGranted, going through the layers of neutron probably don't help here...17:31
xgermanalso the LB vms drain memory so if you try to pack that all on one box it will choke17:32
xgermanaka controller and compute node17:32
johnsomYeah, these are "ghost LBs"  Just the required database records for the API to do it's thing.17:34
johnsomHmm, yeah, the API call is about the same as the client, so I guess formatting wasn't that bad.17:34
johnsomreal    0m16.463s17:35
xgermanmaybe if it has to traverse the network?17:36
xgermanaka goes from one controller node to the other (and not to speak of the load balancer people employ on the REST API)17:36
xgermanso you could easily get two or three roundtrips17:37
johnsom400 LBs: real    0m32.477s17:37
johnsomSo, kind of linear17:37
johnsomThe actual DB query is less than a second.  It's got to be either the sqlalchemy translation or something in the neutron layers....17:38
eanderssonYea - looks like sqlachemy translation17:39
*** armax has joined #openstack-lbaas17:39
eanderssonBut going to dig deeper into that today.17:39
xgermanthanks — let us know17:40
eanderssonWe are running Mitaka btw17:41
xgermanuh, oh17:44
eanderssonCouldn't find any major improvements on this code path though in master.17:44
xgermanyeah, we kept  listing lbs pretty stable17:45
johnsomYeah, in neutorn-lbaas that hasn't changed in a long time.  I see the neutron process pegging out a CPU core, so....17:45
johnsomKind of curious how our new octavia API performs now.  Since I have already trashed this devstack I might try it17:46
*** asdf__ has quit IRC17:47
eanderssonYea - I see a core pegged as well.17:47
eandersson> python-sqlalchemy-1.0.11-1.el7.x86_6417:48
eanderssonGoing to try a newer version of sqlalchemy in the lab as well to rule that out.17:48
johnsomWell, the good news is our new Octavia API is a lot faster....17:54
johnsom400 LBs via Octavia v2 API: real    0m3.524s17:54
*** gcheresh_ has joined #openstack-lbaas17:54
eanderssonWhere is the Octacvia v2 Api? Is it different from Neutron-LBAAS v2 API? :D18:01
johnsomIt is a compatible superset of the LBaaS v2 API.  We are merging neutron-lbaas into octavia.18:02
eanderssonYep, read about that. Excited.18:02
johnsomhttp://docs-draft.openstack.org/57/438757/10/check/gate-octavia-api-ref/125623d//api-ref/build/html/18:03
johnsomIt's WIP18:03
*** ducttape_ has joined #openstack-lbaas18:09
*** armax has quit IRC18:09
*** armax has joined #openstack-lbaas18:18
*** ducttape_ has quit IRC18:22
openstackgerritShashank Kumar Shankar proposed openstack/octavia master: Run Octavia API in a WSGI server  https://review.openstack.org/44093418:28
*** harlowja has quit IRC18:33
*** krypto has quit IRC18:35
*** krypto has joined #openstack-lbaas18:35
*** cody-somerville has joined #openstack-lbaas18:36
*** cody-somerville has quit IRC18:36
*** cody-somerville has joined #openstack-lbaas18:36
*** krypto has quit IRC18:52
*** cody-somerville has quit IRC18:56
*** cody-somerville has joined #openstack-lbaas18:57
*** csomerville has joined #openstack-lbaas19:03
*** cody-somerville has quit IRC19:03
*** armax has quit IRC19:08
*** csomerville has quit IRC19:08
*** csomerville has joined #openstack-lbaas19:09
*** ducttape_ has joined #openstack-lbaas19:12
*** csomerville has quit IRC19:14
*** cody-somerville has joined #openstack-lbaas19:16
*** cody-somerville has quit IRC19:16
*** cody-somerville has joined #openstack-lbaas19:16
*** cody-somerville has quit IRC19:21
*** cody-somerville has joined #openstack-lbaas19:24
*** voelzmo has joined #openstack-lbaas19:25
*** csomerville has joined #openstack-lbaas19:28
*** cody-somerville has quit IRC19:29
openstackgerritShashank Kumar Shankar proposed openstack/octavia master: Run Octavia API in a WSGI server  https://review.openstack.org/44093419:30
*** csomerville has quit IRC19:33
*** csomerville has joined #openstack-lbaas19:35
*** csomerville has quit IRC19:41
*** cody-somerville has joined #openstack-lbaas19:42
*** ducttape_ has quit IRC19:43
*** cody-somerville has quit IRC19:47
*** csomerville has joined #openstack-lbaas19:48
*** voelzmo has quit IRC19:53
johnsomOctavia meeting starting soon on #openstack-meeting-alt19:56
*** harlowja has joined #openstack-lbaas19:58
*** pcaruana has quit IRC20:01
*** aojea has joined #openstack-lbaas20:11
openstackgerritShashank Kumar Shankar proposed openstack/octavia master: Run Octavia API in a WSGI server  https://review.openstack.org/44093420:16
*** armax has joined #openstack-lbaas20:20
*** ducttape_ has joined #openstack-lbaas20:43
*** rcernin has quit IRC20:45
*** ducttape_ has quit IRC20:48
*** gcheresh_ has quit IRC20:50
openstackgerritOpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements  https://review.openstack.org/45389421:05
openstackgerritOpenStack Proposal Bot proposed openstack/octavia master: Updated from global requirements  https://review.openstack.org/45389721:07
*** bbzhao has quit IRC21:22
*** bbzhao has joined #openstack-lbaas21:23
*** aojea has quit IRC21:53
*** catintheroof has quit IRC21:54
*** chlong has quit IRC22:02
eanderssonjohnsom, xgerman the version of sqlalchemy seems to have a farily large impact22:05
*** harlowja has quit IRC22:05
xgermaninteresting22:05
eanderssonin my lab it takes ~25-30s with 1.0.12 and 160 LBs22:05
johnsomBetter results with something newer?22:05
eanderssonwith 1.1.7 it takes about 14-15s22:05
eanderssonThis might be why you guys are getting different numbers than me.22:06
eandersson1.0.12 is the recommended for Mitaka though22:07
johnsomYep.  I can't point to a specific reason for that other than it is what was tested.  You might be able to run your environment with a newer version if you test it out22:09
eandersson14s for 160 LBs is still slow though, but better :p22:11
eanderssonThis is on significantly slower hardware though22:11
*** ducttape_ has joined #openstack-lbaas22:14
eanderssonActually the recommended version for Ocata (1.0.17) is as slow as the recommended version for Mitaka22:15
johnsomI think the Pike recommendation is 1.1.922:16
johnsomhttps://github.com/openstack/requirements/blob/master/upper-constraints.txt#L12722:17
eanderssonNice - as 1.1.x gives a nice boost in performance here22:17
*** gongysh has joined #openstack-lbaas22:19
*** ducttape_ has quit IRC22:21
*** fnaval___ has quit IRC22:51
openstackgerritNir Magnezi proposed openstack/octavia master: Auto detect haproxy user_group  https://review.openstack.org/42939822:53
nmagnezijohnsom, https://review.openstack.org/#/c/429398/31/octavia/amphorae/backends/agent/api_server/listener.py@124 <<-- that was a tricky one (works!)22:54
johnsomOk, will look22:54
eanderssonWow - when doing a single list lbs and this code is hit recursively 3338 times https://github.com/openstack/neutron-lbaas/blob/stable/mitaka/neutron_lbaas/services/loadbalancer/data_models.py#L7423:00
johnsomnmagnezi Tricky indeed.  So with the multiple config file it doesn't ignore the existing setting?23:03
*** gongysh has quit IRC23:20
*** harlowja has joined #openstack-lbaas23:25
*** ducttape_ has joined #openstack-lbaas23:28
eanderssonhttps://github.com/openstack/neutron-lbaas/blob/stable/mitaka/neutron_lbaas/services/loadbalancer/data_models.py#L8423:30
eanderssonThis is the line that takes time.23:30
*** ducttape_ has quit IRC23:39
*** ducttape_ has joined #openstack-lbaas23:40
*** ducttape_ has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!