Friday, 2017-02-10

*** VW has quit IRC00:09
*** david-lyle has joined #openstack-operators00:09
*** karthikpr has quit IRC00:17
*** ducttape_ has quit IRC00:19
*** david-lyle has quit IRC00:20
*** karad has quit IRC00:24
*** saneax-_-|AFK is now known as saneax00:25
*** VW has joined #openstack-operators00:29
*** VW_ has joined #openstack-operators00:33
*** VW has quit IRC00:33
*** VW_ has quit IRC00:36
*** mriedem has quit IRC00:39
*** ducttape_ has joined #openstack-operators00:50
*** notmyname has quit IRC00:57
*** notmyname has joined #openstack-operators01:01
*** jamesdenton has joined #openstack-operators01:05
*** karthikpr has joined #openstack-operators01:07
*** karthikpr has quit IRC01:11
*** B_Smith has quit IRC01:11
*** karthikpr has joined #openstack-operators01:11
*** B_Smith has joined #openstack-operators01:12
*** stupidnic has quit IRC01:13
*** stupidnic has joined #openstack-operators01:17
*** ducttape_ has quit IRC01:22
*** delattec has quit IRC01:44
*** jamesdenton has quit IRC01:57
*** karthikpr has quit IRC01:58
*** karthikpr has joined #openstack-operators02:06
*** emccormickva has joined #openstack-operators02:09
*** karthikpr has quit IRC02:10
*** Apoorva has quit IRC02:11
*** emccormick has quit IRC02:12
*** jamesdenton has joined #openstack-operators02:12
*** ducttape_ has joined #openstack-operators02:15
*** mriedem has joined #openstack-operators02:31
*** ducttape_ has quit IRC02:34
*** jamesdenton has quit IRC02:41
*** harlowja has quit IRC02:53
*** sudipto_ has joined #openstack-operators03:02
*** sudipto has joined #openstack-operators03:02
*** ducttape_ has joined #openstack-operators03:04
*** sdake has quit IRC03:28
*** ducttape_ has quit IRC03:28
*** mriedem has quit IRC03:39
*** ducttape_ has joined #openstack-operators03:42
*** ducttape_ has quit IRC03:52
*** markvoelker has quit IRC03:56
*** markvoelker has joined #openstack-operators03:58
*** sdake has joined #openstack-operators04:07
*** saneax is now known as saneax-_-|AFK04:07
*** karthikpr has joined #openstack-operators04:10
*** karthikpr has quit IRC04:14
*** sudipto has quit IRC04:16
*** sudipto_ has quit IRC04:16
*** karthikpr has joined #openstack-operators04:17
*** faizy has joined #openstack-operators04:28
*** karthikpr has quit IRC04:33
*** karthikpr has joined #openstack-operators04:34
*** karthikpr has quit IRC04:38
*** karthikpr has joined #openstack-operators04:38
*** sudipto has joined #openstack-operators04:42
*** sudipto_ has joined #openstack-operators04:42
*** ducttape_ has joined #openstack-operators04:43
*** karthikpr has quit IRC04:44
*** karthikpr has joined #openstack-operators04:50
*** karthikpr has quit IRC04:55
*** karthikpr has joined #openstack-operators04:57
*** esmiurium has quit IRC05:01
*** karthikpr has quit IRC05:03
*** ducttape_ has quit IRC05:05
*** esmiurium has joined #openstack-operators05:05
*** dmitryme has quit IRC05:10
*** dmitryme has joined #openstack-operators05:10
*** karthikpr has joined #openstack-operators05:15
*** karthikpr has quit IRC06:00
*** karthikpr has joined #openstack-operators06:01
*** karthikpr has quit IRC06:06
*** karthikpr has joined #openstack-operators06:08
*** karthikpr has quit IRC06:14
*** armax has quit IRC06:15
*** karthikpr has joined #openstack-operators06:19
*** admin0_ has joined #openstack-operators06:40
*** admin0_ has quit IRC06:41
*** makowals has quit IRC06:46
*** makowals has joined #openstack-operators06:48
*** yprokule has joined #openstack-operators06:57
*** markvoelker has quit IRC07:02
*** markvoelker has joined #openstack-operators07:02
*** rcernin has joined #openstack-operators07:03
*** ducttape_ has joined #openstack-operators07:05
*** markvoelker has quit IRC07:06
*** ducttape_ has quit IRC07:11
*** faizy has quit IRC07:28
*** pcaruana has joined #openstack-operators07:30
*** faizy has joined #openstack-operators07:31
*** makowals has quit IRC07:33
*** fragatina has quit IRC07:36
*** mnaser has quit IRC07:41
*** makowals has joined #openstack-operators07:45
*** slaweq has quit IRC07:48
*** admin0_ has joined #openstack-operators07:52
*** karthikpr has quit IRC07:52
*** slaweq has joined #openstack-operators07:54
*** jbadiapa has joined #openstack-operators08:00
*** faizy_ has joined #openstack-operators08:02
*** markvoelker has joined #openstack-operators08:03
*** sdake_ has joined #openstack-operators08:04
*** faizy has quit IRC08:05
*** simon-AS559 has joined #openstack-operators08:05
*** faizy__ has joined #openstack-operators08:05
*** sdake has quit IRC08:07
*** markvoelker has quit IRC08:09
*** faizy_ has quit IRC08:09
*** faizy has joined #openstack-operators08:09
*** faizy__ has quit IRC08:12
*** vijaykc4 has joined #openstack-operators08:16
*** saneax-_-|AFK is now known as saneax08:19
*** sdake_ has quit IRC08:37
*** sdake has joined #openstack-operators08:39
*** sdake has quit IRC08:40
*** fragatina has joined #openstack-operators08:46
*** faizy has quit IRC08:46
*** faizy has joined #openstack-operators08:47
*** fragatina has quit IRC08:51
*** paramite has joined #openstack-operators08:55
*** derekh has joined #openstack-operators09:07
*** ducttape_ has joined #openstack-operators09:08
*** ducttape_ has quit IRC09:13
*** admin0_ has quit IRC09:23
*** jsheeren has joined #openstack-operators09:30
*** admin0_ has joined #openstack-operators09:35
*** vijaykc4 has quit IRC09:40
*** vijaykc4 has joined #openstack-operators09:41
*** dbecker has quit IRC09:41
*** faizy_ has joined #openstack-operators09:42
*** dbecker has joined #openstack-operators09:42
*** faizy has quit IRC09:45
*** fragatina has joined #openstack-operators09:49
*** fragatina has quit IRC09:55
*** jsheeren has quit IRC09:57
*** admin0_ has quit IRC09:58
*** markvoelker has joined #openstack-operators10:03
*** admin0_ has joined #openstack-operators10:21
*** twi99y has joined #openstack-operators10:32
*** twi99y_ has quit IRC10:32
*** twi99y has quit IRC10:33
*** twi99y has joined #openstack-operators10:33
*** maishsk has joined #openstack-operators10:34
*** admin0_ has quit IRC10:46
*** electrofelix has joined #openstack-operators10:56
*** admin0_ has joined #openstack-operators11:00
*** vijaykc4 has quit IRC11:01
*** ducttape_ has joined #openstack-operators11:09
*** ducttape_ has quit IRC11:14
*** faizy_ has quit IRC11:20
*** vijaykc4 has joined #openstack-operators11:32
*** admin0_ has quit IRC11:41
*** saneax is now known as saneax-_-|AFK11:49
*** tesseract has joined #openstack-operators11:50
*** vijaykc4 has quit IRC11:58
*** vijaykc4 has joined #openstack-operators11:58
*** saneax-_-|AFK is now known as saneax11:58
*** vijaykc4 has quit IRC11:58
*** cdelatte has joined #openstack-operators12:08
*** sudipto has quit IRC12:09
*** sudipto_ has quit IRC12:09
*** fragatina has joined #openstack-operators12:10
*** fragatina has quit IRC12:14
*** saneax is now known as saneax-_-|AFK12:19
*** karad has joined #openstack-operators12:22
*** ducttape_ has joined #openstack-operators12:23
*** manheim has joined #openstack-operators12:24
*** saneax-_-|AFK is now known as saneax12:29
*** faizy_ has joined #openstack-operators12:31
*** dminer has joined #openstack-operators12:37
*** jamesdenton has joined #openstack-operators12:40
*** vijaykc4 has joined #openstack-operators12:45
*** admin0_ has joined #openstack-operators12:45
*** jamesden_ has joined #openstack-operators12:45
*** markvoelker has quit IRC12:46
*** markvoelker has joined #openstack-operators12:46
*** jamesdenton has quit IRC12:46
*** admin0_ has quit IRC12:47
*** admin0_ has joined #openstack-operators12:52
*** jamesden_ has quit IRC12:52
*** faizy_ has quit IRC12:56
*** sdake has joined #openstack-operators13:06
*** VW has joined #openstack-operators13:06
*** karad has quit IRC13:09
*** ducttape_ has quit IRC13:18
*** faizy has joined #openstack-operators13:19
*** dminer has quit IRC13:24
*** vijaykc4 has quit IRC13:29
*** manheim has quit IRC13:48
*** mriedem has joined #openstack-operators13:50
*** manheim has joined #openstack-operators13:56
*** ducttape_ has joined #openstack-operators13:57
*** sudipto has joined #openstack-operators14:10
*** sudipto_ has joined #openstack-operators14:10
*** fragatina has joined #openstack-operators14:11
*** ducttape_ has quit IRC14:12
*** jamesdenton has joined #openstack-operators14:13
*** fragatina has quit IRC14:16
*** jamesdenton has quit IRC14:23
*** jamesdenton has joined #openstack-operators14:24
*** manheim has quit IRC14:28
*** dansmith is now known as superdan14:29
*** sdake_ has joined #openstack-operators14:36
*** manheim has joined #openstack-operators14:38
*** sdake has quit IRC14:39
*** saneax is now known as saneax-_-|AFK14:39
*** ducttape_ has joined #openstack-operators14:40
*** maishsk has quit IRC14:46
*** ducttape_ has quit IRC14:51
*** ducttape_ has joined #openstack-operators14:53
*** vijaykc4 has joined #openstack-operators15:03
*** manheim has quit IRC15:03
*** sudipto_ has quit IRC15:08
*** sudipto has quit IRC15:08
*** faizy_ has joined #openstack-operators15:14
*** faizy has quit IRC15:17
*** manheim has joined #openstack-operators15:26
*** dims_ has quit IRC15:28
*** manheim has quit IRC15:29
*** manheim has joined #openstack-operators15:33
*** dims has joined #openstack-operators15:33
*** admin0_ has quit IRC15:43
*** admin0_ has joined #openstack-operators15:44
*** marst has quit IRC15:52
*** dims has quit IRC15:55
*** dims has joined #openstack-operators15:55
*** mnaser has joined #openstack-operators15:56
*** dims is now known as Guest1240615:56
*** marst has joined #openstack-operators16:02
*** Guest12406 has quit IRC16:05
*** rcernin has quit IRC16:06
*** karthikpr has joined #openstack-operators16:07
*** vijaykc4 has quit IRC16:07
*** karthik__ has joined #openstack-operators16:09
*** karthikpr has quit IRC16:09
*** Oku_OS is now known as Oku_OS-away16:10
*** vijaykc4 has joined #openstack-operators16:10
*** karthik__ has quit IRC16:10
*** karthikpr has joined #openstack-operators16:11
*** vijaykc4 has quit IRC16:17
*** armax has joined #openstack-operators16:18
*** admin0_ has quit IRC16:19
*** karthikpr has quit IRC16:21
*** admin0_ has joined #openstack-operators16:24
*** dims_ has joined #openstack-operators16:24
*** vijaykc4 has joined #openstack-operators16:25
*** vijaykc4 has quit IRC16:25
*** vijaykc4 has joined #openstack-operators16:27
*** sudipto_ has joined #openstack-operators16:29
*** sudipto has joined #openstack-operators16:29
*** vijaykc4 has quit IRC16:33
*** vijaykc4 has joined #openstack-operators16:34
*** twi99y has quit IRC16:44
*** twi99y_ has joined #openstack-operators16:44
*** tesseract has quit IRC16:45
*** manheim has quit IRC16:46
*** twi99y has joined #openstack-operators16:47
shewlessmdorman: around to talk about rmq getting saturated when nuetron is restarted on a setup with lots of ports?16:49
*** twi99y_ has quit IRC16:50
*** pcaruana has quit IRC16:53
mdormanshewless: sure, i can try :)16:54
*** admin0_ has quit IRC16:54
shewlessmdorman: it was you who mentioned you experience a similar problem right? (Can't remember for sure)16:54
mdormanyes16:55
shewlessdo you remember if you did anything to mitigate the issue?  Otherwise, if you think it's a rmq issue, can you recommend some reading for me to "tweak" rmq?16:56
shewlessmaybe increase some thresholds, set some backoff times, etc16:56
*** VW_ has joined #openstack-operators16:58
*** manheim has joined #openstack-operators17:00
*** VW_ has quit IRC17:00
*** VW__ has joined #openstack-operators17:00
*** VW has quit IRC17:02
*** VW__ has quit IRC17:02
*** VW has joined #openstack-operators17:02
*** VW_ has joined #openstack-operators17:03
*** VW has quit IRC17:03
*** harlowja has joined #openstack-operators17:05
*** harlowja has quit IRC17:05
*** harlowja has joined #openstack-operators17:05
mdormanshewless: in our particular case, the trigger for it was when a dhcp port changed.  inside nuetron this causes a complete reload of all security group rules on all ports on that network.17:09
*** Apoorva has joined #openstack-operators17:09
mdormanthat would generate a bunch of load on neutron, which in turn would cause ovs agents to drop offline (because neutron-server was not processing their checkins/keep alives fast enough.)   so that just exacerbated the problem, now with agents for other networks reconnecting.17:10
shewlessmdorman: For me, IIRC, the trigger is restarting neutron-openvswitch-agent17:10
mdormanyeah if you have a large number of ports behind a single ovs agent i would expect similar behavior.   when the agent comes up it has to fetch all teh port info for all the ports from the neutron-server17:10
shewlessrecommendations for what I can/should add for scale? More ovs agents? Is that even possible?17:11
mdormani think the recommendation i would say is more neutron-server instances.  so there are more workers to process all the rpcs from the agents.  but at some point you can only go so far with that and you’ll hit a scale issue at teh database.17:13
mdormanbut generally i think more neutron-server workers is the next step17:13
*** karthikpr has joined #openstack-operators17:15
*** faizy_ has quit IRC17:17
*** mriedem is now known as mriedem_lunch17:17
klindgren_shewless, how many neutron agents do you have?17:17
shewlesswould that be the api_workers?17:18
shewlessor the rpc_workers17:18
shewlessor something different :)17:18
*** fragatina has joined #openstack-operators17:19
*** fragatina has quit IRC17:20
*** fragatina has joined #openstack-operators17:20
*** VW_ has quit IRC17:23
shewlessklindgren_ and mdorman: I have neutron installed in 1 spot. And I haven't tweaked the workers from default.  hints on where I would add more neutron-server workers?17:24
*** VW has joined #openstack-operators17:24
klindgren_sorry - how many in: neutron agent-list17:25
klindgren_which would be all the things that  RPC (rabbitmq) communication is used for.17:25
klindgren_so iirc its rpc_workers and api_workers in neutron.conf under the default section.17:27
klindgren_search for those under: http://docs.openstack.org/liberty/config-reference/content/section_neutron.conf.html17:27
klindgren_we currently run with 3 neutron servers (1 process per server) with like 20 RPC workers and 20 API workers iirc.17:27
mdormanyaeh at some point though, if you have more workers than you do cpu cores on the box, it’s not going to buy you much.   then you’d need to go to multiple servers for it.17:29
*** twi99y_ has joined #openstack-operators17:29
shewlessklindgren_: ah.. 1 open vswitch agent per compute. 1 DHCP agent, 1 L3 agent, 1 metadata agent17:29
*** twi99y has quit IRC17:30
klindgren_We only started running into dhcp agent issues and RPC saturation around 800+ agents (~12 dhcp agents and ~800 compute nodes)17:30
klindgren_now 0 dhcp agents :-D17:31
shewlessno dhcp?17:31
shewlesswhich agents do I need to increase? the open vswitch ones on the computes?17:32
shewlessof the "others" on the control?17:32
shewlessdo you tweak the "workers" in nova too/17:34
*** karthikpr has quit IRC17:35
*** twi99y has joined #openstack-operators17:36
*** twi99y_ has quit IRC17:37
*** yprokule has quit IRC17:39
*** karthikpr has joined #openstack-operators17:40
*** derekh has quit IRC17:40
*** vijaykc4 has quit IRC17:41
*** karthikpr has quit IRC17:44
*** karthikpr has joined #openstack-operators17:45
*** vijaykc4 has joined #openstack-operators17:49
*** VW has quit IRC18:08
*** VW has joined #openstack-operators18:11
*** manheim has quit IRC18:11
*** sudipto has quit IRC18:16
*** sudipto_ has quit IRC18:16
*** karthikpr has quit IRC18:16
*** karthikpr has joined #openstack-operators18:17
shewlessklindren_, mdorman: the number of agents doesn't seem to change even if I increase the rpc_workers and api_workers in neutron.conf. This is done on the "controller" right?18:19
*** simon-AS559 has quit IRC18:19
*** karthikpr has quit IRC18:22
*** admin0_ has joined #openstack-operators18:26
*** mriedem_lunch is now known as mriedem18:27
*** VW has quit IRC18:32
mdormanyes.  there is adifference between agents and workers.  agents are the things that run out on the hypervisors (ovs agent, l3 agent, etc.).   workers are the neutron-server processes that run on the controller.  that rpc_workers config on the controller should incrase those.18:32
*** admin0_ has quit IRC18:32
*** vijaykc4 has quit IRC18:35
*** slaweq has quit IRC18:40
*** mxburns has joined #openstack-operators18:41
*** ducttape_ has quit IRC18:41
*** slaweq has joined #openstack-operators18:42
*** admin0_ has joined #openstack-operators18:44
*** sdake_ has quit IRC18:48
*** sdake has joined #openstack-operators18:57
*** paramite has quit IRC19:03
shewlessmdorman: and my goal is to increase the workers right? Do I just ps wwaux | grep neutron-server to see how many workers are running/>19:05
*** cdelatte has quit IRC19:09
*** dminer has joined #openstack-operators19:10
*** karthikpr has joined #openstack-operators19:19
*** sdake_ has joined #openstack-operators19:29
*** VW has joined #openstack-operators19:29
*** cdelatte has joined #openstack-operators19:32
*** sdake has quit IRC19:32
*** electrofelix has quit IRC19:35
*** ducttape_ has joined #openstack-operators19:38
*** fragatina has quit IRC19:39
*** twi99y_ has joined #openstack-operators19:40
*** twi99y has quit IRC19:43
mdormanshewless:   yes, increase workers.   and yeah looking at the number of things in the process list is a good way to validate20:05
shewlessmdorman: thanks this gives me hope! I'll try it out and report back probably next week20:15
*** VW has quit IRC20:26
*** karthikpr has quit IRC20:30
*** sdake_ has quit IRC20:31
*** marst has quit IRC20:34
*** karthikpr has joined #openstack-operators20:36
*** VW has joined #openstack-operators20:51
*** ducttape_ has quit IRC20:53
*** marst has joined #openstack-operators20:53
*** twi99y_ has quit IRC20:54
*** VW has quit IRC20:54
*** twi99y has joined #openstack-operators20:55
*** fragatina has joined #openstack-operators20:56
*** VW has joined #openstack-operators20:57
*** racedo has quit IRC20:59
*** jamesdenton has quit IRC21:14
*** ducttape_ has joined #openstack-operators21:15
*** admin0_ has quit IRC21:16
*** jamesdenton has joined #openstack-operators21:16
shewlessmdorman: is rpc_workers the important one for this particular problem (openvswitch, ports, etc)21:19
shewlessI assume api_workers is important for api requests but not a factor in my particular problem21:19
shewlesssame with the rpc_state_workers21:20
*** liverpooler has quit IRC21:24
*** VW has quit IRC21:25
*** sdake has joined #openstack-operators21:28
*** VW has joined #openstack-operators21:29
*** dminer has quit IRC21:29
*** sdake has quit IRC21:32
*** dminer has joined #openstack-operators21:33
*** karthikpr has quit IRC21:45
*** VW has quit IRC21:57
*** fragatina has quit IRC21:59
*** VW has joined #openstack-operators22:10
*** VW_ has joined #openstack-operators22:11
*** VW has quit IRC22:11
*** jamesdenton has quit IRC22:13
*** dminer has quit IRC22:26
*** karthikpr has joined #openstack-operators22:31
*** karthikpr has quit IRC22:36
*** Apoorva has quit IRC22:37
*** VW has joined #openstack-operators22:55
*** VW has quit IRC22:58
*** VW has joined #openstack-operators22:58
*** VW_ has quit IRC22:59
*** karthikpr has joined #openstack-operators22:59
*** VW has quit IRC22:59
*** VW has joined #openstack-operators23:00
*** catintheroof has quit IRC23:03
*** karthikpr has quit IRC23:03
*** karthikpr has joined #openstack-operators23:03
*** catintheroof has joined #openstack-operators23:04
*** Apoorva has joined #openstack-operators23:07
*** catintheroof has quit IRC23:08
*** karthikpr has quit IRC23:08
*** VW has quit IRC23:17
*** marst has quit IRC23:22
*** slaweq has quit IRC23:45
*** slaweq has joined #openstack-operators23:48
mdormanshewless:   yes for rpc_workers re: api_workers.   i am not eaxctly sure what rpc_state_workers is for23:51
*** ducttape_ has quit IRC23:51

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!