Tuesday, 2016-02-09

*** krobzaur has joined #openstack-operators00:01
*** simon-AS559 has quit IRC00:03
*** VW has joined #openstack-operators00:05
*** SimonChung1 has quit IRC00:07
*** SimonChung has joined #openstack-operators00:07
*** VW has quit IRC00:09
*** k_stev has quit IRC00:14
*** armax has quit IRC00:19
*** armax_ has joined #openstack-operators00:19
*** eil397 has quit IRC00:27
*** xarses has quit IRC00:29
*** SimonChung has quit IRC00:58
*** xarses has joined #openstack-operators00:59
*** xarses has quit IRC00:59
*** xarses has joined #openstack-operators01:00
*** xarses has quit IRC01:19
*** l4h has quit IRC01:42
mgagnemfisch effect of memcached for keystonemiddleware on API operations: http://grafana.openstack.org/dashboard/db/nodepool-internap (select 7 days, effect starts at ~2/5)01:46
*** xarses has joined #openstack-operators01:47
*** xarses has quit IRC01:47
*** xarses has joined #openstack-operators01:47
stevemarmgagne: good results for create/list/delete servers01:52
mgagneyea, will see for effect of fernet token later this week01:52
*** ducttape_ has joined #openstack-operators01:53
*** apoorvad has quit IRC01:54
*** l4h has joined #openstack-operators02:00
*** ducttape_ has quit IRC02:05
*** ducttape_ has joined #openstack-operators02:05
*** gyee has quit IRC02:06
*** _ducttape_ has joined #openstack-operators02:08
*** ducttape_ has quit IRC02:09
*** Marga_ has quit IRC02:30
*** Marga_ has joined #openstack-operators02:34
*** l4h has quit IRC02:34
*** l4h has joined #openstack-operators02:34
*** l4h has quit IRC02:34
*** Marga_ has quit IRC02:40
*** fawadkhaliq has joined #openstack-operators02:47
*** armax_ is now known as armax02:54
*** lhcheng has quit IRC02:59
*** _ducttape_ has quit IRC03:02
*** ducttape_ has joined #openstack-operators03:02
*** ducttape_ has quit IRC03:04
*** ducttape_ has joined #openstack-operators03:05
*** vinsh has joined #openstack-operators03:11
*** dims has joined #openstack-operators03:27
*** dims_ has quit IRC03:28
*** dims has quit IRC03:30
*** dims has joined #openstack-operators03:43
*** rawl has joined #openstack-operators03:45
*** rawl has quit IRC03:45
*** dims has quit IRC03:47
*** ducttape_ has quit IRC03:51
*** ducttape_ has joined #openstack-operators03:53
*** Marga_ has joined #openstack-operators03:54
*** ducttape_ has quit IRC04:01
*** ducttape_ has joined #openstack-operators04:09
*** apoorvad has joined #openstack-operators04:13
*** armax_ has joined #openstack-operators04:18
*** apoorvad has quit IRC04:19
*** armax has quit IRC04:21
*** armax_ is now known as armax04:21
*** fawadkhaliq has quit IRC04:42
*** mdorman has joined #openstack-operators04:50
*** krobzaur has quit IRC05:05
*** ducttape_ has quit IRC05:06
*** fawadkhaliq has joined #openstack-operators05:14
*** fawadkhaliq has quit IRC05:14
*** fawadkhaliq has joined #openstack-operators05:14
*** Marga_ has quit IRC05:24
*** mdorman has quit IRC05:34
*** krobzaur has joined #openstack-operators05:34
*** B_Smith_ has quit IRC05:54
*** B_Smith has joined #openstack-operators06:01
*** Marga_ has joined #openstack-operators06:06
*** sgordon has quit IRC06:09
*** sgordon has joined #openstack-operators06:11
*** l4h has joined #openstack-operators06:34
*** berendt has joined #openstack-operators06:44
*** l4h has quit IRC06:48
*** armax has quit IRC06:49
*** simon-AS559 has joined #openstack-operators06:51
*** eddima has joined #openstack-operators07:21
*** simon-AS559 has quit IRC07:39
*** belmoreira has joined #openstack-operators07:41
*** reed_ has quit IRC07:59
*** reed_ has joined #openstack-operators08:00
*** reed_ has quit IRC08:01
*** liverpooler has joined #openstack-operators08:03
*** matrohon has joined #openstack-operators08:31
*** VW has joined #openstack-operators08:33
*** VW has quit IRC08:37
*** ahonda has joined #openstack-operators08:51
*** ahonda has quit IRC08:51
*** ahonda has joined #openstack-operators08:52
*** ahonda has quit IRC08:52
*** ahonda has joined #openstack-operators08:52
*** ahonda has quit IRC08:52
*** simon-AS559 has joined #openstack-operators08:58
*** simon-AS5591 has joined #openstack-operators09:00
*** simon-AS559 has quit IRC09:03
*** derekh has joined #openstack-operators09:39
*** fawadkhaliq has quit IRC09:47
*** fawadkhaliq has joined #openstack-operators09:47
*** fawadkhaliq has quit IRC10:01
*** krobzaur has quit IRC10:07
*** kazuIchikawa has joined #openstack-operators10:10
*** Marga_ has quit IRC10:12
*** electrofelix has joined #openstack-operators10:18
*** prithiv has joined #openstack-operators10:24
*** prithiv has quit IRC10:32
*** krobzaur has joined #openstack-operators10:37
*** dims has joined #openstack-operators10:45
*** dims has quit IRC10:49
*** dims has joined #openstack-operators10:50
*** fawadkhaliq has joined #openstack-operators10:50
*** prithiv has joined #openstack-operators11:01
*** saneax_AFK is now known as saneax11:11
*** fawadkhaliq has quit IRC11:19
*** fawadkhaliq has joined #openstack-operators11:19
*** fawadkhaliq has quit IRC11:19
*** fawadkhaliq has joined #openstack-operators11:20
*** egonzalez has joined #openstack-operators11:34
*** dims_ has joined #openstack-operators11:40
*** dims has quit IRC11:40
*** saneax is now known as saneax_AFK11:57
*** prithiv has quit IRC12:08
*** dims_ has quit IRC12:11
*** dims_ has joined #openstack-operators12:19
*** krobzaur has quit IRC12:24
*** zhangjn has quit IRC12:24
*** zhangjn has joined #openstack-operators12:27
*** openstackgerrit_ has joined #openstack-operators12:39
*** fawadkhaliq has quit IRC12:43
*** ducttape_ has joined #openstack-operators12:53
*** Marga_ has joined #openstack-operators12:55
*** prithiv has joined #openstack-operators13:06
*** ducttape_ has quit IRC13:10
*** fawadkhaliq has joined #openstack-operators13:22
*** VW has joined #openstack-operators13:26
*** VW has quit IRC13:27
*** regXboi has joined #openstack-operators13:31
*** bvandenh has joined #openstack-operators13:39
*** mriedem has joined #openstack-operators13:41
*** openstackgerrit_ has quit IRC13:56
*** dims_ has quit IRC14:03
*** britthou_ has joined #openstack-operators14:03
*** dims has joined #openstack-operators14:10
*** ducttape_ has joined #openstack-operators14:21
*** ducttape_ has quit IRC14:21
*** ferest has joined #openstack-operators14:24
*** k_stev has joined #openstack-operators14:25
*** ferest has quit IRC14:28
*** ferest_ has joined #openstack-operators14:28
*** portante has quit IRC14:30
*** prithiv has quit IRC14:30
*** prithiv has joined #openstack-operators14:32
*** ferest has joined #openstack-operators14:33
*** portante has joined #openstack-operators14:33
*** ferest_ has quit IRC14:34
*** ferest_ has joined #openstack-operators14:35
*** ferest_ has quit IRC14:37
*** ferest has quit IRC14:38
*** krobzaur has joined #openstack-operators14:39
*** prithiv has quit IRC14:51
*** amit213 has quit IRC14:51
*** fawadkhaliq has quit IRC14:52
*** cgross has quit IRC14:56
*** cgross has joined #openstack-operators14:56
*** xarses has quit IRC15:00
*** k_stev has quit IRC15:00
*** k_stev1 has joined #openstack-operators15:00
*** berendt has quit IRC15:01
*** ducttape_ has joined #openstack-operators15:01
*** ducttape_ has quit IRC15:02
*** ducttape_ has joined #openstack-operators15:02
*** cdelatte has joined #openstack-operators15:14
*** piet has joined #openstack-operators15:22
*** dminer has joined #openstack-operators15:23
*** VW has joined #openstack-operators15:39
*** VW has quit IRC15:40
*** VW_ has joined #openstack-operators15:40
*** xarses has joined #openstack-operators15:40
*** VW has joined #openstack-operators15:40
*** mdorman has joined #openstack-operators15:45
*** mdorman has quit IRC15:46
*** mdorman has joined #openstack-operators15:47
*** liverpooler has quit IRC15:53
*** nikhil is now known as nikhil_k15:56
*** esker has joined #openstack-operators16:01
*** k_stev1 has quit IRC16:06
*** armax has joined #openstack-operators16:08
*** k_stev has joined #openstack-operators16:08
*** fawadkhaliq has joined #openstack-operators16:08
mriedemopen question: how frequently do operators pick up stable/liberty point releases (since they are decentralized now), maybe there aren't that many on liberty yet,16:11
mriedemor do ops usually wait until the first few stable branch point releases are out before upgrading to stable+1?16:11
*** klindgren__ is now known as klindgren16:21
*** belmoreira has quit IRC16:21
klindgrenwe use to do stable.1 - but the last few times we moved overwe were at stable.016:21
klindgrenwe haven't moved to liberty yet16:22
*** _ducttape_ has joined #openstack-operators16:26
*** xarses has quit IRC16:27
*** ducttape_ has quit IRC16:29
*** kazuIchikawa has quit IRC16:30
*** _ducttape_ has quit IRC16:36
*** ducttape_ has joined #openstack-operators16:37
yankcrimehistorically we've waited until the first stable point release16:38
yankcrimebut yeah, we've also yet to get to liberty16:38
yankcrime(we being datacentred)16:38
*** piet has quit IRC16:40
*** egonzalez has quit IRC16:41
*** piet has joined #openstack-operators16:46
*** esker has quit IRC16:49
mriedemanyone know if cern is on liberty yet?16:51
mriedemklindgren: yankcrime: i'm assuming picking up stable point releases for the release you're on, e.g. kilo 2015.1.3, is not terribly difficult since there shouldn't be config changes or things removed to worry about16:52
klindgrenits somewhat difficult16:52
klindgrenits a day or more effort16:52
klindgrento go through and rebase all the patches16:53
yankcrimeyeah, we've some custom patches etc. in place that we need to be careful with16:53
* yankcrime nods16:53
mriedemoh right, didn't think about that16:53
mriedemi just meant the actual update of the packages and restarting services16:53
klindgrenmost of the time though point releases aren't *terrible*16:53
klindgrenyea - installing the new package and the like for us in the same practice as doing major upgrades16:54
klindgrenwe have the packages always replace the config files then have puppet reconfigure the config files as needed and puppet restarts the services16:54
klindgrenre: rebaseing of patches - most of the time that stuff is pretty trivial as well - rarely do we have to fix anything most of the time if we do - its moving line numbers16:55
klindgrenbut we do have to test point releases pretty well16:55
*** VW_ has joined #openstack-operators16:55
klindgrenmore than once has a point releases (especially on the keystone side) totally fubared shit16:55
klindgrenLike the utf8 conversion of bianary data in ldap fields - because they weren;t asking for specific enough fields16:56
*** apoorvad has joined #openstack-operators16:56
mriedemok, i did see the bug you guys reported against keystone (or one of the keystone libs) in keystone a couple of weeks ago that led to a bunch of reverts16:56
klindgrenthen the domains issues inder v2 tha they reverted16:56
mriedemklindgren: those were in stable point releases or just moving to kilo.0?16:57
*** zeih_ has joined #openstack-operators16:57
klindgrenI think - nova has been pretty good for us16:57
klindgrenkilo.0 -> kilo.216:57
mriedemah16:57
klindgrenfor the keystone domains issue16:58
klindgrenand the other one iirc was like icehouse .2 -> icehouse.316:58
mriedemhmm, would be interesting to get a post-mortem from stevemar on that, what went wrong and why they weren't caught16:58
mriedemi might be mistaken but i don't think keystone has CI for ldap16:58
klindgrenthey dont - and we are AD16:58
klindgrenso its even more fun16:59
*** VW has quit IRC16:59
klindgren<316:59
mriedemyeah i can imagine16:59
*** VW_ has quit IRC16:59
mriedemo17:00
mriedem*i'll be interested to know how ops people deal with / feel about liberty stable point releases when they get there since the cats release themselves now17:00
mriedemand some projects are releasing rapidly on stable/liberty, like neutron17:00
*** VW has joined #openstack-operators17:00
klindgrenhonestly - I htink I am going to be treating them as one big stable point releae17:01
klindgrenrev everything at the same time - most likely skip a few of the faster ones - depending on what they fix17:02
*** d0ugal has quit IRC17:02
mriedemyeah, i assumed that people would just update on certain schedules17:03
mriedemunless there was like a security fix they needed to pick up right away17:03
*** VW has quit IRC17:03
*** VW has joined #openstack-operators17:04
*** bvandenh has quit IRC17:06
*** piet has quit IRC17:10
*** krobzaur has quit IRC17:11
*** gyee has joined #openstack-operators17:12
*** prithiv has joined #openstack-operators17:15
*** prithiv has quit IRC17:17
*** simon-AS5591 has quit IRC17:17
*** SimonChung has joined #openstack-operators17:18
*** prithiv has joined #openstack-operators17:19
*** matrohon has quit IRC17:25
*** eil397 has joined #openstack-operators17:50
*** derekh has quit IRC17:58
*** lhcheng_ has joined #openstack-operators17:59
*** SimonChung has quit IRC18:00
*** SimonChung1 has joined #openstack-operators18:01
*** l4h has joined #openstack-operators18:05
*** eil397 has quit IRC18:07
*** electrofelix has quit IRC18:07
*** eil397 has joined #openstack-operators18:08
*** zeih_ has quit IRC18:12
*** krobzaur has joined #openstack-operators18:13
*** alop has joined #openstack-operators18:20
*** piet has joined #openstack-operators18:23
*** piet has quit IRC18:23
*** delattec has joined #openstack-operators18:23
*** cdelatte has quit IRC18:26
*** VW has quit IRC18:29
*** krobzaur has quit IRC18:29
*** krobzaur has joined #openstack-operators18:29
*** bvandenh has joined #openstack-operators18:30
*** esker has joined #openstack-operators18:30
*** bvandenh_ has joined #openstack-operators18:31
*** lhcheng_ has quit IRC18:33
*** prithiv has quit IRC18:34
*** bvandenh has quit IRC18:35
*** pilgrimstack has quit IRC18:36
*** bvandenh_ has quit IRC18:37
*** krobzaur has quit IRC18:38
*** krobzaur has joined #openstack-operators18:39
*** eil397 has quit IRC18:41
*** matrohon has joined #openstack-operators18:46
claytonmriedem: I think for most operators upgrading every six months is plenty.  I'm sure plenty of people upgrade between dot releases, btu I doubt it's the common case18:46
claytonespecially if you run multiple services on a single node and use upstream packages, since that means you're going to have to restart ovs agent to upgrade, which is usually customer impacting18:47
*** delatte has joined #openstack-operators18:48
*** mriedem has quit IRC18:49
*** delattec has quit IRC18:51
*** mriedem has joined #openstack-operators18:51
mriedemwhat did i just miss? connection dropped18:52
*** lhcheng has joined #openstack-operators18:54
mgagnemriedem someone upgraded and restarted ovs :D18:55
mgagne"mriedem: I think for most operators upgrading every six months is plenty.  I'm sure plenty of people upgrade between dot releases, btu I doubt it's the common case"18:56
mgagne"especially if you run multiple services on a single node and use upstream packages, since that means you're going to have to restart ovs agent to upgrade, which is usually customer impacting"18:56
claytonfwiw, that's better in liberty, but it's still not fixed.18:56
*** matrohon has quit IRC18:57
*** SimonChung has joined #openstack-operators18:59
*** SimonChung1 has quit IRC18:59
*** SimonChung has quit IRC18:59
*** SimonChung1 has joined #openstack-operators18:59
*** bvandenh_ has joined #openstack-operators19:04
*** fawadkhaliq has quit IRC19:06
*** eil397 has joined #openstack-operators19:06
*** VW has joined #openstack-operators19:13
*** VW has quit IRC19:14
*** pilgrimstack has joined #openstack-operators19:14
*** VW has joined #openstack-operators19:15
*** esker has quit IRC19:16
*** esker has joined #openstack-operators19:17
*** dims_ has joined #openstack-operators19:17
*** dims has quit IRC19:17
*** esker has quit IRC19:19
xavpaiceheck, I thought that was fixed in liberty19:21
xavpaice(restarting the neutron agent used to drop all the flows when it started back up rather than checking them off) - thought that wasn't the case in liberty.  Haven't done the upgrade yet to find out :(19:22
claytonxavpaice: it's better in liberty, I'm actually in the middle of doing testing to quantify the difference19:23
claytonit used to drop all the flows, now the br-int and br-tun flows stay up, but the br-ex flows are dropped19:23
claytonIt looks like the issue is that the br-ex flows are dropped, then all the br-tun flows are repopulated, then all the br-ex flows are setup again.19:23
claytonwhich basically means routers can't reach out during that time...19:24
claytonthere is also a window where the patch port between br-int and br-tun is torn down during OVS agent restart, but that's much smaller, probably not noticable in almost all cases.19:25
xavpaiceouch19:26
xavpaiceso, still customer impacting, therefore preventing us doing maintenance19:27
*** dims_ has quit IRC19:27
xavpaicegood to know :)19:27
claytonthere is a patch in progress for mitaka to fix it19:27
claytonhttps://bugs.launchpad.net/neutron/+bug/151405619:27
openstackLaunchpad bug 1514056 in neutron "Set agent timestamp aka cookie to physical bridges" [Low,In progress] - Assigned to Eugene Nikanorov (enikanorov)19:27
xavpaicemeh, liberty's old hat now.19:27
claytonthis is someting that's hard to test in the gate, so it's hard to detect regressions too19:28
* xavpaice sets watch on that bug19:28
xavpaicefor sure19:28
claytonwhen I was doing my investigation, I put a sleep 10 in rootwrap, so it's pretty obvious then ;)19:28
xavpaiceI'll bet19:29
*** dims has joined #openstack-operators19:30
*** eil397 has quit IRC19:30
*** eil397 has joined #openstack-operators19:31
*** eil397 has quit IRC19:33
*** prithiv has joined #openstack-operators19:34
*** esker has joined #openstack-operators19:34
*** esker has quit IRC19:34
*** esker has joined #openstack-operators19:34
*** eil397 has joined #openstack-operators19:36
*** simon-AS559 has joined #openstack-operators19:37
*** prithiv has quit IRC19:38
*** prithiv has joined #openstack-operators19:38
*** simon-AS5591 has joined #openstack-operators19:39
*** dims has quit IRC19:41
*** dims has joined #openstack-operators19:42
*** simon-AS559 has quit IRC19:42
*** mriedem has quit IRC19:45
*** dims_ has joined #openstack-operators19:46
*** mriedem has joined #openstack-operators19:47
*** dims has quit IRC19:49
*** esker has quit IRC19:56
*** dims_ has quit IRC19:57
*** simon-AS559 has joined #openstack-operators20:00
*** rockyg has joined #openstack-operators20:01
*** simon-AS5591 has quit IRC20:03
*** eil397 has quit IRC20:06
*** rcernin has joined #openstack-operators20:08
*** piet has joined #openstack-operators20:09
*** piet has quit IRC20:13
*** piet has joined #openstack-operators20:13
*** VW has quit IRC20:23
*** Marga_ has quit IRC20:23
*** VW has joined #openstack-operators20:24
*** prithiv has quit IRC20:24
mdormananybody have a working example of using the HTTPProxyToWSGI middleware with neutron (or anything else)?   struggling with it a bit.20:31
mdorman2016-02-09 13:27:53.300 22959 TRACE neutron.common.config LookupError: No loader given in section 'filter:httpproxytowsgi'20:32
mdorman[filter:httpproxytowsgi]20:32
mdormanpaste.filter_factor = oslo_middleware:HTTPProxyToWSGI.factory20:32
*** maishsk has joined #openstack-operators20:43
*** krobzaur_ has joined #openstack-operators20:46
*** piet has quit IRC20:46
*** krobzaur has quit IRC20:47
*** maishsk has quit IRC20:52
*** esker has joined #openstack-operators20:53
*** pilgrimstack has quit IRC20:53
*** maishsk has joined #openstack-operators20:53
*** l4h has quit IRC20:55
*** Marga_ has joined #openstack-operators20:59
*** esker has quit IRC20:59
*** eil397 has joined #openstack-operators21:01
*** nikhil_k is now known as nikhil_21:01
*** nikhil_ is now known as nikhil_k21:02
*** dims has joined #openstack-operators21:03
*** krotscheck is now known as krotscheck_dcm21:03
*** eil397 has quit IRC21:08
*** prithiv has joined #openstack-operators21:10
*** simon-AS559 has quit IRC21:18
*** l4h has joined #openstack-operators21:28
*** matrohon has joined #openstack-operators21:28
*** regXboi has quit IRC21:43
*** nikhil_k is now known as nikhil21:45
*** Marga_ has quit IRC21:49
*** Marga_ has joined #openstack-operators21:50
*** Marga_ has quit IRC21:57
*** Marga_ has joined #openstack-operators21:57
*** esker has joined #openstack-operators22:00
*** matrohon has quit IRC22:03
*** esker has quit IRC22:05
*** SimonChung has joined #openstack-operators22:11
*** SimonChung1 has quit IRC22:11
*** SimonChung1 has joined #openstack-operators22:13
*** SimonChung2 has joined #openstack-operators22:13
*** SimonChung1 has quit IRC22:13
*** SimonChung has quit IRC22:13
*** prithiv has quit IRC22:24
*** prithiv has joined #openstack-operators22:24
*** rockyg has quit IRC22:25
*** ducttape_ has quit IRC22:31
*** esker has joined #openstack-operators22:34
*** dims_ has joined #openstack-operators22:39
*** dims has quit IRC22:40
*** esker has quit IRC22:46
*** esker has joined #openstack-operators22:52
*** krobzaur_ has quit IRC22:52
*** rcernin has quit IRC22:58
*** jaypipes has quit IRC23:00
*** SimonChung has joined #openstack-operators23:03
*** SimonChung2 has quit IRC23:03
*** SimonChung has quit IRC23:04
*** SimonChung has joined #openstack-operators23:04
*** SimonChung1 has joined #openstack-operators23:11
*** SimonChung has quit IRC23:11
*** mriedem has quit IRC23:22
*** alop has quit IRC23:28
*** SimonChung1 has quit IRC23:28
*** SimonChung has joined #openstack-operators23:28
*** SimonChung1 has joined #openstack-operators23:29
*** SimonChung has quit IRC23:29
*** bvandenh_ has quit IRC23:29
*** VW has quit IRC23:30
*** mdorman has quit IRC23:35
*** k_stev has quit IRC23:37
*** prithiv has left #openstack-operators23:39
*** mdorman has joined #openstack-operators23:48
*** eil397 has joined #openstack-operators23:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!