Wednesday, 2017-01-18

*** harlowja has joined #openstack-operators00:06
*** VW has joined #openstack-operators00:09
*** VW has quit IRC00:14
klindgrenhas anyone here ran into an issue on hammer, where when rebalacing the cluster the RGW nodes crash and need to restart the ceph-radosgw service.  THen it keeps crashing?00:49
*** sdake has quit IRC01:21
*** ducttape_ has joined #openstack-operators01:25
*** fifieldt has quit IRC01:27
*** stevemar has quit IRC01:27
*** bsilverman_ has quit IRC01:27
*** basilAB has joined #openstack-operators01:28
*** fifieldt has joined #openstack-operators01:29
*** stevemar has joined #openstack-operators01:29
*** bsilverman_ has joined #openstack-operators01:29
*** B_Smith has quit IRC01:29
*** mriedem1 has joined #openstack-operators01:39
*** mriedem has quit IRC01:41
*** B_Smith has joined #openstack-operators01:52
*** mriedem1 is now known as mriedem01:55
*** Apoorva has quit IRC02:04
*** B_Smith has quit IRC02:10
*** jamesdenton has joined #openstack-operators02:23
*** B_Smith has joined #openstack-operators02:23
*** ducttape_ has quit IRC02:24
*** ducttape_ has joined #openstack-operators02:29
*** harlowja has quit IRC02:36
*** Apoorva has joined #openstack-operators02:44
*** VW has joined #openstack-operators02:58
*** VW has quit IRC03:00
*** VW has joined #openstack-operators03:00
*** ducttape_ has quit IRC03:15
*** jamesdenton has quit IRC03:16
*** B_Smith has quit IRC03:29
*** B_Smith has joined #openstack-operators03:33
*** sudipto_ has joined #openstack-operators03:50
*** sudipto has joined #openstack-operators03:50
*** B_Smith has quit IRC03:52
*** sdake has joined #openstack-operators03:53
*** B_Smith has joined #openstack-operators03:57
*** sudipto has quit IRC04:14
*** sudipto_ has quit IRC04:14
*** B_Smith has quit IRC04:28
*** sudipto_ has joined #openstack-operators04:35
*** sudipto has joined #openstack-operators04:35
*** B_Smith has joined #openstack-operators04:44
*** VW has quit IRC04:50
openstackgerritNam Nguyen Hoai proposed openstack/osops-coda: Remove white space between print ()  https://review.openstack.org/42166604:51
*** Apoorva has quit IRC04:51
*** sudipto_ has quit IRC05:10
*** sudipto has quit IRC05:10
*** tanee_ has joined #openstack-operators05:10
*** sudipto has joined #openstack-operators05:12
*** sudipto_ has joined #openstack-operators05:13
*** Apoorva has joined #openstack-operators05:14
*** sdake has quit IRC05:16
*** sdake has joined #openstack-operators05:25
*** sdake has quit IRC05:46
*** crenguta has joined #openstack-operators05:57
*** Apoorva has quit IRC06:21
*** slaweq_ has joined #openstack-operators06:26
*** crenguta has quit IRC06:35
*** crenguta has joined #openstack-operators06:36
*** B_Smith has quit IRC06:48
*** saneax-_-|AFK is now known as saneax07:00
*** B_Smith has joined #openstack-operators07:03
*** bapalm has quit IRC07:16
*** Oku_OS-away is now known as Oku_OS07:17
*** bapalm has joined #openstack-operators07:18
*** tesseract has joined #openstack-operators07:25
*** tesseract has quit IRC07:25
*** tesseract has joined #openstack-operators07:26
*** dfflanders has quit IRC07:29
*** zioproto has joined #openstack-operators07:44
*** crenguta has quit IRC07:49
*** tanee has quit IRC07:51
*** tanee_ is now known as tanee07:52
*** crenguta has joined #openstack-operators07:59
*** markvoelker has joined #openstack-operators08:26
*** zeih has joined #openstack-operators08:28
*** rmart04 has joined #openstack-operators08:35
*** B_Smith has quit IRC08:36
*** B_Smith has joined #openstack-operators08:40
*** rmart04 has quit IRC08:50
*** rmart04 has joined #openstack-operators08:57
*** simon-AS559 has joined #openstack-operators09:11
*** zioproto has quit IRC09:15
*** racedo has joined #openstack-operators09:17
*** derekh has joined #openstack-operators09:23
*** dfflanders has joined #openstack-operators09:26
*** mattoliverau has quit IRC09:27
*** matt6434 has joined #openstack-operators09:28
*** zeih has quit IRC09:40
*** pcaruana has joined #openstack-operators09:41
*** zeih has joined #openstack-operators09:49
*** zeih has quit IRC10:00
*** zioproto has joined #openstack-operators10:03
*** zeih has joined #openstack-operators10:08
*** B_Smith has quit IRC10:21
*** B_Smith has joined #openstack-operators10:30
*** pcaruana has quit IRC10:38
*** B_Smith has quit IRC10:48
*** paramite has joined #openstack-operators10:54
yankcrimeklindgren: i've seen monitor nodes crash under heavy rebalancing operations, but never rgws10:57
*** racedo has quit IRC10:57
*** racedo has joined #openstack-operators10:58
*** dfflanders has quit IRC11:05
*** sudipto_ has quit IRC11:25
*** sudipto has quit IRC11:25
*** dalees has quit IRC11:27
*** dalees has joined #openstack-operators11:28
*** aojea has quit IRC11:40
*** B_Smith has joined #openstack-operators11:42
*** aojea has joined #openstack-operators11:50
*** pcaruana has joined #openstack-operators11:56
*** cdelatte has joined #openstack-operators12:15
*** zeih has quit IRC12:27
*** zioproto has quit IRC12:44
*** sudipto_ has joined #openstack-operators12:44
*** sudipto has joined #openstack-operators12:44
*** ducttape_ has joined #openstack-operators12:44
*** sdake_ has joined #openstack-operators12:57
*** racedo has quit IRC12:57
*** racedo has joined #openstack-operators12:58
*** sdake__ has joined #openstack-operators12:59
*** sdake_ has quit IRC13:02
*** paramite has quit IRC13:02
*** sdake__ is now known as sdake13:09
*** ducttape_ has quit IRC13:09
mnaseranyone see this type of behaviour on network nodes? DHCPRELEASE(tap5a1402bd-a8) 10.30.48.143 fa:16:3e:44:cd:ad unknown lease13:32
*** B_Smith has quit IRC13:35
mnaserand things like this (those are the more problematic ones): dnsmasq-dhcp[15226]: DHCPDISCOVER(tap22f0da1c-9f) fa:16:3e:43:01:93 no address available13:39
*** liverpooler has joined #openstack-operators13:41
*** catintheroof has joined #openstack-operators13:45
*** B_Smith has joined #openstack-operators13:45
*** Miouge has joined #openstack-operators13:50
*** ducttape_ has joined #openstack-operators13:53
*** liverpooler has quit IRC13:54
*** liverpooler has joined #openstack-operators13:54
erhudyyeah, i've seen it a few times13:58
erhudyi think it's an overly twitchy dhclient in the instance13:58
*** zeih has joined #openstack-operators13:59
erhudyi don't really know how it could mistake itself for another instance but it does the DHCPRELEASE and then dnsmasq blacklists it for like 10 minutes13:59
erhudyklindgren: i've never seen RGW crash during rebalances, occasionally OSDs fall over but that's it14:00
erhudycurrently on 0.94.6, moving to 0.94.914:00
*** mriedem has quit IRC14:01
*** liverpooler has quit IRC14:04
*** liverpooler has joined #openstack-operators14:05
*** electrofelix has joined #openstack-operators14:10
*** zeih has quit IRC14:28
*** B_Smith has quit IRC14:32
*** mriedem has joined #openstack-operators14:34
*** B_Smith has joined #openstack-operators14:40
*** jamesdenton has joined #openstack-operators14:48
*** VW has joined #openstack-operators15:06
*** ducttape_ has quit IRC15:10
*** openstack has joined #openstack-operators15:19
*** sudipto has quit IRC15:19
*** simonmcc has quit IRC15:19
*** notmyname has quit IRC15:19
*** tnarg has quit IRC15:19
*** khappone_ has quit IRC15:19
*** mgagne has quit IRC15:19
*** robputt has quit IRC15:19
*** fyxim has quit IRC15:19
*** DuncanT has quit IRC15:19
*** lukl has quit IRC15:19
*** mj_richardson has quit IRC15:19
*** zeus has quit IRC15:19
*** mrhillsman has quit IRC15:19
*** godog has quit IRC15:19
*** mrhillsman has joined #openstack-operators15:19
*** notmyname has joined #openstack-operators15:19
*** zeih has joined #openstack-operators15:20
*** robputt has joined #openstack-operators15:22
*** mgagne has joined #openstack-operators15:22
*** sudipto has joined #openstack-operators15:22
*** tnarg has joined #openstack-operators15:22
*** khappone_ has joined #openstack-operators15:22
*** lukl has joined #openstack-operators15:22
*** zeus has joined #openstack-operators15:22
*** mgagne has quit IRC15:23
*** mgagne has joined #openstack-operators15:23
*** mgagne is now known as Guest5853115:23
*** godog_ is now known as godog15:26
*** godog is now known as Guest2027315:26
*** DuncanT has joined #openstack-operators15:26
*** Guest20273 has quit IRC15:26
*** Guest20273 has joined #openstack-operators15:26
*** Guest20273 is now known as godog15:27
*** fyxim has joined #openstack-operators15:29
*** zeih has quit IRC15:31
*** saneax is now known as saneax-_-|AFK15:36
*** simonmcc has joined #openstack-operators15:37
*** kencjohnston_ has quit IRC15:42
*** kencjohnston has joined #openstack-operators15:43
*** zeih has joined #openstack-operators15:46
*** B_Smith has quit IRC15:47
*** david-lyle has joined #openstack-operators15:47
*** ducttape_ has joined #openstack-operators15:48
*** zeih has quit IRC15:52
*** marst_ has joined #openstack-operators15:58
*** _ducttape_ has joined #openstack-operators15:58
*** racedo has quit IRC16:00
*** ducttape_ has quit IRC16:01
*** racedo has joined #openstack-operators16:02
*** B_Smith has joined #openstack-operators16:03
*** markvoelker has quit IRC16:11
*** zul has quit IRC16:15
*** zul has joined #openstack-operators16:15
*** rmart04 has quit IRC16:15
*** _ducttape_ has quit IRC16:29
*** ducttape_ has joined #openstack-operators16:30
*** ducttape_ has quit IRC16:36
*** _ducttape_ has joined #openstack-operators16:37
*** harlowja has joined #openstack-operators16:39
*** ddyer has quit IRC16:44
*** ddyer has joined #openstack-operators16:48
*** VW_ has joined #openstack-operators16:54
*** _ducttape_ has quit IRC16:56
*** ducttape_ has joined #openstack-operators16:56
*** VW has quit IRC16:57
*** VW_ has quit IRC16:57
*** VW has joined #openstack-operators16:58
*** VW has quit IRC17:01
*** VW has joined #openstack-operators17:01
*** simon-AS559 has quit IRC17:02
*** sudipto has quit IRC17:07
*** sudipto_ has quit IRC17:07
*** rmart04 has joined #openstack-operators17:09
*** rmart04 has quit IRC17:14
*** tesseract has quit IRC17:21
openstackgerritMerged openstack/osops-coda: Remove white space between print ()  https://review.openstack.org/42166617:30
openstackgerritMerged openstack/osops-tools-contrib: Lint the lampstack apply role  https://review.openstack.org/39403317:30
*** harlowja has quit IRC17:33
*** david-lyle is now known as bailing-wire17:37
*** markvoelker has joined #openstack-operators17:38
*** ddyer has quit IRC17:39
*** ddyer has joined #openstack-operators17:43
*** armax has joined #openstack-operators17:43
*** catintheroof has quit IRC17:44
*** catintheroof has joined #openstack-operators17:44
*** markvoelker_ has joined #openstack-operators17:44
*** catintheroof has quit IRC17:44
*** Apoorva has joined #openstack-operators17:45
*** catintheroof has joined #openstack-operators17:45
*** markvoelker has quit IRC17:46
*** bailing-wire has quit IRC17:46
*** catintheroof has quit IRC17:50
*** racedo has quit IRC17:50
*** armax_ has joined #openstack-operators17:57
*** armax has quit IRC17:57
*** armax_ is now known as armax17:57
*** markvoelker has joined #openstack-operators17:58
*** derekh has quit IRC18:01
*** markvoelker_ has quit IRC18:01
*** B_Smith has quit IRC18:12
*** VW has quit IRC18:12
*** VW has joined #openstack-operators18:14
*** VW has quit IRC18:14
*** VW has joined #openstack-operators18:15
*** B_Smith has joined #openstack-operators18:19
*** shewless has joined #openstack-operators18:27
shewlessHI there. Does anyone here have a moment to help me get SSL working for nova? I've set nova.conf the way I think it needs to be set but I'm getting an oslo_service errror18:28
shewlessThe error is: ERROR oslo_service.service     self._context.load_cert_chain(certfile, keyfile) ERROR oslo_service.service IOError: [Errno 13] Permission denied18:29
shewlessfrom nova-api.log18:29
shewlessI've tried to change the ownership and permissions of my cert files but it doesn't seem to help at all18:29
*** sdake has quit IRC18:30
*** electrofelix has quit IRC18:38
*** sdake has joined #openstack-operators18:40
*** sticker_ has joined #openstack-operators18:51
*** sticker has quit IRC18:54
*** racedo has joined #openstack-operators18:59
*** dminer has joined #openstack-operators19:05
*** racedo has quit IRC19:07
*** sdake_ has joined #openstack-operators19:10
*** sdake has quit IRC19:14
*** Oku_OS is now known as Oku_OS-away19:19
*** racedo has joined #openstack-operators19:22
*** sdake_ is now known as sdkae19:23
*** sdkae is now known as sdake19:23
*** racedo has quit IRC19:24
*** B_Smith has quit IRC19:43
*** VW_ has joined #openstack-operators19:55
*** VW__ has joined #openstack-operators19:57
*** VW_ has quit IRC19:57
*** VW has quit IRC19:58
*** VW__ has quit IRC19:59
*** VW has joined #openstack-operators19:59
*** B_Smith has joined #openstack-operators20:00
*** bailing-wire has joined #openstack-operators20:00
*** bailing-wire is now known as david-lyle20:02
*** markvoelker_ has joined #openstack-operators20:03
*** VW has quit IRC20:04
*** VW has joined #openstack-operators20:05
*** markvoelker has quit IRC20:05
*** markvoelker_ has quit IRC20:32
*** _ducttape_ has joined #openstack-operators20:36
*** ducttape_ has quit IRC20:39
mnasershewless sudo -u nova stat /path/to/ssl/cert20:42
mnaserdoes that work20:42
mnaserif not, fix20:42
shewlessmnaser thanks. I was able to fix the problem by restarting nova-api (:S)20:43
mnaserthats strange20:43
mnaserbut hey, it works it works20:43
shewlesswell.. I fixed the permissions and then restarted nova-api20:44
shewlessI'm getting another error now:  sslv3 alert handshake failure20:44
shewlessand I can't find anyway to tell nova which encryption to use20:45
shewlessI'm thinking of just using an nginx reverse proxy to do SSL20:45
*** aojea has quit IRC21:00
*** crenguta has quit IRC21:09
*** aojea has joined #openstack-operators21:12
mnasershewless i personally run haproxy to do ssl21:21
shewlessmnaser: interesting. Any chance you have an example of how to do that?21:24
*** zul has quit IRC21:25
*** matt6434 is now known as mattoliverau21:26
*** _ducttape_ has quit IRC21:28
*** zul has joined #openstack-operators21:28
*** ducttape_ has joined #openstack-operators21:28
*** sdake_ has joined #openstack-operators21:29
mnasershewless just plain o'l haproxy + ssl forwarding by http to the nova service21:30
shewlesswhat about buffers, headers, etc? no other config?21:31
*** sdake has quit IRC21:32
erhudyi would hate to terminate SSL on the services directly, we terminate all SSL on haproxy21:32
mnaserwhat erhudy said21:32
mnaserand nope, i had no specific needs shewless21:33
erhudy~~just works~~21:33
mnasernginx was a mess because it tries too much to be an http server21:33
mnaserhaproxy is dumb in a good way21:33
mnaserthat it doesn't interfere21:33
mnaserexample glance infront of nginx would cause weird issues because nginx would filter out some headers because 'it didnt like it'21:33
shewlessmnaser and erhudy I needed to talk to you guys a long time ago21:33
shewlessout of curiosity do you haproxy horizon and keystone or do you let apache do that?21:34
erhudyhaproxy is still in front of them for me21:34
*** VW has quit IRC21:35
erhudyapache is responsible for doing mod_wsgi but SSL is still terminated on haproxy21:35
mnaserwe have haproxy infront of all services21:35
*** B_Smith has quit IRC21:35
mnaser^ that21:35
*** VW has joined #openstack-operators21:35
*** sticker_ is now known as sticker21:44
*** B_Smith has joined #openstack-operators21:47
shewlessmnaser and erhudy: to clarify your actual endpoints as added to openstack are http? But anything that's using the apis use https which is reverse proxied to http?21:49
mnaserwe have https everywhere21:49
mnaser(and i dont know why anyone wouldn't)21:50
erhudyall the stuff i publish in the service catalog is https and references the keepalived VIP21:50
erhudyinter-service communication is https21:50
mnaseronly diff we have is21:50
shewlessokay so when you add the actual endpoint (openstack endpoint create) you do so with https21:50
mnaseryeah21:50
mnaserand we use virtualhosts21:50
mnaserso our apis run on port 80/44321:50
shewlessHow does that work with nova?21:51
erhudywe differ from mnaser in that respect, we have haproxy listen on the default API ports21:51
erhudye.g. 8776, 9292, whatever21:52
shewlessokay so nova's endpoint is https://nova.example.com:8774/v2.1/%(tenant_id)s21:53
shewlessbut nova.conf will have NO ssl config at all21:53
shewlesshaproxy will receive https://nova.example.com:8774 and terminate the SSL21:53
erhudyright21:54
shewlessbut how does it jam it into the endpoint?21:54
shewlesslike that part of the connection won't actually be SSL right?21:54
*** slaweq_ has quit IRC22:00
*** slaweq has joined #openstack-operators22:00
*** slaweq has quit IRC22:02
*** slaweq has joined #openstack-operators22:02
*** chyka has joined #openstack-operators22:02
*** slaweq has quit IRC22:07
*** slaweq has joined #openstack-operators22:09
*** haplo37 has quit IRC22:22
*** jamesdenton has quit IRC22:28
*** slaweq has quit IRC22:29
*** slaweq has joined #openstack-operators22:30
*** haplo37 has joined #openstack-operators22:31
*** david-lyle has quit IRC22:32
*** armax has quit IRC22:36
*** sdake_ has quit IRC22:38
*** david-lyle has joined #openstack-operators22:40
*** VW has quit IRC22:45
*** liverpoo1er has quit IRC22:49
*** sdake has joined #openstack-operators23:10
*** saneax-_-|AFK is now known as saneax23:15
*** ducttape_ has quit IRC23:16
*** sdake has quit IRC23:24
*** ddyer has quit IRC23:24
*** furlongm_ has quit IRC23:24
*** tanee has quit IRC23:24
*** fifieldt has quit IRC23:24
*** stevemar has quit IRC23:24
*** bsilverman_ has quit IRC23:24
*** sdake has joined #openstack-operators23:24
*** ddyer has joined #openstack-operators23:24
*** furlongm_ has joined #openstack-operators23:24
*** tanee has joined #openstack-operators23:24
*** fifieldt has joined #openstack-operators23:24
*** stevemar has joined #openstack-operators23:24
*** bsilverman_ has joined #openstack-operators23:24
*** sdake has quit IRC23:25
*** sdake has joined #openstack-operators23:26
*** jamesdenton has joined #openstack-operators23:29
*** jamesdenton has quit IRC23:31
*** B_Smith has quit IRC23:32
*** dmitryme has quit IRC23:34
*** DinaBelova has quit IRC23:34
*** DinaBelova has joined #openstack-operators23:35
*** dmitryme has joined #openstack-operators23:36
*** B_Smith has joined #openstack-operators23:38
*** armax has joined #openstack-operators23:41
*** dmitryme has quit IRC23:57
*** DinaBelova has quit IRC23:57
*** DinaBelova has joined #openstack-operators23:57
*** dmitryme has joined #openstack-operators23:58
*** cdelatte has quit IRC23:59
*** DinaBelova has quit IRC23:59
*** dmitryme has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!