| *** saneax is now known as saneax_AFK | 00:23 | |
| *** vinsh has quit IRC | 00:23 | |
| *** dims has joined #openstack-operators | 00:23 | |
| *** vinsh has joined #openstack-operators | 00:23 | |
| *** vinsh_ has joined #openstack-operators | 00:27 | |
| *** vinsh has quit IRC | 00:27 | |
| *** Marga__ has quit IRC | 00:27 | |
| *** Marga_ has joined #openstack-operators | 00:28 | |
| *** lhcheng has quit IRC | 00:30 | |
| *** mnaser has quit IRC | 00:31 | |
| *** dims has quit IRC | 00:32 | |
| *** furlongm has joined #openstack-operators | 00:35 | |
| *** markvoelker has joined #openstack-operators | 00:40 | |
| *** mnaser has joined #openstack-operators | 00:41 | |
| *** SimonChung has joined #openstack-operators | 00:41 | |
| *** SimonChung has quit IRC | 00:42 | |
| *** SimonChung2 has joined #openstack-operators | 00:42 | |
| *** SimonChung has joined #openstack-operators | 00:43 | |
| *** SimonChung2 has quit IRC | 00:43 | |
| *** SimonChung1 has quit IRC | 00:44 | |
| *** SimonChung has quit IRC | 00:47 | |
| *** dims has joined #openstack-operators | 00:56 | |
| *** ducttape_ has joined #openstack-operators | 01:12 | |
| *** ducttape_ has quit IRC | 01:12 | |
| *** ducttape_ has joined #openstack-operators | 01:14 | |
| *** markvoelker has quit IRC | 01:14 | |
| *** SimonChung has joined #openstack-operators | 01:21 | |
| *** kstev has quit IRC | 01:22 | |
| *** SimonChung1 has joined #openstack-operators | 01:23 | |
| *** SimonChung has quit IRC | 01:25 | |
| *** chlong has quit IRC | 01:26 | |
| *** ducttape_ has quit IRC | 01:36 | |
| *** ducttape_ has joined #openstack-operators | 01:38 | |
| *** harshs has quit IRC | 01:42 | |
| *** ducttape_ has quit IRC | 01:54 | |
| *** rockyg has joined #openstack-operators | 01:54 | |
| *** apoorvad has quit IRC | 01:58 | |
| *** piet has joined #openstack-operators | 02:01 | |
| *** ducttape_ has joined #openstack-operators | 02:02 | |
| *** piet has quit IRC | 02:06 | |
| *** piet has joined #openstack-operators | 02:06 | |
| *** fawadkhaliq has quit IRC | 02:12 | |
| *** ducttape_ has quit IRC | 02:23 | |
| *** rockyg has quit IRC | 02:31 | |
| *** dims has quit IRC | 02:38 | |
| *** armax has quit IRC | 02:38 | |
| *** lhcheng has joined #openstack-operators | 02:44 | |
| *** armax has joined #openstack-operators | 02:58 | |
| *** fawadkhaliq has joined #openstack-operators | 03:06 | |
| *** markvoelker has joined #openstack-operators | 03:11 | |
| *** esker has quit IRC | 03:13 | |
| *** lhcheng_ has joined #openstack-operators | 03:26 | |
| *** lhcheng has quit IRC | 03:29 | |
| *** chlong has joined #openstack-operators | 03:30 | |
| *** haukebruno has quit IRC | 03:38 | |
| *** haukebruno has joined #openstack-operators | 03:40 | |
| *** markvoelker has quit IRC | 03:44 | |
| *** VW has joined #openstack-operators | 03:55 | |
| *** haukebruno has quit IRC | 03:57 | |
| *** Marga_ has quit IRC | 03:59 | |
| *** Marga_ has joined #openstack-operators | 04:00 | |
| *** Marga_ has quit IRC | 04:00 | |
| *** piet has quit IRC | 04:19 | |
| *** piet has joined #openstack-operators | 04:19 | |
| *** armax has quit IRC | 04:20 | |
| *** vsedelnik has joined #openstack-operators | 04:20 | |
| *** Marga_ has joined #openstack-operators | 04:21 | |
| *** vsedelnik has quit IRC | 04:23 | |
| *** armax has joined #openstack-operators | 04:25 | |
| *** Marga_ has quit IRC | 04:25 | |
| *** piet has quit IRC | 04:30 | |
| *** piet has joined #openstack-operators | 04:31 | |
| *** Marga_ has joined #openstack-operators | 04:33 | |
| *** Marga_ has quit IRC | 04:34 | |
| *** Marga_ has joined #openstack-operators | 04:34 | |
| *** piet has quit IRC | 04:44 | |
| *** piet has joined #openstack-operators | 04:53 | |
| *** VW has quit IRC | 05:26 | |
| *** armax has quit IRC | 05:29 | |
| *** lhcheng has joined #openstack-operators | 05:40 | |
| *** markvoelker has joined #openstack-operators | 05:41 | |
| *** piet has quit IRC | 05:44 | |
| *** lhcheng_ has quit IRC | 05:44 | |
| *** lhcheng has quit IRC | 05:45 | |
| *** armax has joined #openstack-operators | 05:48 | |
| *** saneax_AFK is now known as saneax | 06:00 | |
| *** furlongm has quit IRC | 06:00 | |
| *** furlongm has joined #openstack-operators | 06:05 | |
| *** vsedelnik has joined #openstack-operators | 06:07 | |
| *** vsedelnik has quit IRC | 06:13 | |
| *** markvoelker has quit IRC | 06:14 | |
| *** simon-AS559 has joined #openstack-operators | 06:17 | |
| *** lhcheng has joined #openstack-operators | 06:43 | |
| *** lhcheng_ has joined #openstack-operators | 06:59 | |
| *** lhcheng has quit IRC | 07:03 | |
| *** admin0 has joined #openstack-operators | 07:11 | |
| *** tesseract has joined #openstack-operators | 07:13 | |
| *** tesseract is now known as Guest97519 | 07:13 | |
| *** vsedelnik has joined #openstack-operators | 07:19 | |
| *** vsedelnik has quit IRC | 07:19 | |
| *** vsedelnik has joined #openstack-operators | 07:19 | |
| *** liverpooler has joined #openstack-operators | 07:24 | |
| *** simon-AS559 has quit IRC | 07:25 | |
| *** VW has joined #openstack-operators | 07:26 | |
| *** maishsk has joined #openstack-operators | 07:29 | |
| *** fawadkhaliq has quit IRC | 07:30 | |
| *** VW has quit IRC | 07:30 | |
| *** liverpooler has quit IRC | 07:33 | |
| *** liverpooler has joined #openstack-operators | 07:34 | |
| *** rarcea has joined #openstack-operators | 07:35 | |
| *** lhcheng_ has quit IRC | 07:50 | |
| *** belmoreira has joined #openstack-operators | 07:58 | |
| *** Guest97519 has quit IRC | 08:00 | |
| *** rarcea has quit IRC | 08:03 | |
| *** matrohon has joined #openstack-operators | 08:04 | |
| *** simon-AS559 has joined #openstack-operators | 08:05 | |
| *** simon-AS5591 has joined #openstack-operators | 08:05 | |
| *** tesseract has joined #openstack-operators | 08:05 | |
| *** tesseract is now known as Guest91794 | 08:06 | |
| *** admin0 has quit IRC | 08:07 | |
| *** bvandenh has joined #openstack-operators | 08:08 | |
| *** simon-AS559 has quit IRC | 08:10 | |
| *** markvoelker has joined #openstack-operators | 08:12 | |
| *** haukebruno has joined #openstack-operators | 08:29 | |
| *** maishsk_ has joined #openstack-operators | 08:31 | |
| *** fawadkhaliq has joined #openstack-operators | 08:31 | |
| *** maishsk has quit IRC | 08:32 | |
| *** maishsk_ is now known as maishsk | 08:32 | |
| *** chlong has quit IRC | 08:32 | |
| *** markvoelker has quit IRC | 08:44 | |
| *** Guest91794 has quit IRC | 08:45 | |
| *** zeih has joined #openstack-operators | 08:50 | |
| *** admin0 has joined #openstack-operators | 08:58 | |
| *** Guest91794 has joined #openstack-operators | 09:00 | |
| *** admin0 has quit IRC | 09:01 | |
| *** luisfdez1 has joined #openstack-operators | 09:07 | |
| *** luisfdez has quit IRC | 09:07 | |
| *** admin0 has joined #openstack-operators | 09:10 | |
| *** zeih has quit IRC | 09:18 | |
| *** admin0 has quit IRC | 09:19 | |
| *** admin0 has joined #openstack-operators | 09:22 | |
| *** rcernin has joined #openstack-operators | 09:31 | |
| *** admin0 has quit IRC | 09:32 | |
| *** admin0 has joined #openstack-operators | 09:35 | |
| *** lhcheng has joined #openstack-operators | 09:38 | |
| *** mathlin has joined #openstack-operators | 09:40 | |
| *** maishsk has quit IRC | 09:40 | |
| *** lhcheng has quit IRC | 09:43 | |
| *** derekh has joined #openstack-operators | 09:47 | |
| *** luisfdez has joined #openstack-operators | 10:11 | |
| *** liverpooler has quit IRC | 10:12 | |
| *** liverpooler has joined #openstack-operators | 10:12 | |
| *** luisfdez1 has quit IRC | 10:14 | |
| *** luisfdez has quit IRC | 10:18 | |
| *** electrofelix has joined #openstack-operators | 10:21 | |
| *** admin0 has quit IRC | 10:28 | |
| *** fwdit has joined #openstack-operators | 10:37 | |
| *** admin0 has joined #openstack-operators | 10:42 | |
| *** markvoelker has joined #openstack-operators | 10:42 | |
| *** luisfdez has joined #openstack-operators | 10:47 | |
| *** admin0 has quit IRC | 10:53 | |
| *** maishsk has joined #openstack-operators | 11:02 | |
| *** dims has joined #openstack-operators | 11:12 | |
| *** markvoelker has quit IRC | 11:15 | |
| *** dims has quit IRC | 11:20 | |
| *** dims has joined #openstack-operators | 11:21 | |
| *** dims has quit IRC | 11:21 | |
| *** dims has joined #openstack-operators | 11:21 | |
| clayton | The OVS restart bug got pushed back to newton: https://bugs.launchpad.net/neutron/+bug/1514056 | 11:38 |
|---|---|---|
| openstack | Launchpad bug 1514056 in neutron "Restarting OVS agent drops VMs traffic when using VLAN provider bridges" [High,In progress] - Assigned to Hynek Mlnarik (hmlnarik-s) | 11:38 |
| *** admin0 has joined #openstack-operators | 11:49 | |
| *** lhcheng has joined #openstack-operators | 11:51 | |
| *** lhcheng has quit IRC | 11:55 | |
| *** eddima has joined #openstack-operators | 12:03 | |
| *** admin0_ has joined #openstack-operators | 12:05 | |
| *** bvandenh has quit IRC | 12:06 | |
| *** belmoreira has quit IRC | 12:07 | |
| *** admin0 has quit IRC | 12:08 | |
| *** admin0_ is now known as admin0 | 12:08 | |
| *** ducttape_ has joined #openstack-operators | 12:10 | |
| raginbajin | clayton: Any reason why they pushed it back? | 12:11 |
| clayton | my guess was just that it was taking too long | 12:12 |
| *** eddima has quit IRC | 12:17 | |
| *** saneax is now known as saneax_AFK | 12:17 | |
| *** liverpooler has quit IRC | 12:18 | |
| *** liverpooler has joined #openstack-operators | 12:19 | |
| *** eddima has joined #openstack-operators | 12:19 | |
| *** ducttape_ has quit IRC | 12:23 | |
| *** dims has quit IRC | 12:25 | |
| *** dims has joined #openstack-operators | 12:26 | |
| *** piet has joined #openstack-operators | 12:29 | |
| *** vsedelnik has quit IRC | 12:29 | |
| *** vsedelnik has joined #openstack-operators | 12:30 | |
| *** armax has quit IRC | 12:33 | |
| *** markvoelker has joined #openstack-operators | 12:53 | |
| *** vsedelnik has quit IRC | 12:57 | |
| *** armax has joined #openstack-operators | 13:05 | |
| *** maishsk has quit IRC | 13:08 | |
| *** maishsk has joined #openstack-operators | 13:08 | |
| *** ctrath has joined #openstack-operators | 13:14 | |
| *** ig0r__ has joined #openstack-operators | 13:19 | |
| *** piet has quit IRC | 13:20 | |
| *** armax has quit IRC | 13:21 | |
| *** ig0r_ has quit IRC | 13:22 | |
| *** chlong has joined #openstack-operators | 13:25 | |
| *** mriedem_away is now known as mriedem | 13:30 | |
| *** piet has joined #openstack-operators | 13:34 | |
| *** luisfdez has quit IRC | 13:34 | |
| *** piet has quit IRC | 13:34 | |
| *** luisfdez has joined #openstack-operators | 13:35 | |
| krobzaur | mjrichardson: Hey, thanks for the pointers guys. I just rebooted the box that was misbehaving and corosync/pacemaker services restarted properly during system init. | 13:37 |
| *** krobzaur has quit IRC | 13:37 | |
| *** ducttape_ has joined #openstack-operators | 13:38 | |
| *** dminer has joined #openstack-operators | 13:50 | |
| *** VW has joined #openstack-operators | 13:54 | |
| *** VW has quit IRC | 13:55 | |
| *** VW has joined #openstack-operators | 13:56 | |
| *** saneax_AFK is now known as saneax | 14:01 | |
| *** krobzaur has joined #openstack-operators | 14:02 | |
| *** Guest41345 has quit IRC | 14:03 | |
| *** admin0 has quit IRC | 14:04 | |
| *** piet has joined #openstack-operators | 14:08 | |
| *** Marga_ has quit IRC | 14:08 | |
| *** armax has joined #openstack-operators | 14:10 | |
| *** armax has left #openstack-operators | 14:11 | |
| *** vsedelnik has joined #openstack-operators | 14:11 | |
| yankcrime | clayton: :( | 14:14 |
| *** bvandenh has joined #openstack-operators | 14:14 | |
| *** VW has quit IRC | 14:15 | |
| *** VW has joined #openstack-operators | 14:16 | |
| *** krobzaur has quit IRC | 14:16 | |
| *** krobzaur has joined #openstack-operators | 14:18 | |
| *** kstev has joined #openstack-operators | 14:20 | |
| *** VW has quit IRC | 14:22 | |
| *** Guest91794 has quit IRC | 14:31 | |
| *** vsedelnik has quit IRC | 14:32 | |
| *** VW has joined #openstack-operators | 14:32 | |
| *** bvandenh has quit IRC | 14:33 | |
| *** vsedelnik has joined #openstack-operators | 14:33 | |
| *** VW has quit IRC | 14:34 | |
| *** VW has joined #openstack-operators | 14:34 | |
| *** regXboi has joined #openstack-operators | 14:47 | |
| *** bvandenh has joined #openstack-operators | 14:49 | |
| *** maishsk has quit IRC | 14:54 | |
| *** maishsk has joined #openstack-operators | 14:56 | |
| *** markvoelker has quit IRC | 15:00 | |
| *** kstev has quit IRC | 15:00 | |
| *** bvandenh has quit IRC | 15:00 | |
| *** SimonChung1 has quit IRC | 15:01 | |
| klindgren | krobzaur, what all are you running under pacemaker? We are running everything active-active-active | 15:04 |
| *** kstev has joined #openstack-operators | 15:04 | |
| *** tesseract has joined #openstack-operators | 15:06 | |
| notmyname | cross-post from the mailing list... | 15:07 |
| *** harshs has joined #openstack-operators | 15:07 | |
| *** tesseract is now known as Guest71467 | 15:07 | |
| notmyname | Swift recently added an ops runbook upstream: http://docs.openstack.org/developer/swift/ops_runbook/index.html | 15:07 |
| notmyname | I hope it will be helpful to anyone running Swift clusters | 15:07 |
| krobzaur | klindgren: A whole bunch of stuff. I'm using Fuel to deploy and so almost everything is configured and installed automatically. But I have 3 controllers. Most of my pacemaker resources are active-active-active (neutron vpn,metadata, and dhcp agents. mysql. heat-engine and ceilometer agent-central and alarm-evaluator) | 15:11 |
| krobzaur | Then rabbitmq and conntrackd are in a master-slave setup | 15:12 |
| krobzaur | Like I said, Fuel did all this configuration for me so I take absolutely no credit for making all this work. I'm just now starting to pick the HA aspect of things apart and try to understand how Fuel is making it all work | 15:13 |
| wasmum | ok, think HA. I have keystone behind lb (f5), all endpoint urls are the vips configured in the LB. I have two "controllers" running keystone - mixed mode ad for ident sql for assignment. I also have two neutron nodes the configs on those and the neutron-server service on the controller use the keystone vips for auth. | 15:13 |
| krobzaur | I also got 2 VIPs for my virtual routers on my public and openstack management network, and the controllers themselves share a vip for their interface on the public and management networks | 15:14 |
| wasmum | when one neutron-server auths, sees the agent list quiers the agents all ok | 15:14 |
| wasmum | second neutron-server node does same thing, list the agents in down state | 15:15 |
| wasmum | keystone is reissueing tokens each request, i suspect | 15:15 |
| wasmum | suggestions....? | 15:15 |
| *** admin0 has joined #openstack-operators | 15:15 | |
| klindgren | We run a clustered rabbitmq setup, with each service configured for list of rabbitmq hosts, seems to work pretty well. Older versions of openstack did not handle rabbitmq behind a vip well at all | 15:16 |
| *** mdorman has joined #openstack-operators | 15:16 | |
| wasmum | we also run rabbitmq cluster with all queues mirrored. then specify the nodes with rabbit_hosts vs. rabbit_host | 15:16 |
| klindgren | wasmum, are you running memcache? | 15:16 |
| wasmum | na, token going into sql | 15:17 |
| krobzaur | klindgren: Gotcha. So the config files for all your openstack services all you to specify a list of rabbit hosts to puul messages from? | 15:17 |
| krobzaur | allow* | 15:17 |
| wasmum | should I switch to memcache and throw that port into a pool behind a vip? | 15:17 |
| wasmum | @krobzaur rabbit_hosts = host:a5672, hostb:5672 | 15:18 |
| *** admin0 has quit IRC | 15:19 | |
| *** admin0 has joined #openstack-operators | 15:19 | |
| *** piet has quit IRC | 15:20 | |
| krobzaur | wasmum: Sweet, thanks. I've done very little configuring by hand so my understanding of the config files for each service and the multitude options they contain is also pretty limited. I guess my lack of understanding stands as a testament to Fuel as a deployment method | 15:20 |
| klindgren | so we run bascially the same setup as you wasmum tokens going to db - never had an issue that you were having | 15:20 |
| krobzaur | Just curious, when you guys deploy openstack how do you go about things? | 15:20 |
| klindgren | a while back neutron added support for memcache and we turned that on to address some issues we were seeing | 15:21 |
| wasmum | klindgren: thanks I have something configed incorrectly on a node then. I'll comb through this second controller | 15:21 |
| *** fwdit has quit IRC | 15:21 | |
| krobzaur | My boss put me on this and told me he wanted to use Mirantis/Fuel so thats what I did but using it seriously limits your ability to adapt/customize your deployment for different use cases | 15:21 |
| klindgren | wasmum, but we do have memcache for nova/neutron/heatwahtever else will use it. But keystone tokens still go to dob. | 15:22 |
| wasmum | krobzaur: first openstack install by hand, taught me a lot about openstack. Then we sent it to our puppet team to puppetize | 15:22 |
| klindgren | db* | 15:22 |
| klindgren | We started with packstack then quickly moved to a non-packstack config (within a week). The puppetized our configis that way | 15:23 |
| klindgren | We run masterless puppet and use ansible to performance actions (such as kicking off a puppet deployment on compute nodes) | 15:23 |
| krobzaur | wasmum: That makes sense. I know Fuel uses puppet behind the scenes to do all its configuration and installation. I've had to poke through some puppet manifests a few times to troubleshoot some deployment errors. I've yet to make my journey through the manual installation process though | 15:24 |
| krobzaur | klindgren: Gotcha. How do you guys handle deploying the OS to hardware. What are you running for a base OS? Ubuntu? CentOS? | 15:25 |
| haukebruno | 1 year ago we hit a limit in token generation vs. galera commit latency where we used to switch to memcache as token backend. worked super well in icehouse, but I don't have any idea how the newer releases look like | 15:25 |
| haukebruno | for now we are fine running galera as token backend for our kilo platforms + looking forward to use fernet tokens after upgrading to liberty | 15:26 |
| haukebruno | just my 2 cents ;) | 15:26 |
| krobzaur | klindgren: I know Fuel use Cobbler for bare-metal booting | 15:26 |
| *** harshs has quit IRC | 15:29 | |
| klindgren | We have an internal system for on boarding servers into the DC. But we are currently working on doing up an ironic install. We Use CentOS. | 15:29 |
| *** armax has joined #openstack-operators | 15:29 | |
| krobzaur | Gotcha. So what are you guys using your openstack deployments for? I'm just an intern right now doing some research/testing for the company I work for, but I studied physics in school and I am hugely interested in how Openstack and be leveraged for HPC use cases | 15:33 |
| *** chlong has quit IRC | 15:33 | |
| krobzaur | I've done some work using heat to deploy TORQUE clusters "on-demand", I think it would be really sweet to provide pre-packaged infrastructure for scientists to use in a "one-click" fashion | 15:34 |
| haukebruno | krobzaur, just virtual private or private cloud here. | 15:35 |
| *** eddima has quit IRC | 15:35 | |
| *** eddima has joined #openstack-operators | 15:36 | |
| haukebruno | krobzaur, I am curious what you HPC guys would expect from an openstack cloud. would you just need "big" and new CPUs? | 15:36 |
| krobzaur | It really depends entirely on the application. My physics sims would just need really high clock speed CPU's (the sim is single threaded). Protein folding sims need huge amounts of memory. Astrophysics guys need tons of parallel processing (high core count) | 15:37 |
| krobzaur | Genome sequencing guys need to process enormous amounts of data (object storage would be absolutely perfect for them) | 15:37 |
| haukebruno | krobzaur, thanks for pointing out. really interesting | 15:38 |
| *** admin0 has quit IRC | 15:39 | |
| krobzaur | So I'm envisioning building tons of heat "stacks" that provide the virtual infrastructure for all these use cases and scientists and just use Horizon to deploy a virtual cluster catered to their use case | 15:39 |
| krobzaur | scientists can* | 15:39 |
| krobzaur | The one thing I'm a little bummed about it the apparent lack of GPU virtualization. But for special GPU accelerated apps you could just use Ironic to deploy an environment on a special subset of bare metal servers equipped with the proper GPUs | 15:40 |
| *** fawadkhaliq has quit IRC | 15:40 | |
| *** saneax is now known as saneax_AFK | 15:40 | |
| *** fawadkhaliq has joined #openstack-operators | 15:40 | |
| krobzaur | Anyway I'm kinda rambling at this point because I get excited about these things =). But thanks for the insight guys, I really appreciate the opportunity to talk to guys who are using openstack in production environments. Nobody at my company seems to think openstack works or is worth anything (thats why they put the intern on it I suppose) | 15:42 |
| *** maishsk has quit IRC | 15:43 | |
| wasmum | haukebruno: thanks! | 15:44 |
| wasmum | krobzaur: keep excited about openstack - there are many more na sayers than you think. i find that the na sayers are those that really need a CD to install things and a support agreement to coddle them | 15:45 |
| *** fawadkhaliq has quit IRC | 15:45 | |
| haukebruno | lol | 15:45 |
| haukebruno | fullack. | 15:45 |
| haukebruno | of course, running an openstack platform needs a lot of learning but imho it is still worth it | 15:46 |
| wasmum | hell's yes. that knowledge translates into a ton of areas as well. | 15:46 |
| wasmum | to forsake that is just fraking dumb | 15:46 |
| wasmum | *freaking | 15:46 |
| wasmum | I guess you can tell where I am at:) | 15:47 |
| klindgren | GPU virt is more of a function of the chosen HV than openstack. But I thought you could enable pass through and have openstack support that. I also though I saw some purposals to make that more configurable inside openstack | 15:48 |
| haukebruno | wasmum, fully agree. it is also the community thing that is pretty awesome for all the different projects and stuff. I never felt that comfortable or familiar inside a community /o/ | 15:51 |
| *** gyee has joined #openstack-operators | 15:51 | |
| *** maishsk has joined #openstack-operators | 15:53 | |
| wasmum | haukebruno: agree 100% | 15:54 |
| *** eddima1 has joined #openstack-operators | 15:55 | |
| *** kstev has quit IRC | 15:57 | |
| *** eddima has quit IRC | 15:57 | |
| *** eddima1 is now known as eddima | 15:57 | |
| *** kstev has joined #openstack-operators | 16:00 | |
| *** kstev1 has joined #openstack-operators | 16:01 | |
| *** maishsk has quit IRC | 16:03 | |
| *** kstev has quit IRC | 16:04 | |
| *** armax has quit IRC | 16:16 | |
| *** kazuIchikawa has joined #openstack-operators | 16:20 | |
| *** permalac has joined #openstack-operators | 16:22 | |
| *** permalac has quit IRC | 16:22 | |
| *** permalac has joined #openstack-operators | 16:23 | |
| *** permalac has quit IRC | 16:23 | |
| *** markvoelker has joined #openstack-operators | 16:25 | |
| *** permalac_ has joined #openstack-operators | 16:31 | |
| *** permalac_ has quit IRC | 16:31 | |
| *** Marga_ has joined #openstack-operators | 16:35 | |
| *** rcernin has quit IRC | 16:35 | |
| *** eil397 has joined #openstack-operators | 16:38 | |
| *** SimonChung has joined #openstack-operators | 16:39 | |
| *** eddima has quit IRC | 16:41 | |
| *** Guest71467 has quit IRC | 16:42 | |
| *** harshs has joined #openstack-operators | 16:43 | |
| *** piet has joined #openstack-operators | 16:43 | |
| *** apoorvad has joined #openstack-operators | 16:51 | |
| *** fawadkhaliq has joined #openstack-operators | 17:00 | |
| *** SimonChung1 has joined #openstack-operators | 17:01 | |
| *** SimonChung has quit IRC | 17:01 | |
| *** fawadkhaliq has quit IRC | 17:10 | |
| *** admin0 has joined #openstack-operators | 17:14 | |
| *** openstackgerrit has quit IRC | 17:14 | |
| *** openstackgerrit_ is now known as openstackgerrit | 17:14 | |
| *** openstackgerrit has quit IRC | 17:15 | |
| *** openstackgerrit_ has joined #openstack-operators | 17:15 | |
| *** openstackgerrit_ is now known as openstackgerrit | 17:16 | |
| *** openstackgerrit_ has joined #openstack-operators | 17:16 | |
| *** fawadkhaliq has joined #openstack-operators | 17:18 | |
| *** pcaruana has quit IRC | 17:24 | |
| *** fawadkhaliq has quit IRC | 17:25 | |
| *** fawadkhaliq has joined #openstack-operators | 17:25 | |
| *** ctrath has quit IRC | 17:28 | |
| *** matrohon has quit IRC | 17:31 | |
| *** ctrath has joined #openstack-operators | 17:34 | |
| *** SimonChung1 has quit IRC | 17:36 | |
| *** SimonChung has joined #openstack-operators | 17:36 | |
| *** pcaruana has joined #openstack-operators | 17:36 | |
| *** simon-AS5591 has quit IRC | 17:37 | |
| *** lhcheng has joined #openstack-operators | 17:39 | |
| krobzaur | Yeah. I will lend them some credit and say that openstack does require a fair bit of expertise and a broad skill set to deploy, manage, and monitor effectively. But the platform itself is absolutely mind-blowing in its current capability and that doesn't even account for the untapped potential of the newer projects out there | 17:41 |
| krobzaur | A lot of the guys I talk to that are anti-openstack are pretty hardcore VMware guys (also a great product), which seems a bit easier to manage from an ops perspective | 17:42 |
| *** mriedem1 has joined #openstack-operators | 17:43 | |
| *** mriedem has quit IRC | 17:44 | |
| krobzaur | klindgren: I'll definitely have to look into that. I didn't find anything that really stood out to me during some casual research but perhaps I'll have to dig a little deeper | 17:45 |
| *** mriedem1 has quit IRC | 17:46 | |
| *** ducttape_ has quit IRC | 17:49 | |
| *** ducttape_ has joined #openstack-operators | 17:49 | |
| *** derekh has quit IRC | 17:49 | |
| *** mriedem has joined #openstack-operators | 17:53 | |
| haukebruno | krobzaur, lol. I also saw that vmware guys | 17:53 |
| haukebruno | luckily my vmware guy is highly interested in openstack too + play around with VIO and stuff | 17:53 |
| *** vsedelnik has quit IRC | 17:54 | |
| *** kazuIchikawa has quit IRC | 17:57 | |
| *** ducttape_ has quit IRC | 18:02 | |
| krobzaur | I wish the VMware pro at my company wasn't so anti openstack because I think we could play around with a lot of really cool integrations between the two cloud platforms | 18:02 |
| krobzaur | I know its possible to use esxi as the hypervisor in openstack, that might be interesting to try | 18:03 |
| krobzaur | VMware also has some excellent automation tools that handle hybrid cloud environments | 18:03 |
| *** dims_ has joined #openstack-operators | 18:03 | |
| *** dims has quit IRC | 18:03 | |
| wasmum | klindgren: caching tokens - how long. Seems like one neutron node will auth for a call, the second neutron node will auth for a call - new tokens for each, then the following calls fail. Say `openstack server create` one portion of that network call works with original token, but continuing on down the chain, the next network api call gets auth with the other ks server in the lb pool = new token. I have a | 18:06 |
| wasmum | suspicion that these back and forths are breaking it. | 18:06 |
| klindgren | iirc we cache for 12 hours | 18:06 |
| haukebruno | 1 hour here | 18:07 |
| *** bsilverman has joined #openstack-operators | 18:08 | |
| klindgren | so in my xp trying to trace timeouts and the like, neutron should use whatever token was passed into it for all further calls related to the original request. Unless you are talking about nova -> neutron interactions for call backs and stuff? | 18:08 |
| klindgren | also new tokens should not invalidate old tokens | 18:09 |
| mgagne | using a common memcache server for keystonemiddleware? | 18:13 |
| *** flwang has quit IRC | 18:16 | |
| klindgren | IIRC we aren't specifically configuring anything in the paste files. I believe it will pull from [DEFAULT}/memcache_servers | 18:16 |
| mgagne | klindgren: [keystone_authtoken]/memcached_servers does the job | 18:17 |
| klindgren | then nopes | 18:18 |
| mgagne | and works fine for us. no more request to KS for API calls proxied from Nova to Neutron. | 18:18 |
| klindgren | and I assume glance as well? | 18:18 |
| klindgren | that will reduce load on keystone a bit | 18:18 |
| mgagne | yes, glance too | 18:18 |
| mgagne | in fact, we are looking into disabling catalog retrieval across the board. it's overloading our firewall | 18:19 |
| mgagne | [keystone_authtoken]/include_service_catalog = False | 18:19 |
| mgagne | we added this config to Ceilometer and it dropped ~20% of trafic going to keystone across all our regions. | 18:20 |
| mgagne | we have system constantly poking Ceilometer and this single config helped a lot | 18:20 |
| *** haukebruno has quit IRC | 18:21 | |
| wasmum | mgagne: using mysql backend (galera cluster) | 18:22 |
| *** ducttape_ has joined #openstack-operators | 18:22 | |
| mgagne | wasmum: not yet, will be 2nd part of our fernet token migration | 18:22 |
| mgagne | wasmum: time being the lacking resource atm =) | 18:23 |
| wasmum | understand | 18:23 |
| mgagne | but yes, we are looking into it | 18:23 |
| mgagne | I'm sure it will greatly reduce our trafic across regions | 18:23 |
| mgagne | plan is to have galera nodes for keystone in each region | 18:24 |
| wasmum | I like the disabled catalog, good find | 18:24 |
| wasmum | thats a huge increase in performance | 18:24 |
| mgagne | yea, really I don't know why it's not disabled by default | 18:24 |
| mgagne | I don't worry much about disabling it for Ceilometer, Neutron and Glance since AFAIK they don't contact other services | 18:25 |
| mgagne | not sure the impact if I disable it for Nova, Heat or Cinder. | 18:25 |
| mgagne | because service could rely on catalog being available in auth context for some operations | 18:26 |
| wasmum | be nice to see that flow chart somewhere on the interwez | 18:27 |
| klindgren | mgagne, under your cells config, do you run a large memcache setup for all the cells or do you run a memcache setup per cell? | 18:27 |
| *** piet has quit IRC | 18:28 | |
| mgagne | klindgren: one memcache per region | 18:28 |
| mgagne | klindgren: but problem is: if memcache is down, API doesn't work anymore | 18:28 |
| mgagne | klindgren: cache code looks to not like unreachable memcache | 18:28 |
| klindgren | we are api cell (also handles glance/nova-api/neutron/keystone) which has a memcache and nova-cell also has its own memcache | 18:28 |
| *** piet has joined #openstack-operators | 18:28 | |
| klindgren | re: authtokens or in general? | 18:28 |
| mgagne | authtokens | 18:29 |
| *** dims_ has quit IRC | 18:29 | |
| mgagne | we don't have memcache for the rest, only authtoken | 18:29 |
| *** flwang has joined #openstack-operators | 18:29 | |
| klindgren | unsing memache to handle metadata lookups under nova-metadata helped reduce conductor workload a bit | 18:29 |
| *** SimonChung has quit IRC | 18:31 | |
| *** SimonChung1 has joined #openstack-operators | 18:31 | |
| klindgren | since metadata lookups from services (puppet) are typically ~8 calls in a few seconds. It means that we fetch metadata once fro mthe db/conductor vs's 8 times | 18:31 |
| *** eil397 has left #openstack-operators | 18:34 | |
| *** simon-AS559 has joined #openstack-operators | 18:37 | |
| *** admin0 has quit IRC | 18:39 | |
| *** simon-AS5591 has joined #openstack-operators | 18:41 | |
| *** simon-AS559 has quit IRC | 18:44 | |
| *** electrofelix has quit IRC | 18:53 | |
| *** admin0 has joined #openstack-operators | 18:56 | |
| mgagne | klindgren: we don't have metadata service yet =) | 19:04 |
| *** rarcea has joined #openstack-operators | 19:05 | |
| *** dminer has quit IRC | 19:09 | |
| *** dims has joined #openstack-operators | 19:12 | |
| *** ctrath has quit IRC | 19:13 | |
| *** krotscheck is now known as kro_death_by_sin | 19:14 | |
| *** kro_death_by_sin is now known as kro_sinus_hell | 19:14 | |
| *** ctrath has joined #openstack-operators | 19:15 | |
| *** rarcea has quit IRC | 19:17 | |
| *** dims has quit IRC | 19:18 | |
| klindgren | so under the api services you also disabled the metadata api? or was that more of a you dont run a seperate nova-metadata service? | 19:21 |
| *** rarcea has joined #openstack-operators | 19:23 | |
| *** dims has joined #openstack-operators | 19:26 | |
| *** vsedelnik has joined #openstack-operators | 19:27 | |
| *** harshs has quit IRC | 19:32 | |
| *** fawadkhaliq has quit IRC | 19:35 | |
| *** ctina has joined #openstack-operators | 19:36 | |
| *** busterswt has joined #openstack-operators | 19:42 | |
| *** dims has quit IRC | 19:45 | |
| *** vsedelnik has quit IRC | 19:46 | |
| *** vsedelnik has joined #openstack-operators | 19:50 | |
| *** amitry has quit IRC | 19:53 | |
| *** amitry has joined #openstack-operators | 19:54 | |
| *** dminer has joined #openstack-operators | 19:54 | |
| *** rockyg has joined #openstack-operators | 19:58 | |
| *** kazuIchikawa has joined #openstack-operators | 19:58 | |
| *** kazuIchikawa has quit IRC | 20:03 | |
| *** ig0r_ has joined #openstack-operators | 20:04 | |
| *** delattec has quit IRC | 20:05 | |
| *** ig0r__ has quit IRC | 20:06 | |
| *** dims has joined #openstack-operators | 20:11 | |
| *** rarcea has quit IRC | 20:12 | |
| *** esker has joined #openstack-operators | 20:14 | |
| *** rarcea has joined #openstack-operators | 20:16 | |
| *** Marga_ has quit IRC | 20:19 | |
| *** ctina has quit IRC | 20:20 | |
| *** VW has quit IRC | 20:22 | |
| *** matrohon has joined #openstack-operators | 20:23 | |
| *** piet has quit IRC | 20:30 | |
| *** piet has joined #openstack-operators | 20:30 | |
| *** regXboi has quit IRC | 20:34 | |
| *** cdelatte has joined #openstack-operators | 20:35 | |
| *** maishsk has joined #openstack-operators | 20:36 | |
| *** rockyg has quit IRC | 20:37 | |
| *** esker has quit IRC | 20:38 | |
| *** rockyg has joined #openstack-operators | 20:38 | |
| *** esker has joined #openstack-operators | 20:38 | |
| *** fawadkhaliq has joined #openstack-operators | 20:42 | |
| *** VW has joined #openstack-operators | 20:43 | |
| *** VW has quit IRC | 20:48 | |
| *** VW has joined #openstack-operators | 20:50 | |
| *** harshs has joined #openstack-operators | 20:55 | |
| *** f1ller is now known as filler | 21:03 | |
| *** maishsk has quit IRC | 21:15 | |
| *** maishsk_ has joined #openstack-operators | 21:15 | |
| *** Guest93664 is now known as med_ | 21:17 | |
| *** med_ has joined #openstack-operators | 21:17 | |
| *** esker has quit IRC | 21:20 | |
| *** esker has joined #openstack-operators | 21:20 | |
| *** vsedelnik has quit IRC | 21:25 | |
| *** esker has quit IRC | 21:26 | |
| *** busterswt has quit IRC | 21:30 | |
| *** kstev has joined #openstack-operators | 21:31 | |
| *** kstev1 has quit IRC | 21:32 | |
| *** pcaruana has quit IRC | 21:38 | |
| *** cdelatte has quit IRC | 21:44 | |
| *** maishsk_ has quit IRC | 21:45 | |
| *** cdelatte has joined #openstack-operators | 21:45 | |
| *** matrohon has quit IRC | 21:47 | |
| *** VW has quit IRC | 21:48 | |
| *** fawadkhaliq has quit IRC | 21:51 | |
| *** fawadkhaliq has joined #openstack-operators | 21:51 | |
| *** pcaruana has joined #openstack-operators | 21:52 | |
| *** krobzaur has quit IRC | 21:53 | |
| *** mriedem has quit IRC | 21:55 | |
| *** Marga_ has joined #openstack-operators | 21:56 | |
| *** ctrath has quit IRC | 21:57 | |
| *** kstev has quit IRC | 22:00 | |
| *** kazuIchikawa has joined #openstack-operators | 22:00 | |
| *** fawadkhaliq has quit IRC | 22:00 | |
| *** fawadkhaliq has joined #openstack-operators | 22:01 | |
| *** harlowja has joined #openstack-operators | 22:01 | |
| *** kstev has joined #openstack-operators | 22:03 | |
| *** kazuIchikawa has quit IRC | 22:05 | |
| *** rockyg has quit IRC | 22:10 | |
| *** VW has joined #openstack-operators | 22:11 | |
| *** dims_ has joined #openstack-operators | 22:11 | |
| *** david-lyle has quit IRC | 22:13 | |
| *** dims has quit IRC | 22:14 | |
| *** VW has quit IRC | 22:15 | |
| *** piet has quit IRC | 22:16 | |
| *** dims has joined #openstack-operators | 22:16 | |
| *** dims_ has quit IRC | 22:17 | |
| *** ducttape_ has quit IRC | 22:20 | |
| *** vsedelnik has joined #openstack-operators | 22:26 | |
| *** piet has joined #openstack-operators | 22:29 | |
| *** vsedelnik has quit IRC | 22:30 | |
| *** shakamunyi has joined #openstack-operators | 22:32 | |
| *** mdorman has quit IRC | 22:53 | |
| *** VW has joined #openstack-operators | 22:56 | |
| *** saneax_AFK is now known as saneax | 22:56 | |
| *** dminer has quit IRC | 22:56 | |
| *** piet has quit IRC | 22:56 | |
| *** VW has quit IRC | 23:04 | |
| *** dims has quit IRC | 23:06 | |
| *** saneax is now known as saneax_AFK | 23:08 | |
| *** VW has joined #openstack-operators | 23:09 | |
| *** admin0 has quit IRC | 23:09 | |
| *** rarcea has quit IRC | 23:25 | |
| *** vsedelnik has joined #openstack-operators | 23:27 | |
| *** chlong has joined #openstack-operators | 23:28 | |
| *** VW has quit IRC | 23:32 | |
| *** vsedelnik has quit IRC | 23:33 | |
| *** unicell has joined #openstack-operators | 23:46 | |
| *** harlowja has quit IRC | 23:47 | |
| *** kstev has quit IRC | 23:47 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!