| *** dims has joined #openstack-ha | 00:07 | |
| *** hoonetorg has quit IRC | 00:29 | |
| *** hoonetorg has joined #openstack-ha | 00:41 | |
| *** hoangcx has joined #openstack-ha | 00:47 | |
| *** rossella_s has quit IRC | 01:03 | |
| *** rossella_s has joined #openstack-ha | 01:03 | |
| *** masahito has joined #openstack-ha | 01:13 | |
| *** dims has quit IRC | 01:21 | |
| *** poki has quit IRC | 02:16 | |
| *** poki has joined #openstack-ha | 02:16 | |
| *** hoangcx has quit IRC | 02:28 | |
| *** hoangcx has joined #openstack-ha | 02:34 | |
| *** haukebruno has joined #openstack-ha | 02:44 | |
| *** markvoelker has joined #openstack-ha | 02:55 | |
| *** markvoelker has quit IRC | 03:01 | |
| *** dileepr has quit IRC | 04:09 | |
| *** markvoelker has joined #openstack-ha | 04:47 | |
| *** markvoelker has quit IRC | 04:51 | |
| *** rossella_s has quit IRC | 05:03 | |
| *** smoriya_ has quit IRC | 05:03 | |
| *** rossella_s has joined #openstack-ha | 05:03 | |
| *** haukebruno has quit IRC | 05:37 | |
| *** masahito has quit IRC | 05:48 | |
| *** haukebruno has joined #openstack-ha | 06:03 | |
| *** Deng has joined #openstack-ha | 06:10 | |
| *** pcaruana has quit IRC | 06:18 | |
| *** nkrinner has joined #openstack-ha | 06:19 | |
| *** markvoelker has joined #openstack-ha | 06:48 | |
| *** markvoelker has quit IRC | 06:52 | |
| ddeja | good news, Mistral in Mitaka realase have a bugfix that make it more HA https://bugs.launchpad.net/mistral/+bug/1502120 | 07:53 |
|---|---|---|
| openstack | Launchpad bug 1502120 in Mistral mitaka "Task stalls in state RUNNING in case of service dies" [High,Fix released] - Assigned to Nikolay Makhotkin (nmakhotkin) | 07:53 |
| *** pcaruana has joined #openstack-ha | 07:57 | |
| aspiers | \o/ | 08:39 |
| haukebruno | morning aspiers \o/ | 08:47 |
| aspiers | morning :) | 08:48 |
| haukebruno | just from reading the guide + 1 regarded question on ask.openstack.org -> is the collapsed setting a common one for production? maybe I don't understand it right, but it sounds like all services (haproxy, nova-*, also galera) are on 1 node | 09:01 |
| haukebruno | ah, and also another question :p the guide point out that someone should disable writes to all galera nodes except one. we were also facing thos multi-master write issues but I switched to different ports pointing to different galera nodes so that are nova controllers could write to 1 node, all keystones to another and so on. could be an improvement (worked for us quiet well) | 09:03 |
| *** rossella_s has quit IRC | 09:03 | |
| *** rossella_s has joined #openstack-ha | 09:03 | |
| ddeja | haukebruno: for the first question on should setup at least 3 controllers with galera, nova-* services etc. when you want to have HA | 09:11 |
| haukebruno | ddeja, yes of course, but I just wondering why to put haproxy and also galera nodes on the same node where the openstack services live | 09:12 |
| haukebruno | is it just for figuring stuff out or a real production scenario? | 09:13 |
| aspiers | haukebruno: IMHO you can have a small production cloud with a 2-node cluster running everything | 09:13 |
| ddeja | haukebruno: oh, ok I get it now | 09:13 |
| aspiers | we support that in SUSE OpenStack Cloud | 09:13 |
| haukebruno | aspiers, ok. support wins here :) just wondering, because from my experiences someone could have lot of trouble with even a galera cluster or rabbitmqs eating up inodes or RAM. so I ever seperate them | 09:15 |
| haukebruno | but yeah, it depends on the use-case I guess. | 09:16 |
| aspiers | well, of course the nodes have to be sized correctly :) | 09:17 |
| aspiers | btw we use postgresql in SOC | 09:17 |
| ddeja | haukebruno: I think it depends on number of servers that you have. I have small production-like setup and if I would want to separate galera cluster from Openstack controll cluster I will end up with 1 or 2 compute nodes ;) | 09:17 |
| aspiers | right :) | 09:18 |
| haukebruno | ddeja, I see | 09:18 |
| haukebruno | aspiers, why postgres over mariadb? | 09:18 |
| aspiers | because of SUSE's relationship with postgres I think | 09:18 |
| haukebruno | I also took a look on postgres some time ago because one customer asked about it. but the documentary especially for all that cluster stuff is hard to find | 09:19 |
| aspiers | we have relationships with both companies of course, but IIRC we are closer with postgres business | 09:19 |
| aspiers | ICBW though | 09:19 |
| haukebruno | makes sense | 09:19 |
| haukebruno | for a smaller env it also make sense to put the other galera nodes to backup. I think I got it now, not every folk has the ability to spin up big and spreaded clusters for the control plane | 09:22 |
| aspiers | with SUSE OpenStack Cloud, we let the operator choose the exact number of clusters they want, and the size of each | 09:23 |
| aspiers | and which services go where | 09:23 |
| aspiers | it's totally flexible | 09:23 |
| aspiers | via drag and drop :) | 09:24 |
| haukebruno | I even split the balancers for galera and rest. fear of DB connection resets is huge :p | 09:24 |
| haukebruno | aspiers, that drag and drop thing sounds a lot better than the "apt-get install foo" stuff I am currently doing /o\ | 09:24 |
| aspiers | ugh yes | 09:24 |
| aspiers | we can build HA clouds from bare metal in ~30 minutes | 09:25 |
| aspiers | for the next release it will be more like 15 minutes :) | 09:25 |
| haukebruno | +1! | 09:25 |
| aspiers | haukebruno: interesting, can you expand on that? I am currently looking at DB connection resets | 09:25 |
| haukebruno | aspiers, because we still want the flexibility to change the config for all openstack services, for example after adding more keystone nodes or adding new services. from what I see even a haproxy reload will cause some connection resets for the mariadb/galera stuff | 09:27 |
| haukebruno | thats the only reason why I seperated them. (also we use a lot of smaller timeout values on the openstack balancers than on the DB ones, but that isn't the big deal) | 09:27 |
| aspiers | adding more nodes / services in SOC is also a few clicks of the mouse :) | 09:28 |
| * haukebruno cries | 09:28 | |
| aspiers | hehe sorry | 09:28 |
| aspiers | I'll stop now | 09:28 |
| haukebruno | just curious, what about the new stuff or maybe the smaller projects like tap as a service? does SOC also supports that or the other way around: is it possible for an operator to manually add openstack projects to a SOC that aren't in SOC itself? | 09:29 |
| aspiers | it's possible | 09:30 |
| aspiers | also, Crowbar is extensible | 09:30 |
| aspiers | and we gladly accept pull requests for new plugins :) | 09:30 |
| haukebruno | seems that it all comes down to coding /o\ | 09:31 |
| aspiers | well you can deploy manually | 09:31 |
| aspiers | but as Florian points out, that's a really bad idea | 09:31 |
| ddeja | haukebruno: you are deploying openstack HA cloud manually? | 09:31 |
| ddeja | I'm not sure if I understand correctly | 09:32 |
| haukebruno | ddeja, yep | 09:33 |
| haukebruno | we started investigating and exploring openstack and business things came quicker than we could handle the automation stuff | 09:34 |
| *** smoriya_ has joined #openstack-ha | 09:34 | |
| ddeja | I see | 09:35 |
| aspiers | haukebruno: you could have deployed in one or two days with SOC ... | 09:36 |
| haukebruno | /o\ | 09:36 |
| aspiers | (or even <1 hour with someone who has experience of the product) | 09:36 |
| ddeja | and to have some choice, Fuel probably would also do the work in 1-2 days | 09:36 |
| ddeja | :) | 09:37 |
| haukebruno | well in 2 days with enough coffee and beer I could do it also :p | 09:37 |
| ddeja | haukebruno: Been there, done this :) | 09:38 |
| haukebruno | imho openstack distros have a lot of advantages but also some disadvantages. I am looking forward to consider things out (tm) | 09:38 |
| haukebruno | currently without too much pressure on myself I would say deploying our standard platform needs 4 or 5 days. (at least about 30 nodes for the control plane stuff) | 09:39 |
| haukebruno | but time isn't the big deal for me/us. human errors are the most annoying stuff in here | 09:39 |
| ddeja | When I was very inexperienced, configuring pacemaker was really painfull for me and distros do it automatically | 09:39 |
| haukebruno | the chances aren't that small to miss something while doing everything from scratch | 09:40 |
| * ddeja spend once 2 days looking for typo in confs... | 09:40 | |
| haukebruno | ddeja, yes I can understand that. I used pacemaker/corosync/haproxy a lot before I started to play with openstack, that was definitively a plus | 09:41 |
| aspiers | yes the main point about automation being important is to avoid human error, typos etc. | 09:41 |
| ddeja | haukebruno: oh, so you have much better start | 09:42 |
| aspiers | the fact that it's a lot quicker is just a bonus | 09:42 |
| haukebruno | ack | 09:42 |
| *** dims has joined #openstack-ha | 09:43 | |
| haukebruno | but anyway, things becoming bigger in the company. the next platform will have about 700 compute nodes in the beginning, means I need regions or cells to seperate everthing and doing _that_ from scratch would be hard pain | 09:43 |
| haukebruno | a collegue is now looking into the ansible project | 09:43 |
| aspiers | haukebruno: please consider SOC too :) | 09:50 |
| haukebruno | aspiers, do you support running SOC on ubuntu? :-D it is hard for me (personally) to work on non-apt distros :p | 09:52 |
| haukebruno | but that wouldn't be a showstopper at all, because it is just me. other collegues are fine with suse, red hat also | 09:53 |
| aspiers | haukebruno: why isit hard for you? | 09:54 |
| aspiers | maybe we can help make it easier | 09:54 |
| ddeja | haukebruno: if you are stick to ubuntu, you can try juju (I have very small expertise with it) or Fuel (it also works with Centos; I can't tell if it handles 700 servers setup) | 09:54 |
| aspiers | I use Ubuntu too, and personally I find zypper much nicer UI | 09:54 |
| haukebruno | aspiers, I never used other distros than debian or ubuntu (for my work stuff). it is always hard to break that old relationships :p | 09:55 |
| aspiers | haukebruno: nah, it's easy :) be brave ;-) | 09:55 |
| haukebruno | hrhr | 09:55 |
| ddeja | at this point I have to agree with aspiers - it's always good to have some new experience | 09:56 |
| haukebruno | ddeja, the scalability is a must-have. 700 is just the first order we did. more are coming for sure | 09:56 |
| aspiers | ddeja: exactly, learning multiple distros really helps improve your Linux knowledge | 09:56 |
| ddeja | haukebruno: I understand. I just want to give you some hitns what to look at but I can't recommend it's fully since I never run it in such scale | 09:57 |
| haukebruno | sorry for asking, but since I ever used any openstack distro: are there any downsides? regarding root access to the maschines for debugging, logging et ceterea? | 09:57 |
| aspiers | haukebruno: no, you get full access | 10:00 |
| haukebruno | aspiers, does config changes need to perform from the gui or is it also possible to do it directly on the nodes? | 10:02 |
| aspiers | haukebruno: strongly recommend https://youtu.be/LM1ANSge01g if you didn't watch the whole thing | 10:02 |
| haukebruno | I really need to check out a demo or something | 10:03 |
| aspiers | haukebruno: Crowbar also has CLI and REST API | 10:03 |
| aspiers | haukebruno: http://crowbar.github.io/ | 10:03 |
| haukebruno | ah. that talk is on my notebook, but didn't see it so far :p | 10:03 |
| haukebruno | is crowbar the deployment part of SOC? | 10:04 |
| aspiers | yes | 10:04 |
| haukebruno | ah cool | 10:04 |
| aspiers | that website is very very new though | 10:04 |
| haukebruno | I will see the talk, I guess the most of my questions could be answered then | 10:05 |
| aspiers | we are still working on making it more helpful / welcoming | 10:05 |
| aspiers | there are also two videos on that website | 10:05 |
| haukebruno | let me check that out, I will come back for questions for sure ;) | 10:06 |
| aspiers | cool | 10:06 |
| *** hoangcx has quit IRC | 10:13 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ha-guide: Updated from openstack-manuals https://review.openstack.org/294524 | 11:22 |
| openstackgerrit | Merged openstack/ha-guide: Updated from openstack-manuals https://review.openstack.org/294524 | 11:35 |
| *** markvoelker has joined #openstack-ha | 12:32 | |
| *** openstackgerrit has quit IRC | 12:48 | |
| *** openstackgerrit has joined #openstack-ha | 12:49 | |
| *** Deng has quit IRC | 12:53 | |
| *** cleong has joined #openstack-ha | 13:35 | |
| *** smoriya_ has quit IRC | 13:35 | |
| haukebruno | aspiers, firstly -> crowbar looks amazing! but now I have a lot of questions! | 13:39 |
| aspiers | :) | 13:39 |
| haukebruno | well, two: a) it works just with suse distros right? and b) what about config options in nova.conf for instance? could someone push it via crowbar into the controller nodes? | 13:39 |
| aspiers | a) currently yes, it used to work with Ubuntu and RHEL but noone was maintaining support so we removed that code. however we would love to add it back | 13:40 |
| aspiers | in a working fashion | 13:40 |
| aspiers | a lot as changed since then | 13:40 |
| *** kgaillot has joined #openstack-ha | 13:43 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:12 | |
| *** nkrinner has quit IRC | 14:12 | |
| *** pcaruana has quit IRC | 14:38 | |
| *** cleong has quit IRC | 15:01 | |
| *** aspiers has quit IRC | 15:59 | |
| *** aspiers has joined #openstack-ha | 16:00 | |
| *** dims_ has joined #openstack-ha | 16:33 | |
| *** dims has quit IRC | 16:34 | |
| *** pcaruana has joined #openstack-ha | 16:47 | |
| *** haukebruno has quit IRC | 16:53 | |
| *** dileepr has joined #openstack-ha | 17:08 | |
| *** dgurtner has joined #openstack-ha | 18:25 | |
| *** dgurtner has quit IRC | 19:32 | |
| *** rossella_s has quit IRC | 21:03 | |
| *** rossella_s has joined #openstack-ha | 21:03 | |
| *** dgurtner has joined #openstack-ha | 21:14 | |
| *** xenolog has quit IRC | 21:25 | |
| *** dgurtner has quit IRC | 21:25 | |
| *** furlongm_ has joined #openstack-ha | 21:25 | |
| *** xenolog has joined #openstack-ha | 21:25 | |
| *** dgurtner has joined #openstack-ha | 21:25 | |
| *** dgurtner has joined #openstack-ha | 21:25 | |
| *** furlongm has quit IRC | 21:26 | |
| *** vuntz has quit IRC | 21:26 | |
| *** vuntz has joined #openstack-ha | 21:26 | |
| *** vuntz has quit IRC | 21:26 | |
| *** vuntz has joined #openstack-ha | 21:26 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:00 | |
| *** dileepr has quit IRC | 22:19 | |
| *** dileepr has joined #openstack-ha | 23:08 | |
| *** kgaillot has quit IRC | 23:09 | |
| *** markvoelker has quit IRC | 23:44 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!