| *** zhurong has quit IRC | 00:01 | |
| *** david-lyle has joined #openstack-kolla | 00:02 | |
| *** jascott1_tmp has joined #openstack-kolla | 00:02 | |
| *** mewald has quit IRC | 00:03 | |
| openstackgerrit | Prithiv proposed openstack/kolla: Support Networking-SFC Container https://review.openstack.org/344930 | 00:15 |
|---|---|---|
| v1k0d3n | is anyone having issues with docker v1.11? | 00:16 |
| v1k0d3n | prechecks are failing with docker v1.11.2 / API 1.23 | 00:17 |
| *** david-lyle has quit IRC | 00:18 | |
| v1k0d3n | ah, i see a bug already out there for it :( crap. https://bugs.launchpad.net/kolla/+bug/1602712 | 00:19 |
| openstack | Launchpad bug 1602712 in kolla "Docker Version Check fails in error" [Undecided,New] | 00:19 |
| *** jascott1_tmp has quit IRC | 00:31 | |
| *** diogogmt has quit IRC | 00:33 | |
| *** banix has joined #openstack-kolla | 00:36 | |
| *** mbound has joined #openstack-kolla | 00:38 | |
| *** mbound has quit IRC | 00:43 | |
| *** Liuqing has joined #openstack-kolla | 00:45 | |
| *** rhallisey has quit IRC | 00:54 | |
| *** zhurong has joined #openstack-kolla | 01:11 | |
| *** banix has quit IRC | 01:11 | |
| *** Jeffrey4l has joined #openstack-kolla | 01:16 | |
| *** inc0 has quit IRC | 01:30 | |
| *** diogogmt has joined #openstack-kolla | 01:32 | |
| *** harlowja has joined #openstack-kolla | 01:35 | |
| *** harlowja has quit IRC | 01:35 | |
| *** harlowja has joined #openstack-kolla | 01:36 | |
| *** daneyon has joined #openstack-kolla | 01:39 | |
| *** csj has joined #openstack-kolla | 01:42 | |
| *** daneyon has quit IRC | 01:43 | |
| *** salv-orl_ has joined #openstack-kolla | 02:02 | |
| *** salv-orl_ has quit IRC | 02:02 | |
| *** salv-orl_ has joined #openstack-kolla | 02:03 | |
| *** salv-orlando has quit IRC | 02:05 | |
| *** salv-orl_ has quit IRC | 02:11 | |
| *** unicell has quit IRC | 02:14 | |
| *** csj has quit IRC | 02:21 | |
| *** david-lyle has joined #openstack-kolla | 02:24 | |
| *** david-lyle has quit IRC | 02:29 | |
| *** signed8bit_Zzz is now known as signed8bit | 02:29 | |
| *** signed8b_ has joined #openstack-kolla | 02:32 | |
| *** signed8bit has quit IRC | 02:32 | |
| *** Jeffrey4l has quit IRC | 02:38 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 02:38 | |
| *** harlowja has quit IRC | 02:46 | |
| *** yuanying has quit IRC | 02:46 | |
| *** signed8b_ has quit IRC | 03:07 | |
| *** salv-orlando has joined #openstack-kolla | 03:09 | |
| *** salv-orlando has quit IRC | 03:19 | |
| *** aernhart has quit IRC | 03:19 | |
| *** inc0 has joined #openstack-kolla | 03:19 | |
| *** inc0 has quit IRC | 03:25 | |
| *** dave-mccowan has quit IRC | 03:27 | |
| *** mark-casey has joined #openstack-kolla | 03:32 | |
| *** yuanying has joined #openstack-kolla | 03:48 | |
| *** yuanying_ has joined #openstack-kolla | 04:03 | |
| *** yuanying has quit IRC | 04:06 | |
| *** diga has joined #openstack-kolla | 04:09 | |
| *** salv-orlando has joined #openstack-kolla | 04:18 | |
| *** bdaca has joined #openstack-kolla | 04:23 | |
| *** salv-orlando has quit IRC | 04:28 | |
| *** diga has quit IRC | 04:29 | |
| *** Liuqing has quit IRC | 04:36 | |
| *** Liuqing has joined #openstack-kolla | 04:36 | |
| *** salv-orlando has joined #openstack-kolla | 04:59 | |
| *** yuanying has joined #openstack-kolla | 05:03 | |
| *** michauds has joined #openstack-kolla | 05:05 | |
| *** yuanying_ has quit IRC | 05:06 | |
| *** kfox1111 has quit IRC | 05:06 | |
| *** michauds has quit IRC | 05:10 | |
| *** kfox1111 has joined #openstack-kolla | 05:15 | |
| *** salman has joined #openstack-kolla | 05:17 | |
| *** unicell has joined #openstack-kolla | 05:24 | |
| openstackgerrit | rahul sharma proposed openstack/kolla: updates default registry tag to latest version https://review.openstack.org/345166 | 05:30 |
| *** unicell has quit IRC | 05:31 | |
| *** unicell has joined #openstack-kolla | 05:31 | |
| *** harlowja has joined #openstack-kolla | 05:39 | |
| *** dmueller-evoila has joined #openstack-kolla | 05:43 | |
| *** mark-casey has quit IRC | 05:58 | |
| *** unicell has quit IRC | 05:59 | |
| *** unicell has joined #openstack-kolla | 06:02 | |
| *** harlowja has quit IRC | 06:04 | |
| *** daneyon has joined #openstack-kolla | 06:09 | |
| *** daneyon has quit IRC | 06:14 | |
| *** bjolo has joined #openstack-kolla | 06:21 | |
| *** bjolo has quit IRC | 06:25 | |
| *** bjolo has joined #openstack-kolla | 06:25 | |
| bjolo | morning | 06:25 |
| *** mewald has joined #openstack-kolla | 06:28 | |
| bdaca | hi | 06:33 |
| *** mewald1 has joined #openstack-kolla | 06:37 | |
| *** mewald has quit IRC | 06:39 | |
| openstackgerrit | Dennis Mueller proposed openstack/kolla: Added neutron-metering-agent https://review.openstack.org/344808 | 06:41 |
| mewald1 | moirning | 06:43 |
| mewald1 | morning | 06:43 |
| *** BernhardW has joined #openstack-kolla | 06:48 | |
| *** BernhardW is now known as bwe_ | 06:48 | |
| *** lulei has joined #openstack-kolla | 06:50 | |
| *** mfalatic has quit IRC | 06:51 | |
| lulei | hi, all. I'm new here. Any suggestion for me to start? Thanks. | 06:52 |
| *** salv-orlando has quit IRC | 06:52 | |
| *** cfarquhar has quit IRC | 06:55 | |
| *** jascott1_tmp has joined #openstack-kolla | 06:56 | |
| *** cfarquhar has joined #openstack-kolla | 06:58 | |
| *** cfarquhar has quit IRC | 06:58 | |
| *** cfarquhar has joined #openstack-kolla | 06:58 | |
| *** salman has quit IRC | 06:59 | |
| *** bootsha has joined #openstack-kolla | 07:00 | |
| mewald1 | What do you guys think about limiting cpu and memory for each container? | 07:02 |
| *** cfarquhar has quit IRC | 07:03 | |
| *** athomas has joined #openstack-kolla | 07:05 | |
| *** Serlex has joined #openstack-kolla | 07:10 | |
| *** Serlex has quit IRC | 07:13 | |
| *** Serlex has joined #openstack-kolla | 07:17 | |
| *** b_bezak has joined #openstack-kolla | 07:18 | |
| *** Serlex has quit IRC | 07:18 | |
| bjolo | hmmm horizon and plugins? | 07:19 |
| bjolo | lulei, what do you want to try? | 07:20 |
| bjolo | for example, containers tab is not showing up. | 07:20 |
| *** cfarquhar has joined #openstack-kolla | 07:20 | |
| *** cfarquhar has quit IRC | 07:20 | |
| *** cfarquhar has joined #openstack-kolla | 07:20 | |
| bjolo | mewald1, when i enable rgw/swift i was kinda expecting that tab to show up in horizon. i guess i have to enable that manually in horizon or? | 07:21 |
| bjolo | mewald1, limiting... anything in particular your are thinking of? | 07:22 |
| mewald1 | I have never tried that in Kolla but typically Horizon discovers the plugins to load by looking at the service endpoints registered in Keystone | 07:22 |
| mewald1 | I would check if everything is set up correctly there first | 07:22 |
| mewald1 | then try using them via CLI, if all of this works but the tab in Horizon is still not showing we should continue looking at horizon | 07:23 |
| mewald1 | any reviewers for https://review.openstack.org/#/c/344705/ ? | 07:27 |
| patchbot | mewald1: patch 344705 - kolla - External Ceph - Implementation Nova | 07:27 |
| bjolo | mewald1, would love to give +1 but i dont know the code base well enough to approve yet. | 07:37 |
| bjolo | mewald1, i can only have opinions on the structure | 07:37 |
| mewald1 | no worries, it was primarily addressing core reviewers :) | 07:38 |
| bjolo | but i do feel globals.conf is getting kinda messy and hard to understand | 07:38 |
| bjolo | to continue the discusssion we had yesterday.... | 07:39 |
| mewald1 | can you go into details? maybe we can work on that without losing the option to enable / disable ceph per individual service | 07:39 |
| bjolo | enable_ceph means today that kolla should deploy ceph right? | 07:39 |
| mewald1 | correct | 07:39 |
| bjolo | i think we should make that a seperate type of playbook. i.e. break that out | 07:40 |
| mewald1 | it already is | 07:40 |
| bjolo | and name it something like kolla_ceph_deploy | 07:40 |
| mewald1 | https://github.com/openstack/kolla/tree/master/ansible/roles/ceph | 07:40 |
| mewald1 | but that makes in inconsistent with all the other enable_* options | 07:40 |
| mewald1 | https://github.com/openstack/kolla/blob/master/etc/kolla/globals.yml#L108 | 07:41 |
| bjolo | yes it is a seperate code tree, but i mean logically | 07:41 |
| bjolo | from an operator and workflow perspective | 07:42 |
| mewald1 | I dont get it | 07:42 |
| bjolo | hold, let me check something | 07:42 |
| mewald1 | The only thing that is inconsistent atm is https://github.com/openstack/kolla/blob/master/etc/kolla/globals.yml#L130 but I'll fix that in a later patch when my Nova patch is through | 07:43 |
| bjolo | kolla ceph deploy options are scattererd out in globals | 07:44 |
| mewald1 | I think the structure is as follows: enable_* parameters are flags for Kolla, "Ceph options" are those options that configure the _deployment_ of Ceph, the "Cinder options" section configures Cinder when _using_ Ceph. It makes sense to me, but if you feel it is messed up propose a patch :) | 07:47 |
| *** Serlex has joined #openstack-kolla | 07:47 | |
| *** salv-orlando has joined #openstack-kolla | 07:53 | |
| bjolo | http://paste.openstack.org/show/538933/ | 07:54 |
| bjolo | kolla means glue | 07:54 |
| bjolo | kolla depoying ceph is an exception i believe for real use cases | 07:55 |
| bjolo | so i would like to see all kolla ceph options grouped together in globals.conf | 07:55 |
| bjolo | but... if we do that. what does enable_ceph mean in the opstack options section? | 07:56 |
| bjolo | openstack options section | 07:57 |
| mewald1 | I dont know man, I am indifferent to this. My suggestion is: Put together a patch and submit it for review and we'll see what people thing | 07:58 |
| mewald1 | -g +k | 07:58 |
| *** daneyon has joined #openstack-kolla | 07:58 | |
| *** shardy has joined #openstack-kolla | 07:59 | |
| bjolo | http://paste.openstack.org/show/538935/ | 08:01 |
| *** daneyon has quit IRC | 08:02 | |
| bjolo | kolla deploy ceph or not? doesn't that need a new flag? | 08:03 |
| *** salv-orlando has quit IRC | 08:05 | |
| *** jascott1_tmp has quit IRC | 08:06 | |
| *** tyrola has joined #openstack-kolla | 08:12 | |
| *** Serlex has quit IRC | 08:12 | |
| bjolo | mewald1, when using your external ceph, where do you feed kolla the ceph info that it needs? | 08:15 |
| bjolo | for cinder_backend_ceph etc | 08:16 |
| mewald1 | bjolo: https://github.com/openstack/kolla/blob/master/doc/external-ceph-guide.rst should explain it in great depth :) | 08:16 |
| mewald1 | its not the most recent version. check my proposed patch for updated version | 08:17 |
| bjolo | ok i see | 08:27 |
| bjolo | when kolla deploys ceph, where does it keep the keys etc? | 08:30 |
| mewald1 | same location: /etc/ceph/ | 08:30 |
| bjolo | i mean on the deployer node | 08:31 |
| mewald1 | not at all I think | 08:31 |
| mewald1 | ansible creates the cephx credentials and installs keyring files in the containers | 08:31 |
| bjolo | hmm i just added one more node to my kolla ceph and kolla-ansible deploy worked | 08:32 |
| *** jascott1_tmp has joined #openstack-kolla | 08:33 | |
| mewald1 | wouldnt you expect that? | 08:33 |
| *** gfidente|afk has joined #openstack-kolla | 08:34 | |
| bjolo | yes, but doesnt kolla deployer need to know some stuff to be able to do that? | 08:34 |
| bjolo | (need to review my ceph deploy knowledge :) ) | 08:34 |
| mewald1 | All you need are bootstrap keys for OSDs I think. If you are interested in finding out how it works: My best guess is to start looking at https://github.com/openstack/kolla/blob/master/ansible/roles/ceph/tasks/distribute_keyrings.yml | 08:36 |
| bjolo | and when kolla deploys new / upgrade a compute node, it needs to know the keys | 08:44 |
| bjolo | ok from what i can see in the code, it fetches the keys from the mons | 08:44 |
| bjolo | correct? | 08:44 |
| mewald1 | looks like it, yes | 08:44 |
| bjolo | so loosing all my mons, means loosing my ceph? | 08:45 |
| bjolo | well it usually does that anyway right but... | 08:45 |
| mewald1 | correct, but thats always the case | 08:45 |
| bjolo | but i mean i cannot deploy new ones with kolla | 08:45 |
| mewald1 | hmm not so sure, because at some point Kolla deploys the first mon without any others available | 08:46 |
| bjolo | lets say big maintenance, i through away my mons containers to deploy new ones | 08:46 |
| mewald1 | to that must work somehow | 08:46 |
| bjolo | whoopsi ceph gone | 08:47 |
| mewald1 | If you loose all mons Ceph is gone by definition. I dont think it makes any different whether you are able to build new mons with the same config or not. Recovering will always be pain in the ass :D | 08:49 |
| bjolo | naw ceph will be offline if i loose my mons | 08:49 |
| bjolo | but if i deploy a new one with right keys, ceph is back | 08:50 |
| mewald1 | you might be right, I have no idea how Kolla handles it. As said: have never tried it :) If you try, please report back :) | 08:50 |
| *** jascott1_tmp has quit IRC | 08:54 | |
| bjolo | have no nodes to play with right now, handed off to testing yesterday | 08:54 |
| *** Serlex has joined #openstack-kolla | 09:01 | |
| *** gfidente|afk is now known as gfidente | 09:03 | |
| *** bootsha has quit IRC | 09:03 | |
| *** salv-orlando has joined #openstack-kolla | 09:03 | |
| bjolo | enable_ceph_rgw does not seem to create an object store API endpoint | 09:05 |
| bjolo | kolla 2.0.2 | 09:05 |
| *** ntpttr has quit IRC | 09:08 | |
| *** ntpttr has joined #openstack-kolla | 09:08 | |
| *** bjolo has quit IRC | 09:08 | |
| *** bjolo has joined #openstack-kolla | 09:08 | |
| mewald1 | I dont think it should | 09:11 |
| mewald1 | swift should, though | 09:11 |
| bjolo | ? | 09:11 |
| bjolo | rgw is to me a replacement for swift | 09:12 |
| mewald1 | ah sorry, got confused :D | 09:12 |
| mewald1 | too much multitasking | 09:12 |
| *** salv-orlando has quit IRC | 09:13 | |
| bdaca | bjolo: to do this, rgw should connect with keystone I think | 09:20 |
| *** wznoinsk_ has joined #openstack-kolla | 09:20 | |
| *** sean-k-m1 has joined #openstack-kolla | 09:20 | |
| bdaca | and it is not implemented i think | 09:20 |
| *** wznoinsk- has joined #openstack-kolla | 09:22 | |
| *** wznoinsk has quit IRC | 09:22 | |
| *** sean-k-mooney has quit IRC | 09:22 | |
| *** wznoinsk has joined #openstack-kolla | 09:22 | |
| *** wznoinsk| has joined #openstack-kolla | 09:23 | |
| *** wznoinsk- has quit IRC | 09:24 | |
| *** wznoinsk has quit IRC | 09:25 | |
| *** wznoinsk_ has quit IRC | 09:25 | |
| *** wznoinsk has joined #openstack-kolla | 09:30 | |
| *** tyrola has quit IRC | 09:31 | |
| *** tyrola has joined #openstack-kolla | 09:31 | |
| *** wznoinsk| has quit IRC | 09:32 | |
| openstackgerrit | Merged openstack/kolla: Configurable policy.json for keystone https://review.openstack.org/344273 | 09:41 |
| *** salv-orlando has joined #openstack-kolla | 09:43 | |
| *** salv-orl_ has joined #openstack-kolla | 09:56 | |
| bdaca | bjolo, mewald1: inspired by your ceph-related discussion i tested your scenario with all ceph_mon removals | 09:57 |
| bdaca | and it breaks everything :D | 09:58 |
| *** bootsha has joined #openstack-kolla | 09:58 | |
| mewald1 | nice! :D | 09:59 |
| *** salv-orlando has quit IRC | 10:00 | |
| bjolo | bdaca, not implemented in kolla or not in general? | 10:00 |
| bjolo | have to check, but our fuel based clouds are runing rados_gw and is intergrated with keystone | 10:01 |
| bjolo | bdaca, cool! (or not) | 10:01 |
| *** zhurong has quit IRC | 10:02 | |
| *** salv-orlando has joined #openstack-kolla | 10:04 | |
| bjolo | bdaca, mewald1 and i continued seperatly on that. if you want to read more, you can read it here | 10:05 |
| bjolo | http://paste.openstack.org/show/538959/ | 10:05 |
| bjolo | time for proteins, fat and carbs. | 10:06 |
| *** dwalsh_ has joined #openstack-kolla | 10:06 | |
| *** dwalsh has joined #openstack-kolla | 10:07 | |
| *** salv-orl_ has quit IRC | 10:08 | |
| *** Liuqing has quit IRC | 10:08 | |
| *** salv-orlando has quit IRC | 10:08 | |
| *** Liuqing has joined #openstack-kolla | 10:09 | |
| bdaca | bjolo: not implemented in kolla, on the ceph rgw guide there is a description what to do to connect rgw with keystone | 10:12 |
| bjolo | aha ok missed that. tnx for pointing out | 10:13 |
| *** wznoinsk_ has joined #openstack-kolla | 10:15 | |
| *** wznoinsk has quit IRC | 10:17 | |
| *** mbound has joined #openstack-kolla | 10:26 | |
| *** mewald1 has quit IRC | 10:35 | |
| *** rhallisey has joined #openstack-kolla | 10:50 | |
| openstackgerrit | Prithiv proposed openstack/kolla: Support Networking-SFC Container https://review.openstack.org/344930 | 10:54 |
| *** chopmann has joined #openstack-kolla | 10:56 | |
| *** bootsha has quit IRC | 10:58 | |
| *** bootsha has joined #openstack-kolla | 11:04 | |
| openstackgerrit | Dennis Mueller proposed openstack/kolla: Added neutron-metering-agent https://review.openstack.org/344808 | 11:16 |
| *** Liuqing has quit IRC | 11:19 | |
| *** dave-mccowan has joined #openstack-kolla | 11:23 | |
| *** dwalsh has quit IRC | 11:25 | |
| bjolo | https://github.com/ceph/ceph-docker | 11:25 |
| *** dwalsh_ has quit IRC | 11:25 | |
| bjolo | sebastian haan is a contributer. says a lot | 11:25 |
| bjolo | oops sorry wrong channel :) | 11:26 |
| *** zhubingbing has joined #openstack-kolla | 11:30 | |
| *** aernhart has joined #openstack-kolla | 11:36 | |
| *** zhurong has joined #openstack-kolla | 11:37 | |
| *** zhurong has quit IRC | 11:42 | |
| *** aernhart_ has joined #openstack-kolla | 11:43 | |
| *** zhurong has joined #openstack-kolla | 11:44 | |
| *** Serlex has quit IRC | 11:44 | |
| *** aernhart has quit IRC | 11:46 | |
| *** Serlex has joined #openstack-kolla | 11:51 | |
| *** bdaca has quit IRC | 11:58 | |
| v1k0d3n | does anyone know if there's anything special i need to do for haproxy with multinode vs aio? i am getting an error on play "Waiting for virtual IP to appear" because there's a timeout. do i need to have separate ip addresses for haproxy on different hosts or am i missing a keepalive flag maybe? | 12:10 |
| *** mewald has joined #openstack-kolla | 12:18 | |
| bjolo | v1k0d3n, you need to have an UNUSED ip for haproxy. i.e. that ip cannot be configure on any of the nodes | 12:19 |
| bjolo | v1k0d3n, sometimes if you do repeated deployments. i.e. deploy, tear down. the IP is still configured on one of the haproxy nodes | 12:20 |
| bjolo | do an ip addr | grep <VIP> on all nodes to see if it is ther | 12:20 |
| bjolo | alternatively run: kolla-ansible -i <inventory file> prechecks | 12:21 |
| openstackgerrit | Mathias proposed openstack/kolla: External Ceph - Implementation Nova https://review.openstack.org/344705 | 12:23 |
| *** athomas has quit IRC | 12:27 | |
| *** banix has joined #openstack-kolla | 12:28 | |
| *** banix has quit IRC | 12:33 | |
| *** banix has joined #openstack-kolla | 12:34 | |
| *** athomas has joined #openstack-kolla | 12:34 | |
| *** diogogmt has quit IRC | 12:36 | |
| *** banix has quit IRC | 12:37 | |
| *** diogogmt has joined #openstack-kolla | 12:38 | |
| *** zhubingbing has quit IRC | 12:41 | |
| *** salv-orlando has joined #openstack-kolla | 12:45 | |
| *** lyncos has joined #openstack-kolla | 12:46 | |
| *** diogogmt has quit IRC | 12:46 | |
| *** zhubingbing has joined #openstack-kolla | 12:47 | |
| v1k0d3n | bjolo: it's an unused ip, but i am running all roles on all nodes in this deployment, so haproxy is trying to install on each node using the same ip address. | 12:48 |
| v1k0d3n | so it passes on the first, fails on the rest. | 12:49 |
| v1k0d3n | prechecks succeed each time though. | 12:49 |
| mewald | v1k0d3n: I dont think that's what's happening. Its not haproxy who sets the IP but keepalived | 12:51 |
| *** salv-orlando has quit IRC | 12:51 | |
| v1k0d3n | ^^ right...sorry. | 12:51 |
| mewald | v1k0d3n: if you connect to each node and do ip addr show in each: Do you see the VIP on each node or multiple times? | 12:52 |
| v1k0d3n | i wrote the wrong thing...mind was thinking one, hands were typing the other (confusing sorry about that). | 12:52 |
| mewald | I've seen it a couple of times that when you redeploy everything the VIP is not removed from the interface | 12:53 |
| mewald | as a result the next run of prechecks fails as the IP responds to ping | 12:53 |
| v1k0d3n | ok, so...i the VIP get's placed on one...node04...but it is not being placed on the others. | 12:53 |
| v1k0d3n | should vip be the same on each? | 12:53 |
| mewald | no, the VIP should only be carried by a single node! | 12:54 |
| v1k0d3n | right, that makes sense... | 12:55 |
| mewald | at the moment of running the prechecks playbook, NOBODY should carry the IP | 12:55 |
| mewald | otherwise the check will fail | 12:55 |
| v1k0d3n | so that happened...let me explain again....starting from scratch. | 12:55 |
| v1k0d3n | ip is 10.254.254.25/32 > prechecks succeed. | 12:55 |
| v1k0d3n | then run...playbook try to apply...get's applied to node04 >> success. | 12:56 |
| v1k0d3n | now it tries to apple to the OTHERS... | 12:56 |
| v1k0d3n | bad | 12:56 |
| v1k0d3n | that fails... | 12:56 |
| v1k0d3n | then memcache/mariadb fails...bombs out. | 12:56 |
| v1k0d3n | one sec...let me paste playbook progression for you... | 12:56 |
| mewald | was going to ask for that :) | 12:57 |
| v1k0d3n | mewald: https://gist.github.com/v1k0d3n/28fbeb16f291aa236139f4bf30e5fcd6 | 12:58 |
| v1k0d3n | see what i mean now? | 12:59 |
| v1k0d3n | so my assumption is that keepalive is either acting wonky / the same ip is being applied to all nodes and that there's a configuration limitation i am missing (my guess here) | 13:00 |
| *** mewald1 has joined #openstack-kolla | 13:02 | |
| mewald1 | well that theory is something you can validate within 30 sec | 13:02 |
| mewald1 | another idea would be: The database nodes dont come up for some reason and therefore Haproxy doesnt see them as UP. | 13:03 |
| mewald1 | https://github.com/openstack/kolla/blob/6f17a71d7fe30422ee96c9e159b45ebd72b7d563/ansible/roles/mariadb/tasks/check.yml | 13:03 |
| mewald1 | see what happens here? The play runs mysql client in the mariadb container against the kolla_internal_fqdn which points to the VIP | 13:04 |
| mewald1 | but you can easily troubleshoot all that | 13:04 |
| *** mewald has quit IRC | 13:05 | |
| openstackgerrit | Dave Walker proposed openstack/kolla: Add Watcher ansible roles and templates https://review.openstack.org/337556 | 13:06 |
| *** JianqingJiang has joined #openstack-kolla | 13:12 | |
| *** jtriley has joined #openstack-kolla | 13:13 | |
| *** godleon has joined #openstack-kolla | 13:16 | |
| *** neouf has quit IRC | 13:19 | |
| *** rmart04 has joined #openstack-kolla | 13:20 | |
| *** neouf has joined #openstack-kolla | 13:21 | |
| *** neouf has quit IRC | 13:39 | |
| *** Liuqing has joined #openstack-kolla | 13:40 | |
| *** neouf has joined #openstack-kolla | 13:40 | |
| *** Liuqing has quit IRC | 13:41 | |
| bjolo | mewald1, ok so now i have configure swift and as you said, it now shows up in horizon | 13:45 |
| bjolo | mewald1, however, i have also built murano and magnum and deployed, but those does not show in horizon. Should they or do i need to do that manually? or not implemented in kolla? | 13:47 |
| bjolo | endpoints are there | 13:47 |
| *** lyncos_ has joined #openstack-kolla | 13:52 | |
| *** haplo37_ has joined #openstack-kolla | 13:57 | |
| *** salv-orlando has joined #openstack-kolla | 14:01 | |
| *** mgoddard_ has joined #openstack-kolla | 14:02 | |
| *** salv-orl_ has joined #openstack-kolla | 14:02 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Bind the ceilometer-api process to the api_interface only https://review.openstack.org/345451 | 14:04 |
| *** zhurong has quit IRC | 14:04 | |
| *** mgoddard has quit IRC | 14:05 | |
| *** salv-orlando has quit IRC | 14:07 | |
| *** salv-orlando has joined #openstack-kolla | 14:09 | |
| *** bjolo has quit IRC | 14:11 | |
| mewald1 | bjolo: I believe they need horizon plugins which are not shipped by default | 14:11 |
| *** salv-orl_ has quit IRC | 14:14 | |
| *** neouf has quit IRC | 14:17 | |
| *** bootsha has quit IRC | 14:18 | |
| v1k0d3n | mewald1: interesting. ok. so i may have to rerun with a different vip. i used the same as with an AIO, which at first was a 169.254.0.25 | 14:18 |
| v1k0d3n | and even on that node, i am getting connection refused :( | 14:19 |
| v1k0d3n | ERROR 2003 (HY000): Can't connect to MySQL server on '169.254.0.25' (111 "Connection refused") | 14:19 |
| *** neouf has joined #openstack-kolla | 14:19 | |
| v1k0d3n | i think the deployment is stuck in some halfway state...which really sucks | 14:20 |
| v1k0d3n | mewald1: so when i try to access the container, the mariadb command doesn't exist | 14:23 |
| mewald1 | the sql client command should be "mysql" | 14:24 |
| mewald1 | or are you looking for the server binary? | 14:24 |
| *** Mr_Broken has joined #openstack-kolla | 14:24 | |
| v1k0d3n | sorry...having issues thinking apparently this morning. | 14:24 |
| v1k0d3n | using mysql, no databases returned. | 14:25 |
| v1k0d3n | (working on little sleep). | 14:25 |
| v1k0d3n | so within the container on localhost... | 14:26 |
| v1k0d3n | ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused") | 14:26 |
| *** aernhart__ has joined #openstack-kolla | 14:26 | |
| v1k0d3n | (mariadb)[mysql@os-node04 /]$ mysql -h localhost -P 3306 -u haproxy -e show databases; | 14:26 |
| mewald1 | try with -u root -h 127.0.0.1 -p<PW> | 14:26 |
| mewald1 | ah | 14:26 |
| mewald1 | well it never got to the point where any databases could be created | 14:26 |
| mewald1 | can you verify the galera cluster works? | 14:27 |
| mewald1 | SHOW STATUS LIKE 'wsrep_cluster%' | 14:27 |
| *** aernhart_ has quit IRC | 14:27 | |
| v1k0d3n | i know this sounds bad...but i am really new to galera clusters. | 14:29 |
| v1k0d3n | learning still. | 14:29 |
| v1k0d3n | so that's a direct command? | 14:30 |
| mewald1 | mysql -h localhost -P 3306 -u haproxy -e "SHOW STATUS LIKE 'wsrep_cluster%';" | 14:30 |
| mewald1 | that should work | 14:30 |
| v1k0d3n | can't connect to socket | 14:31 |
| v1k0d3n | 2.0.1 is a good build version for kolla, right? | 14:31 |
| mewald1 | ah shit, sorry. use root user | 14:31 |
| mewald1 | yes | 14:31 |
| *** Mr_Broken has quit IRC | 14:31 | |
| mewald1 | mysql -h 127.0.0.1 -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster%';" | 14:31 |
| mewald1 | does that do? | 14:31 |
| v1k0d3n | looking for a mysql password in the passwords.yml file.... | 14:33 |
| v1k0d3n | db got it | 14:33 |
| v1k0d3n | still connection refused. | 14:34 |
| v1k0d3n | maybe i need to rebuild the containers again or something? | 14:34 |
| *** huikang has joined #openstack-kolla | 14:34 | |
| *** ljjjustin has quit IRC | 14:40 | |
| *** bwe_ has quit IRC | 14:42 | |
| *** ljjjustin has joined #openstack-kolla | 14:43 | |
| *** michauds has joined #openstack-kolla | 14:44 | |
| *** aernhart_ has joined #openstack-kolla | 14:47 | |
| *** JianqingJiang has quit IRC | 14:48 | |
| *** bdaca has joined #openstack-kolla | 14:48 | |
| *** Jeffrey4l_ has quit IRC | 14:48 | |
| *** zhubingbing has quit IRC | 14:49 | |
| *** zhubingbing has joined #openstack-kolla | 14:49 | |
| *** aernhart__ has quit IRC | 14:50 | |
| *** dmueller-evoila has quit IRC | 14:52 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 14:53 | |
| mewald1 | v1k0d3n: just found out mariadb is not listening on 127.0.0.1 | 14:53 |
| mewald1 | you have to use the IP of your controller host | 14:54 |
| *** inc0 has joined #openstack-kolla | 14:55 | |
| inc0 | good morning | 14:55 |
| mewald1 | hi | 14:56 |
| *** michauds has quit IRC | 14:57 | |
| *** inc0_ has joined #openstack-kolla | 14:57 | |
| *** diogogmt has joined #openstack-kolla | 14:59 | |
| *** pece has joined #openstack-kolla | 15:01 | |
| *** b_bezak has quit IRC | 15:01 | |
| *** inc0 has quit IRC | 15:01 | |
| *** b_bezak has joined #openstack-kolla | 15:01 | |
| *** mgoddard_ has quit IRC | 15:02 | |
| *** haplo37_ has quit IRC | 15:02 | |
| *** david-lyle has joined #openstack-kolla | 15:04 | |
| *** b_bezak has quit IRC | 15:06 | |
| *** mgoddard has joined #openstack-kolla | 15:08 | |
| *** daneyon has joined #openstack-kolla | 15:11 | |
| openstackgerrit | Merged openstack/kolla: Change rgw frontend interface https://review.openstack.org/344410 | 15:13 |
| *** sdake has joined #openstack-kolla | 15:13 | |
| inc0_ | hey sdake, you | 15:14 |
| inc0_ | you've pinged me yesterday'\ | 15:14 |
| *** haplo37_ has joined #openstack-kolla | 15:14 | |
| sdake | morning | 15:14 |
| sdake | ya | 15:14 |
| sdake | need an email t the l related to osic cluster | 15:14 |
| sdake | key points should be dates, link to testing, and contributors so you can get them set up | 15:14 |
| sdake | since that starts friday iirc | 15:15 |
| sdake | or monday | 15:15 |
| inc0_ | on that note, we'll move 1 week forward, got info from osic operators | 15:15 |
| inc0_ | but will get more time in return | 15:15 |
| inc0_ | they had some problems with setup | 15:15 |
| inc0_ | I'll send over details as soon as I get them myself:) | 15:15 |
| *** daneyon has quit IRC | 15:16 | |
| *** salv-orlando has quit IRC | 15:16 | |
| *** sdake_ has joined #openstack-kolla | 15:17 | |
| sdake_ | rhallisey its best practice to put a 1 week window on voting and tell how many votes are needed to pass a vote and place [vote] in the tag | 15:19 |
| *** sdake has quit IRC | 15:19 | |
| sdake_ | that way people don't forget to vote - know its a vote -and whether they need to care enough to vote | 15:19 |
| rhallisey | k | 15:20 |
| rhallisey | I can update it | 15:22 |
| inc0_ | sdake_, did you review customizations prototype? | 15:24 |
| openstackgerrit | Merged openstack/kolla: Change rgw frontend interface https://review.openstack.org/344411 | 15:24 |
| sdake_ | man i can't believe kolla has 4365 commits | 15:27 |
| inc0_ | well we did have Sam in our team | 15:27 |
| inc0_ | he had his way to push these numbers to unrealistic levels | 15:27 |
| *** unicell1 has joined #openstack-kolla | 15:29 | |
| *** bootsha has joined #openstack-kolla | 15:30 | |
| *** unicell has quit IRC | 15:30 | |
| *** mliima has quit IRC | 15:32 | |
| *** michauds has joined #openstack-kolla | 15:35 | |
| *** bootsha has quit IRC | 15:35 | |
| *** michauds has quit IRC | 15:40 | |
| *** salv-orlando has joined #openstack-kolla | 15:43 | |
| pece | hi, any tips how to tcpdump MySQL traffic from single container in Kolla? | 15:43 |
| *** slagle has quit IRC | 15:45 | |
| inc0_ | pece you can't do this on a single container per se | 15:45 |
| inc0_ | but you can do this for all of them, and you can filter out using tcpdump | 15:45 |
| inc0_ | like you normally would | 15:46 |
| inc0_ | also you can run tcpdump on host and it won't affect anything | 15:46 |
| *** slagle has joined #openstack-kolla | 15:47 | |
| *** rmart04 has quit IRC | 15:49 | |
| *** zhubingbing has quit IRC | 15:50 | |
| *** zhubingbing has joined #openstack-kolla | 15:51 | |
| sdake_ | inc0_ kolla has same number of commits in its lifecycle that heat did | 16:12 |
| *** jascott1_tmp has joined #openstack-kolla | 16:19 | |
| *** godleon has quit IRC | 16:19 | |
| inc0_ | gerrit is down? | 16:22 |
| *** ayoung has joined #openstack-kolla | 16:24 | |
| *** ayoung has quit IRC | 16:25 | |
| *** ayoung has joined #openstack-kolla | 16:25 | |
| *** derekjhyang has joined #openstack-kolla | 16:29 | |
| derekjhyang | Hey folks, I have a question: when I built nova-libvirt container image with tools/build.py , I met an error in libvirt-bin installation(I used ubuntu-based image) | 16:36 |
| Mech422 | Morning | 16:36 |
| *** lyncos has quit IRC | 16:39 | |
| *** zhubingbing has quit IRC | 16:41 | |
| *** david-lyle has quit IRC | 16:41 | |
| *** mewald1 has quit IRC | 16:43 | |
| Mech422 | sdake_, inc0_: I think the recent 'external ceph' discussions really are more about pluggability then ceph... | 16:44 |
| Mech422 | sdake_, inc0_: so many of the openstack components have multiple drivers/options/etc now that as sdake mentioned, it will be impossible to cater to them all in kolla core | 16:45 |
| inc0_ | yeah, I agree | 16:46 |
| inc0_ | but I think it's summit-worthy topic | 16:46 |
| *** sdake_ has quit IRC | 16:46 | |
| Mech422 | inc0_: yeah - above my pay grade... | 16:46 |
| inc0_ | nah | 16:46 |
| inc0_ | just we need lots of eyes/brains on this one | 16:47 |
| Mech422 | inc0_: but I don't think we should be 'all or nothing' on stuff like this (eg 'ceph is enabled so everything uses ceph' ) | 16:47 |
| Mech422 | inc0_: _deploying_ stuff should be orthogonal to _using_ it | 16:47 |
| inc0_ | yeah, I agree, we always planned to do this plugabbility | 16:47 |
| Mech422 | inc0_: btw, my boss is wanting to know about the upgrade process with kolla... | 16:48 |
| inc0_ | what does he want to know? | 16:49 |
| *** sdake has joined #openstack-kolla | 16:49 | |
| Mech422 | inc0_: do we have compelling upgrade/maintenance stories yet ? | 16:49 |
| *** mark-casey has joined #openstack-kolla | 16:49 | |
| Mech422 | inc0_: oh just the usual - can we use kolla to stay on 'current' os (eg will kolla/netwon be able to in-place upgrade to kolla/o) | 16:50 |
| Mech422 | inc0_: can we add nodes with kolla, can we delete nodes with kolla, etc etc | 16:50 |
| inc0_ | so, stories are hard, we need production deployments to hear the stories;) but featureset is there | 16:55 |
| inc0_ | we support orchiestrated upgrades between major and minor releases | 16:55 |
| inc0_ | kolla-ansible upgrade will take tyou from L->M and M->N | 16:55 |
| Mech422 | inc0_: I'll be doing rolling upgrades of our small cluster starting a week from friday | 16:55 |
| inc0_ | kolla tries to do rolling whenever it's possible | 16:55 |
| Mech422 | inc0_: oh nice! | 16:55 |
| inc0_ | but I still suggest to call a maintenance window | 16:56 |
| inc0_ | as neutron for example is crap to do rolling upgrade | 16:56 |
| inc0_ | still, 60 nodes took 10mins so that's not that bad | 16:56 |
| inc0_ | as for adding/deleting ndoes | 16:56 |
| inc0_ | compute nodes are super easy | 16:56 |
| inc0_ | controller nodes are a bit harder as galera and rabbitmq doesn't like chanign of cluster size that much | 16:57 |
| inc0_ | but apart from that, to add node just add it into inventory and rerun the deployment | 16:57 |
| inc0_ | delete node - just remove containers and volumes | 16:57 |
| *** unicell has joined #openstack-kolla | 16:57 | |
| *** unicell1 has quit IRC | 16:58 | |
| Mech422 | inc0_: cool ...This should also give us some feedback as to how kolla/Newton is working overall, as our dev/qa group is supposed to start using it | 16:58 |
| Mech422 | inc0_: very small, and not mission critical - but live users nonetheless :-) | 16:59 |
| inc0_ | yeah, make sure to tell us any painpoints you encounter | 16:59 |
| inc0_ | brb, changing vpns | 17:00 |
| *** inc0_ has quit IRC | 17:00 | |
| *** inc0 has joined #openstack-kolla | 17:00 | |
| inc0 | back | 17:01 |
| rahuls | sdake: should I update the version to 2.0.3 in stable/mitaka? | 17:03 |
| rahuls | I missed the commit in ansible/groups_vars/all.yml | 17:04 |
| rahuls | https://review.openstack.org/#/c/345166/ | 17:04 |
| patchbot | rahuls: patch 345166 - kolla (stable/mitaka) - updates default registry tag to latest version | 17:04 |
| rahuls | I hope its ok if I update it to 2.0.3? | 17:04 |
| rahuls | when I built my images yesterday, they were built with tag 2.0.3 and then deployment failed at it was looking for 2.0.2 | 17:04 |
| *** jascott1_tmp has quit IRC | 17:05 | |
| *** chopmann has quit IRC | 17:08 | |
| Mech422 | inc0: I have a patch to fix the ceph-in-partition stuff that was just merged - should I open a new bug/gerrit or can we just use the existing one ? | 17:10 |
| Mech422 | inc0: its literally like 2 lines of code | 17:10 |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: WIP(do not merge) Add status command for Workflow Engine https://review.openstack.org/345102 | 17:13 |
| inc0 | TrivialFix | 17:13 |
| inc0 | just use this | 17:13 |
| *** inc0 has quit IRC | 17:13 | |
| *** inc0 has joined #openstack-kolla | 17:14 | |
| *** wrouesnel1 has quit IRC | 17:17 | |
| *** athomas has quit IRC | 17:19 | |
| *** Jeffrey4l_ has quit IRC | 17:19 | |
| openstackgerrit | rahul sharma proposed openstack/kolla: updates default registry tag to latest version https://review.openstack.org/345166 | 17:23 |
| *** athomas has joined #openstack-kolla | 17:30 | |
| *** harlowja has joined #openstack-kolla | 17:44 | |
| *** bjolo has joined #openstack-kolla | 17:45 | |
| inc0 | wirehead_, rhallisey I might have to cancel next weeks meeting | 17:54 |
| *** mewald has joined #openstack-kolla | 17:54 | |
| rhallisey | inc0, move it later? | 17:57 |
| *** Serlex has quit IRC | 17:57 | |
| ccesario | hi guys... does someone is having problem on kolla-build with the current git code ? http://paste.openstack.org/show/539099/ | 17:58 |
| *** jascott1_tmp has joined #openstack-kolla | 17:58 | |
| *** mliima has joined #openstack-kolla | 17:58 | |
| Mech422 | ccesario: looks like your missing the 'https' driver for apt | 17:59 |
| ccesario | Mech422, maybe any code did this..... because before I update my local rep the build it was working :) | 18:01 |
| Mech422 | ccesario: something might have switched to using a https repo? | 18:01 |
| Mech422 | ccesario: I'm building now - give me 20 minutes and I can see if I get errors | 18:01 |
| *** inc0 has quit IRC | 18:02 | |
| Mech422 | ccesario: you could try install apt-transport-https and rebuilding if you don't want to wait | 18:02 |
| ccesario | Mech422, sure... sure | 18:02 |
| ccesario | let me check with apt-transport-https installed | 18:03 |
| *** david-lyle has joined #openstack-kolla | 18:04 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Adds ansible support to deploy skydns service https://review.openstack.org/335677 | 18:04 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Adds ansible support to deploy skydns service https://review.openstack.org/335677 | 18:05 |
| *** david-lyle_ has joined #openstack-kolla | 18:05 | |
| *** david-lyle has quit IRC | 18:05 | |
| *** jascott1_tmp has quit IRC | 18:07 | |
| ccesario | Mech422, the hosts already has apt-transport-https pack ... let try include it in base container | 18:07 |
| Mech422 | ccesario: sorry boss grabbed me - I'll take a look in a bit... | 18:13 |
| ccesario | Mech422, no problem :) | 18:14 |
| *** david-lyle_ has quit IRC | 18:22 | |
| ccesario | Mech422, only to notify you.... the same error happen even with apt-transport-https pack into docker/base/Dockerfile.j2 file | 18:27 |
| *** salv-orlando has quit IRC | 18:28 | |
| *** salv-orlando has joined #openstack-kolla | 18:30 | |
| *** bjolo_ has joined #openstack-kolla | 18:32 | |
| *** salv-orlando has quit IRC | 18:34 | |
| *** harlowja has quit IRC | 18:35 | |
| *** harlowja has joined #openstack-kolla | 18:38 | |
| bjolo | good evening | 18:40 |
| *** daneyon has joined #openstack-kolla | 18:47 | |
| *** bjolo has quit IRC | 18:50 | |
| *** michauds has joined #openstack-kolla | 18:51 | |
| *** derekjhyang has quit IRC | 18:52 | |
| *** daneyon has quit IRC | 18:53 | |
| Mech422 | ccesario: I get the same error | 18:57 |
| ccesario | Mech422, O_o | 18:57 |
| ccesario | thanks to let me know :) | 18:58 |
| ccesario | but does not exists https:// entry to apt configs | 18:59 |
| *** harlowja has quit IRC | 18:59 | |
| *** banix has joined #openstack-kolla | 19:06 | |
| *** banix has quit IRC | 19:07 | |
| Mech422 | ccesario: I'm getting weird xml responses from elasticsearch.co : https://packages.elastic.co/elasticsearch/2.x/debian | 19:12 |
| Mech422 | sdake: Is the wifey re-arranging repos ? | 19:13 |
| Mech422 | sdake: spring cleaning maybe ? :-D | 19:14 |
| *** bjolo has joined #openstack-kolla | 19:14 | |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: WIP(do not merge) Add status command for Workflow Engine https://review.openstack.org/345102 | 19:21 |
| *** jascott1_tmp has joined #openstack-kolla | 19:30 | |
| v1k0d3n | hey all. having issues with a multi-node deployment, and i can't seem to get over them. | 19:35 |
| bjolo | whats the error? | 19:35 |
| v1k0d3n | it always passes prechecks, but fails at haproxy | Waiting for virtual IP to appear | 19:35 |
| v1k0d3n | i have all nodes be both compute and controller | 19:36 |
| v1k0d3n | 5 node cluster. | 19:36 |
| bjolo | can you paste your multinode and globals.conf? | 19:36 |
| v1k0d3n | it works on the first node that ansible grabs, applies the ip address for the keepalive, and bombs out on all the rest. | 19:36 |
| bjolo | what version of kolla? | 19:36 |
| bjolo | pip list | grep kolla | 19:36 |
| *** ByPasS has joined #openstack-kolla | 19:37 | |
| v1k0d3n | https://gist.github.com/v1k0d3n/3fe9f71129bc52d82c038dbdc9e0094a | 19:38 |
| v1k0d3n | https://gist.github.com/v1k0d3n/06033dbeece0ab61dfd956c8efee1bd3 | 19:39 |
| *** sdake_ has joined #openstack-kolla | 19:39 | |
| v1k0d3n | kolla is 2.0.1 | 19:39 |
| *** sdake has quit IRC | 19:40 | |
| v1k0d3n | anything glaring there? | 19:42 |
| bjolo | not the error, but fyi. when building your own images, it is recommended to add a 4 digit to openstack_release. i.e 2.0.1.1 for ex | 19:42 |
| v1k0d3n | oh, interesting...i thought that had to be the same as kolla...didn't realize that. | 19:42 |
| v1k0d3n | so anything on the error though? | 19:43 |
| v1k0d3n | been stuck on this, and really want to find a way through it. | 19:44 |
| *** shardy has quit IRC | 19:45 | |
| bjolo | nope *should* work | 19:45 |
| bjolo | but since it is not working, i would simplify | 19:45 |
| bjolo | disable TLS for now | 19:45 |
| bjolo | only deploy 1 controler, 1 network, 1 compute | 19:45 |
| bjolo | you only have IP configured on eth0 right? | 19:46 |
| bjolo | do you have kolla tools on each node so you can clean them properly? | 19:47 |
| v1k0d3n | do kolla tools need to be install on each node? | 19:48 |
| v1k0d3n | lol | 19:48 |
| bjolo | nope | 19:48 |
| v1k0d3n | ok...but just for the cleanup, is that right? | 19:48 |
| bjolo | but the kolla repo does have a tools folder that ... | 19:48 |
| bjolo | exactly | 19:48 |
| bjolo | i assume you have a couple of failed installs by now right? | 19:48 |
| v1k0d3n | tis true...definitely. | 19:49 |
| bjolo | easiest is just to clone the kolla repo on each node | 19:49 |
| bjolo | clean up each host. cleanup-containers; cleanup-host; cleanup-images -f | 19:51 |
| bjolo | make sure VIP is not lingering on any node | 19:51 |
| bjolo | simply the deployment as suggested above | 19:51 |
| bjolo | there is a 2.0.2 tag out now as well | 19:52 |
| bjolo | but i dont think there are any 2.0.2 images on docker hub, so you need to do another build if you wanna try that | 19:52 |
| bjolo | personally i pip install stable-mitaka latest branch | 19:58 |
| bjolo | what version of ansible and docker are you running? | 20:03 |
| *** bdaca has quit IRC | 20:04 | |
| *** lyncos_ has quit IRC | 20:04 | |
| rahuls | bjolo: stable/mitaka do has 2.0.2 tag. I guess the new images get built with 2.0.3 tag | 20:05 |
| rahuls | my bad, I read half line. | 20:05 |
| rahuls | is there any build issue going on with ubuntu based images? | 20:06 |
| bjolo | i run centos images, but i have done quite a few ubuntu builds lately and not seen any issues | 20:07 |
| *** inc0 has joined #openstack-kolla | 20:07 | |
| rahuls | I am seeing them today | 20:07 |
| bjolo | rahuls, on mitaka branch? | 20:08 |
| rahuls | yes | 20:08 |
| rahuls | I guess earlier today, ccesario pointed out the same | 20:09 |
| bjolo | hmm | 20:09 |
| rahuls | http://logs.openstack.org/66/345166/2/check/gate-kolla-dsvm-build-ubuntu-binary/fd13ed5/console.html#_2016-07-21_19_33_37_623781 | 20:09 |
| openstackgerrit | Mathias proposed openstack/kolla: External Ceph - Implementation Nova https://review.openstack.org/344705 | 20:09 |
| rahuls | This is the output of jenkins autobuild for one of my patches | 20:09 |
| rahuls | it passes everything except ubuntu | 20:09 |
| *** inc0 has quit IRC | 20:09 | |
| rahuls | it looks like there is some issue going on with ubuntu based images | 20:10 |
| *** mewald has left #openstack-kolla | 20:10 | |
| *** gfidente is now known as gfidente|afk | 20:11 | |
| *** inc0 has joined #openstack-kolla | 20:11 | |
| *** shardy has joined #openstack-kolla | 20:12 | |
| *** salv-orlando has joined #openstack-kolla | 20:14 | |
| bjolo | im having kolla swift access issues | 20:14 |
| bjolo | did a small setup of swift and it works for admin, but a normal users have no access | 20:16 |
| bjolo | anyone experienced this? | 20:18 |
| *** tyrola has quit IRC | 20:24 | |
| *** huikang has quit IRC | 20:31 | |
| *** mliima has quit IRC | 20:34 | |
| *** jtriley has quit IRC | 20:36 | |
| openstackgerrit | Ildar proposed openstack/kolla: Fixing typo in the documentation https://review.openstack.org/345659 | 20:37 |
| Mech422 | rahuls: Can you access the elastic.co repos? I think thats the issue biting ccesario and I | 20:37 |
| Mech422 | anyone else having issues with the elastic.co repos ? http://packages.elastic.co/kibana/4.4/debian shows an xml error | 20:38 |
| bjolo | hmm admin also has issues when going through the public apis | 20:38 |
| bjolo | whats going on here | 20:39 |
| Mech422 | bjolo: are you using different public/admin endpoints? | 20:39 |
| *** harlowja has joined #openstack-kolla | 20:39 | |
| bjolo | yes, using TLS so public is not the same as admin/internal | 20:40 |
| Mech422 | bjolo: I have to export OS_INTERFACE=admin and OS_ENDPOINT=admin | 20:40 |
| Mech422 | bjolo: it default to trying to hit the public endpoint IIRC | 20:40 |
| *** lyncos has joined #openstack-kolla | 20:40 | |
| bjolo | ok | 20:42 |
| Mech422 | bjolo: you can just add that to your admin-openrc.sh | 20:42 |
| bjolo | admin-openrc.sh works fine using os client | 20:44 |
| bjolo | horizon works fine when using admin tenant | 20:44 |
| inc0 | anyone seen issue with apt-transport-https missing? | 20:44 |
| Mech422 | anyone else having issues with the elastic.co repos ? http://packages.elastic.co/kibana/4.4/debian shows an xml error | 20:45 |
| bjolo | adding admin to a the sandbox tenant as _member_ | 20:45 |
| bjolo | swift not working | 20:45 |
| *** tyrola has joined #openstack-kolla | 20:45 | |
| Mech422 | inc0: check scrollback | 20:45 |
| inc0 | Mech422, I just tried to build and it failed on elastic | 20:45 |
| Mech422 | inc0: yeah - repo is fubar | 20:45 |
| Mech422 | inc0: I tried asking here and in #elasticsearch but not getting any answer | 20:46 |
| Mech422 | inc0: I figured sdake_ could let his wife know | 20:46 |
| inc0 | shes working with Ansible, not elasti | 20:46 |
| inc0 | c | 20:46 |
| Mech422 | inc0: oh - my bad | 20:47 |
| Mech422 | inc0: anyway - if you hit the url directly, it spits out an xml error saying key 'kibana/4.4/debian' not found | 20:48 |
| *** ccesario has quit IRC | 20:49 | |
| inc0 | for me it failed on build | 20:49 |
| Mech422 | inc0: same for me and ccesario | 20:49 |
| inc0 | that https method not found | 20:49 |
| *** tyrola has quit IRC | 20:49 | |
| Mech422 | inc0: I think that's a red-herring | 20:49 |
| inc0 | and here I was, doing demo how cool is kolla...;) | 20:49 |
| Mech422 | inc0: oh - I see you met Mr. Murphy | 20:50 |
| Mech422 | :-) | 20:50 |
| Mech422 | inc0: we should probaby update /etc/kolla/admin-openrc.sh at some point to specify the admin endpoint... | 20:53 |
| Mech422 | inc0: we're starting to see people install with seperate public/admin endpoints, and everything seems to default to hitting the 'public' endpoint | 20:53 |
| inc0 | yeah, good call | 20:54 |
| Mech422 | I export OS_INTERFACE and OS_ENDPOINT='admin' ...I think that covers nova, and neutron? there was one I couldn't find the override for ? cinder maybe ? | 20:55 |
| *** salv-orlando has quit IRC | 20:56 | |
| bjolo | Mech422, yes +1 on that | 20:56 |
| bjolo | TLS is a must for production | 20:56 |
| bjolo | implies seperate admin and public endpoints afa understand it | 20:57 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: POC - Ansible support for Mariadb https://review.openstack.org/334255 | 20:57 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Adds ansible support to deploy skydns service https://review.openstack.org/335677 | 20:57 |
| *** haplo37_ has quit IRC | 21:00 | |
| *** tyrola has joined #openstack-kolla | 21:03 | |
| bjolo | has anyone attempted to make a docker image of kolla deployer node? | 21:04 |
| bjolo | docker run kolla-master:2.0.2 | 21:04 |
| bjolo | with all deps sorted out of course | 21:06 |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: WIP(do not merge) Add status command for Workflow Engine https://review.openstack.org/345102 | 21:06 |
| Mech422 | inc0: <warkolm> just tested with LS and it works ok? | 21:08 |
| Mech422 | inc0: maybe its not the elastic.co repos (warkolm says they're not supposed to be browseable) | 21:09 |
| inc0 | Mech422, what hepled me is to install apt-get -y install apt-transport-https in container | 21:10 |
| Mech422 | inc0: ccessario mentioned trying that and not having any luck... | 21:11 |
| Mech422 | inc0: I'll try that when I get back - I gotta take the GF to look at some houses | 21:11 |
| Mech422 | inc0: did you put that in base image ? or just elasticsearch container ? | 21:12 |
| Mech422 | inc0: oh - elastic.co is in the base image sources.list, so needs to be there I guess | 21:12 |
| *** banix has joined #openstack-kolla | 21:20 | |
| bjolo | v1k0d3n, hows it going? any improvements? | 21:24 |
| v1k0d3n | bjolo: had a really long call with a big wig on our side.... | 21:27 |
| bjolo | ok you gotta take care of that first :) | 21:28 |
| v1k0d3n | getting back to it. while doing that i noticed that pip installing kolla failed on one of the other nodes | 21:28 |
| bjolo | oh you dont have to do pip install to run the tools | 21:28 |
| bjolo | just git clone ...; cd kolla/tools; ./cleanup-containers | 21:28 |
| v1k0d3n | the clean that you're talking about... | 21:29 |
| v1k0d3n | just the cleanup stull in /kolla, right | 21:29 |
| v1k0d3n | ? | 21:29 |
| v1k0d3n | containers/host/etc | 21:29 |
| bjolo | yes | 21:29 |
| bjolo | resets the target node | 21:29 |
| v1k0d3n | ah! on the pip...should've really thought about that! | 21:30 |
| *** daneyon has joined #openstack-kolla | 21:30 | |
| bjolo | only master node needs pip install | 21:32 |
| v1k0d3n | assuming if i do that i need to redeploy passwords, certs etc... | 21:32 |
| v1k0d3n | btw...thanks for the help and staying around/asking again | 21:32 |
| *** tyrola has quit IRC | 21:33 | |
| bjolo | that will be done automatically when you do your next kolla-ansible -i <inventory file> deploy | 21:33 |
| bjolo | no worries :) | 21:33 |
| *** tyrola has joined #openstack-kolla | 21:33 | |
| mark-casey | bjolo: v1k0d3n: you may not have to clone on nodes. ansible -i /usr/local/share/kolla/ansible/inventory/ -m script -a '/kolla/tools/cleanup-containers' all | 21:33 |
| v1k0d3n | even the certs? i thought certs needed to be done manually... | 21:33 |
| mark-casey | that'll push the script from the deployer and run it | 21:34 |
| mark-casey | out of $tmpdir | 21:34 |
| *** daneyon has quit IRC | 21:35 | |
| bjolo | mark-casey, cool. | 21:35 |
| v1k0d3n | \o/ mark-casey! awesome trick | 21:36 |
| mark-casey | :) | 21:37 |
| *** dave-mccowan has quit IRC | 21:37 | |
| *** inc0 has quit IRC | 21:37 | |
| bjolo | so i still have this swift issue when accessing it as a normal _member_ user for sandbox tenant | 21:39 |
| bjolo | Forbidden 403 | 21:39 |
| *** shardy has quit IRC | 21:39 | |
| bjolo | is there like a quota or permission that needs to be enabled? | 21:39 |
| bjolo | keystone policy.json? | 21:40 |
| *** harlowja has quit IRC | 21:41 | |
| v1k0d3n | shoot now complaining that /etc/kolla/certificates/haproxy.pem doesn't exist (which is doesnt) | 21:43 |
| bjolo | did you comment out the TLS stuff in globals.conf? | 21:43 |
| bjolo | oh did you run ./cleanup-host on the deployer node? | 21:44 |
| v1k0d3n | yes | 21:44 |
| bjolo | sorry if i was vague. only do the cleanup on the target nodes | 21:45 |
| bjolo | i meant | 21:45 |
| *** pece has quit IRC | 21:45 | |
| v1k0d3n | well, they're all target nodes in this case... | 21:45 |
| v1k0d3n | all doing everything, if that's ok? | 21:45 |
| v1k0d3n | was told it should be though... | 21:45 |
| bjolo | ? | 21:45 |
| bjolo | please go over your setup | 21:45 |
| bjolo | you have a seperate master node and 5 target nodes? | 21:46 |
| v1k0d3n | 5 hosts. all controller, compute and network. it's for a test environment. | 21:46 |
| bjolo | yes, but since you have issues, i would recommend to scale down and simplify till you get it working | 21:47 |
| bjolo | i.e. basic basic default deployment | 21:47 |
| bjolo | no TLS | 21:47 |
| *** harlowja has joined #openstack-kolla | 21:48 | |
| bjolo | 1 node as the master node. git clone ...; pip install; tox -e genconfig; cp etc /etc/kolla; etc etc | 21:48 |
| *** harlowja has quit IRC | 21:48 | |
| bjolo | edit inventory file and specify one node for controller, one for network and one for compute | 21:49 |
| v1k0d3n | i just noticed something...... | 21:49 |
| v1k0d3n | never mind...can't be in use. | 21:50 |
| v1k0d3n | but when i make changes and then pip install....i have my git kolla repo in /opt/kolla, but realized that what mark-casey sent works which uses an inventory file of /usr/local/share/kolla/ansible/inventory/multinode and not /opt/kolla.... | 21:51 |
| v1k0d3n | the two are throwing me off...now i'm wondering if there are two globals or someting too? | 21:51 |
| *** jascott1_tmp has quit IRC | 21:52 | |
| bjolo | channel please correct me if im wrong, but default location it looks in should be /etc/kolla | 21:53 |
| bjolo | in my /etc/kolla i have globals.conf passwords.yml and kolla-build.conf | 21:53 |
| bjolo | i have also copied the default inventory file /etc/kolla | 21:54 |
| bjolo | so when i run anything, i do: kolla-ansible -i /etc/kolla/multinode deploy | 21:54 |
| bjolo | you only do the pip install once on the master node | 21:55 |
| v1k0d3n | is the globals config correct? when i limited to three nodes it started complaining about ceph now... | 21:56 |
| bjolo | disable ceph as well | 21:56 |
| v1k0d3n | this is starting to get strange. | 21:56 |
| bjolo | not needed for basic deploy | 21:56 |
| v1k0d3n | so most of the stuff below isn't required right? ironic etc...? | 21:56 |
| bjolo | no | 21:57 |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: Add resource status command for Workflow Engine https://review.openstack.org/345102 | 21:57 |
| *** banix has quit IRC | 21:57 | |
| bjolo | v1k0d3n, the globals.conf you pasted has all you need. Just disable TLS in that one | 21:59 |
| *** jascott1_tmp has joined #openstack-kolla | 21:59 | |
| v1k0d3n | ok | 22:01 |
| v1k0d3n | thanks | 22:01 |
| v1k0d3n | running again. simple...one compute, one network, one controller. easy. | 22:02 |
| v1k0d3n | no TLS | 22:02 |
| v1k0d3n | going to be frustrated if this works and don't understand why wouldn't work with multi :( | 22:02 |
| bjolo | you are still runing multi | 22:03 |
| v1k0d3n | so true. sorry...with all i mean. | 22:03 |
| bjolo | it will still deploy haproxy etc, but only one of each type | 22:03 |
| v1k0d3n | think i may still get hung up on the mariadb issue. | 22:03 |
| bjolo | but here is the cool thing with kolla | 22:03 |
| bjolo | if this works | 22:03 |
| bjolo | then you can edit your inventory file and add one more controller etc | 22:04 |
| bjolo | and run the deploy again | 22:04 |
| v1k0d3n | yeah, so i think this is back to where i was at one point with mariadb... | 22:05 |
| v1k0d3n | think it's going to fail at: Waiting for MariaDB service to be ready through VIP | 22:05 |
| v1k0d3n | where mysql command should be issued to the container. | 22:05 |
| v1k0d3n | i think this container may be faulty during build. | 22:05 |
| v1k0d3n | and 2.0.1 is known to be good? | 22:06 |
| bjolo | yes, centos binary worked for me | 22:06 |
| v1k0d3n | i know for sure 2.0.0 works...wrote a blog about that. and i seem to recall the container being built at one point via 2.0.1 | 22:06 |
| v1k0d3n | ah, you're using centos binaries. i was using ubuntu source. | 22:07 |
| bjolo | i was. now i run centos source | 22:07 |
| v1k0d3n | and centos source containers work well for you? | 22:07 |
| bjolo | so far...im also pretty much starting out and just testing things | 22:08 |
| *** salv-orlando has joined #openstack-kolla | 22:08 | |
| v1k0d3n | i wonder if there some cruft from running a previous registry on the same build host...does removing a registry container also blow away the storage it was using previously? | 22:08 |
| *** tyrola has quit IRC | 22:09 | |
| *** jascott1_tmp has quit IRC | 22:10 | |
| bjolo | if you do a docker rm registry i believe so | 22:11 |
| bjolo | docker inspect registry | 22:11 |
| *** salv-orl_ has joined #openstack-kolla | 22:14 | |
| bjolo | but bump up your openstack_version in kolla-build.conf and globals.conf and do a new build | 22:17 |
| *** salv-orlando has quit IRC | 22:17 | |
| bjolo | i.e. change from 2.0.1.1 to 2.0.1.2 | 22:18 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Adds ansible support to deploy skydns service https://review.openstack.org/335677 | 22:19 |
| *** sdake_ has quit IRC | 22:24 | |
| *** tyrola has joined #openstack-kolla | 22:25 | |
| *** tyrola has quit IRC | 22:27 | |
| v1k0d3n | so that just tells kolla what to build and fetch from the repository? | 22:32 |
| bjolo | na it is just a version number | 22:33 |
| bjolo | further down in kolla-build.conf it actually specifies what sources to fetch | 22:33 |
| *** jascott1_tmp has joined #openstack-kolla | 22:34 | |
| bjolo | search for "location" | 22:34 |
| bjolo | kolla hardcodes a specific version for each project | 22:35 |
| v1k0d3n | getting worried...hanging again at the mariadb command again, and this is with a centos source deployment. | 22:35 |
| v1k0d3n | i don't think it's going to work again...wait and see though. | 22:36 |
| bjolo | in my kolla-build.conf i have changed most of them to <project>-stable-mitaka.tar.gz | 22:36 |
| *** gfidente|afk has quit IRC | 22:38 | |
| *** salv-orl_ has quit IRC | 22:41 | |
| *** salv-orlando has joined #openstack-kolla | 22:43 | |
| *** lyncos has quit IRC | 22:43 | |
| v1k0d3n | yeah, so failed...same spot | 22:44 |
| bjolo | geee whats going on | 22:44 |
| bjolo | lets take this from the top | 22:44 |
| bjolo | on your deployer node, what version of ansible and docker is installed | 22:45 |
| v1k0d3n | one sec. let me load something on git. gonna be around for another 15-20 minutes. this will answer a lot of questions. | 22:45 |
| bjolo | maybe... im in sweden so its alrady 1AM here | 22:47 |
| v1k0d3n | oh damn | 22:48 |
| v1k0d3n | well, you'll be around here tomorrow? | 22:48 |
| v1k0d3n | 1am...haha...that's like my sort of night! | 22:48 |
| bjolo | do your thing, if im still here i help | 22:48 |
| *** salv-orlando has quit IRC | 22:48 | |
| *** sdake has joined #openstack-kolla | 22:49 | |
| v1k0d3n | well, let me just answer the questions... | 22:49 |
| v1k0d3n | had issues with ansible 2.2, then 2.1. then noticed 2.0 tried that...nope...1.9.4 | 22:49 |
| v1k0d3n | so i'm on 1.9.4 right now | 22:49 |
| v1k0d3n | i have started building a repo that builds out hosts from scratch, loads prerequisites for kolla, and deploys the project. have drifted from that for troubleshooting obviously, but would want to eventually pin the right versions of packages once i figure out what is causing this issue. | 22:50 |
| v1k0d3n | in an AIO, it just worked. my blog http://www.jinkit.com/openstack-dockerized/ | 22:51 |
| v1k0d3n | needed to clean it up, and i want to have a repo i can refer to and just say "here...load, run, run kolla...done". | 22:51 |
| v1k0d3n | one thing that has drifted...i didn't do the wily kernel upgrade because i noticed the kernel is already 4.2 on these hosts. | 22:52 |
| *** banix has joined #openstack-kolla | 22:52 | |
| *** jascott1_tmp has quit IRC | 23:12 | |
| *** daneyon has joined #openstack-kolla | 23:18 | |
| *** daneyon has quit IRC | 23:23 | |
| *** chopmann has joined #openstack-kolla | 23:25 | |
| *** tyrola has joined #openstack-kolla | 23:39 | |
| *** tyrola has quit IRC | 23:43 | |
| *** jascott1_tmp has joined #openstack-kolla | 23:54 | |
| *** aernhart_ has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!