*** tyrola has joined #openstack-kolla | 00:07 | |
*** tyrola has quit IRC | 00:13 | |
*** ravig has quit IRC | 00:25 | |
*** daneyon_ has joined #openstack-kolla | 00:30 | |
*** daneyon_ has quit IRC | 00:34 | |
*** ssurana has joined #openstack-kolla | 00:41 | |
*** ravig has joined #openstack-kolla | 00:52 | |
*** ravig has quit IRC | 00:59 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: debug the publicURL endpoint for compute service https://review.openstack.org/327146 | 01:09 |
---|---|---|
*** ssurana has quit IRC | 01:15 | |
*** zhiwei has joined #openstack-kolla | 01:21 | |
*** tfukushima has joined #openstack-kolla | 01:24 | |
*** cfarquhar has quit IRC | 01:24 | |
*** cfarquhar has joined #openstack-kolla | 01:29 | |
*** cfarquhar has quit IRC | 01:29 | |
*** cfarquhar has joined #openstack-kolla | 01:29 | |
*** david-lyle has quit IRC | 01:34 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: debug the publicURL endpoint for compute service https://review.openstack.org/327146 | 01:36 |
*** tfukushima has quit IRC | 01:52 | |
*** rhallisey has quit IRC | 02:06 | |
*** ssurana has joined #openstack-kolla | 02:07 | |
*** ssurana has quit IRC | 02:13 | |
*** ssurana has joined #openstack-kolla | 02:13 | |
*** daneyon_ has joined #openstack-kolla | 02:18 | |
*** tfukushima has joined #openstack-kolla | 02:18 | |
*** daneyon_ has quit IRC | 02:23 | |
*** sacharya has joined #openstack-kolla | 02:27 | |
*** ravig has joined #openstack-kolla | 02:47 | |
*** ayoung has quit IRC | 02:48 | |
*** klint has joined #openstack-kolla | 02:53 | |
*** williamcaban has quit IRC | 02:58 | |
*** ravig has quit IRC | 03:12 | |
*** ravig has joined #openstack-kolla | 03:13 | |
*** coolsvap has joined #openstack-kolla | 03:20 | |
*** ravig has quit IRC | 03:48 | |
*** williamcaban has joined #openstack-kolla | 03:50 | |
*** ravig has joined #openstack-kolla | 03:52 | |
*** ravig has quit IRC | 03:55 | |
*** williamcaban has quit IRC | 03:56 | |
*** ravig has joined #openstack-kolla | 03:56 | |
*** ravig has quit IRC | 04:03 | |
*** daneyon_ has joined #openstack-kolla | 04:07 | |
*** ssurana has quit IRC | 04:09 | |
*** ssurana has joined #openstack-kolla | 04:10 | |
*** sacharya has quit IRC | 04:10 | |
*** ssurana has quit IRC | 04:10 | |
*** ravig has joined #openstack-kolla | 04:10 | |
*** daneyon_ has quit IRC | 04:11 | |
*** ssurana has joined #openstack-kolla | 04:12 | |
*** ssurana has quit IRC | 04:14 | |
*** ssurana has joined #openstack-kolla | 04:14 | |
*** ssurana has quit IRC | 04:23 | |
*** ssurana has joined #openstack-kolla | 04:23 | |
*** ssurana has quit IRC | 04:23 | |
*** tfukushima has quit IRC | 05:10 | |
*** salv-orlando has joined #openstack-kolla | 05:21 | |
*** salv-orl_ has quit IRC | 05:21 | |
*** tfukushima has joined #openstack-kolla | 05:21 | |
*** dave-mccowan has quit IRC | 05:27 | |
*** salv-orl_ has joined #openstack-kolla | 05:29 | |
*** salv-orlando has quit IRC | 05:31 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: debug the publicURL endpoint for compute service https://review.openstack.org/327146 | 06:05 |
mdnadeem_ | coolsvap, I face an issue with nova nova-novncproxy container, the command nova-novncproxy gets fail with the <WARNING: no 'numpy' module, HyBi protocol will be slower> | 06:13 |
coolsvap | mdnadeem_, it fails with warning? | 06:14 |
mdnadeem_ | yes | 06:14 |
coolsvap | mdnadeem_, not sure atm, need to verify | 06:15 |
mdnadeem_ | coolsvap, here is log http://pastebin.com/qTPnYTTH | 06:15 |
mdnadeem_ | coolsvap, nova.conf >> http://pastebin.com/D3Qvt4r4 | 06:15 |
coolsvap | mdnadeem_, i will check and get back to you on this, currently occupied with something else | 06:19 |
mdnadeem_ | coolsvap, Thanks | 06:20 |
*** ravig has quit IRC | 06:23 | |
*** daneyon_ has joined #openstack-kolla | 06:49 | |
*** daneyon_ has quit IRC | 06:54 | |
mdnadeem_ | coolsvap, Here is the traceback logs for nova-novncproxy container : http://pastebin.com/yL0dMQ38, please check once get free | 07:01 |
mdnadeem_ | >> http://pastebin.com/yL0dMQ38 | 07:01 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH https://review.openstack.org/294906 | 07:03 |
*** salv-orl_ has quit IRC | 07:07 | |
*** salv-orlando has joined #openstack-kolla | 07:08 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the openstack_release variable in all.yml https://review.openstack.org/328873 | 07:08 |
*** Jeffrey4l has joined #openstack-kolla | 07:09 | |
*** matrohon has joined #openstack-kolla | 07:16 | |
*** sbezverk_ has joined #openstack-kolla | 07:16 | |
*** sbezverk has quit IRC | 07:17 | |
*** athomas has joined #openstack-kolla | 07:25 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the novnc configure in the nova.conf https://review.openstack.org/328882 | 07:38 |
*** mikelk has joined #openstack-kolla | 07:38 | |
*** mikelk has quit IRC | 07:41 | |
*** jamie_h has joined #openstack-kolla | 07:42 | |
*** mikelk has joined #openstack-kolla | 07:47 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Remove the duplication of the inventory https://review.openstack.org/328891 | 08:06 |
*** tyrola has joined #openstack-kolla | 08:17 | |
*** shardy has joined #openstack-kolla | 08:18 | |
*** shardy has quit IRC | 08:21 | |
*** salv-orlando has quit IRC | 08:23 | |
*** salv-orlando has joined #openstack-kolla | 08:24 | |
*** Serlex has joined #openstack-kolla | 08:34 | |
*** daneyon_ has joined #openstack-kolla | 08:37 | |
*** daneyon_ has quit IRC | 08:42 | |
*** dmk0202 has joined #openstack-kolla | 08:44 | |
*** tfukushima has quit IRC | 08:50 | |
*** tfukushima has joined #openstack-kolla | 08:51 | |
*** fragatin_ has quit IRC | 08:52 | |
*** SiRiuS__ has joined #openstack-kolla | 09:12 | |
*** jmccarthy has quit IRC | 10:04 | |
*** athomas has quit IRC | 10:10 | |
*** pbourke has quit IRC | 10:14 | |
*** pbourke has joined #openstack-kolla | 10:15 | |
*** athomas has joined #openstack-kolla | 10:17 | |
*** jmccarthy has joined #openstack-kolla | 10:18 | |
openstackgerrit | Merged openstack/kolla: Fix the openstack_release variable in all.yml https://review.openstack.org/328873 | 10:20 |
*** daneyon_ has joined #openstack-kolla | 10:26 | |
*** daneyon_ has quit IRC | 10:30 | |
*** mikelk has quit IRC | 10:41 | |
*** tfukushima has quit IRC | 10:41 | |
openstackgerrit | Tuan proposed openstack/kolla: Fix typos in Kolla https://review.openstack.org/328943 | 10:48 |
*** alyson_ has joined #openstack-kolla | 10:56 | |
mdnadeem_ | pbourke, Can you please re-visit this ps : https://review.openstack.org/#/c/285138/ | 11:00 |
patchbot | mdnadeem_: patch 285138 - kolla - Added unit test for kolla_docker | 11:00 |
*** williamcaban has joined #openstack-kolla | 11:02 | |
mdnadeem_ | pbourke, Thanks :) | 11:03 |
pbourke | np | 11:03 |
*** mikelk has joined #openstack-kolla | 11:04 | |
openstackgerrit | Merged openstack/kolla: Added unit test for kolla_docker https://review.openstack.org/285138 | 11:05 |
*** alyson_ has left #openstack-kolla | 11:06 | |
*** salv-orlando has quit IRC | 11:07 | |
*** salv-orlando has joined #openstack-kolla | 11:08 | |
*** williamcaban has quit IRC | 11:11 | |
*** rhallisey has joined #openstack-kolla | 11:25 | |
openstackgerrit | Md Nadeem proposed openstack/kolla: Added test document https://review.openstack.org/328069 | 11:26 |
*** salv-orl_ has joined #openstack-kolla | 11:28 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: debug the publicURL endpoint for compute service https://review.openstack.org/327146 | 11:31 |
*** salv-orlando has quit IRC | 11:32 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the rabbitmq crash on CentOS binary and source deploy https://review.openstack.org/328960 | 11:38 |
*** williamcaban has joined #openstack-kolla | 11:40 | |
*** mliima has joined #openstack-kolla | 11:47 | |
*** Jeffrey4l has quit IRC | 11:48 | |
*** Jeffrey4l has joined #openstack-kolla | 11:49 | |
*** williamcaban has quit IRC | 12:00 | |
*** coolsvap has quit IRC | 12:25 | |
*** prithiv has joined #openstack-kolla | 12:31 | |
*** ppowell has joined #openstack-kolla | 12:49 | |
*** tyrola has quit IRC | 12:54 | |
*** stvnoyes has quit IRC | 12:59 | |
*** stvnoyes has joined #openstack-kolla | 13:00 | |
*** klint has quit IRC | 13:00 | |
*** sgordon` has joined #openstack-kolla | 13:01 | |
*** dave-mccowan has joined #openstack-kolla | 13:03 | |
*** daneyon_ has joined #openstack-kolla | 13:08 | |
*** ayoung has joined #openstack-kolla | 13:09 | |
*** daneyon_ has quit IRC | 13:12 | |
*** jtriley has joined #openstack-kolla | 13:16 | |
*** diogogmt has quit IRC | 13:26 | |
*** prithiv has quit IRC | 13:27 | |
*** prithiv has joined #openstack-kolla | 13:29 | |
*** ravig has joined #openstack-kolla | 13:30 | |
rhallisey | wirehead_, when you're around, I got the keystone bootstrap working | 13:37 |
*** diogogmt has joined #openstack-kolla | 13:39 | |
*** ravig has quit IRC | 13:40 | |
*** prithiv has quit IRC | 13:51 | |
*** prithiv has joined #openstack-kolla | 13:52 | |
*** diogogmt has quit IRC | 13:53 | |
*** Liuqing has joined #openstack-kolla | 13:58 | |
*** diogogmt has joined #openstack-kolla | 14:02 | |
openstackgerrit | Patrick Powell proposed openstack/kolla: The quickstart guide still says to use Anisble 1.9.4 which no longer works with Kolla. https://review.openstack.org/329015 | 14:02 |
*** SiRiuS__ has quit IRC | 14:03 | |
*** SiRiuS__ has joined #openstack-kolla | 14:04 | |
wirehead_ | rhallisey: Awesome. | 14:05 |
rhallisey | wirehead_, I left a comment | 14:06 |
rhallisey | I'll show you what it should look like in a patch to glance | 14:06 |
openstackgerrit | Merged openstack/kolla: Fix typos in Kolla https://review.openstack.org/328943 | 14:07 |
rhallisey | wirehead_, unless you'd rather I push what I have | 14:08 |
*** prithiv has quit IRC | 14:10 | |
*** prithiv has joined #openstack-kolla | 14:10 | |
*** jtriley has quit IRC | 14:12 | |
*** diogogmt has quit IRC | 14:12 | |
*** dwalsh has joined #openstack-kolla | 14:12 | |
wirehead_ | rhallisey, so I just needed to execute that in a shell? | 14:15 |
wirehead_ | rhallisey, or was there more? | 14:15 |
rhallisey | that's all. There's a format you need to follow | 14:16 |
wirehead_ | rhallisey: OK, whichever works easier for you. | 14:16 |
wirehead_ | rhallisey: I was biking up a mountain when you asked how far I was. | 14:17 |
rhallisey | I changed to format a little to make it look neater | 14:17 |
rhallisey | ha | 14:17 |
openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Convert Keystone to HostNetwork=False, Fix 2/3rds of bootstrap https://review.openstack.org/328567 | 14:20 |
rhallisey | wirehead_, try that out^ | 14:21 |
openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Convert Keystone to HostNetwork=False, Fix 2/3rds of bootstrap https://review.openstack.org/328567 | 14:22 |
*** jtriley has joined #openstack-kolla | 14:26 | |
*** inc0 has joined #openstack-kolla | 14:37 | |
inc0 | good morning | 14:37 |
rhallisey | inc0, hey :) | 14:39 |
*** sacharya has joined #openstack-kolla | 14:47 | |
*** inc0 has quit IRC | 14:51 | |
*** esharao has joined #openstack-kolla | 14:53 | |
openstackgerrit | Patrick Powell proposed openstack/kolla: The quickstart guide still says to use Anisble 1.9.4 https://review.openstack.org/329015 | 14:54 |
*** inc0 has joined #openstack-kolla | 14:59 | |
*** jtriley has quit IRC | 15:00 | |
*** Liuqing has quit IRC | 15:00 | |
wirehead_ | rhallisey: so, I was thinking that, given that the three steps of the workflow are somewhat serialized, there's not nearly the same benefit on putting them as three individual containers. Might be easier to just join them with a && | 15:02 |
*** diogogmt has joined #openstack-kolla | 15:02 | |
openstackgerrit | Ryan Hallisey proposed openstack/kolla: Change the search path to look locally https://review.openstack.org/329044 | 15:04 |
*** coolsvap has joined #openstack-kolla | 15:05 | |
rhallisey | wirehead_, might make it hard to read | 15:05 |
rhallisey | the cost of 3 containers isn't really much | 15:05 |
*** fragatina has joined #openstack-kolla | 15:11 | |
*** david-lyle has joined #openstack-kolla | 15:12 | |
*** ravig has joined #openstack-kolla | 15:17 | |
*** vhosakot has joined #openstack-kolla | 15:17 | |
*** jtriley has joined #openstack-kolla | 15:21 | |
openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Look in kolla-kubernetes directory instead of kolla https://review.openstack.org/329058 | 15:30 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Get package from cdn rather than github https://review.openstack.org/329060 | 15:34 |
*** harlowja_at_home has joined #openstack-kolla | 15:35 | |
*** prithiv has quit IRC | 15:36 | |
*** mikelk has quit IRC | 15:37 | |
*** jtriley has quit IRC | 15:39 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Get packages from cdn rather than github https://review.openstack.org/329060 | 15:41 |
*** gmmaha has quit IRC | 15:44 | |
*** mark-casey has joined #openstack-kolla | 15:45 | |
*** gmmaha has joined #openstack-kolla | 15:45 | |
*** gmmaha has quit IRC | 15:46 | |
*** gmmaha has joined #openstack-kolla | 15:47 | |
*** vhosakot has quit IRC | 15:49 | |
*** vhosakot has joined #openstack-kolla | 15:49 | |
*** daneyon_ has joined #openstack-kolla | 15:50 | |
mag009_ | busy morning here :P | 15:51 |
inc0 | mag009_, oh? | 15:51 |
mag009_ | morning y'all | 15:51 |
*** dmk0202 has quit IRC | 15:51 | |
inc0 | hey | 15:51 |
*** sacharya_ has joined #openstack-kolla | 15:55 | |
*** daneyon_ has quit IRC | 15:55 | |
inc0 | mag009_, here goes nothing, trying to deploy your calico;) | 15:57 |
mag009_ | lol cool | 15:57 |
*** sacharya has quit IRC | 15:58 | |
*** esharao has quit IRC | 15:58 | |
mag009_ | multinode? | 15:58 |
mag009_ | I haven't tested in single | 15:58 |
inc0 | wouldn't have it any other way | 15:58 |
*** esharao has joined #openstack-kolla | 15:59 | |
inc0 | would there be any point of testing it on single node? | 15:59 |
inc0 | one network tho | 15:59 |
inc0 | I mean no switches in between | 15:59 |
mag009_ | not really but technically it should work | 15:59 |
inc0 | well you wouldnt use bgp at all | 16:00 |
inc0 | that being said..I don't use bgp too | 16:00 |
inc0 | not in full in any case | 16:00 |
inc0 | anyway, lets see | 16:00 |
inc0 | we can go fancy and write a heat template later to setup calico with kolla on multiple vms connected by cumulus vx;) | 16:01 |
mag009_ | yep | 16:03 |
*** ayoung has quit IRC | 16:04 | |
*** jtriley has joined #openstack-kolla | 16:06 | |
openstackgerrit | Patrick Powell proposed openstack/kolla: The quickstart guide still says to use Anisble 1.9.4 https://review.openstack.org/329015 | 16:08 |
*** prithiv has joined #openstack-kolla | 16:08 | |
*** matrohon has quit IRC | 16:08 | |
*** jtriley has quit IRC | 16:12 | |
Lyncos | good morning guys | 16:15 |
*** zhiwei has quit IRC | 16:17 | |
*** prithiv has quit IRC | 16:19 | |
*** ayoung has joined #openstack-kolla | 16:19 | |
inc0 | hey | 16:22 |
*** matrohon has joined #openstack-kolla | 16:22 | |
vhosakot | hello all! | 16:23 |
mark-casey | hello | 16:24 |
openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Check for a local kolla directory https://review.openstack.org/329090 | 16:24 |
*** salv-orl_ has quit IRC | 16:25 | |
*** salv-orlando has joined #openstack-kolla | 16:25 | |
Mech422 | morning | 16:29 |
*** jtriley has joined #openstack-kolla | 16:30 | |
*** mummer has joined #openstack-kolla | 16:37 | |
*** jtriley has quit IRC | 16:42 | |
*** athomas has quit IRC | 16:43 | |
*** harlowja_at_home has quit IRC | 16:59 | |
*** ssurana has joined #openstack-kolla | 17:00 | |
*** Serlex has quit IRC | 17:04 | |
*** ayoung has quit IRC | 17:04 | |
*** fragatina has quit IRC | 17:06 | |
*** fragatina has joined #openstack-kolla | 17:06 | |
*** sacharya has joined #openstack-kolla | 17:13 | |
inc0 | harlowja, mag009_ Lyncos soo, it seems you all will try to deploy kolla on 1k+ nodes | 17:13 |
*** sacharya_ has quit IRC | 17:13 | |
Lyncos | yeah eventually 1k | 17:14 |
inc0 | just saying, talk to each other if needs be;) we need to start thinking about this cell stuff | 17:14 |
inc0 | support in kolla | 17:14 |
*** ravig has quit IRC | 17:15 | |
*** sacharya_ has joined #openstack-kolla | 17:16 | |
*** ayoung has joined #openstack-kolla | 17:18 | |
*** sacharya has quit IRC | 17:19 | |
*** jtriley has joined #openstack-kolla | 17:19 | |
*** ssurana has left #openstack-kolla | 17:24 | |
*** jamie_h has quit IRC | 17:27 | |
*** ssurana has joined #openstack-kolla | 17:28 | |
*** salv-orl_ has joined #openstack-kolla | 17:29 | |
*** ssurana has left #openstack-kolla | 17:29 | |
*** jtriley has quit IRC | 17:30 | |
*** salv-orl_ has quit IRC | 17:32 | |
*** salv-orlando has quit IRC | 17:32 | |
*** salv-orlando has joined #openstack-kolla | 17:32 | |
*** vhosakot has quit IRC | 17:36 | |
*** vhosakot has joined #openstack-kolla | 17:41 | |
harlowja | ya, sadly cells in nova is in weird state (afaik) | 17:42 |
*** ravig has joined #openstack-kolla | 17:44 | |
*** Mech422 has quit IRC | 17:45 | |
*** Mech422 has joined #openstack-kolla | 17:46 | |
Mech422 | Hmm - I get a wierd error with mitaka where I can't launch VM instances - the 'provisioning block device' spinner never goes away | 17:47 |
Mech422 | but the volume show up via the 'volumes' tab and api | 17:47 |
Mech422 | it's wierd - only seems to happen on like 50% of my re-builds | 17:47 |
Mech422 | (cluster rebuilds/redeploys that is) | 17:48 |
*** ravig has quit IRC | 17:49 | |
*** jtriley has joined #openstack-kolla | 17:49 | |
openstackgerrit | Mauricio Lima proposed openstack/kolla: Remove Heat CLI deprecated commands https://review.openstack.org/329125 | 17:51 |
openstackgerrit | David Wang proposed openstack/kolla: Enable kolla k8s to override bind api bind address in genconfig https://review.openstack.org/327925 | 17:51 |
wirehead_ | Every open-stack project, like Green Day and The Offspring, must eventually learn that being a cell-out is inevitable. | 17:53 |
*** fragatina has quit IRC | 17:53 | |
*** ravig has joined #openstack-kolla | 17:54 | |
*** mbound has joined #openstack-kolla | 17:55 | |
*** dwalsh has quit IRC | 17:55 | |
*** coolsvap has quit IRC | 17:56 | |
*** coolsvap has joined #openstack-kolla | 17:57 | |
Mech422 | wirehead_: *groan* | 17:59 |
*** fragatina has joined #openstack-kolla | 18:00 | |
Mech422 | inc0: so, Jeffrey4l pointed out we can get around the ceph-stale-partition names stuff by calling 'udevadm trigger' after re-writing the partition names | 18:04 |
Mech422 | inc0: which would get around your dis-taste for scraping command output | 18:04 |
Mech422 | inc0: (Not sure I have 100% faith in it doing the right thing - but I guess that would be a udev problem if it fails?) | 18:05 |
*** fragatina has quit IRC | 18:06 | |
*** dwalsh has joined #openstack-kolla | 18:07 | |
*** ravig has quit IRC | 18:09 | |
*** coolsvap has quit IRC | 18:13 | |
*** Jeffrey4l has quit IRC | 18:17 | |
*** mbound has quit IRC | 18:19 | |
*** mbound has joined #openstack-kolla | 18:22 | |
*** sdake_ has joined #openstack-kolla | 18:23 | |
sdake_ | morning | 18:24 |
sean-k-mooney | sdake_: good morning | 18:24 |
*** mbound has quit IRC | 18:25 | |
*** mbound has joined #openstack-kolla | 18:26 | |
*** berendt has joined #openstack-kolla | 18:33 | |
*** daneyon_ has joined #openstack-kolla | 18:33 | |
openstackgerrit | Christian Berendt proposed openstack/kolla: Load the metadata definitions during Glance bootstrap https://review.openstack.org/322415 | 18:37 |
*** daneyon_ has quit IRC | 18:37 | |
*** fragatina has joined #openstack-kolla | 18:39 | |
*** ravig has joined #openstack-kolla | 18:40 | |
*** ravig has quit IRC | 18:45 | |
*** ravig has joined #openstack-kolla | 18:47 | |
*** jtriley has quit IRC | 18:51 | |
*** jtriley has joined #openstack-kolla | 18:59 | |
*** The_Ball has quit IRC | 18:59 | |
*** salv-orlando has quit IRC | 19:04 | |
*** salv-orlando has joined #openstack-kolla | 19:05 | |
sean-k-mooney | hi not sure who is about but does anyone have any guidence on how to do hostlevel configuration with kolla? | 19:06 |
sean-k-mooney | im think in relation to https://blueprints.launchpad.net/kolla/+spec/kernel-hugepage-config | 19:07 |
sean-k-mooney | i see too options to configuring kernel hugepages on the host. | 19:07 |
sean-k-mooney | 1 the common playbook | 19:07 |
sean-k-mooney | 2 the install-deps playbook i am also working on | 19:08 |
sean-k-mooney | is there any preference? | 19:08 |
sean-k-mooney | as the install-deps playbook is intended to be optional that approch would involve documenting how to configur it manually if it is not used. | 19:10 |
sean-k-mooney | sdake_: inc0: any taughts ? | 19:12 |
inc0 | just came back | 19:15 |
inc0 | reading log | 19:15 |
inc0 | sean-k-mooney, preconf of host is not something we normally do\ | 19:16 |
sean-k-mooney | inc0: no worries | 19:16 |
inc0 | but I'm ok with it as long as it will be separate play | 19:16 |
sean-k-mooney | ok | 19:17 |
inc0 | something like kolla-ansible prep-host or whatever | 19:17 |
sean-k-mooney | i think in that case then ill ammend the blueprint to suggest the install-deps playbook instead of common then | 19:17 |
sean-k-mooney | strictly speaking configuring hugepages can be done by the operator today before running kolla but i would like to provide a automated solution too | 19:19 |
inc0 | sean-k-mooney, so our current policy is "do not mess with host" | 19:20 |
sean-k-mooney | yes in the main playbooks | 19:21 |
inc0 | but having easier prep-host playbook would be valuable | 19:21 |
inc0 | yeah, that's my point, if you're up to prep playbook "this is host messing playbook", I'm ok with it | 19:21 |
sean-k-mooney | ya i started on this https://review.openstack.org/#/c/325631/ | 19:21 |
patchbot | sean-k-mooney: patch 325631 - kolla - [WIP] add baremetal role to install kolla deps. | 19:21 |
*** The_Ball has joined #openstack-kolla | 19:22 | |
inc0 | ahh I missed this one | 19:22 |
inc0 | yeah, just plz make it separate play | 19:22 |
inc0 | instead of role | 19:22 |
sean-k-mooney | the intent being that it will do all of the host level config to pass the pre checks | 19:22 |
sean-k-mooney | ah ok you would like it to be an entirly different playbook | 19:22 |
sean-k-mooney | i can do that | 19:23 |
inc0 | yeah | 19:23 |
inc0 | it will prep to pass the prechecks and also I'm ok with doing optimization stuff | 19:23 |
inc0 | like this kernel thing | 19:23 |
inc0 | it's like "if you run this play, we do stuff to your host." | 19:23 |
sean-k-mooney | ya so what i was thinking of doing was documenting how to configure hugpages in one of the deployment guide then automating it as part of the install-deps playbook | 19:24 |
mark-casey | inc0: can the baremetal prep be a play that calls a role like cleanup or mariadb_recovery? because otherwise where do the templates in the current change-set go? | 19:26 |
inc0 | so as far as I'm concerned, just name it correctly in task | 19:26 |
sean-k-mooney | oh hi mark-casey just reading your comments now | 19:27 |
inc0 | well, you can add new role for baremetal | 19:27 |
inc0 | it just wont be in normal playbook | 19:28 |
mark-casey | inc0: right, ok. yes we're thinking the same I think | 19:28 |
mark-casey | sean-k-mooney: hello :) | 19:28 |
sean-k-mooney | am yes what ill do is refactor the current patchset to create a new playbook with a new role | 19:29 |
mark-casey | sean-k-mooney: cool. also regarding kernel hugepages, any change for an explicit "off" setting? hugepages (or at least transparent ones) are fairly highly recommended against for applications like redis and mongo if you don't have large amounts of ram | 19:30 |
sean-k-mooney | mark-casey: sure can you add that to https://blueprints.launchpad.net/kolla/+spec/kernel-hugepage-config so that i dont forget | 19:31 |
mark-casey | or.. wait. maybe this doesn't get so far as inside the VMs. I may be off base | 19:31 |
*** salv-orlando has quit IRC | 19:31 | |
sean-k-mooney | yes this dose not effect inside the guest | 19:31 |
*** salv-orlando has joined #openstack-kolla | 19:31 | |
sean-k-mooney | it would just be allocating them on the host | 19:31 |
sean-k-mooney | you may want to disable transparent hugepages on the host though for some reason | 19:32 |
sean-k-mooney | e.g. mongo does not like them? | 19:32 |
inc0 | huge pages will also affect LM right? | 19:35 |
inc0 | live migration | 19:35 |
mark-casey | yea I do inside those hosts. I'll add that to the bp in case someone does want that, but I don't think it matters to me | 19:35 |
mark-casey | inc0: yea i think there's been a nova thread on that in the ML | 19:35 |
mark-casey | or, someone trying to get certain page sizes on certain instances of nova | 19:36 |
sean-k-mooney | inc0: if you have hugepaged backed guest you have to livemigrate to another host with hugepage memory or it will fail | 19:36 |
sean-k-mooney | inc0: other then that i dont belive there is a restriction | 19:36 |
mark-casey | sean-k-mooney: what are the upsides in this case? just compute nodes with tons of ram? | 19:36 |
inc0 | sean-k-mooney, restriction - no, but performance will be affected | 19:37 |
inc0 | LM has to copy dirtied pages | 19:37 |
inc0 | so if you have small number of large pages, there will be more dirtying and copying | 19:37 |
sean-k-mooney | am hugepages would be required to enable ovs with dpdk. its also is a prefromanc boost to the guest | 19:37 |
sean-k-mooney | mark-casey: ^^ | 19:38 |
inc0 | all I'm saying, it'll be tradeoff | 19:38 |
inc0 | so an idea | 19:38 |
inc0 | instead of setting it up | 19:38 |
inc0 | add check in prechecs | 19:38 |
*** dwalsh has quit IRC | 19:39 | |
inc0 | if enable_dpdk: yes, then check if hugepages are on | 19:39 |
inc0 | and leave kernel config to ops | 19:39 |
inc0 | Mech422, you were one having perm denied for ceph? | 19:40 |
sean-k-mooney | well i could do both i guess? since it wont be part of the main playbook i can add a precheck for that | 19:40 |
inc0 | or was it mag009_ ? | 19:40 |
sean-k-mooney | inc0: in the install-dpes playbook it would be optional and disabled by default so ops would not have to use it if they dont want too | 19:40 |
*** mliima has quit IRC | 19:41 | |
mark-casey | sean-k-mooney: cool. def wanting to investigate dpdk here | 19:43 |
mark-casey | inc0: I think at some level of ram (128gb+?) not having hugepages degrades performance cause the TLB gets unwieldy | 19:44 |
mark-casey | too many pages to track | 19:44 |
sean-k-mooney | mark-casey: ya its been on my todo list for far too long at this point. | 19:44 |
sean-k-mooney | you will see performace degrdates vs with hugepages well before that for your guest but yes if you have hosts with lots of ram hugepages are become more important overall | 19:45 |
*** esharao_ has joined #openstack-kolla | 19:47 | |
*** dwalsh has joined #openstack-kolla | 19:50 | |
*** esharao has quit IRC | 19:50 | |
openstackgerrit | Ken Wronkiewicz proposed openstack/kolla: Keystone interface address and memcached override https://review.openstack.org/328569 | 19:55 |
inc0 | mag009_, Mech422 one of you had problems with permission denied during ceph deploy right? | 19:57 |
inc0 | ansible did change sth | 19:57 |
Mech422 | inc0: umm - about 50% of the time, Horizon seems to 'lose' the completion message when provisioning a new volume | 19:57 |
Mech422 | inc0: Not sure if its a perms problem though | 19:58 |
inc0 | not that | 19:58 |
inc0 | during deploy ansible raises perm denied | 19:59 |
Mech422 | inc0: no - it just doesn't see the partition renames | 19:59 |
*** esmiurium has quit IRC | 19:59 | |
mag009_ | inc0: yes | 20:00 |
Mech422 | inc0: bootstrap_osd rewrites the partition names, but the kernel/udev doesn't update, so start_osd sees 'stale' partition names and thinks there's nothing to do | 20:00 |
mag009_ | it was me who repported it | 20:00 |
mag009_ | at the boostrap it tried to modify the fstab | 20:01 |
mag009_ | first it tried to create /var/lib/ceph | 20:01 |
mag009_ | why you ask ? | 20:01 |
inc0 | I just ran into it | 20:02 |
inc0 | added become=true to inventory and still no bueno | 20:03 |
mag009_ | not there | 20:03 |
mag009_ | roles/ceph/tasks/start_osd.yml | 20:04 |
mag009_ | at the mount | 20:04 |
Mech422 | oh - I added passwordless root sudo to my 'kolla' user on deployment host, and it seems to work fine | 20:04 |
mag009_ | Mech422: your using ansible 2.X ? | 20:04 |
Mech422 | start_osd works fine for me ATM - yeah ansible 2.1 I think? | 20:04 |
inc0 | I use ansible_user=root | 20:05 |
Mech422 | I think I had to fart around with the ansible_version though | 20:05 |
sdake_ | morning folks | 20:05 |
sdake_ | or afternoon I suppose | 20:05 |
sdake_ | does anyone know if v2 and v3 of keystone can coexist in one deployment? | 20:05 |
inc0 | sdake_, it can.. | 20:05 |
sdake_ | for example, I want v3 of heat and magnum | 20:06 |
inc0 | it usually does | 20:06 |
sdake_ | but v2 of everything else | 20:06 |
mag009_ | inc0: i have a user ubuntu and passwordless in group sudo but i still had to add this become=true at the mount to get pass that permission denied | 20:06 |
inc0 | it's not that | 20:06 |
inc0 | it's "Looking up disks" | 20:06 |
mag009_ | different error ? | 20:06 |
sdake_ | inc0 are there any gotchas? | 20:06 |
*** sdake_ is now known as sdake | 20:06 | |
mag009_ | ah ok | 20:06 |
inc0 | so from within container | 20:06 |
Mech422 | mag009_: odd - I don't think I had to do anything for that - my fstab does have a UUID mount for /var/lib/ceph/osd | 20:07 |
sdake | inc0 do yu end up with separate openrc files for accessing magnum in teh above use case vs say nova (which would be on keystone v2) | 20:07 |
inc0 | I suppose drop root stuff messed this up | 20:08 |
sdake | inc0 and does nova, cinder, and other heat dependencies require v3 if ehat is on v3? | 20:08 |
sdake | inc0 i'd take an answer of "I dont know" over a guess :) | 20:08 |
Mech422 | inc0: the first play in start_osds.yml is (-m find_disks) blows up with stale partition data... | 20:09 |
Mech422 | inc0: you sure you have a perm error ? | 20:09 |
sean-k-mooney | sdake: i dont know the answer to your question but how would you configure different keystone versions today? in the config of each of the services directly? | 20:09 |
sdake | sean-k-mooney this is for our cloud services cloud at cisco (an internal cloud providere) | 20:10 |
inc0 | pretty sure ;) OSError: [Errno 13] Permission denied: '/home/ansible/.ansible | 20:10 |
sdake | sean-k-mooney and they are not on kolla at all | 20:10 |
sean-k-mooney | sdake: oh ok | 20:10 |
inc0 | however I see changed version in ansible toolbox | 20:10 |
sean-k-mooney | sdake: so that was a general query not kolla specific | 20:11 |
inc0 | 2 days ago | 20:11 |
openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Add documentation for api_interface_address https://review.openstack.org/327961 | 20:11 |
inc0 | and lookup is called from within container | 20:11 |
inc0 | can somebody do full rebuild and try to deploy ceph please? | 20:11 |
Mech422 | inc0: I did about 8 hours ago - deployed fine | 20:11 |
inc0 | did you rebuild toolbox from master? | 20:11 |
Mech422 | inc0: yeah | 20:11 |
inc0 | hmm | 20:12 |
Mech422 | inc0: but I include my ceph-in-partition patch | 20:12 |
inc0 | can I have link plz? | 20:12 |
Mech422 | inc0: that adds a sudoers entry for sgdisk though - so it shouldnt affect anything | 20:12 |
sean-k-mooney | Mech422: ceph-in-partition patch e.g. using partitions for OSDs? | 20:13 |
inc0 | actually it might | 20:13 |
inc0 | sean-k-mooney, yeah | 20:13 |
Mech422 | inc0: https://review.openstack.org/#/c/326609/4 | 20:13 |
patchbot | Mech422: patch 326609 - kolla - Read partition tables directly, bypassing stale ke... | 20:13 |
sean-k-mooney | ok cool that would be useful. | 20:13 |
inc0 | -2 from Jeffrey tho, let me see | 20:14 |
Mech422 | inc0: there's crap in your scroll back today about this too - Jeffrey determined we can run 'udevadm trigger' instead of using sgdisk | 20:14 |
Mech422 | inc0: and pbourke said something about it might be related to a problem he was having... | 20:15 |
Mech422 | but I haven't seen him since | 20:15 |
sdake | sean-k-mooney ya general query | 20:15 |
sdake | sean-k-mooney do you know? | 20:15 |
sdake | sean-k-mooney note I'd like to convert these folks over to kolla, but its brownfield | 20:15 |
sdake | and thats harder then greenfield :) | 20:15 |
sdake | sean-k-mooney inc0 an answer of "I dont know" is suitable if you dont' know, if you do know, a confirmation would be great | 20:16 |
Mech422 | sean-k-mooney: yeah - sam yapple put everything in place to run ceph on partitons - its just the kernel gets bitchy about updating the partition table data | 20:16 |
inc0 | sdake, they always coexist in same install | 20:17 |
inc0 | try keystone_host:5000/v2 and v3 | 20:17 |
inc0 | both should be there | 20:17 |
inc0 | Mech422, didn't help | 20:18 |
inc0 | http://paste.fedoraproject.org/378700/49077146/ | 20:18 |
openstackgerrit | Merged openstack/kolla-kubernetes: Accidently added a file on keystone merge https://review.openstack.org/328761 | 20:18 |
sean-k-mooney | Mech422: ya the first time i tried to deploy ceph with kolla was before that and i tred to use a partition on the same disk as my boot partition | 20:19 |
inc0 | hmm I think I see the issue | 20:19 |
sean-k-mooney | Mech422: i was quickly reminded of the fact that linux will hapily nuke the partion table on the disk it is running form. havent tried it since | 20:20 |
Mech422 | sean-k-mooney: actually - since Sam re-writes partition name on full disk - it could be an isssue there too - its just unlikely as the disk probably isn't in-use/mounted | 20:20 |
sdake | inc0 so openrc file specifies the keystone endpoint | 20:24 |
sdake | inc0 wuld you needd separate openrc files for communicating with heat vs nova in my use case? | 20:24 |
sdake | inc0 keeep in mind this is liberty version of openstack that is hand deployed | 20:25 |
sdake | rather hand upgraded | 20:25 |
sdake | i htink they recently upgraded from kilo | 20:25 |
sdake | like 6 mo ago | 20:25 |
inc0 | yeah different openrc | 20:25 |
inc0 | should do the trick | 20:26 |
sdake | so heat uses nova | 20:26 |
sean-k-mooney | Mech422: if i want to use a partition for an osd wont this still zap the entire disk the partion is on https://github.com/openstack/kolla/blob/master/docker/ceph/ceph-osd/extend_start.sh#L22 | 20:26 |
sdake | will it just use the v2 endpoint then? | 20:26 |
sdake | because heat only knwos the v3 endpoint, will it look up the v2 endpoint? | 20:26 |
inc0 | no, use different openrc to hear | 20:26 |
inc0 | heat | 20:26 |
Mech422 | sean-k-mooney: there's a guard in there...use seperate journal and it won't zap | 20:27 |
inc0 | or use v3 for everyone | 20:27 |
sdake | inc0 they dont want t o move everythign to 3 | 20:27 |
sdake | jusst the bar eminimum required to get magnum to wokr | 20:27 |
inc0 | then use separate openrc for magnum | 20:27 |
sdake | ok so magnum calls heat with v3 credentials, then heat calls nova with v3 credentials, but nova only has v2 | 20:28 |
sean-k-mooney | Mech422: true but if i dont use the external journal and just spcify a partition do we do any checks to make sure it does not zap every thing? | 20:28 |
sdake | this is where my brain geets stuck | 20:28 |
sean-k-mooney | Mech422: the fact that zaping the disk is contolled by USE_EXTERNAL_JOURNAL just seams a little odd | 20:29 |
Mech422 | sean-k-mooney: Umm - Blue! actually - I gotta look at the code - its bootstrap_osd that does the dirty deeds | 20:29 |
*** jtriley has quit IRC | 20:31 | |
Mech422 | sean-k-mooney: yeah - no external journal assumes full disk and zaps it | 20:32 |
rhallisey | anyone mind reviewing: https://review.openstack.org/#/c/320744/ | 20:32 |
patchbot | rhallisey: patch 320744 - kolla - Add kolla-kubernetes bootstrap capability to mariadb | 20:32 |
rhallisey | and it's attached patches | 20:32 |
*** ravig has quit IRC | 20:32 | |
sean-k-mooney | Mech422: looking at https://github.com/openstack/kolla/blob/master/doc/ceph-guide.rst i do not see any reference to use external jornals either or using partitions | 20:32 |
Mech422 | sean-k-mooney: yeah - I found the info on sam's blog | 20:32 |
Mech422 | sean-k-mooney: doc patches welcome :-P | 20:32 |
sean-k-mooney | hehe ya i might open a bug for it and see if i can get to it later in the week. | 20:33 |
sean-k-mooney | do you have a pointer to sams blog on the topic? | 20:33 |
Mech422 | sean-k-mooney: but if you want to use partitions - just use 2, 1 data, 1 journal... | 20:34 |
Mech422 | sean-k-mooney: err...lemme check my google fu real quick | 20:34 |
sean-k-mooney | Mech422: no worries if you dont have it to hand i can find it with google too | 20:35 |
Mech422 | sean-k-mooney: https://yaple.net/2016/03/20/deploying-openstack-mitaka-with-kolla-docker-and-ansible/ | 20:35 |
Mech422 | sean-k-mooney: right under the first inlined console block | 20:35 |
*** ravig has joined #openstack-kolla | 20:36 | |
sean-k-mooney | Mech422: cool thanks | 20:36 |
Mech422 | sean-k-mooney: and you just thought I was here for my charm and good looks :-) | 20:37 |
*** mark-casey has quit IRC | 20:37 | |
Mech422 | inc0: the .ansible directory for my deployment user is owned by root - run kolla via sudo | 20:39 |
Mech422 | inc0: err...I run kolla via sudo that is | 20:39 |
inc0 | Mech422, issue is /home/ansible is owned bu root inside kolla-toolbox container | 20:40 |
inc0 | I think | 20:40 |
Mech422 | inc0: Hmm - yeah - mine is too...but it works.. | 20:41 |
wirehead_ | rhallisey: so, it looks like it’s bootstrapping, but when I try to actually run keystone, it’s not really responding to traffic. | 20:41 |
inc0 | at first it couldn't even access .ansible.cfg | 20:41 |
*** ravig has quit IRC | 20:42 | |
Mech422 | inc0: thats wierd - I rebuild all the containers a couple of times a night... never seen this problem | 20:42 |
Mech422 | (full process from foreman bootstrap to running cluster is about 2 hours) | 20:43 |
sdake | Mech422 how many nodes | 20:43 |
sdake | Mech422 in my enviornment a kolla deploy takes about 6 minutes from registry all in one | 20:43 |
*** ravig has joined #openstack-kolla | 20:43 | |
sdake | is the foreman bootstrap automated? | 20:43 |
rhallisey | wirehead_, how are you contacting it? | 20:43 |
wirehead_ | logged into the running container, did a curl http://127.0.0.1:5000/ | 20:44 |
Mech422 | sdake: currently, it bootstraps 7 nodes including deployment host | 20:44 |
Mech422 | sdake: just formating drives/loading ubuntu takes me more then 6 minutes | 20:44 |
sdake | what part takes 2 hours? | 20:44 |
Mech422 | sdake: all of it - blank box to working cluster == 2 hours | 20:44 |
sdake | a cat at INtel deployed 64 nodes in 24 minutes | 20:45 |
sdake | Mech422 are you includding imgee building in that 2 hours? | 20:45 |
Mech422 | sdake: yes, doing EVERYTHING from scratch - zero to cluster | 20:45 |
sdake | so yes on image building? | 20:45 |
Mech422 | sdake: and the killer is I don't have 10GigE at home | 20:45 |
Mech422 | yes, I build images 2x night on average | 20:46 |
rhallisey | wirehead_, hmm | 20:46 |
Mech422 | and I install a private registry to pull them from | 20:46 |
sdake | ok so most of the 2 hours is likely spent image bulding | 20:46 |
wirehead_ | now it’s working. | 20:46 |
sdake | Mech422 if you want to speed that up, just use the docker hub registry | 20:46 |
Mech422 | sdake: not really - deploy is the longest part | 20:46 |
rhallisey | wirehead_, oh really? | 20:46 |
rhallisey | what did you do? | 20:46 |
Mech422 | sdake: I think I'm saturating my 1G link | 20:46 |
sdake | Mech422 ya th eintel cat had 10gige | 20:46 |
sdake | and i have 10gige | 20:47 |
wirehead_ | Well, I pointed it at http://127.0.0.1:5000/v3/ | 20:47 |
Mech422 | sdake: and I run a docker registry on the deployement host | 20:47 |
sdake | the other bottleneck is often the disk | 20:47 |
Mech422 | sdake: so the deploy doesn't have to go off switch | 20:47 |
Mech422 | sdake: eh - I'm building on SSD..but not nvme | 20:47 |
sdake | nvme is 3-4 x speed of regular sata ssd | 20:47 |
rhallisey | wirehead_, nice :) | 20:48 |
sean-k-mooney | even on old server with an ssd build is quick provided your upstream internet connection is quick i find | 20:48 |
*** matrohon has quit IRC | 20:48 | |
sdake | build takes me 15 minutes on gige to the internet | 20:48 |
rhallisey | wirehead_, lately I've been having issues with my DB. | 20:48 |
sdake | push takes 12 minutes | 20:48 |
Mech422 | sdake: I build and push in one step - takes about 30ish minutes | 20:49 |
*** ravig has quit IRC | 20:49 | |
*** ravig has joined #openstack-kolla | 20:49 | |
wirehead_ | rhallisey: might also need a bit of time to come back up. | 20:49 |
sdake | Mech422 you could speed things up by using docker pull to pull to your local registry from the docker hub, then using docker push to push to your local registry | 20:49 |
sdake | Mech422 but we don't provide any tooling around that - perhaps you could make some :) | 20:49 |
sean-k-mooney | sdake: i site arount the 20-30 min mark which is still fine. less i i use an apt/pip cache to eliminate upstream bandwidth limits | 20:49 |
rhallisey | ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use | 20:49 |
rhallisey | ugh | 20:49 |
Mech422 | sdake: my internet is slower then my lan - how would hitting the net be faster then local? | 20:50 |
sean-k-mooney | s/stie arount/sit around/ | 20:50 |
sdake | Mech422 when you build lots of networking is used | 20:50 |
sdake | Mech422 much much mor enetworking hhen a docker pull | 20:50 |
Mech422 | sdake: oh wait - you mean instead of building ? yeah - but I like building the containers from scratch...gives me faith in the process :-) | 20:50 |
sdake | ok sounds good :) | 20:51 |
sdake | if your happy with 2 hours wfm ;) | 20:51 |
rhallisey | wirehead_, have you rebuilt your mariadb container recently? | 20:51 |
sdake | 2 hours would drive me crazeeee | 20:51 |
wirehead_ | friday. | 20:51 |
Mech422 | sdake: I don't wanna wake up one day and find out my stuff only worked because of a magic container someone uploaded.... | 20:51 |
wirehead_ | I’ve been zapping my env fairly frequently. | 20:51 |
sdake | Mech422 instead you get to wake up to diferent changes in all the upstream packages :) | 20:51 |
wirehead_ | Yeah, it looks like it just takes a *while* to come up. | 20:51 |
Mech422 | sdake: yeah - so I know what broke :-) | 20:52 |
rhallisey | wirehead_, the db? | 20:52 |
wirehead_ | rhallisey: Keystone itself | 20:52 |
sdake | mech so accounting for the time | 20:52 |
sdake | you said 30 minutes to build and push | 20:52 |
sean-k-mooney | sdake: isnt that an argument for using the source build instead of the binary builds | 20:52 |
sdake | 1 hr to bulid the nodes, and 30 mins to deploy kolla? | 20:52 |
rhallisey | wirehead_, how long would you say? | 20:52 |
sdake | sean-k-mooney isnt which an argument | 20:52 |
sdake | I made several :) | 20:52 |
Mech422 | sdake: nah.. more like 30 minutes to install metal (3 reboots - dam slow boot times) | 20:53 |
sean-k-mooney | all of the above i guess but i was refering the the upstream packages changing and breaking stuff | 20:53 |
*** cfarquhar has quit IRC | 20:53 | |
Mech422 | sdake: about 30 minutes to run my 'prep' and 'build containers' | 20:53 |
sdake | sean-k-mooney RPM or DEB packaging is slower moving and more conservative | 20:53 |
wirehead_ | rhallisey: 3 minutes. | 20:53 |
sdake | so I think from binary packaging would give most reliable results | 20:54 |
Mech422 | sdake: an maybe 30-45 minutes to do deploy | 20:54 |
Mech422 | sdake: plus my fuck-off time when I go get a smoke/sandwich/etc | 20:54 |
sdake | for 9 nodes - what storagee graph driver are you using? | 20:54 |
Mech422 | sdake: storage graph? | 20:54 |
sean-k-mooney | aufs/ btrfs/device mapper? | 20:55 |
sdake | --storage-driver option to docker | 20:55 |
Mech422 | sdake: oh - aufs | 20:55 |
sdake | I dont know anything about aufs performance | 20:55 |
Mech422 | on XFS | 20:55 |
sdake | but device mapper is terrible | 20:55 |
sean-k-mooney | i generally use the btrfs storage driver | 20:56 |
Mech422 | sdake: hmm - actually - I think host root is ext4 - ceph osds are XFS | 20:56 |
sean-k-mooney | seams to work well | 20:56 |
sdake | Mech422 if your host root is ext4, then you are using devicemapper | 20:56 |
Mech422 | sean-k-mooney: I've been waiting to use btrfs - but ceph backed off recommending it like last year, and I haven't seen if they changed their minds yet ? | 20:56 |
sdake | Mech422 you can tell by running docker --somecommand | 20:56 |
inc0 | sdake, you can use aufs too | 20:57 |
inc0 | he can* | 20:57 |
sdake | Mech422 for btrfs, just use /var/lib/docker | 20:57 |
sdake | Mech422 rather mount btrfs or aufs as /var/lib/docker | 20:57 |
inc0 | Mech422, ubuntu 16.04 on host? | 20:57 |
sdake | Mech422 my bet is your using lvm | 20:57 |
inc0 | or 14.04? | 20:57 |
Mech422 | inc0: 14.04 | 20:57 |
sdake | what is the diagnostics command for docker | 20:57 |
sdake | its a -- option | 20:57 |
sean-k-mooney | sdake: i dont think the defult is lvm anymore though | 20:57 |
sdake | it tells the configured state of docker | 20:57 |
inc0 | Mech422, show me you docker options in /var/lib/docker plz | 20:58 |
Mech422 | none 75G 7.8G 64G 11% /var/lib/docker/aufs/mnt/289c7550cc7573f305c809934ef49865e9d0acbb49791fc76d354d06b7 | 20:58 |
inc0 | sorry | 20:58 |
inc0 | /etc/default/docker | 20:58 |
Mech422 | DOCKER_OPTS="--insecure-registry 192.168.101.11:4000" | 20:58 |
inc0 | Mech422, try to add -s aufs | 20:58 |
sdake | sean-k-mooney i see - didn't know that | 20:58 |
Mech422 | inc0: yeah - that was for sdake - looks like it is aufs | 20:58 |
inc0 | 14.04 uses devicemapper as default afair | 20:59 |
sdake | there is a command which will tell you | 20:59 |
inc0 | and you didn't seem to change it | 20:59 |
sean-k-mooney | docker info will tell you the storage driver in use too | 20:59 |
sdake | docker --diagnostics | 20:59 |
sdake | or something | 20:59 |
Mech422 | inc0: oh - dunno about DM | 20:59 |
sdake | sean-k-mooney what is the command - i am in flagstaf atm and don't have access to my lab | 20:59 |
Mech422 | inc0: I meant no lvm | 20:59 |
inc0 | dm is equally bad | 20:59 |
sdake | Mech422 docker uses lvm - you don't | 20:59 |
sdake | ya dm is pos with docker | 20:59 |
inc0 | try to add -s aufs and see if it changes stuff | 21:00 |
Mech422 | sdake: Umm - lvm is not installed on box ? | 21:00 |
sean-k-mooney | sdake: docker info will shoul you what storage backend is in use | 21:00 |
sdake | Mech422 run docker info | fpaste | 21:00 |
sdake | or paste it | 21:00 |
Mech422 | Storage Driver: aufs | 21:00 |
Mech422 | Root Dir: /var/lib/docker/aufs | 21:00 |
Mech422 | Backing Filesystem: extfs | 21:00 |
Mech422 | Dirs: 301 | 21:00 |
Mech422 | Dirperm1 Supported: true | 21:00 |
Mech422 | Logging Driver: json-file | 21:00 |
sdake | that settles that | 21:00 |
Mech422 | so I don't need the -s aufs flag right, and I'm not using DM/LVM ? | 21:01 |
sdake | overlayfs is fastest, but it eats babiess | 21:01 |
sdake | Mech422 no | 21:01 |
sdake | not using dm/lvm | 21:01 |
sdake | once overlayfs matures, it will be th way to go | 21:02 |
sean-k-mooney | btrfs is about as fast as overlayfs | 21:02 |
sdake | Mech422 i'd recommend putting /var/lib/docker on its own partition so yu can have whatever filesystem type you want on it | 21:02 |
sean-k-mooney | i belive overlayfs out performs btrfs for small files but other then that they are close | 21:03 |
openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Chown /home/ansible to ansible https://review.openstack.org/329185 | 21:03 |
Mech422 | sdake: yeah - I'm more concerned about reliability then build time though... aufs/ext/xfs are a lil safer then overlay/btrfs/etc aren't they ? | 21:03 |
*** ravig has quit IRC | 21:03 | |
openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Convert Keystone to HostNetwork=False, Fix bootstrap. https://review.openstack.org/328567 | 21:04 |
inc0 | btrfs is pretty stable now | 21:04 |
sdake | btw, meeting time is changed | 21:05 |
Mech422 | inc0: I lost track of it like last year... there were some issues with it and ceph... | 21:05 |
*** ChanServ sets mode: +o sdake | 21:05 | |
Mech422 | inc0: if thats fixed, I should play with it - it seems pretty cool | 21:05 |
*** sdake changes topic to "Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA)" | 21:06 | |
wirehead_ | rhallisey: OK, I pushed another rev that should get things just about right. | 21:07 |
*** ravig has joined #openstack-kolla | 21:07 | |
Mech422 | *sigh* now you guys got me wanting a 10GigE switch... | 21:08 |
sdake | Mech422 don't put ceph on btrfs | 21:08 |
sdake | Mech422 put ceph on raw volumes | 21:08 |
Mech422 | kinda hard to justify for a home lab though | 21:08 |
sdake | 850 bucks | 21:08 |
wirehead_ | I looked at the price of 10 GigE switches and decided I can wait. | 21:08 |
sdake | not a killer :) | 21:08 |
Mech422 | wirehead_: yeah - I need like 24 ports | 21:09 |
sean-k-mooney | sdake: really? | 21:09 |
Mech422 | sdake: best I saw was $1K for arrista on ebay...what you using ? | 21:09 |
sean-k-mooney | i know you said you got it fairly cheaply set up but thats less then i taught | 21:09 |
sdake | https://www.amazon.com/NETGEAR-ProSAFE-XS708E-Ethernet-XS708E-100NES/dp/B00B46AEE6/ref=sr_1_sc_1?s=electronics&ie=UTF8&qid=1465852172&sr=1-1-spell&keywords=10+gig+sswitch | 21:09 |
sdake | its 8 port | 21:10 |
sdake | they make a 12 and 24 port model as well | 21:10 |
sdake | 8/12/16/28/48 | 21:10 |
sean-k-mooney | thats not bad what nics are you using? | 21:11 |
sdake | intel 10gig nics | 21:11 |
sean-k-mooney | onboard or ading im guess niantic but dont recall the external name | 21:11 |
Mech422 | sdake: Ouch - 16 port version is $1500 | 21:11 |
sdake | https://www.amazon.com/Intel-Ethernet-Converged-Network-X540T2/dp/B0077CS9UM/ref=pd_sim_sbs_229_1?ie=UTF8&dpID=41MW9E%2BrmrL&dpSrc=sims&preST=_AC_UL320_SR320%2C320_&refRID=FRMMP456PWBR9QNJ875C | 21:12 |
sean-k-mooney | ya thats the one i was thinking of | 21:12 |
sdake | dual port | 21:12 |
wirehead_ | Dig the giant heatsink. | 21:12 |
Mech422 | sean-k-mooney: you can get 10GigE mellanox for like $20 on ebay now | 21:12 |
*** ChanServ sets mode: -o sdake | 21:12 | |
sdake | wirehead_ its nice because its fanless | 21:12 |
sdake | fans break | 21:12 |
wirehead_ | Yeah, I’m not a fan of ‘em. | 21:13 |
sdake | its bad because it geneateas alot of heat | 21:13 |
Mech422 | sean-k-mooney: I got the fiber ones...forget what they're called | 21:13 |
sean-k-mooney | Mech422: yep i know i was tempted by them in the past | 21:13 |
sdake | and if you dont have good airflow the nics overheat and your box locks up | 21:13 |
*** zhiwei has joined #openstack-kolla | 21:13 | |
sean-k-mooney | the conectx2 right? | 21:13 |
Mech422 | sean-k-mooney: yeah.. | 21:13 |
wirehead_ | Yeah, but then you can at least optimize the case fans. Large slow-spinning fans. | 21:14 |
sean-k-mooney | http://www.ebay.ie/itm/Lot-of-2-Mellanox-ConnectX-2-Single-Port-SFP-10GBE-Network-Card-MNPA19-XTR-/301689994552?hash=item463e1ffd38:g:hEAAAOSwgQ9VqDx2 | 21:14 |
sdake | sean-k-mooney i wanted copper | 21:14 |
sean-k-mooney | the problem with those is they use qsfp+ connectors instead of sfp+ since they are also for infiniband | 21:14 |
sdake | right | 21:14 |
sdake | i also wanted intel | 21:15 |
sdake | intel drivers for nics are the best in the world | 21:15 |
sdake | intel consisstenly has the best edrivers for all their gear compared to alternatives imo | 21:15 |
sean-k-mooney | well as an intel employee that make me happy to hear you say :) | 21:15 |
rhallisey | wirehead_, man have to apply like 10 patches across 2 repos to get things working .. | 21:15 |
sdake | i wouldn't buy anything but intel | 21:15 |
*** daneyon_ has joined #openstack-kolla | 21:15 | |
wirehead_ | Some jobs ago, in the dark days when I was both Lord High Fixer and Senior Software Engineer, we had to rip out all of the Broadcom NICs and replace them with Intel NICs. | 21:15 |
sdake | ya broadcom linux driver pile of steaming shit | 21:16 |
wirehead_ | One day, after a mostly unrelated outage that had working a day that started at 2 AM, I went on a bike ride. And I biked past some folks with Broadcom jackets and had to restrain my animalistic urge to violence. | 21:16 |
sdake | mellanox drivers are ok - but overly complex | 21:16 |
wirehead_ | Well, even worse than Broadcom Linux drivers is Broadcom FreeBSD drivers. | 21:17 |
*** ravig has quit IRC | 21:17 | |
*** zhiwei has quit IRC | 21:17 | |
inc0 | sooo....something is broken. | 21:17 |
wirehead_ | rhallisey: yeah, we need to get some patches merged. :) | 21:18 |
inc0 | neutron l3 didnt mount config volumes | 21:18 |
inc0 | Mech422, did you see sth like that? | 21:18 |
sdake | wirehead_ freebsd, is that a thing :) | 21:18 |
Mech422 | inc0: Umm - no - my cluster works pretty much... | 21:18 |
rhallisey | wirehead_, now I'm co-authored on that patch there's no one to merge it :/ . Fail | 21:18 |
Mech422 | inc0: except sometime when it builds - horizon loses the 'volume built' completion message when trying to spin up a VM | 21:19 |
wirehead_ | rhallisey: I guess it’s dcwangmit01’s turn to go power-mad with his +2 bit. | 21:19 |
rhallisey | hehe | 21:19 |
wirehead_ | sdake: in certain audiences, that can get you lynched and/or surrounded by a mob of weeping nerds. | 21:20 |
*** daneyon_ has quit IRC | 21:20 | |
wirehead_ | sdake: Y! had a very baroque and bespoke FreeBSD setup. | 21:20 |
inc0 | nvm will look at it tomorrow | 21:21 |
inc0 | bye | 21:21 |
wirehead_ | Complete with a custom package manager with built-in containers. | 21:21 |
*** inc0 has quit IRC | 21:21 | |
Mech422 | wirehead_: we actually have a couple of FBSD kernel committer on staff... | 21:21 |
Mech422 | wirehead_: only place I've ever worked that uses FBSD :-) | 21:21 |
wirehead_ | I mean, FreeBSD Jails were stable long before LXC. | 21:21 |
sdake | wirehead_ sounds like firedrills :) | 21:23 |
sean-k-mooney | o/ time for food. talk to people tomorow. maybe some day we will deploy kolla/openstack on bsd... | 21:23 |
wirehead_ | Ever since I’ve gotten over my irrational infantile love of microkernels, I’ve been pretty pleased to not care about the underlying operating system. | 21:24 |
wirehead_ | Because I recognize that if I care, I’d probably write my own. | 21:24 |
wirehead_ | (one of my ex-coworkers has written his own OS along with designing his own CPU and hardware architecture, so I know where this rabbit hole leads) | 21:25 |
*** dwalsh has quit IRC | 21:26 | |
sdake | microkernel is a good idea | 21:29 |
sdake | but in reality, what adds valaue to the kernel is a rich set of driver support | 21:29 |
sdake | and nobody really cares aboutanything but mac, windows, and linux | 21:29 |
Mech422 | wirehead_: heh | 21:29 |
Mech422 | wirehead_: if I was to play with some other OS - I'd probably go with one of the BeOS clones | 21:30 |
sdake | in a microkernel world, you barely would need a userspace to run docker with | 21:30 |
Mech422 | haiku or atheros or something | 21:30 |
*** d_code_ has joined #openstack-kolla | 21:31 | |
sdake | its also a proven fact that monolithic kernels outperform (but are harder to maintain) microkernels | 21:31 |
wirehead_ | There was some interesting stuff done with the L4 microkernel. | 21:32 |
*** d_code has quit IRC | 21:33 | |
*** d_code_ is now known as d_code | 21:33 | |
Mech422 | wirehead_: heh - pretty much everything was done to the poor L4 kernel - weren't there like 12+ projects using it ? | 21:33 |
sdake | wow msft buying inked in | 21:33 |
Mech422 | sdake: yeah - and symantec is buying blue coat | 21:34 |
wirehead_ | Pretty much, and then it petered out. | 21:34 |
Mech422 | all your SSL are belong to us | 21:34 |
Mech422 | wirehead_: yeah - same with the plan9 stuff - people maintained it a while, then it sorta...fizzled | 21:34 |
Mech422 | wirehead_: QNX looked really neat...I was disappointed to see their open source stuff vanish | 21:35 |
wirehead_ | Well, a lot of the same things that were neat about microkernels can be moved one layer up the stack with a hypervisor. | 21:36 |
wirehead_ | e.g. https://www.qubes-os.org/ | 21:37 |
Mech422 | wirehead_: no..no new toys! too much in the 'I want to play with this' pile already | 21:39 |
*** rhallisey has quit IRC | 21:39 | |
*** cfarquhar has joined #openstack-kolla | 21:40 | |
*** cfarquhar has quit IRC | 21:40 | |
*** cfarquhar has joined #openstack-kolla | 21:40 | |
wirehead_ | My brother was actually a serious BeOS developer for a while in college. | 21:46 |
*** dmk0202 has joined #openstack-kolla | 21:47 | |
Mech422 | wirehead_: yeah? it was a neat system.. | 21:52 |
Mech422 | wirehead_: sorta like a consumer oriented Next box :-) | 21:52 |
sdake | inc0 can you ahve a look at my response in https://review.openstack.org/#/c/306662/4 | 21:54 |
patchbot | sdake: patch 306662 - kolla - Make Cinder access glance in round robin fashion | 21:54 |
sdake | and either chagne your vote or provide an alternative approach :) | 21:55 |
*** jtriley has joined #openstack-kolla | 21:57 | |
wirehead_ | Our college ACM chapter had one of the real BeBox Towers. Complete with the blue sides and the LEDs that monitored CPU usage and GeekPort and all. | 21:58 |
*** SiRiuS__ has quit IRC | 22:03 | |
openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Switching Horizon over to work via Kubernetes network https://review.openstack.org/329204 | 22:08 |
openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Switching Horizon over to work via Kubernetes network https://review.openstack.org/329204 | 22:08 |
openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Add documentation for api_interface_address https://review.openstack.org/327961 | 22:09 |
wirehead_ | My brother just had a Motorola-branded PowerPC Mac Clone that he mostly used to run BeOS. | 22:10 |
Mech422 | wirehead_: I couldn't afford PowerPC :-P | 22:12 |
Mech422 | wirehead_: I think I was running amd/cyrix 486 clones back then | 22:12 |
*** ravig has joined #openstack-kolla | 22:12 | |
*** rhallisey has joined #openstack-kolla | 22:14 | |
wirehead_ | Well, he got an internship at Motorola and thus a discount. | 22:15 |
wirehead_ | Also, this was farther along. The days when most folks were on Pentium II and so. | 22:16 |
*** esharao_ has quit IRC | 22:17 | |
openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Add documentation for api_interface_address https://review.openstack.org/327961 | 22:21 |
openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Add documentation for api_interface_address https://review.openstack.org/327961 | 22:22 |
openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Switching Horizon over to work via Kubernetes network https://review.openstack.org/329204 | 22:23 |
sdake | i tried to geet an internship at motorola | 22:26 |
sdake | they turned me down | 22:26 |
sdake | but intel gave me one instead | 22:26 |
sdake | i htink it had something to do with me asking their chief engineer what happens if two satellites fail in iridium | 22:26 |
sdake | in front of our entire engineering university :) | 22:27 |
*** ayoung has quit IRC | 22:27 | |
*** ravig has quit IRC | 22:31 | |
*** ravig has joined #openstack-kolla | 22:32 | |
*** dwalsh has joined #openstack-kolla | 22:33 | |
wirehead_ | rhallisey: So, the underscore in configmap isn’t going to be in 1.3 according to most recent PR comments. | 22:33 |
rhallisey | ok | 22:34 |
wirehead_ | rhallisey: so I’m not sure where that puts us with the configmap and underscores. e.g. I’m not sure what to do about service_definition.py in Horizon and if there’s even a way to make it find that path. | 22:36 |
sdake | my suggestion would be to make it work for what is available | 22:37 |
sdake | and when 1.3 comes out change it | 22:37 |
wirehead_ | Well, Horizon and Nova both have some underscores. | 22:39 |
wirehead_ | sdake: I don’t think we’ll see underscores till 1.4. | 22:42 |
wirehead_ | sdake: the most recent github comment was that underscores don’t make it till 1.3 | 22:43 |
wirehead_ | till after 1.3 | 22:43 |
*** jtriley has quit IRC | 22:43 | |
*** dave-mccowan has quit IRC | 22:47 | |
wirehead_ | Hm. | 22:49 |
wirehead_ | Okay, I have a solution for 1.2 | 22:51 |
*** salv-orlando has quit IRC | 22:51 | |
*** dmk0202 has quit IRC | 22:52 | |
*** dave-mccowan has joined #openstack-kolla | 22:52 | |
*** salv-orlando has joined #openstack-kolla | 22:52 | |
wirehead_ | rhallisey / sdake: What we need to do is forcibly change all of the underscore paths to dashes in the kubectl command. And then when you mount the configmap, you have to provide a file mapping to restore the underscores. | 22:52 |
*** ravig has quit IRC | 22:55 | |
*** beekhof has quit IRC | 22:59 | |
*** ravig has joined #openstack-kolla | 23:00 | |
*** daneyon_ has joined #openstack-kolla | 23:04 | |
*** beekhof has joined #openstack-kolla | 23:07 | |
*** daneyon_ has quit IRC | 23:08 | |
*** ravig has quit IRC | 23:10 | |
*** dwalsh has quit IRC | 23:16 | |
*** mbound has quit IRC | 23:23 | |
harlowja | if i was to make https://github.com/openstack/kolla/blob/master/docker/nova/nova-base/Dockerfile.j2#L52-L53 not happen (so that i can do this later via a bind mount so that puppet configuration of those files still works); would that be ok with folks? | 23:27 |
harlowja | same question i guess for the other services config files | 23:27 |
harlowja | (not to have them mved and copied around like that in the dockerfile) | 23:27 |
*** salv-orl_ has joined #openstack-kolla | 23:28 | |
openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Switching Horizon over to work via Kubernetes network https://review.openstack.org/329204 | 23:29 |
*** zhiwei has joined #openstack-kolla | 23:29 | |
*** salv-orlando has quit IRC | 23:31 | |
openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Switching Horizon over to work via Kubernetes network https://review.openstack.org/329204 | 23:33 |
*** ayoung has joined #openstack-kolla | 23:34 | |
*** zhiwei has quit IRC | 23:34 | |
harlowja | so whats the plan on a kolla-ansible split out? | 23:43 |
harlowja | then have kolla just be the images + docker stuffs? | 23:43 |
wirehead_ | Well, it’s quite interesting that Kolla-kubernetes is mostly re-using the ansible templates, thus making a lot of the split-out less obviously useful. | 23:50 |
rhallisey | wirehead_, it uses it for config though | 23:51 |
rhallisey | it could use anything for config | 23:51 |
wirehead_ | Yeah, like we’re kinda joined at the hip unless we re-implement configs on the kubernetes side. | 23:51 |
rhallisey | it's attached to the dockerfiles more though | 23:51 |
wirehead_ | rhallisey: https://blueprints.launchpad.net/kolla-kubernetes/+spec/fix-underscores-in-paths | 23:52 |
wirehead_ | yeah | 23:52 |
rhallisey | harlowja, it will be discussed at the mid cycle | 23:53 |
harlowja | when's that again | 23:53 |
rhallisey | mid july | 23:53 |
harlowja | k | 23:53 |
wirehead_ | Ansible HQ (Durhum, NC) July 11th, 12th | 23:54 |
*** mummer has quit IRC | 23:54 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!