| *** janem has quit IRC | 00:03 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Configure log path for kibana https://review.openstack.org/386980 | 00:04 |
|---|---|---|
| *** zhubingbing has joined #openstack-kolla | 00:08 | |
| *** portdirect has quit IRC | 00:15 | |
| *** portdirect has joined #openstack-kolla | 00:16 | |
| *** haplo37 has quit IRC | 00:27 | |
| *** g3ek has quit IRC | 00:27 | |
| *** haplo37 has joined #openstack-kolla | 00:36 | |
| *** g3ek has joined #openstack-kolla | 00:36 | |
| *** huikang has joined #openstack-kolla | 00:38 | |
| *** huikang has quit IRC | 00:43 | |
| *** janem has joined #openstack-kolla | 00:44 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Configure log path for kibana https://review.openstack.org/386980 | 00:49 |
| *** neilus has joined #openstack-kolla | 00:52 | |
| *** neilus has quit IRC | 00:57 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 01:07 |
| *** sdake_ has joined #openstack-kolla | 01:13 | |
| *** sdake has quit IRC | 01:17 | |
| *** Pavo has quit IRC | 01:23 | |
| openstackgerrit | Merged openstack/kolla: change source to . in gnocchi-base dockerfile https://review.openstack.org/385328 | 01:24 |
| *** Pavo has joined #openstack-kolla | 01:28 | |
| *** huikang has joined #openstack-kolla | 01:29 | |
| *** Pavo_ has joined #openstack-kolla | 01:31 | |
| *** Pavo has quit IRC | 01:33 | |
| *** Pavo_ is now known as Pavo | 01:33 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 01:44 |
| *** huikang has quit IRC | 02:06 | |
| *** huikang has joined #openstack-kolla | 02:07 | |
| *** huikang has quit IRC | 02:11 | |
| *** huikang has joined #openstack-kolla | 02:22 | |
| *** HyperJohnGraham_ has joined #openstack-kolla | 02:29 | |
| HyperJohnGraham_ | sdake_: sbezverk: do you recommend I blow away my external ceph that was installed using ./kolla-ansible -i ../ansible/inventory/multinode deploy -t ceph -vvv | 02:31 |
| HyperJohnGraham_ | start over with everything ? | 02:31 |
| sdake_ | i'd keep that external ceph in place | 02:31 |
| sdake_ | what i'd recommend doing is waiting until these cats have working multinode install directions :) | 02:31 |
| sdake_ | Jeffrey4l there are some patches in teh quque | 02:32 |
| Jeffrey4l | sdake_ yep. | 02:32 |
| kfox1111 | sdake_: I'm working on that a bit more with the Modular gate job. | 02:32 |
| Jeffrey4l | will review it afternoon ;) | 02:32 |
| kfox1111 | should make it easier to reuse some of the ceph stuff. | 02:33 |
| *** salv-orlando has joined #openstack-kolla | 02:45 | |
| *** salv-orlando has quit IRC | 02:50 | |
| HyperJohnGraham_ | sdake_: ok cool. I still have ~30 days till SC16 :) | 02:51 |
| HyperJohnGraham_ | i would like to clean out the kubernetes scraps still running on my external ceph server | 02:53 |
| *** huikang has quit IRC | 03:02 | |
| *** huikang has joined #openstack-kolla | 03:03 | |
| *** huikang has quit IRC | 03:03 | |
| *** huikang has joined #openstack-kolla | 03:03 | |
| *** janem has quit IRC | 03:09 | |
| *** sdake_ has quit IRC | 03:19 | |
| *** huikang has quit IRC | 03:43 | |
| *** huikang has joined #openstack-kolla | 03:44 | |
| *** salv-orlando has joined #openstack-kolla | 03:47 | |
| *** huikang has quit IRC | 03:48 | |
| *** g3ek has quit IRC | 03:49 | |
| *** g3ek has joined #openstack-kolla | 03:51 | |
| *** salv-orlando has quit IRC | 03:51 | |
| *** dave-mcc_ has quit IRC | 03:54 | |
| *** janem has joined #openstack-kolla | 04:05 | |
| *** janem has quit IRC | 04:12 | |
| *** huikang has joined #openstack-kolla | 04:14 | |
| *** huikang has quit IRC | 04:18 | |
| *** neilus has joined #openstack-kolla | 04:28 | |
| *** neilus has quit IRC | 04:33 | |
| *** salv-orlando has joined #openstack-kolla | 04:48 | |
| *** salv-orlando has quit IRC | 04:52 | |
| *** msimonin has joined #openstack-kolla | 05:04 | |
| *** msimonin has quit IRC | 05:05 | |
| *** msimonin has joined #openstack-kolla | 05:06 | |
| *** janem has joined #openstack-kolla | 05:08 | |
| *** janem has quit IRC | 05:19 | |
| *** msimonin has quit IRC | 05:24 | |
| *** msimonin has joined #openstack-kolla | 05:42 | |
| *** salv-orlando has joined #openstack-kolla | 05:48 | |
| *** salv-orlando has quit IRC | 05:53 | |
| *** msimonin has quit IRC | 06:03 | |
| *** janem has joined #openstack-kolla | 06:15 | |
| *** neilus has joined #openstack-kolla | 06:17 | |
| *** neilus has quit IRC | 06:22 | |
| *** janem has quit IRC | 06:22 | |
| *** salv-orlando has joined #openstack-kolla | 06:49 | |
| bjolo_ | Jeffrey4l: | 06:52 |
| bjolo_ | Could you reproduce the tls issue? | 06:52 |
| bjolo_ | https://review.openstack.org/386975 | 06:53 |
| *** salv-orlando has quit IRC | 06:54 | |
| *** tonanhngo has quit IRC | 07:16 | |
| *** janem has joined #openstack-kolla | 07:18 | |
| *** g3ek has quit IRC | 07:22 | |
| *** tonanhngo has joined #openstack-kolla | 07:24 | |
| *** janem has quit IRC | 07:25 | |
| *** tonanhngo has quit IRC | 07:25 | |
| *** g3ek has joined #openstack-kolla | 07:28 | |
| *** msimonin has joined #openstack-kolla | 07:33 | |
| *** msimonin has quit IRC | 07:39 | |
| *** tonanhngo has joined #openstack-kolla | 07:44 | |
| *** tonanhngo has quit IRC | 07:47 | |
| *** salv-orlando has joined #openstack-kolla | 07:50 | |
| *** msimonin has joined #openstack-kolla | 07:53 | |
| *** salv-orlando has quit IRC | 07:54 | |
| *** msimonin has quit IRC | 08:00 | |
| *** msimonin has joined #openstack-kolla | 08:01 | |
| *** msimonin has quit IRC | 08:01 | |
| *** neilus has joined #openstack-kolla | 08:05 | |
| *** neilus has quit IRC | 08:10 | |
| *** janem has joined #openstack-kolla | 08:21 | |
| *** janem has quit IRC | 08:28 | |
| *** tonanhngo has joined #openstack-kolla | 08:53 | |
| *** tonanhngo has quit IRC | 08:54 | |
| *** tonanhngo has joined #openstack-kolla | 09:14 | |
| *** neilus has joined #openstack-kolla | 09:14 | |
| *** tonanhngo has quit IRC | 09:17 | |
| *** msimonin has joined #openstack-kolla | 09:18 | |
| *** neilus has quit IRC | 09:19 | |
| *** neilus has joined #openstack-kolla | 09:23 | |
| *** janem has joined #openstack-kolla | 09:24 | |
| *** janem has quit IRC | 09:31 | |
| *** tonanhngo has joined #openstack-kolla | 09:33 | |
| *** tonanhngo has quit IRC | 09:35 | |
| *** msimonin has quit IRC | 09:38 | |
| *** tonanhngo has joined #openstack-kolla | 09:54 | |
| *** tonanhngo has quit IRC | 09:57 | |
| *** lamt has quit IRC | 10:05 | |
| *** tonanhngo has joined #openstack-kolla | 10:14 | |
| *** tonanhngo has quit IRC | 10:16 | |
| *** janem has joined #openstack-kolla | 10:27 | |
| *** salv-orlando has joined #openstack-kolla | 10:27 | |
| *** janem has quit IRC | 10:34 | |
| *** tonanhngo has joined #openstack-kolla | 10:35 | |
| *** salv-orlando has quit IRC | 10:36 | |
| *** tonanhngo has quit IRC | 10:36 | |
| *** tonanhngo has joined #openstack-kolla | 10:55 | |
| *** tonanhngo has quit IRC | 10:58 | |
| *** sdake has joined #openstack-kolla | 11:11 | |
| *** tonanhngo has joined #openstack-kolla | 11:13 | |
| *** tonanhngo has quit IRC | 11:14 | |
| *** sdake_ has joined #openstack-kolla | 11:18 | |
| *** sdake has quit IRC | 11:20 | |
| *** neilus has quit IRC | 11:22 | |
| *** janem has joined #openstack-kolla | 11:30 | |
| *** salv-orlando has joined #openstack-kolla | 11:32 | |
| *** tonanhngo has joined #openstack-kolla | 11:33 | |
| *** tonanhngo has quit IRC | 11:35 | |
| *** janem has quit IRC | 11:37 | |
| *** salv-orlando has quit IRC | 11:37 | |
| *** spsingh has joined #openstack-kolla | 11:42 | |
| *** tonanhngo has joined #openstack-kolla | 11:55 | |
| *** tonanhngo has quit IRC | 11:57 | |
| *** tonanhngo has joined #openstack-kolla | 12:13 | |
| *** tonanhngo has quit IRC | 12:16 | |
| *** salv-orlando has joined #openstack-kolla | 12:33 | |
| *** janem has joined #openstack-kolla | 12:33 | |
| *** salv-orlando has quit IRC | 12:37 | |
| *** janem has quit IRC | 12:38 | |
| sdake_ | morning | 12:42 |
| sdake_ | sbezverk around? | 12:54 |
| sdake_ | Jeffrey4l around? | 12:54 |
| Jeffrey4l | sdake_, yep | 12:54 |
| sdake_ | what did you think of that tls patch | 12:54 |
| Jeffrey4l | it is fine if it fixed the tls issue. | 12:55 |
| sdake_ | i dont know if it did | 12:56 |
| sdake_ | but it looks goood to me | 12:56 |
| Jeffrey4l | did bjolo_ tried that? he is testing tls. | 12:56 |
| sdake_ | Jeffrey4l i also run tls - but my env is busted atm | 12:58 |
| sdake_ | i asked bjolo in the bug log to test the patch | 12:58 |
| Jeffrey4l | i can test this, now. working | 12:58 |
| Jeffrey4l | working on it. | 12:58 |
| *** msimonin has joined #openstack-kolla | 13:11 | |
| sdake_ | Jeffrey4l have you setup tls before? | 13:11 |
| sdake_ | we need m0ar gates | 13:12 |
| Jeffrey4l | i setup one env several days ago. just for test the tls bug. | 13:13 |
| *** msimonin has quit IRC | 13:22 | |
| sdake_ | Jeffrey4l oh nice! | 13:22 |
| sdake_ | Jeffrey4l were you able to confirm it? | 13:22 |
| sdake_ | Jeffrey4l that was what i was after until both my laptops cratered | 13:22 |
| sdake_ | now i'm on my 3rd and final backup laptop | 13:22 |
| Jeffrey4l | sdake_, i confirmed the issue exist. and trying to deploy with the fix patch. | 13:23 |
| sdake_ | cool | 13:23 |
| Jeffrey4l | you have lots of compute ;) | 13:23 |
| *** tonanhngo has joined #openstack-kolla | 13:23 | |
| *** tonanhngo has quit IRC | 13:26 | |
| *** zhurong has joined #openstack-kolla | 13:27 | |
| *** salv-orlando has joined #openstack-kolla | 13:34 | |
| sbezverk | sdake_: still around? | 13:35 |
| sdake_ | yes sir | 13:35 |
| sdake_ | so - looking at the system, you are using the mflex device | 13:36 |
| sdake_ | for storage | 13:36 |
| sdake_ | i think that is going to be a problem | 13:36 |
| sdake_ | also your using the built in raid device | 13:36 |
| sbezverk | sdake_ it is working now | 13:36 |
| sdake_ | what did you do to fix it | 13:36 |
| sbezverk | sdake_ I am using registry on a different node build-node | 13:36 |
| sbezverk | it seems the issue is with btrfs it cannot handle properly registry and building on the same box.. | 13:37 |
| *** salv-orlando has quit IRC | 13:39 | |
| sdake_ | sbezverk i doubt thats the issue | 13:42 |
| sdake_ | sbezverk but if your unblocked wfm - have a bunch of shit to do myself | 13:42 |
| sdake_ | sbezverk also please dont use flexflash on further installs :) | 13:42 |
| sdake_ | i'm not sure what its for but I think it wears out easily | 13:43 |
| sdake_ | i mean you can use it to install | 13:43 |
| sdake_ | but don't install to it | 13:43 |
| *** huikang has joined #openstack-kolla | 13:45 | |
| kfox1111 | morning. | 13:51 |
| sbezverk | sdake_: sure I am not using flexflash, I partitioned raid for this purpose | 13:51 |
| sdake_ | sbezverk there is a sda which is 64gb | 13:51 |
| sdake_ | i think that is flexflash no? | 13:51 |
| sbezverk | sdake_ nope, sda is raid array | 13:51 |
| sdake_ | sbezverk the reports i have are that the ucs raid is sluggish | 13:51 |
| sdake_ | although i have conflciting rpeorts saying its faster | 13:52 |
| sdake_ | so who knows ;) | 13:52 |
| sbezverk | do not care | 13:52 |
| sdake_ | but if you raid 10 16 disks, bad things probably happen in either cndition | 13:52 |
| kfox1111 | raid performance can be a tricky thing to measure. one bum drive in the set can mess it all up. | 13:52 |
| sbezverk | about raid performance | 13:53 |
| kfox1111 | we found a particular sub model was slower then other ones with the same model number once. | 13:53 |
| sdake_ | sbezverk the reason you care is it may be the caus eof the crater of the docker build | 13:53 |
| *** tonanhngo has joined #openstack-kolla | 13:54 | |
| *** tonanhngo has quit IRC | 13:55 | |
| sbezverk | sdake_: sorry but I disagree, this hardware is awsome :-) using it for years now nothing will change it ;-) | 13:55 |
| Jeffrey4l | sdake_, tls works with that PS. | 14:01 |
| sdake_ | Jeffrey4l nice | 14:01 |
| sdake_ | lets get er merged then :) | 14:01 |
| Jeffrey4l | yep | 14:02 |
| *** tonanhngo has joined #openstack-kolla | 14:17 | |
| *** tonanhngo has quit IRC | 14:18 | |
| *** salv-orlando has joined #openstack-kolla | 14:21 | |
| *** dwalsh has joined #openstack-kolla | 14:23 | |
| sdake_ | everyone congratulate rbergeron on 1 year at rht ! | 14:25 |
| *** sdake has joined #openstack-kolla | 14:29 | |
| openstackgerrit | Merged openstack/kolla: OpenStack Services Should Use keystone_internal_url for auth https://review.openstack.org/386975 | 14:30 |
| sdake | Jeffrey4l cherrypick plz: https://review.openstack.org/#/c/387051/1 | 14:31 |
| sdake | rather +2 the cherrypick | 14:31 |
| Jeffrey4l | done +w | 14:32 |
| *** sdake_ has quit IRC | 14:32 | |
| *** huikang has quit IRC | 14:32 | |
| *** huikang has joined #openstack-kolla | 14:33 | |
| *** tonanhngo has joined #openstack-kolla | 14:33 | |
| *** tonanhngo has quit IRC | 14:36 | |
| *** huikang has quit IRC | 14:37 | |
| *** dwalsh has quit IRC | 14:38 | |
| *** rhallisey has joined #openstack-kolla | 14:38 | |
| sdake | Jeffrey4l 1 less bug :) | 14:41 |
| sdake | 3 days left ;( | 14:41 |
| Jeffrey4l | all bug has fix now. | 14:42 |
| sdake | Jeffrey4l i guess i just woke up and missed that | 14:45 |
| *** dave-mccowan has joined #openstack-kolla | 14:47 | |
| *** zhurong has quit IRC | 14:48 | |
| sdake | hey dave-mccowan a+ on the tls work :) | 14:48 |
| dave-mccowan | sdake thanks | 14:48 |
| sdake | say did you catch my last email | 14:49 |
| sdake | probably not since i'm not on vpn | 14:49 |
| sdake | did you inspect all services for this error? | 14:49 |
| dave-mccowan | sdake i searched for all cases of keystone_internal_url and keystone_external_url, and fixed the two i found. i didn't look at every template to see if someone found a different way to code it wrong. | 14:50 |
| dave-mccowan | at the time i did the original tls work, i verified that all were correct at that time. | 14:51 |
| sdake | ok | 14:51 |
| sdake | wonder how that neutron one got chagned around | 14:51 |
| *** tonanhngo has joined #openstack-kolla | 14:53 | |
| dave-mccowan | from git history, it looks like some of the kolla-kubernetes commits affected it | 14:53 |
| *** tonanhngo has quit IRC | 14:54 | |
| dave-mccowan | be back later | 14:56 |
| *** dave-mccowan has quit IRC | 14:56 | |
| sdake | Jeffrey4l your run change - did you test that? | 15:08 |
| Jeffrey4l | sdake, yep. it works with /run:/run:shared | 15:08 |
| sdake | hmm let me look at your change | 15:09 |
| Jeffrey4l | ok. the old fix is use specify /run/xxx folder rather than /run directly. the latest fix is use /run:/run:shared instead | 15:10 |
| sdake | Jeffrey4l right ij ust looked at your change | 15:11 |
| sdake | what does :shared do? | 15:11 |
| sdake | i fear that kubernetes has no :shared option | 15:11 |
| sdake | sbezverk ^^ | 15:11 |
| Jeffrey4l | it should have. if so neutron-l3 dhcp should not work at all right now. | 15:12 |
| sdake | Jeffrey4l did you catch all occurances of run? | 15:13 |
| *** tonanhngo has joined #openstack-kolla | 15:13 | |
| hogepodge | I'm having a build problem. Building from source on xenial for xenial for some reason apt refuses to install ceph-common because it can't install python-cephfs. Which is weird, because on the host everything goes just fine. No amount of updating or purging of apt and the docker cache helps | 15:14 |
| Jeffrey4l | sdake, what's that mean? who hold the /run folder? it should be kernel issue/does. | 15:14 |
| sdake | Jeffrey4l did you check the code base for every example of run and make sure it has shared attached to it | 15:14 |
| *** tonanhngo has quit IRC | 15:14 | |
| Jeffrey4l | yes. i did. changed all /run | 15:15 |
| sdake | rhallisey around? | 15:15 |
| sdake | hogepodge do you have build logs? | 15:15 |
| Jeffrey4l | got some explanation here https://github.com/docker/docker/issues/27277#issuecomment-253972146 | 15:16 |
| sdake | rhallisey please ack https://review.openstack.org/#/c/385559/7 | 15:16 |
| hogepodge | Aside from what's output to stdout? I can probably dig them up | 15:17 |
| hogepodge | I also have the scroll back from the build | 15:17 |
| sdake | think you want private mounts not shared ones | 15:18 |
| sdake | Jeffrey4l from that issue tracker, he says to use private mounts yet your patch uses shared mounts | 15:19 |
| * sdake confused :) | 15:19 | |
| sdake | used and abused too :) | 15:19 |
| Jeffrey4l | but we need shared mount. so i change both /run and /run/netns to shared. | 15:19 |
| sdake | /run/netns was shared, and /run was private prior? | 15:21 |
| Jeffrey4l | without shared mount, openvswitch won't see the namespace. the namespace should be saw by openvswitch-agent l3-agent and openvswitch_db | 15:21 |
| Jeffrey4l | correct. | 15:21 |
| sdake | i see - seems like a docker bug still | 15:21 |
| Jeffrey4l | hmm. it should be a kernel mount issue. | 15:21 |
| sdake | docker shouldn't permit a condition in which a crater occurs in the entire operating system | 15:21 |
| *** msimonin has joined #openstack-kolla | 15:22 | |
| sdake | hogepodge do this | 15:22 |
| Jeffrey4l | docker should use kernel mount `mount a b ` directly. | 15:22 |
| sdake | hogepodge i don't have a ubuntu system handy - my laptops are busted | 15:22 |
| sdake | hogepodge run script | 15:22 |
| sdake | do the build | 15:22 |
| sdake | type texit | 15:22 |
| sdake | this will save a file clled typescript | 15:22 |
| sdake | paste that | 15:22 |
| *** neilus has joined #openstack-kolla | 15:23 | |
| hogepodge | sdake ok, will do. I'll need to report back later. Thanks | 15:23 |
| sdake | Jeffrey4l - does that problem affect our documentation? | 15:23 |
| sdake | like systemd setup for eg | 15:23 |
| Jeffrey4l | sdake, for ubuntu 14.04, yes. for os with systemd: no | 15:24 |
| Jeffrey4l | for ubuntu 14.04 the user should add mount --make-share /run, too. | 15:24 |
| sdake | Jeffrey4l ok thanks | 15:24 |
| Pavo | morning | 15:25 |
| sdake | sup pavo | 15:25 |
| Pavo | about to get my second server up | 15:25 |
| Pavo | then its kolla time | 15:25 |
| Pavo | once thats done, will work on figuring out how to get cobbler to pxe UEFI | 15:25 |
| Pavo | spent to much time on it | 15:26 |
| sdake | pxe UEFI cobbler | 15:26 |
| sdake | 3 oxymorons all in a row :) | 15:26 |
| *** neilus has quit IRC | 15:27 | |
| *** huikang has joined #openstack-kolla | 15:29 | |
| Pavo | lol | 15:29 |
| Pavo | what do you use for bare metal provisioning sdake? | 15:30 |
| sdake | pavo we have implementation of bifrost available | 15:30 |
| sdake | not sure on its status wrt uefi | 15:30 |
| Pavo | yeah might have to look into how to use that, never seen it before but I need something that does work with UEFI though | 15:31 |
| Pavo | our RAID controller will not work in legacy mode | 15:31 |
| *** sbezverk has quit IRC | 15:33 | |
| *** tonanhngo has joined #openstack-kolla | 15:33 | |
| *** janem has joined #openstack-kolla | 15:34 | |
| *** msimonin has quit IRC | 15:35 | |
| *** tonanhngo has quit IRC | 15:35 | |
| *** msimonin has joined #openstack-kolla | 15:36 | |
| *** janem has quit IRC | 15:38 | |
| *** sbezverk has joined #openstack-kolla | 15:38 | |
| sdake | Jeffrey4l any other bugs we should bring into rc3? | 15:39 |
| sdake | Jeffrey4l (as in critical bugs) | 15:39 |
| Jeffrey4l | no i think. | 15:39 |
| *** sbezverk_ has joined #openstack-kolla | 15:40 | |
| sdake | Jeffrey4l what are your thoughtso n this bug and the gate https://review.openstack.org/#/c/385586/3 | 15:42 |
| *** sbezverk has quit IRC | 15:43 | |
| sdake | http://logs.openstack.org/86/385586/3/check/gate-kolla-dsvm-deploy-centos-binary-centos-7-nv/d6e2336/console.html#_2016-10-13_19_31_56_247741 | 15:43 |
| Jeffrey4l | i prefer not use memcached as default. | 15:43 |
| openstackgerrit | Merged openstack/kolla: Cinder endpoint wrong fix https://review.openstack.org/386965 | 15:44 |
| sdake | https://review.openstack.org/#/c/387078/ | 15:46 |
| sdake | Jeffrey4l that patch doesn't enable memcache by default | 15:46 |
| Jeffrey4l | sdake, memcached is enabled in default. so token use memcached in default too. | 15:48 |
| sdake | right - so why is that patch wrong? | 15:49 |
| sdake | why is the gate jammed up on deploy | 15:49 |
| Jeffrey4l | re error log: i doubt memcache token drive can not handle multi memcached proproly. | 15:50 |
| Jeffrey4l | ^^ no. the CI is one node env. | 15:51 |
| sdake | Jeffrey4l re https://bugs.launchpad.net/kolla/+bug/1632811 | 15:52 |
| openstack | Launchpad bug 1632811 in kolla "Keystone token table filling up" [Critical,In progress] - Assigned to Mathias Ewald (mewald) | 15:52 |
| sdake | Jeffrey4l can you add some commentary there | 15:52 |
| sdake | Jeffrey4l we may need to jetison this bug | 15:52 |
| Jeffrey4l | ok. | 15:52 |
| sdake | i'm not really sure | 15:52 |
| sdake | seems like a DOS is triggerable by hammering keystone with auth requests | 15:52 |
| sdake | Jeffrey4l hogepodge reported a bug with ceph + ubuntu not building | 15:54 |
| sdake | Jeffrey4l do you have an ubuntu test bed? | 15:54 |
| *** msimonin has quit IRC | 15:55 | |
| Jeffrey4l | hogepodge, have any logs? or the same with CI? the ubuntu gate is red now. | 15:55 |
| sdake | Jeffrey4l just ubuntu binary gate | 15:56 |
| sdake | not ubuntu source | 15:56 |
| sdake | hogepodge arey ou running rc2? | 15:57 |
| Jeffrey4l | hmm. if it is ceph + ubuntu issue, it should happen on both source and binary. because the logic for build ceph is the same in source and binary. | 15:58 |
| sdake | ohnm it is both source and binary | 15:58 |
| Jeffrey4l | do not use ubuntu normally. let me try. | 15:59 |
| sdake | Jeffrey4l http://logs.openstack.org/65/386965/3/check/gate-kolla-dsvm-build-ubuntu-source-ubuntu-xenial-nv/cdcf580/console.html#_2016-10-15_19_57_09_516368 | 15:59 |
| Jeffrey4l | i doubt the apt prefernces in ubuntu. | 16:00 |
| Jeffrey4l | ya. i saw that. | 16:00 |
| sdake | i dont know what that means | 16:00 |
| sdake | but it doesn't look good | 16:00 |
| sdake | this may be a problem introduced by the ceph.com outage | 16:00 |
| sdake | possibly they had to respin their keys | 16:00 |
| sdake | or packages | 16:01 |
| sdake | or both | 16:01 |
| sdake | for centos we use the storage sig's bits | 16:01 |
| sdake | not ceph.com | 16:01 |
| Pavo | second node up woot | 16:01 |
| *** salv-orl_ has joined #openstack-kolla | 16:03 | |
| *** msimonin has joined #openstack-kolla | 16:03 | |
| *** salv-orl_ has quit IRC | 16:03 | |
| Jeffrey4l | no. ceph.com and ubuntu cloudarchive both have ceph 10.2.x. i think we should remove ceph.com now. sdake | 16:03 |
| Jeffrey4l | i will push a PS for test. | 16:03 |
| sdake | here i have a bug for you to use | 16:04 |
| Jeffrey4l | cool | 16:04 |
| sdake | https://bugs.launchpad.net/kolla/+bug/1633888 | 16:04 |
| openstack | Launchpad bug 1633888 in kolla "after ceph outage, ceph ubuntu totally broken" [Undecided,New] | 16:04 |
| *** tonanhngo has joined #openstack-kolla | 16:04 | |
| sdake | well that one is critical | 16:05 |
| sdake | https://bugs.launchpad.net/kolla/+bug/1633888 | 16:05 |
| openstack | Launchpad bug 1633888 in kolla newton "after ceph outage, ceph ubuntu totally broken" [Critical,Confirmed] | 16:05 |
| *** salv-orlando has quit IRC | 16:05 | |
| *** sdake_ has joined #openstack-kolla | 16:08 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Remove ceph repo for Ubuntu install https://review.openstack.org/387081 | 16:10 |
| sdake_ | Jeffrey4l i think you are missing a key for ceph that needs to be deleted too | 16:11 |
| *** sdake has quit IRC | 16:11 | |
| Jeffrey4l | yes.. | 16:12 |
| sdake_ | https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L234-L242 | 16:13 |
| Jeffrey4l | i am finding which one is ceph's | 16:14 |
| Jeffrey4l | seem there are some issue in ubuntu packaging. http://paste.openstack.org/show/585829/ | 16:18 |
| Jeffrey4l | still debugging | 16:18 |
| Jeffrey4l | OK. it is ceph.com issue. | 16:19 |
| Jeffrey4l | confirmed. | 16:19 |
| *** huikang has quit IRC | 16:19 | |
| *** huikang has joined #openstack-kolla | 16:20 | |
| *** neilus has joined #openstack-kolla | 16:23 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Remove ceph repo for Ubuntu install https://review.openstack.org/387081 | 16:24 |
| *** huikang has quit IRC | 16:25 | |
| Jeffrey4l | need go to bed. | 16:25 |
| Jeffrey4l | bye sdake_ | 16:25 |
| sdake_ | night | 16:25 |
| Jeffrey4l | re ubuntu ceph issue: the root cause is now ubuntu cloud archive provides higher ceph version than ceph.com. but ceph.com has a higher priority apt number than ubuntu cloud archive. | 16:26 |
| sdake_ | i see | 16:27 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Remove ceph repo for Ubuntu install https://review.openstack.org/387081 | 16:28 |
| Jeffrey4l | night | 16:28 |
| *** Guest96828 is now known as zeus | 16:38 | |
| *** zeus has joined #openstack-kolla | 16:38 | |
| *** msimonin has quit IRC | 16:42 | |
| *** ankush has joined #openstack-kolla | 16:48 | |
| *** neilus has quit IRC | 16:57 | |
| HyperJohnGraham_ | sdake_: sounds like you need a new laptop or two ;) | 16:57 |
| sdake_ | i comandeered my wife's | 16:58 |
| HyperJohnGraham_ | i can give you guys access to my blade server for testing if it would help | 16:58 |
| HyperJohnGraham_ | or i could run some tests for you | 16:59 |
| *** dave-mccowan has joined #openstack-kolla | 16:59 | |
| *** neilus has joined #openstack-kolla | 17:01 | |
| sdake_ | i've got decks to prepare for summit | 17:02 |
| sdake_ | unfortunately | 17:02 |
| *** salv-orlando has joined #openstack-kolla | 17:04 | |
| kfox1111 | Jeffrey4l: I ran into a similar issue before with centos. giant had preference to hammer and caused a huge mess when trying to upgrade. :/ | 17:05 |
| *** salv-orlando has quit IRC | 17:08 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 17:14 |
| *** zhubingbing has quit IRC | 17:16 | |
| *** neilus has quit IRC | 17:17 | |
| *** msimonin has joined #openstack-kolla | 17:19 | |
| *** mandre has quit IRC | 17:20 | |
| *** msimonin has quit IRC | 17:20 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 17:25 |
| *** neilus has joined #openstack-kolla | 17:26 | |
| *** huikang has joined #openstack-kolla | 17:26 | |
| *** huikang has quit IRC | 17:37 | |
| *** huikang has joined #openstack-kolla | 17:37 | |
| *** mandre has joined #openstack-kolla | 17:37 | |
| *** huikang has quit IRC | 17:41 | |
| *** huikang has joined #openstack-kolla | 17:41 | |
| *** sdake_ has quit IRC | 17:45 | |
| *** neilus has quit IRC | 17:50 | |
| bjolo | good evening | 17:54 |
| *** neilus has joined #openstack-kolla | 17:59 | |
| *** salv-orlando has joined #openstack-kolla | 18:05 | |
| *** salv-orlando has quit IRC | 18:10 | |
| *** sdake has joined #openstack-kolla | 18:14 | |
| *** mandre has quit IRC | 18:16 | |
| *** msimonin has joined #openstack-kolla | 18:16 | |
| *** mandre has joined #openstack-kolla | 18:27 | |
| *** msimonin has quit IRC | 18:27 | |
| bjolo | sdake, just tested the TLS PS and it works. weird, i can almost swear i tried that config manually and it did not work?! | 18:28 |
| bjolo | but anyhow. happy it got resolved :) | 18:29 |
| bjolo | dave-mccowan, kudos on the TLS :) | 18:30 |
| sdake | bjolo thats good news, becaus eits already been mreged :) | 18:33 |
| bjolo | i noticed that when my cherrypick did not work :) | 18:39 |
| *** ankush has quit IRC | 18:41 | |
| *** pbourke_ has quit IRC | 18:45 | |
| *** pbourke_ has joined #openstack-kolla | 18:46 | |
| bjolo | sdake, can i trouble you with a q | 18:49 |
| sdake | no trouble shoot | 18:49 |
| bjolo | trying to find a PS in gerrit | 18:49 |
| bjolo | im pretty sure i saw a PS fly by the other day for this bug | 18:50 |
| bjolo | https://bugs.launchpad.net/kolla/+bug/1631944 | 18:50 |
| openstack | Launchpad bug 1631944 in kolla "cron container fails to start. missing sahara.conf" [Undecided,New] | 18:50 |
| bjolo | but im not able to find it in gerrit | 18:50 |
| bjolo | any good search trick? | 18:50 |
| *** msimonin has joined #openstack-kolla | 18:54 | |
| sdake | bjolo https://review.openstack.org/Documentation/user-search.html | 18:56 |
| *** jmccarthy has quit IRC | 18:57 | |
| sdake | bjolo perhaps this one https://review.openstack.org/#/c/371670/4/ansible/roles/common/templates/cron.json.j2 | 18:59 |
| *** jmccarthy has joined #openstack-kolla | 18:59 | |
| *** huikang has quit IRC | 19:03 | |
| *** huikang has joined #openstack-kolla | 19:03 | |
| bjolo | could be | 19:04 |
| bjolo | but the bug is still there | 19:04 |
| bjolo | template file yes | 19:04 |
| *** huikang has quit IRC | 19:04 | |
| bjolo | but copy is not | 19:04 |
| *** huikang has joined #openstack-kolla | 19:04 | |
| hogepodge | sdake: Jeffrey4l The gate is failing with the same failures I'm seeing | 19:04 |
| hogepodge | (I was afk for a while, family things) | 19:04 |
| sdake | hogepodge ya we got a fix for that | 19:05 |
| sdake | hogepodge https://review.openstack.org/#/c/387081/ | 19:05 |
| hogepodge | sdake: I'll apply the patch and try it out. If things go ok I'll +1 the review | 19:07 |
| sdake | wfm | 19:07 |
| sdake | although ubuntu is now atleasat buildilng | 19:07 |
| sdake | although our gate doesn't test ceph | 19:07 |
| hogepodge | sdake: I'll be doing full deploys today, so I'll be able to verify | 19:08 |
| kfox1111 | sdake: it would be good to get the kolla-kubernetes gate hooked up somehow to the kolla containers build part. then we could test container changes against ceph. | 19:19 |
| openstackgerrit | bjorn lofdahl proposed openstack/kolla: fix cron logrotate for sahara https://review.openstack.org/387092 | 19:19 |
| bjolo | sdake, ^^ | 19:21 |
| bjolo | can we consider this critical and merge it for newton rc3? | 19:21 |
| *** huikang has quit IRC | 19:26 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 19:26 |
| *** huikang has joined #openstack-kolla | 19:26 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 19:41 |
| Pavo | hmmm getting some errors on kolla-build | 19:42 |
| Pavo | http://pastebin.com/tc24YFi5 | 19:42 |
| Pavo | mind taking alook sdake | 19:42 |
| Pavo | main erros are | 19:43 |
| Pavo | errors | 19:43 |
| Pavo | INFO:kolla.cmd.build:base:Cannot open: http://repo.percona.com/release/7/RPMS/x86_64/percona-release-0.1-3.noarch.rpm. Skipping. | 19:43 |
| Pavo | INFO:kolla.cmd.build:base:Error: Nothing to do | 19:43 |
| Pavo | then a ton of ERROR:kolla.cmd.build:mesosphere-base:Parent image error'd with message "error" | 19:43 |
| Pavo | what is INFO:kolla.cmd.build:base: ---> [Warning] IPv4 forwarding is disabled. Networking will not work. | 19:46 |
| *** ankush has joined #openstack-kolla | 19:51 | |
| Pavo | think I figured it out | 19:53 |
| Pavo | had to add net.ipv4.ip_forward = 1 to /etc/sysctl.conf | 19:54 |
| dave-mccowan | bjolo thanks! i'm glad you're using TLS. please let me know if you run into any other services with TLS issues. | 19:57 |
| *** tonanhngo has quit IRC | 19:57 | |
| Pavo | do I need to add net.ipv4.ip_forward = 1 to all nodes? | 19:59 |
| *** salv-orlando has joined #openstack-kolla | 20:07 | |
| *** salv-orlando has quit IRC | 20:11 | |
| *** tonanhngo has joined #openstack-kolla | 20:14 | |
| *** tonanhngo has quit IRC | 20:16 | |
| *** huikang has quit IRC | 20:16 | |
| *** awiddersheim has quit IRC | 20:34 | |
| *** salv-orlando has joined #openstack-kolla | 20:41 | |
| *** neilus has quit IRC | 20:46 | |
| *** quasar__ is now known as kwazar | 20:49 | |
| *** ankush has quit IRC | 20:56 | |
| sbezverk_ | kfox1111: ping | 20:56 |
| *** salv-orl_ has joined #openstack-kolla | 21:03 | |
| *** salv-orlando has quit IRC | 21:03 | |
| wznoinsk | Pavo: did you have to add it on all compute nodes? (I'd imagine when kolla-build runs it spawn containers which have some dockerish IPs hence need masq/frowarding) | 21:09 |
| Pavo | dunno yet, build is almost done | 21:10 |
| Pavo | once thats done I deploy to nodes and find out | 21:10 |
| wznoinsk | if you're using registry you'll be pulling registry (on deployment/controller node usually) -> compute/storage/etc_nodes so no building on these so forwarding shouldn't be needed | 21:11 |
| wznoinsk | s/pulling registry/pulling from registry | 21:11 |
| Pavo | yeah I was thinking the same | 21:12 |
| Pavo | I still think you guys need to make a repo container at first build then have all there other images pull from that container for building, would make building go extremey faster imo | 21:13 |
| Pavo | any reason these failed? | 21:16 |
| Pavo | http://pastebin.com/UynikaCc | 21:16 |
| Pavo | and do they matter? | 21:17 |
| wznoinsk | murano-base failed hence the rest | 21:18 |
| wznoinsk | don't matter "normally" if you don't use mistral/murano, what's the error for murano? | 21:18 |
| Pavo | if not using murano does it matter? | 21:18 |
| wznoinsk | murano-base | 21:18 |
| Pavo | ok | 21:18 |
| *** awiddersheim has joined #openstack-kolla | 21:24 | |
| *** neilus has joined #openstack-kolla | 21:27 | |
| *** neilus has quit IRC | 21:31 | |
| *** salv-orl_ has quit IRC | 21:36 | |
| *** salv-orlando has joined #openstack-kolla | 21:43 | |
| *** sbezverk_ has quit IRC | 21:44 | |
| *** sbezverk has joined #openstack-kolla | 21:44 | |
| Pavo | ok deployment done | 21:45 |
| Pavo | what IP do I go to again? | 21:45 |
| *** spsingh has quit IRC | 21:45 | |
| Pavo | ok the VIP that I set in globals | 21:47 |
| Pavo | now need to find the dashboard password lol | 21:47 |
| Pavo | shouldn't that be same as keystone_admin_password? | 21:48 |
| hogepodge | kfox1111: you around? | 21:52 |
| Pavo | ok deployment done | 21:58 |
| Pavo | but not showing all the storage I have | 21:58 |
| Pavo | and in multinode ansible file I have storage set for both controller and compute nodes | 21:59 |
| Pavo | shouldn't it have the max of both added together? | 21:59 |
| Pavo | this is why I ask | 21:59 |
| Pavo | https://www.dropbox.com/s/77eguzeax7n4od8/Screenshot%202016-10-16%2017.55.38.png?dl=0 | 21:59 |
| HyperJohnGraham_ | did you label all your drives ? | 22:00 |
| Pavo | only showing 49Gb | 22:00 |
| Pavo | and I have over 30Tb of storage on the compute node alone | 22:00 |
| *** salv-orl_ has joined #openstack-kolla | 22:03 | |
| *** salv-orlando has quit IRC | 22:05 | |
| HyperJohnGraham_ | Pavo: did you label with KOLLA_CEPH_OSD_BOOTSTRAP http://docs.openstack.org/developer/kolla/ceph-guide.html#preparation | 22:08 |
| Pavo | not using ceph afaik | 22:10 |
| *** tonanhngo has joined #openstack-kolla | 22:12 | |
| *** tonanhngo has quit IRC | 22:12 | |
| kfox1111 | hogepodge: yeah. whats up? | 22:14 |
| kfox1111 | sbezverk: ping | 22:14 |
| hogepodge | kfox1111: I want to enable some horizon plugins, and saw you had a mailing list post about it earlier this year. I've installed the source using the source plugin architecture in kolla, but was wondering if you had any suggestions on how to get the enabled files over to horizon | 22:15 |
| kfox1111 | hogepodge: never done it in kolla. :/ | 22:15 |
| kfox1111 | was wondering how to do it at the time but never got to the point where I actually was able to. | 22:16 |
| hogepodge | kfox1111: I figure with the source installed it's just adding the right code to the right block in the horizon docker script, but I'm pretty weak at docker | 22:16 |
| kfox1111 | hogepodge: yeah. you could tweak the docker file to copy the enable file into the right dir. it should enable it then. | 22:17 |
| hogepodge | kfox1111: as far as I can tell the source plugins are working, now just need to set the enabled bits | 22:17 |
| kfox1111 | or in kolla-kubernetes, you could do it in the template. | 22:17 |
| kfox1111 | probably a way to do it in ansible too, but don't know it very well. | 22:17 |
| kfox1111 | should be able to cp the files into the container at /usr/share/openstack-dashboard/openstack_dashboard/local/enabled | 22:19 |
| hogepodge | kfox1111: kolla lets you override the templates at the block entry points, I'm just trying to work through the abstraction layers to see exactly where I need to do it | 22:19 |
| kfox1111 | well, at least on rdo based packaging. never done ubuntu, so not sure exactly where it places its enabled files. | 22:19 |
| kfox1111 | hogepodge: hmm... | 22:20 |
| kfox1111 | you could do the enable at runtime by moving the extended start script out of the way and put your own there that copies the enable scripts in place and then call the origional extended start. | 22:21 |
| kfox1111 | seems like there should be a better way to do it then that though. | 22:21 |
| kfox1111 | I kind of think the horizon code needs an extended start that takes in a list of enable files to load based on an env var. | 22:22 |
| kfox1111 | then each user can enable them at will on their own cloud without rebuilding. | 22:22 |
| kfox1111 | that would probably be best long term. | 22:22 |
| sbezverk | kfox1111:pong | 22:23 |
| kfox1111 | then we could ship a horizon with all the plugins preloaded but disabled, and the user could selectively load the ones they want. | 22:23 |
| kfox1111 | sbezverk: whats up? | 22:23 |
| sbezverk | kfox1111: I saw your ping a bit earlier | 22:23 |
| kfox1111 | sbezverk: responding to your ping. | 22:24 |
| kfox1111 | from 16:56 | 22:24 |
| sbezverk | kfox1111: Ah right from this morning.. | 22:24 |
| sbezverk | kfox1111: that problem is resolved :-) | 22:24 |
| kfox1111 | ok. :) | 22:24 |
| kfox1111 | did you get your ceph going? | 22:25 |
| sbezverk | kfox1111: kind of.. lots of troubles :-( now kubelet on ths same server as monitor/osd refuses to bind pv/pvc | 22:26 |
| sbezverk | kfox1111: but from remote server I can attach image without problem.. | 22:27 |
| kfox1111 | sbezverk: interesting. I heard that is unsupported but always kind of just worked. | 22:27 |
| kfox1111 | maybe they check for it in jewel. | 22:28 |
| sbezverk | kfox1111: really bad timing :-( | 22:28 |
| * kfox1111 nods | 22:29 | |
| kfox1111 | how many nodes do you have? | 22:29 |
| sbezverk | well, I got 5 nodes in total.. I thought ro use 1 as ceph mon + kube-apiserver | 22:30 |
| sbezverk | the rest as compute nodes | 22:30 |
| sbezverk | but I guess I will have to use 1 node as dedicated cluster | 22:30 |
| kfox1111 | I think the restriction only is on having an osd and an rbd mount on the same host. | 22:31 |
| kfox1111 | so that would only affect the rabbit and mariadb containers. | 22:31 |
| sbezverk | ok then it should work | 22:31 |
| sbezverk | as I assigned kolla_control label on different hosts | 22:31 |
| kfox1111 | yeah | 22:33 |
| sbezverk | kfox1111: then it must be something else | 22:33 |
| kfox1111 | your osds are seperate from kolla_control? | 22:33 |
| hogepodge | kfox1111: the horizon dockerfile template installs the lbaas plugin, so I'm just following that model | 22:36 |
| sbezverk | kfox1111: there is juts 1 osd and it is separate from kolla_controller labeled host | 22:37 |
| kfox1111 | sry. computer locked up. | 22:44 |
| kfox1111 | sbezverk: ah. and it can't mout the rbd volume? so something else for sure then. | 22:45 |
| kfox1111 | anything in dmesg? | 22:45 |
| kfox1111 | hogepodge: ah. ok. I haven't looked at that yet, but that sounds like a good example. | 22:45 |
| hogepodge | the problem is there's not a good place to drop it in, as far as I can tell | 22:46 |
| hogepodge | if I do it in the footer kolla_extend_start has already run, but maybe that's not a problem? | 22:46 |
| kfox1111 | if you tweak the kolla_extend_start to do it before starting horizon? | 22:47 |
| *** unicell1 has quit IRC | 22:47 | |
| Pavo | ok so the Note about Cinder on http://docs.openstack.org/developer/kolla/cinder-guide.html | 22:48 |
| Pavo | I take it I can not use Cinder as my storage currently correct | 22:48 |
| Pavo | requires me to use Ceph correct? | 22:49 |
| Pavo | gonna be kind of hard since I only have 2 nodes | 22:49 |
| kfox1111 | Pavo: lvm is not a very production ready backend. | 22:49 |
| kfox1111 | IMHO, its only default in cinder as they don't want to make the other storage vendors mad by defaulting ceph on. | 22:50 |
| hogepodge | it doesn't look like extend_start is doing anything that will interfere | 22:50 |
| Pavo | I understand that but now you are requiring a minimum 3 node deployment | 22:50 |
| kfox1111 | Pavo: you can do a single node ceph. | 22:50 |
| Pavo | to have correct storage | 22:50 |
| kfox1111 | I don't think its less stable then lvm. | 22:50 |
| Pavo | mind telling me how because all my storage is on 1 node which is also my compute and network node | 22:51 |
| kfox1111 | your trying to do an all in one setup? | 22:51 |
| Pavo | no | 22:51 |
| Pavo | I have 2 nodes | 22:51 |
| kfox1111 | oh. | 22:51 |
| Pavo | 1 controller with 512Gb RAM, 72 cpus but only has 1.5 Tb storage | 22:51 |
| Pavo | 1 compute node with 512GB Ram, 72 CPUS, and 30Tb Storage | 22:52 |
| kfox1111 | can you move some of your storage to the ohter node? | 22:52 |
| kfox1111 | would be more fault tollerent then. | 22:52 |
| Pavo | so I figured I would use the compute node as networking and storage since it has the most storage and 4 SFP 10Gb | 22:52 |
| Pavo | yeah I can move 5 of the 1.5 SAS SSDs to the controller if needed | 22:53 |
| kfox1111 | If you can dig up a 3rd machine small enough just to run a ceph mon, and you split your storage between the two existing nodes, | 22:54 |
| kfox1111 | you could archetect it so that you could turn off either of those servers and still keep the cloud running | 22:54 |
| Pavo | I could run a VM on one of the nodes | 22:54 |
| kfox1111 | run neutron ha network nodes, one per node. | 22:54 |
| Pavo | but can not add another physical node | 22:54 |
| Pavo | our require is very specific | 22:55 |
| kfox1111 | not even something really small like a nuc? | 22:55 |
| Pavo | nope | 22:55 |
| Pavo | we have a requirement for 3 nodes only | 22:56 |
| Pavo | 1 master node, 1 compute node and 1 controller node | 22:56 |
| kfox1111 | you have a 3rd already? | 22:56 |
| kfox1111 | ah. so can you run a ceph mon on all 3? | 22:56 |
| kfox1111 | that woudl give you fault tollerence. | 22:56 |
| Pavo | the master node is only a laptop with 8 cores, 64Gb ram | 22:56 |
| kfox1111 | a mon only needs about a gig of ram. | 22:56 |
| Pavo | but its only to manage the cloud | 22:56 |
| Pavo | can that mon be ran as a VM on the master? | 22:57 |
| Pavo | or as a docker container? | 22:57 |
| kfox1111 | it is just a tie breaker. | 22:57 |
| kfox1111 | the issue is this: | 22:57 |
| kfox1111 | if you only have 2 machines runnign ceph, and one dies, how does the remaining machine know if he's just cut off from the network switch, or the other node really did die? | 22:57 |
| kfox1111 | you need to have a third on the network to ensure that the cluster doesn't go split brained. | 22:58 |
| Pavo | true I understand the need for HA | 22:58 |
| Pavo | but with our deployment its blown away after each mission anyways | 22:58 |
| Pavo | and rebuilt before each mission | 22:58 |
| kfox1111 | k. so, if you don't really care about ha, you can get most of ha by doing what you where thinging. | 22:59 |
| Pavo | thats why we do not use Fuel | 22:59 |
| Pavo | because it requires 3 nodes minimum | 22:59 |
| kfox1111 | if you realy really don't care about ha, then using lvm should still work. | 23:00 |
| Pavo | ok so lets say I use the master node as the mon for ceph | 23:00 |
| Pavo | yeah lvm would be ideal for our use case | 23:00 |
| kfox1111 | the master node would work too though, I think. | 23:01 |
| kfox1111 | it only needs to be up as a tie breaker. so you can reboot it if the other two nodes are up without issue. | 23:01 |
| Pavo | and the ceph mon is only a container correct? | 23:02 |
| Pavo | that would be running on the master node | 23:02 |
| kfox1111 | you can run it that way. | 23:02 |
| Pavo | would that require the master node to have more than 1 NIC? | 23:02 |
| Pavo | because thats also another requirement | 23:03 |
| Pavo | master node can only use 1 NIC | 23:03 |
| kfox1111 | should be able to use the one nice for that. | 23:03 |
| sbezverk | kfox1111: it was somehting else, not it is working | 23:03 |
| sbezverk | now it is working | 23:03 |
| *** mtaylor22 has joined #openstack-kolla | 23:03 | |
| sbezverk | there is a bug in canal.yaml installation.. | 23:03 |
| kfox1111 | ah. | 23:05 |
| sbezverk | kfox1111: they do not all necessary binaries to /opt/cni/bin | 23:06 |
| sbezverk | I meant do not copy | 23:06 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 23:06 |
| kfox1111 | sbezverk: really? did that break recently? | 23:06 |
| kfox1111 | cause I just deployed it a few hours ago and it seemed to be working. | 23:07 |
| kfox1111 | though I didn't try it multinode I guess. | 23:07 |
| sbezverk | hmmm, check /opt/cni/bin and see if it has loopback binary file | 23:08 |
| *** salv-orl_ has quit IRC | 23:08 | |
| kfox1111 | I don't have easy access to the gate. :/ | 23:08 |
| kfox1111 | I'm working on breaking up the gate into smaller chunks, so I can deploy the same code in minikube though. | 23:09 |
| kfox1111 | so should be able to manually test again soon. | 23:09 |
| *** msimonin has quit IRC | 23:10 | |
| *** msimonin has joined #openstack-kolla | 23:10 | |
| sbezverk | kfox1111: ok, if you see loopback is not there, you need to get latest cni binaries archive and dump them into /opt/cni/bin folder | 23:11 |
| kfox1111 | hmm.. did you intall the kubernetes-cni package? | 23:12 |
| kfox1111 | rpm -qf /opt/cni/bin/loopback | 23:13 |
| kfox1111 | kubernetes-cni-0.3.0.1-0.07a8a2.x86_64 | 23:13 |
| sbezverk | yes | 23:13 |
| Pavo | whats the requiresments to use kubernetes with kolla? | 23:14 |
| sdake | hogepodge enabling plugins in horizon ??? PROFIT | 23:14 |
| kfox1111 | Pavo: any 1.4 should worm. | 23:14 |
| kfox1111 | work. | 23:14 |
| Pavo | I mean as how many nodes? | 23:14 |
| *** bjolo__ has joined #openstack-kolla | 23:14 | |
| sdake | pavo mesosphere should error | 23:15 |
| kfox1111 | Pavo: 1+ | 23:15 |
| sdake | percona not so much | 23:15 |
| Pavo | sdake I fixed that issue | 23:15 |
| sdake | bjolo are other services affected? | 23:15 |
| sbezverk | kfox1111: servers I have use different nic names, I will use that feature with placing nic name in the file for neutron-openvswitch-agent | 23:16 |
| Pavo | kfox1111 couldn't I just switch to using kolla-kubernetes and beable to have full HA with just 2 nodes? | 23:16 |
| sbezverk | sdake: your servers are gradually come to new life ;-) | 23:16 |
| sdake | bjolo https://review.openstack.org/#/c/387092/ is failing the gate | 23:16 |
| kfox1111 | sbezverk: cool. :) | 23:16 |
| sdake | sbezverk all my servers belong to us | 23:17 |
| kfox1111 | Pavo: full ha can't be done without 3 nodes. | 23:17 |
| kfox1111 | Pavo: I'm rebuilding my home system to be 2 big boxes for storage/services, and one small intel nuc. super cheep just to run a ceph-mon/etcd for full ha support. | 23:17 |
| *** bjolo has quit IRC | 23:18 | |
| *** bjolo_ is now known as bjolo | 23:18 | |
| *** msimonin has quit IRC | 23:18 | |
| Pavo | ok | 23:18 |
| kfox1111 | it doesn't take very much of a 3rd machine to get to full ha. you just need something to be a tie breaker. | 23:19 |
| *** unicell has joined #openstack-kolla | 23:19 | |
| Pavo | guess I could start using the master node as the ceph mon | 23:19 |
| kfox1111 | that should work I think. | 23:19 |
| Pavo | can you still use the underline OS with LVM? | 23:20 |
| Pavo | or not | 23:20 |
| kfox1111 | yeah. underlying os can be lvm | 23:20 |
| Pavo | ok | 23:20 |
| *** unicell has quit IRC | 23:21 | |
| Pavo | git an example globals to achieve this? | 23:21 |
| Pavo | got* | 23:21 |
| sbezverk | sdake: it is so much better to use real hardware than VMs ;-) | 23:22 |
| kfox1111 | I don't. I haven't played with kolla ceph much yet. | 23:22 |
| sbezverk | much faster | 23:22 |
| kfox1111 | one of my co-workers usually deploys our ceph's. | 23:22 |
| *** sdake_ has joined #openstack-kolla | 23:22 | |
| kfox1111 | they have been rock solid though. | 23:22 |
| kfox1111 | I'm getting the parts for my personal ceph in tuesday. | 23:23 |
| Pavo | I have never played with it at all | 23:23 |
| kfox1111 | I managed to forklift a ceph while the production cloud running on it was live. no one noticed. | 23:24 |
| kfox1111 | moved each storage node from one rack to another, one at a time. | 23:24 |
| kfox1111 | ceph's pretty awesome. :) | 23:25 |
| sbezverk | kfox1111: when it let's to install itself ;-) | 23:25 |
| *** sdake has quit IRC | 23:26 | |
| kfox1111 | sdake_: what do I need to do to backport the patch? | 23:27 |
| *** unicell has joined #openstack-kolla | 23:29 | |
| *** unicell1 has joined #openstack-kolla | 23:29 | |
| *** unicell has quit IRC | 23:29 | |
| *** lamt has joined #openstack-kolla | 23:35 | |
| sdake_ | kfox1111 link master patch that needs backport | 23:43 |
| kfox1111 | sdake_: https://review.openstack.org/#/c/386965/ | 23:46 |
| sbezverk | kfox1111: have you tried ipmitool with cisco CIMC? | 23:59 |
| *** lamt has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
