| *** yangyapeng has joined #openstack-kolla | 00:00 | |
| *** yangyapeng has quit IRC | 00:05 | |
| *** jascott1 has joined #openstack-kolla | 00:09 | |
| *** yangyapeng has joined #openstack-kolla | 00:09 | |
| *** manheim has joined #openstack-kolla | 00:09 | |
| *** jascott1_ has joined #openstack-kolla | 00:10 | |
| *** jascott1 has quit IRC | 00:10 | |
| *** manheim has quit IRC | 00:14 | |
| *** gema has quit IRC | 00:16 | |
| *** jascott1_ has quit IRC | 00:19 | |
| *** jascott1 has joined #openstack-kolla | 00:19 | |
| *** yangyapeng has quit IRC | 00:20 | |
| *** jascott1 has quit IRC | 00:20 | |
| *** jascott1_ has joined #openstack-kolla | 00:20 | |
| *** gema has joined #openstack-kolla | 00:21 | |
| *** duonghq has joined #openstack-kolla | 00:33 | |
| duonghq | morning guys | 00:33 |
|---|---|---|
| *** goldyfruit has joined #openstack-kolla | 00:38 | |
| *** zhangfei has joined #openstack-kolla | 01:02 | |
| *** mdnadeem has joined #openstack-kolla | 01:08 | |
| *** goldyfruit has quit IRC | 01:25 | |
| *** ducttap__ has joined #openstack-kolla | 02:02 | |
| *** ducttap__ has quit IRC | 02:12 | |
| *** ducttape_ has joined #openstack-kolla | 02:19 | |
| *** ducttap__ has joined #openstack-kolla | 02:21 | |
| *** ducttape_ has quit IRC | 02:21 | |
| *** ducttap__ has quit IRC | 02:22 | |
| *** ducttape_ has joined #openstack-kolla | 02:23 | |
| *** ducttape_ has quit IRC | 02:24 | |
| *** ducttap__ has joined #openstack-kolla | 02:24 | |
| *** ducttap__ has quit IRC | 02:28 | |
| *** tovin07_ has joined #openstack-kolla | 02:33 | |
| *** ducttap__ has joined #openstack-kolla | 02:34 | |
| *** ducttap__ has quit IRC | 02:38 | |
| *** ducttape_ has joined #openstack-kolla | 02:38 | |
| *** ducttap__ has joined #openstack-kolla | 02:40 | |
| *** ducttape_ has quit IRC | 02:40 | |
| *** ducttap__ has quit IRC | 02:44 | |
| *** sbezverk has quit IRC | 02:44 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' to necessary tasks (general roles) https://review.openstack.org/398682 | 02:48 |
| *** zhangfei has quit IRC | 03:06 | |
| *** bmace has quit IRC | 03:10 | |
| *** ducttape_ has joined #openstack-kolla | 03:19 | |
| *** zhangfei has joined #openstack-kolla | 03:19 | |
| *** eaguilar is now known as eaguilar|afk | 03:19 | |
| spsurya | morning all | 03:22 |
| spsurya | duonghq: morning | 03:22 |
| duonghq | morning spsurya | 03:22 |
| *** bmace has joined #openstack-kolla | 03:23 | |
| *** ducttape_ has quit IRC | 03:31 | |
| *** ramishra has joined #openstack-kolla | 03:39 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' for only necessary tasks (default roles) https://review.openstack.org/398684 | 03:41 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' for only neccesary tasks (all other roles) https://review.openstack.org/398685 | 03:42 |
| duonghq | spsurya, can you give me some suggestion on https://review.openstack.org/398682 | 03:42 |
| *** ducttape_ has joined #openstack-kolla | 03:52 | |
| *** ducttap__ has joined #openstack-kolla | 03:54 | |
| *** ducttape_ has quit IRC | 03:54 | |
| *** ducttap__ has quit IRC | 03:55 | |
| spsurya | duonghq: sure, I will take a look on this and will ping you in EOD if i would have any | 04:00 |
| duonghq | thanks | 04:00 |
| *** japestinho has joined #openstack-kolla | 04:01 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' to necessary tasks (general roles) https://review.openstack.org/398682 | 04:04 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' for only necessary tasks (default roles) https://review.openstack.org/398684 | 04:13 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' for only neccesary tasks (all other roles) https://review.openstack.org/398685 | 04:14 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' for only necessary tasks (default roles) https://review.openstack.org/398684 | 04:19 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Ansible strategy for Rolling upgrade https://review.openstack.org/482863 | 04:27 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Apply neutron database migration https://review.openstack.org/407922 | 04:29 |
| *** prateek has joined #openstack-kolla | 04:48 | |
| *** sbezverk has joined #openstack-kolla | 04:51 | |
| *** ducttape_ has joined #openstack-kolla | 04:53 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Disable grafana Getting Started panel https://review.openstack.org/485932 | 04:56 |
| *** ducttape_ has quit IRC | 04:58 | |
| *** tonanhngo has joined #openstack-kolla | 05:10 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Support customizing grafana home dashboard https://review.openstack.org/485931 | 05:10 |
| *** mewald has quit IRC | 05:10 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Support customizing grafana home dashboard https://review.openstack.org/485931 | 05:11 |
| *** jascott1_ has quit IRC | 05:12 | |
| *** jascott1 has joined #openstack-kolla | 05:13 | |
| *** unicell has joined #openstack-kolla | 05:13 | |
| *** jascott1 has quit IRC | 05:20 | |
| *** janki has joined #openstack-kolla | 05:25 | |
| *** gema has quit IRC | 05:25 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add vmware DVS support to kolla-ansible https://review.openstack.org/459270 | 05:40 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add vmware DVS support to kolla-ansible https://review.openstack.org/459270 | 05:42 |
| *** jascott1 has joined #openstack-kolla | 05:57 | |
| *** jascott1 has quit IRC | 06:02 | |
| *** ducttape_ has joined #openstack-kolla | 06:03 | |
| *** mewald has joined #openstack-kolla | 06:04 | |
| *** omenv has joined #openstack-kolla | 06:06 | |
| *** ducttape_ has quit IRC | 06:08 | |
| *** jbadiapa has joined #openstack-kolla | 06:08 | |
| *** japestinho has quit IRC | 06:10 | |
| *** ansmith has quit IRC | 06:17 | |
| *** coolsvap has joined #openstack-kolla | 06:19 | |
| *** ansmith has joined #openstack-kolla | 06:29 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Ansible strategy for Rolling upgrade https://review.openstack.org/482863 | 06:38 |
| *** jascott1 has joined #openstack-kolla | 06:39 | |
| *** jascott1 has quit IRC | 06:43 | |
| *** ducttape_ has joined #openstack-kolla | 06:57 | |
| *** manheim has joined #openstack-kolla | 07:00 | |
| *** ducttape_ has quit IRC | 07:02 | |
| *** manheim has quit IRC | 07:04 | |
| *** dixiaoli has joined #openstack-kolla | 07:04 | |
| *** dixiaoli has quit IRC | 07:06 | |
| *** zhangfei has quit IRC | 07:10 | |
| *** zhangfei has joined #openstack-kolla | 07:10 | |
| *** tonanhngo has quit IRC | 07:13 | |
| *** zhangfei has quit IRC | 07:15 | |
| *** japestinho has joined #openstack-kolla | 07:17 | |
| *** jascott1 has joined #openstack-kolla | 07:20 | |
| *** mewald1 has joined #openstack-kolla | 07:26 | |
| *** magicboiz has joined #openstack-kolla | 07:27 | |
| *** jascott1 has quit IRC | 07:27 | |
| *** zhangfei has joined #openstack-kolla | 07:28 | |
| *** rmart04 has joined #openstack-kolla | 07:29 | |
| *** mewald has quit IRC | 07:29 | |
| *** magicboiz has quit IRC | 07:32 | |
| *** rmart04 has quit IRC | 07:34 | |
| *** janki is now known as janki|lunch | 07:37 | |
| *** magicboiz has joined #openstack-kolla | 07:39 | |
| *** rmart04 has joined #openstack-kolla | 07:43 | |
| *** egonzalez has joined #openstack-kolla | 07:47 | |
| *** david-lyle has quit IRC | 07:50 | |
| *** dklyle has joined #openstack-kolla | 07:50 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 07:56 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 07:57 |
| *** tonanhngo has joined #openstack-kolla | 08:04 | |
| *** jascott1 has joined #openstack-kolla | 08:04 | |
| openstackgerrit | Helena proposed openstack/kolla-ansible master: Enabled additional .conf files to be read by collectd https://review.openstack.org/477580 | 08:07 |
| *** tonanhngo has quit IRC | 08:08 | |
| *** jascott1 has quit IRC | 08:09 | |
| *** athomas has joined #openstack-kolla | 08:09 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Added note https://review.openstack.org/485727 | 08:11 |
| openstackgerrit | Merged openstack/kolla-ansible master: Dynamically retrieve the location of ARA to work on both py2 & py3 https://review.openstack.org/486362 | 08:11 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 08:22 |
| openstackgerrit | Merged openstack/kolla master: Update the documentation link for doc migration https://review.openstack.org/485151 | 08:26 |
| *** tonanhngo has joined #openstack-kolla | 08:32 | |
| *** dklyle has quit IRC | 08:32 | |
| *** dklyle has joined #openstack-kolla | 08:33 | |
| *** dklyle has quit IRC | 08:36 | |
| egonzalez | Jeffrey4l, tested mariadb upgrade with your patch and worked fine, awesome!!! | 08:37 |
| *** tonanhngo has quit IRC | 08:37 | |
| kolla-slack | <jeffrey4l> cool | 08:40 |
| *** janki|lunch is now known as janki | 08:40 | |
| *** jascott1 has joined #openstack-kolla | 08:42 | |
| *** jascott1 has quit IRC | 08:47 | |
| *** david-lyle has joined #openstack-kolla | 08:47 | |
| kolla-slack | <jeffrey4l> did anyone see the photo i posted? | 08:47 |
| egonzalez | Jeffrey4l, what photo? the workflow in the review? | 08:48 |
| kolla-slack | <jeffrey4l> openstack days china | 08:49 |
| *** david-lyle has quit IRC | 08:49 | |
| *** david-lyle has joined #openstack-kolla | 08:50 | |
| kolla-slack | <jeffrey4l> https://kubernetes.slack.com/files/jeffrey4l/F6CM1JDB6/img_20170724_164020.jpg | 08:50 |
| openstackgerrit | Pavel Gluschak (scsnow) proposed openstack/kolla master: Ensure interface exists before adding OVS port https://review.openstack.org/477884 | 08:50 |
| kolla-slack | <jeffrey4l> seems post photo in slack do not work for irc. | 08:52 |
| *** dklyle has joined #openstack-kolla | 08:53 | |
| *** david-lyle has quit IRC | 08:53 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Optimize reconfiguration for mariadb https://review.openstack.org/433480 | 08:57 |
| *** manheim has joined #openstack-kolla | 08:57 | |
| *** ducttape_ has joined #openstack-kolla | 08:58 | |
| *** manheim has quit IRC | 09:00 | |
| *** manheim has joined #openstack-kolla | 09:00 | |
| *** jaosorior has joined #openstack-kolla | 09:02 | |
| *** ducttape_ has quit IRC | 09:03 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 09:11 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 09:15 |
| *** sambetts|afk is now known as sambetts | 09:16 | |
| openstackgerrit | Juan Badia Payno proposed openstack/kolla master: Changed fluentd repository for CentOS/RHEL/Oraclelinux https://review.openstack.org/467219 | 09:19 |
| *** tonanhngo has joined #openstack-kolla | 09:24 | |
| *** tonanhngo has quit IRC | 09:28 | |
| egonzalez | mewald1, re prometheus images, finally was able to read ENVs from p*-base image? | 09:35 |
| mewald1 | egonzalez: yes it worked | 09:37 |
| numans | egonzalez, Hi, i have a question (which seems to be stupid), how are the images pushed to the kolla docker hub (https://hub.docker.com/u/kolla/) ? | 09:37 |
| numans | in kolla docker hub, i don't see neutron-server-odl or neutron-server-ovn ? Is it possible to have them in the docker hub ? if so, how ? | 09:38 |
| duonghq | egonzalez, ping, | 09:38 |
| egonzalez | numans, those images are only master branch (not pushed yet) | 09:39 |
| *** jascott1 has joined #openstack-kolla | 09:39 | |
| numans | egonzalez, oh ok. so only on the stable branch get pushed ? | 09:39 |
| egonzalez | numans, images for now are pushed manually, however there is work in progress to do it daily for master and stable branchs | 09:40 |
| numans | egonzalez, ok. is it possible to have the neutron-server-odl/ovn pushed. | 09:40 |
| egonzalez | numans, if want to use master images for testing there are registries built on each commit https://tarballs.openstack.org/kolla/images/ | 09:41 |
| egonzalez | numans, not now because only stable are pushed | 09:41 |
| egonzalez | duonghq, sup | 09:41 |
| numans | egonzalez, ack. got it. | 09:41 |
| numans | thanks | 09:41 |
| duonghq | egonzalez, if you have free time, can you review the direction of ansible-become ps(s): https://review.openstack.org/#/c/398682/12 | 09:42 |
| egonzalez | duonghq, yep, have you read my latest comment in keystone upgrade? https://review.openstack.org/#/c/425446/ | 09:43 |
| duonghq | yes, I'll add one more variable for prevent it run at all action | 09:43 |
| duonghq | only upgrade is needed | 09:43 |
| egonzalez | if we cannot make it with serial/strategy, we can play with handler notify order | 09:43 |
| *** jascott1 has quit IRC | 09:44 | |
| duonghq | ya, I'll try both | 09:44 |
| duonghq | thank you | 09:44 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 09:45 |
| *** japestinho has quit IRC | 09:47 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' to necessary tasks (general roles) https://review.openstack.org/398682 | 09:52 |
| *** serlex has joined #openstack-kolla | 09:56 | |
| *** ducttape_ has joined #openstack-kolla | 10:04 | |
| *** tovin07_ has quit IRC | 10:05 | |
| *** ducttap__ has joined #openstack-kolla | 10:07 | |
| manheim | oh, rdo is such a nightmare | 10:08 |
| *** ducttape_ has quit IRC | 10:08 | |
| *** ducttap__ has quit IRC | 10:12 | |
| egonzalez | manheim, rdo != packstack ;) | 10:14 |
| *** mdnadeem has quit IRC | 10:16 | |
| manheim | true, true | 10:17 |
| *** tonanhngo has joined #openstack-kolla | 10:18 | |
| *** tonanhngo_ has joined #openstack-kolla | 10:20 | |
| *** jascott1 has joined #openstack-kolla | 10:21 | |
| *** tonanhngo has quit IRC | 10:23 | |
| *** tonanhngo_ has quit IRC | 10:25 | |
| *** jascott1 has quit IRC | 10:25 | |
| *** duonghq has quit IRC | 10:34 | |
| *** ansmith has quit IRC | 10:35 | |
| *** eaguilar|afk is now known as eaguilar | 10:37 | |
| *** eaguilar is now known as eaguilar|afk | 10:37 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove warning during kolla_docker execution https://review.openstack.org/483544 | 10:42 |
| mewald1 | Someone told me before but I forgot to take a note: how can I generate new release notes? | 10:55 |
| *** tonanhngo has joined #openstack-kolla | 11:02 | |
| *** kristian__ has joined #openstack-kolla | 11:06 | |
| mewald1 | egonzalez: would you mind? I could push a patchset for redis now if I knew how to create a new release note | 11:06 |
| *** tonanhngo has quit IRC | 11:07 | |
| *** kristia__ has joined #openstack-kolla | 11:09 | |
| *** kristian__ has quit IRC | 11:09 | |
| *** zhangfei has quit IRC | 11:10 | |
| *** dciabrin has quit IRC | 11:11 | |
| *** dciabrin has joined #openstack-kolla | 11:12 | |
| openstackgerrit | Mathias Ewald proposed openstack/kolla-ansible master: Fix grafana post-config check https://review.openstack.org/486378 | 11:13 |
| sean-k-mooney | egonzalez: i think i have made all the changes you requested in https://review.openstack.org/#/c/408872/ except refactoring into the main roles. ill be trying to complete that today | 11:20 |
| sean-k-mooney | egonzalez: centos packageversions fo docker, runc and systemd are too old though. | 11:21 |
| sean-k-mooney | egonzalez: the centos host support now works but systemd and docker conflict in how they create cgroups. this has been fixed on ubuntu but not centos | 11:22 |
| *** jascott1 has joined #openstack-kolla | 11:23 | |
| *** jascott1 has quit IRC | 11:27 | |
| egonzalez | mewald1, reno new <title> | 11:29 |
| egonzalez | sean-k-mooney, cool, thanks! is something that need to be fixed in centos docker packages, nothing to do in our side? | 11:30 |
| *** dciabrin has quit IRC | 11:31 | |
| *** dciabrin has joined #openstack-kolla | 11:32 | |
| *** tonanhngo has joined #openstack-kolla | 11:34 | |
| *** awiddersheim has joined #openstack-kolla | 11:36 | |
| *** tonanhngo has quit IRC | 11:39 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 11:40 |
| *** awidders_ has quit IRC | 11:40 | |
| *** pbourke_ has joined #openstack-kolla | 11:45 | |
| *** iniazi_ has joined #openstack-kolla | 11:45 | |
| egonzalez | sean-k-mooney, when rework the role to be in ansible/roles, please add upgrade, check, prechecks, reconfigure files empty with just three dashes --- | 11:46 |
| egonzalez | or if doing one of those actions with dpdk enabled will fail, also pull.yml | 11:47 |
| *** iniazi has quit IRC | 11:48 | |
| egonzalez | sean-k-mooney, actually reconfigure use - include: deploy.yml and upgrade dont know if something different than deploy is needed | 11:51 |
| *** ducttape_ has joined #openstack-kolla | 12:03 | |
| sean-k-mooney | egonzalez: to fix the conflict they need to update teh version of systemd as far as i understand. we cant work around it on our end. | 12:04 |
| *** Tom has joined #openstack-kolla | 12:05 | |
| *** jascott1 has joined #openstack-kolla | 12:05 | |
| egonzalez | sean-k-mooney, roger | 12:05 |
| *** Tom is now known as Guest22197 | 12:05 | |
| egonzalez | maybe add a warning in the docs | 12:05 |
| openstackgerrit | Merged openstack/kolla master: Add collectd support to telegraf https://review.openstack.org/485923 | 12:05 |
| openstackgerrit | Merged openstack/kolla master: Tweaks to allow Horizon dev mode https://review.openstack.org/474598 | 12:06 |
| sean-k-mooney | egonzalez: sure ill added the other files. yes ill add a disclaimer to the docs | 12:06 |
| Guest22197 | hi, why does the installer trying to get 4.0.2 version while docker hub includes only 4.0.0? is this a known issue? | 12:06 |
| sean-k-mooney | egonzalez: for reconfigure i may or may not need to tweek the ovs-dpdkctl helper tool. i cant remember if i supprot for example changing bridge names or interfaces. so that might need another patch | 12:08 |
| *** ducttape_ has quit IRC | 12:08 | |
| egonzalez | sean-k-mooney, sure, always that are documented limitations or not tested behaviour is OK with me | 12:09 |
| mewald1 | I have a problem in my sensu-client containers: To build the sensu plugins ruby >=2.1.0 is required which is there in ubuntu image by default. CentOS doesn't have it. Any suggestions? | 12:09 |
| *** jascott1 has quit IRC | 12:09 | |
| *** Guest22197 is now known as tomkolla | 12:10 | |
| sean-k-mooney | egonzalez: upgrade is also a little tricky, upgradeding form ovs 2.7 to 2.8 should just be replacing the container. same for 2.5->2.6. 2.6->2.7 on the other hand requires different handeling beauce of backwards incompatible changes in ovs so i would prefer to support 2.7+ upgrades only since ubuntu already ships 2.7 | 12:11 |
| egonzalez | sean-k-mooney, upgrades also apply to dpdk image 5.0.1 to 5.0.2 or to custom tag. ovs may or not may change during that upgrade | 12:12 |
| egonzalez | sean-k-mooney, i mean, ovs version is controlled in dpdk images, i dont mean to upgrade ovs, mean about upgrade actions | 12:13 |
| sean-k-mooney | egonzalez: the ovs-dpdk image is staticly linked to dpdk is included in the ovs-dpdk image | 12:13 |
| sean-k-mooney | egonzalez: yes i understand. my assumtion is that you wont change the dpdk/ovs wersion as part of a z stream upgrade which should just replace the containers without modifying the configs | 12:14 |
| egonzalez | sean-k-mooney, yep, maybe just include deploy.yml as with reconfigure action, os same as ovs role: | 12:15 |
| egonzalez | - include: config.yml | 12:15 |
| egonzalez | - name: Flush Handlers | 12:15 |
| egonzalez | meta: flush_handlers | 12:15 |
| sean-k-mooney | yep will do. i have to admit i had not spend much time on the reconfiure or upgrade aspects. mind if i put the implementaion of those in followup patches and just have them blank in the base one. https://review.openstack.org/#/c/408872/ is already larger then i would like. | 12:18 |
| tomkolla | hi, why does the installer trying to get 4.0.2 version while docker hub includes only 4.0.0? is this a known issue? | 12:18 |
| egonzalez | sean-k-mooney, yep,. leave it blank, we can add bugs for missing tasks | 12:19 |
| sean-k-mooney | im also planning to add a destory-ovs-dpdk command to handel tear down. in addtion to the normal destroy actions you need to run ovs-dpdkctl uninstall before you delete the config files on the host | 12:19 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix grafana post-config check https://review.openstack.org/486378 | 12:19 |
| egonzalez | tomkolla, ansible is trying to pull latest images 4.0.2, but in dockerhub there is only 4.0.0. you have two options. build your own 4.0.2 images, or change openstack_release: 4.0.0 to use that images | 12:20 |
| sean-k-mooney | egonzalez: ovs-dpdkctl uninstall basicaly removes the systemd service files that were installed when deploying and bind the dpdk nics back to the kernel | 12:20 |
| *** jascott1 has joined #openstack-kolla | 12:27 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: DNM: test master branch https://review.openstack.org/456140 | 12:27 |
| *** rhallisey has joined #openstack-kolla | 12:30 | |
| *** ansmith has joined #openstack-kolla | 12:30 | |
| *** jascott1 has quit IRC | 12:32 | |
| *** hrw has quit IRC | 12:32 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: DNM: test master branch https://review.openstack.org/456140 | 12:32 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: DNM: test master branch https://review.openstack.org/456140 | 12:34 |
| *** hrw has joined #openstack-kolla | 12:34 | |
| *** eaguilar|afk has quit IRC | 12:40 | |
| *** eaguilar has joined #openstack-kolla | 12:41 | |
| hawi | hi there. is there something i shuld know when upgrading from 4.0.0 -> 4.0.2? mariadb fails during upgrade | 12:53 |
| kolla-slack | <jeffrey4l> egonzalez , how about backport the reconfigure mariadb patch? :D | 12:55 |
| openstackgerrit | zhouya proposed openstack/kolla-ansible master: Support assigning HA traffic to dedicated interface https://review.openstack.org/481503 | 12:55 |
| egonzalez | Jeffrey4l, kind of feature right? but yeah, newton-ocata works fine, bug ocata-ocata and ocata-pike fails so also bug | 12:56 |
| *** eaguilar has quit IRC | 12:59 | |
| *** tonanhngo has joined #openstack-kolla | 13:00 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: DNM: test master branch https://review.openstack.org/456140 | 13:00 |
| *** eaguilar has joined #openstack-kolla | 13:00 | |
| *** rwallner has joined #openstack-kolla | 13:00 | |
| *** rwallner has quit IRC | 13:01 | |
| *** rwallner has joined #openstack-kolla | 13:01 | |
| egonzalez | hawi, atm https://bugs.launchpad.net/kolla-ansible/+bug/1692507 | 13:02 |
| openstack | Launchpad bug 1692507 in kolla-ansible "mariadb multinode upgrade broken ocata-pike" [Critical,Fix released] | 13:02 |
| egonzalez | hawi, though only affect master, but also affect ocata-ocata upgrade in multinode | 13:03 |
| egonzalez | hawi, is that your case? multinode ocata-ocata upgrade? | 13:03 |
| egonzalez | hawi, mind sharing mariadb.log from a couple of nodes, should be some message as "cannot connect to gcomm: openstack , connection refused" or sth similar | 13:05 |
| *** pbourke has quit IRC | 13:06 | |
| *** pbourke has joined #openstack-kolla | 13:07 | |
| *** jascott1 has joined #openstack-kolla | 13:09 | |
| *** lucasxu has joined #openstack-kolla | 13:11 | |
| *** jascott1 has quit IRC | 13:13 | |
| *** ducttape_ has joined #openstack-kolla | 13:15 | |
| *** goldyfruit has joined #openstack-kolla | 13:15 | |
| hawi | egonzalez: yes, multinode ocata-ocata upgrade | 13:16 |
| *** rwallner has quit IRC | 13:17 | |
| *** rwallner has joined #openstack-kolla | 13:17 | |
| *** Pedro_Pacheco has joined #openstack-kolla | 13:20 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 13:22 |
| Pedro_Pacheco | Hello folks! I very noob in deploy openstack with kolla-ansible. Here is the correct place to take doubts (deploy questions)? Thks! | 13:22 |
| egonzalez | Pedro_Pacheco, yeah, also can ask in ask.openstack.org | 13:23 |
| *** prateek has quit IRC | 13:24 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: DNM: test master branch https://review.openstack.org/456140 | 13:25 |
| sdake | morning folks | 13:26 |
| Pedro_Pacheco | Thank you, engozalez! I go to started there. | 13:26 |
| *** srnbckr has joined #openstack-kolla | 13:26 | |
| egonzalez | hawi, if can share logs or check https://bugs.launchpad.net/kolla-ansible/+bug/1692507 is your same bug to triage and potentially backport to ocata | 13:29 |
| openstack | Launchpad bug 1692507 in kolla-ansible "mariadb multinode upgrade broken ocata-pike" [Critical,Fix released] | 13:29 |
| hawi | yes, i can | 13:29 |
| *** ducttape_ has quit IRC | 13:29 | |
| hawi | just a sec | 13:29 |
| openstackgerrit | Mathias Ewald proposed openstack/kolla master: Add Redis Sentinel https://review.openstack.org/486299 | 13:29 |
| *** zhangfei has joined #openstack-kolla | 13:30 | |
| *** Pedro_Pacheco has quit IRC | 13:31 | |
| openstackgerrit | Mathias Ewald proposed openstack/kolla master: Add prometheus and related containers https://review.openstack.org/484882 | 13:32 |
| *** kristia__ has quit IRC | 13:40 | |
| *** kristian__ has joined #openstack-kolla | 13:41 | |
| *** ducttape_ has joined #openstack-kolla | 13:43 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 14:05 |
| openstackgerrit | Chason Chan proposed openstack/kolla master: Rearrange existing documentation to fit the new standard layout https://review.openstack.org/486423 | 14:11 |
| *** emccormick has joined #openstack-kolla | 14:14 | |
| egonzalez | pbourke, around? | 14:16 |
| *** zhangfei has quit IRC | 14:19 | |
| sdake | egonzalez sup dude how u been | 14:32 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 14:32 |
| *** zhangfei has joined #openstack-kolla | 14:32 | |
| egonzalez | hey sdake , fine, you going to PTG? | 14:33 |
| sdake | egonzalez quick q - is to to late to merge a change like this considering we have 1 mo to go: https://review.openstack.org/#/c/468632/ | 14:33 |
| sdake | egonzalez yes ptg and australia | 14:34 |
| sdake | board meetings at a minimum have to attend | 14:34 |
| egonzalez | sdake, cool, we'll see us at ptg, got TSP for denver | 14:35 |
| egonzalez | sdake, re mariadb, I would not upgrade version now, we just barely fixed a broken upgrade from ocata-master, i dont think we'll have enough time for testing | 14:36 |
| egonzalez | sdake, if some database expert raises help maybe we can make it for pike | 14:36 |
| *** athomas has quit IRC | 14:37 | |
| sdake | feels dangeorus not to use rdo's mariadb | 14:37 |
| sdake | but i understand the feeling of risk | 14:37 |
| *** kristian__ has quit IRC | 14:39 | |
| *** emccormick has quit IRC | 14:39 | |
| *** kristian__ has joined #openstack-kolla | 14:39 | |
| *** athomas has joined #openstack-kolla | 14:41 | |
| *** kristian__ has quit IRC | 14:43 | |
| *** mewald1 has quit IRC | 14:49 | |
| *** jamesbenson has joined #openstack-kolla | 14:57 | |
| openstackgerrit | James Benson proposed openstack/kolla-ansible master: Added note https://review.openstack.org/485727 | 14:57 |
| *** zhangfei has quit IRC | 14:59 | |
| jamesbenson | thanks egonzalez, gerrit passed and I have since rebased it again. :-) | 14:59 |
| jamesbenson | morning all :-) | 14:59 |
| *** kristian__ has joined #openstack-kolla | 15:01 | |
| *** kristian__ has quit IRC | 15:02 | |
| *** kristian__ has joined #openstack-kolla | 15:02 | |
| *** kristian__ has quit IRC | 15:02 | |
| *** kristian__ has joined #openstack-kolla | 15:02 | |
| *** lucasxu has quit IRC | 15:03 | |
| openstackgerrit | Merged openstack/kolla master: Remove python-wsme and python-pecan packages for centos https://review.openstack.org/486063 | 15:03 |
| *** kristian__ has quit IRC | 15:07 | |
| *** kristian__ has joined #openstack-kolla | 15:07 | |
| rwellum | ping sbezverk | 15:11 |
| inc0 | good morning | 15:12 |
| sbezverk | rwellum pong | 15:13 |
| sbezverk | good morning inc0 | 15:13 |
| *** rmart04 has quit IRC | 15:13 | |
| *** dklyle is now known as david-lyle | 15:14 | |
| rwellum | sbezverk - can you look at this nova piece from my cloud.yaml and see if it's correct for 5.x? I'm still having some issues. https://www.irccloud.com/pastebin/qXtAERfw/ | 15:14 |
| jamesbenson | morning inc0 | 15:14 |
| rwellum | sbezverk - this is what I have for 4.x https://www.irccloud.com/pastebin/46m9V29t/ | 15:14 |
| egonzalez | morning inc0 | 15:14 |
| egonzalez | inc0, noticed ceph gates randonmly fails due lack of disk, testing gates notices sometimes VMs have 90GB disk and other 30GB https://review.openstack.org/#/c/456140/ | 15:15 |
| *** jascott1 has joined #openstack-kolla | 15:16 | |
| inc0 | egonzalez: hmm, good to know, infra folk said that /opt will have more stor | 15:16 |
| inc0 | I'll dig into thins, thanks | 15:16 |
| sbezverk | rwellum it looks ok, not sure it does not work in your test bed.. I will be on training for a couple of days | 15:16 |
| sbezverk | so will not be able to help much.. | 15:17 |
| rwellum | sbezverk: thanks - my major concern was removing: placement_api: | 15:18 |
| *** jascott1 has quit IRC | 15:20 | |
| *** emccormick has joined #openstack-kolla | 15:27 | |
| *** kristia__ has joined #openstack-kolla | 15:27 | |
| *** kristian__ has quit IRC | 15:28 | |
| sbezverk | rwellum it should not matter, absence in cloud.yaml does not mean it will not be started.. | 15:29 |
| jamesbenson | inc0: silly question, but I'm a bit stuck on a concept and having trouble fully wrapping my head around it (or something). Ceph is distributed block/object storage. VM's live in block storage in ceph. So when you boot a VM, the VM lives in ceph but the VM itself in openstack might be listed as on a compute node without an OSD. So in essence, this brings CPU to storage? Is that correct? | 15:29 |
| rwellum | sbezverk: ack | 15:29 |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: Rearrange existing documentation to fit the new standard layout https://review.openstack.org/486659 | 15:29 |
| inc0 | egonzalez: change you linked seems to fail on mariadb | 15:31 |
| egonzalez | inc0, there are multiple failures, mariadb, horizon, and lack of disk :( | 15:32 |
| egonzalez | i think mariadb is related to disk, horizon is because a recent change | 15:32 |
| *** janki has quit IRC | 15:35 | |
| egonzalez | inc0, this fail due lack of disk http://logs.openstack.org/40/456140/15/check/gate-kolla-ansible-dsvm-deploy-ceph-centos-source-centos-7-2-node-nv/1930a51/console.html | 15:36 |
| *** coolsvap has quit IRC | 15:39 | |
| *** eaguilar is now known as eaguilar|afk | 15:40 | |
| *** mewald has joined #openstack-kolla | 15:43 | |
| *** iniazi has joined #openstack-kolla | 15:45 | |
| *** kristia__ has quit IRC | 15:48 | |
| *** iniazi_ has quit IRC | 15:49 | |
| *** kristian__ has joined #openstack-kolla | 15:49 | |
| *** eaguilar|afk is now known as eaguilar | 15:50 | |
| *** eaguilar is now known as eaguilar|afk | 15:50 | |
| *** prateek has joined #openstack-kolla | 15:50 | |
| kfox1111 | jamesbenson: it depends on how you configure ceph. | 15:50 |
| kfox1111 | ceph is just remote storage. | 15:50 |
| kfox1111 | you can configure vm's to host their root drive on ceph. | 15:51 |
| kfox1111 | or you can have the vm's still stored locally on each compute node. | 15:51 |
| kfox1111 | cinder volumes are always remote though with ceph. | 15:51 |
| jamesbenson | kfox1111: yeah, I'm just trying to understand that better. I know on Friday we were chatting about my setup a bit. And trying to understand how the storage nodes will work with the compute nodes since the computes dont have the 10G | 15:52 |
| *** kristian__ has quit IRC | 15:54 | |
| *** eaguilar|afk is now known as eaguilar | 15:55 | |
| *** eaguilar is now known as eaguilar|afk | 15:55 | |
| *** vhosakot has joined #openstack-kolla | 15:56 | |
| *** kristian__ has joined #openstack-kolla | 15:56 | |
| inc0 | egonzalez: so issue is | 15:56 |
| inc0 | we're not using devstack gate | 15:57 |
| inc0 | so while we have access to volume | 15:57 |
| inc0 | it's not getting mounted because we don't use devstack gate node prep | 15:57 |
| emccormick | jamesbenson your hypervisor (probably KVM) uses ceph libraries to mount a volume directly from a given compute host | 15:58 |
| emccormick | so hypervisor process is on a nova node and it mounts the disk from ceph | 15:58 |
| egonzalez | inc0, roger, thanks! | 15:59 |
| emccormick | kind of like if you stuck /var/lib/nova/instances on an NFS server, except in this case nothing actually gets mounted to the compute host itself | 15:59 |
| inc0 | I'll try to fix it | 15:59 |
| egonzalez | inc0, will see errors in binary rpm based gates | 16:00 |
| egonzalez | *horizon | 16:00 |
| *** serlex has quit IRC | 16:07 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Fix horizon secret_key error https://review.openstack.org/486681 | 16:07 |
| *** egonzalez has quit IRC | 16:08 | |
| *** omenv has quit IRC | 16:08 | |
| *** jascott1 has joined #openstack-kolla | 16:10 | |
| kristian__ | hey, could someone help me with networking? When I assign any network (Public and private) to a router with admin state up the port is down. L3 agent logs http://paste.openstack.org/show/616326/. ovs-vsctl http://paste.openstack.org/show/616328/. Router namespace http://paste.openstack.org/show/616329/ | 16:11 |
| *** jamesbenson_ has joined #openstack-kolla | 16:17 | |
| inc0 | kristian__: hmm...that's curious, can you show me your globals.yml? | 16:18 |
| kristian__ | cure | 16:18 |
| kristian__ | *sure | 16:18 |
| vhosakot | inc0, Jeffrey4l: Any reason why we are not running the init-runonce script in the kolla repo to check end-to-end OpenStack sanity at kolla gate. I see it was removed in https://review.openstack.org/#/c/401891/3/tools/deploy_aio.sh. | 16:19 |
| vhosakot | inc0, Jeffrey4l: https://github.com/openstack/kolla/commit/cc6d491c397efc50b96cdc33bebac1d2c2f07f00#diff-d33ddea4b52f4b549350318d57aee9b1L52 | 16:19 |
| kristian__ | inc0: http://paste.openstack.org/show/616331/ | 16:20 |
| inc0 | we run it in kolla-ansible code vhosakot | 16:20 |
| inc0 | and kolla gate runs kolla-ansible code | 16:20 |
| *** jascott1 has quit IRC | 16:20 | |
| *** jascott1 has joined #openstack-kolla | 16:21 | |
| inc0 | kristian__: br0, where do you create br0? | 16:21 |
| kristian__ | will paste network interfaces | 16:21 |
| inc0 | please | 16:21 |
| vhosakot | inc0: right, I know we run in kolla-ansible :) but I see init-runonce not run in the kolla gate logs.. let me check more. | 16:21 |
| kristian__ | I have only one nic on my node | 16:21 |
| *** mewald has left #openstack-kolla | 16:21 | |
| vhosakot | br | 16:22 |
| inc0 | vhosakot: ah you might be right | 16:22 |
| inc0 | I guess that's a bug in gates:) | 16:23 |
| kristian__ | inc0: http://paste.openstack.org/show/616333/ | 16:23 |
| vhosakot | inc0: I'm starting to work on upgrade gate.. and wanted to know if we need to run upgrade in kolla gate too (for which I need to check if end-to-end OpenStack sanity is first run in kolla gate, and it looks like we don't).. I'll dig more logs and check,. | 16:23 |
| vhosakot | brb 10 mins... rebooting | 16:23 |
| inc0 | vhosakot: just k-ans | 16:24 |
| inc0 | for now | 16:24 |
| vhosakot | inc0: cool cool, thanks for the confirmation | 16:24 |
| *** vhosakot has quit IRC | 16:24 | |
| *** jascott1_ has joined #openstack-kolla | 16:24 | |
| *** jascott1 has quit IRC | 16:24 | |
| inc0 | kristian__: hmm so I honestly don't know how will ovs behave with linuxbridge like that | 16:24 |
| kristian__ | it worked | 16:24 |
| inc0 | mind doing docker exec -it openvswitch_vswitchd vsctl show | 16:25 |
| kristian__ | I have rebooted the node and redeployed | 16:25 |
| *** jascott1_ has quit IRC | 16:25 | |
| kristian__ | inc0: looks like it hung | 16:25 |
| inc0 | right | 16:26 |
| *** jascott1 has joined #openstack-kolla | 16:26 | |
| inc0 | so we need to debug this | 16:26 |
| kristian__ | should I redeploy with linuxbridge? | 16:26 |
| inc0 | linuxbridge is not well tested | 16:26 |
| kristian__ | ok | 16:26 |
| inc0 | you cna try, but I really don't know how it behaves, all our gates and everything runs ovs | 16:26 |
| kristian__ | inc0: how could I redeploy it with ovs? | 16:27 |
| inc0 | kristian__: with master we allow to configure your own ovs | 16:27 |
| kristian__ | also I will be here in maybe 2 hours, but I should be here for 30 min | 16:28 |
| kristian__ | Im AIO | 16:28 |
| inc0 | single iface is issue:/ | 16:28 |
| kristian__ | I know | 16:29 |
| kristian__ | ovh | 16:29 |
| kristian__ | I dont have vrack compactible server | 16:29 |
| inc0 | it will be much easier in Pike | 16:30 |
| *** jascott1 has quit IRC | 16:31 | |
| *** jascott1 has joined #openstack-kolla | 16:32 | |
| *** jascott1 has quit IRC | 16:33 | |
| kristian__ | then I cannt wait | 16:33 |
| kristian__ | will it be working day0? | 16:33 |
| inc0 | well, what will happen is it will allow you to turn off deployment of ovs and use existing ovs | 16:34 |
| inc0 | which you could configure yourself | 16:34 |
| inc0 | however, let's get this fixed shall we | 16:34 |
| *** jascott1 has joined #openstack-kolla | 16:35 | |
| kristian__ | jamesbenson: hey james, you here | 16:35 |
| kristian__ | inc0: james helped me to set it up | 16:35 |
| jamesbenson | I'm here | 16:35 |
| jamesbenson | I'm not sure if you've modified your setup since we last chat, but inc0, kristian__ also did some funky stuff with mac spoofing/cloning, I wasn't sure about :-/ | 16:36 |
| kristian__ | jamesbenson: Im having problems with router ports being down int and ext networks | 16:36 |
| jamesbenson | yeah I've been following :-) | 16:36 |
| kristian__ | jamesbenson: I maybe found that solution right in openstack | 16:36 |
| jamesbenson | k | 16:36 |
| kristian__ | and its redeployed | 16:37 |
| jamesbenson | awesome | 16:37 |
| jamesbenson | Yeah, I haven't had a chance to really test a single eth in my setup here yet... | 16:37 |
| kristian__ | let me destroy it, disable lbaas and that stuff | 16:37 |
| kristian__ | I have only enabled that worked | 16:37 |
| jamesbenson | k | 16:37 |
| inc0 | kristian__: sooo | 16:37 |
| inc0 | if you're willing to do some manual hacking | 16:37 |
| inc0 | try installing and configuring ovs | 16:38 |
| inc0 | manually | 16:38 |
| kristian__ | and also the thing why on the first place I redeployed is I couldnt reach the public subnet from the vm, but I could reach it from anywhere else | 16:38 |
| inc0 | as in, add veth to ovs and use veth as api_iface | 16:38 |
| kristian__ | inc0: I will need to go soon, last ditch for now I will try redeploying with default kolla package config | 16:39 |
| inc0 | so | 16:39 |
| inc0 | try thins later: | 16:39 |
| kristian__ | I will ping you | 16:39 |
| inc0 | ok | 16:39 |
| inc0 | jamesbenson: that might interest you too in fact | 16:40 |
| kristian__ | and jamesbenson if the router ports will work, I will ping you with the mac spoofing in horizon | 16:40 |
| jamesbenson | yeah, I wanted to finish writing the doc with 1 eth | 16:40 |
| jamesbenson | k | 16:40 |
| kristian__ | you can put cidrs in there | 16:40 |
| inc0 | so what we can test out is: | 16:40 |
| inc0 | set https://github.com/openstack/kolla-ansible/blob/stable/ocata/ansible/roles/neutron/defaults/main.yml#L28 this var to false | 16:41 |
| inc0 | and manually configure ovs | 16:41 |
| kristian__ | inc0: we will, is it possible to deploy pike?? | 16:41 |
| kristian__ | stable in b2? | 16:41 |
| inc0 | well we test it all the time | 16:42 |
| inc0 | but it's not as tested as stable would be | 16:42 |
| inc0 | for kolla or other projects | 16:42 |
| kristian__ | I just need base kolla, serial nova proxy, lbaas and maybe metering | 16:43 |
| kfox1111 | jamesbenson: some of it depends on if you are building pets vs cattle. | 16:43 |
| kfox1111 | I usually do local storage for vm's as my use case is totally pets. | 16:43 |
| kfox1111 | totally cattle I mean. | 16:43 |
| *** eaguilar|afk has quit IRC | 16:43 | |
| kfox1111 | I put any data I really care about in cinder volumes for state. | 16:43 |
| kfox1111 | blowing out a single vm isn't horible. | 16:43 |
| kfox1111 | if you are caring about pets though, | 16:44 |
| kfox1111 | if you put the vm in cinder/ceph, its easier to mgirate from host to host. | 16:44 |
| kfox1111 | but then your relying on network attached storage for the pet. | 16:44 |
| jamesbenson | kfox1111: your analogy is very confusing! | 16:44 |
| inc0 | yeah if you want live migration at all, use network storage | 16:44 |
| inc0 | jamesbenson: you know pets vs cattle analogy? | 16:45 |
| jamesbenson | :-/ I'm thinking not, right now... | 16:46 |
| jamesbenson | live migration might be helpful for us | 16:46 |
| inc0 | no worries, it's been used in conferences a lot | 16:46 |
| inc0 | but unless you go to conferences, you might've missed it | 16:46 |
| jamesbenson | (only been to one, in austin) | 16:46 |
| inc0 | pets - vm's you don't want to die, you fix them like you cure sick pet | 16:47 |
| kfox1111 | jamesbenson: basically, | 16:47 |
| jamesbenson | got it | 16:47 |
| kfox1111 | sysadmins precloud are use to giving names to their machines. | 16:47 |
| jamesbenson | cattle can be slaughtered... | 16:47 |
| inc0 | cattle - vms which, when fail, you just destroy and move on, like you shoot sick cows | 16:47 |
| kfox1111 | lovingly nursing them back to health when they get sick. etc. | 16:47 |
| jamesbenson | yeah | 16:48 |
| kfox1111 | but users don't really care about machines. they care about services. | 16:48 |
| jamesbenson | qq: where is /var/lib/nova/instances? this path doesn't exists for me... | 16:48 |
| kfox1111 | in the same way, a user doesn't care if their hamburger comes from one cow, or 10. they just want a tastsy burger. :) | 16:48 |
| jamesbenson | lol... | 16:48 |
| kfox1111 | so in the cloud, you have lots of little vm's, and you try not to care about them so much as you care about the service that runs across many of them stays up. | 16:49 |
| inc0 | jamesbenson: /var/lib/docker/volumes/nova_compute ... | 16:49 |
| jamesbenson | thanks | 16:49 |
| *** jascott1 has quit IRC | 16:49 | |
| kfox1111 | if one gets sick, you don't spend much time on it, because there are a lot of them. you just replace it. | 16:49 |
| inc0 | it's docker volume | 16:49 |
| kfox1111 | jamesbenson: does that help? :) | 16:50 |
| kfox1111 | the cloud really is the industrial revolution of the sysadmin world. | 16:50 |
| kfox1111 | rather then hand crafting every machine, you build a production line. | 16:50 |
| kfox1111 | but a lot of folks gain a cloud and then treat it as a cheep place to run pets. (which its pretty bad at) | 16:51 |
| kfox1111 | because they see terminology/technology that they think feels comfortable from the old ways and don't reevaluate what it means to run in a cloud. | 16:51 |
| mgkwill | inc0: when is pike feature freeze? | 16:52 |
| jamesbenson | yeah I get it | 16:52 |
| inc0 | regular projects has it this week, we're 2 weeks later | 16:52 |
| inc0 | mgkwill: let's merge odl asap;) | 16:52 |
| mgkwill | inc0: thanks - looking to get odl in before - agreed | 16:52 |
| inc0 | it's in merge conflict now | 16:52 |
| mgkwill | inc0: yeah i know, will push update today | 16:53 |
| inc0 | cool, thanks Marcus | 16:53 |
| *** unicell has quit IRC | 16:54 | |
| inc0 | https://review.openstack.org/#/c/408872/ check dpdk change too | 16:54 |
| jamesbenson | kfox1111: If I run: rbd -p vms ls and it shows me my <<server-id>>_disk does that mean it is backed up on ceph then? treated as a pet? | 16:56 |
| *** vhosakot has joined #openstack-kolla | 16:56 | |
| kfox1111 | jamesbenson: fyi, I haven't gotten live migrate reliably working. I've kind of written it off. but it has been a while since I tried last. so maybe its better now. | 16:56 |
| kfox1111 | jamesbenson: yeah, I think that means the vm's ephemeral storage is in ceph. | 16:56 |
| *** omenv has joined #openstack-kolla | 16:57 | |
| jamesbenson | kfox1111: I think this is all of the info: http://paste.openstack.org/show/616339/ | 16:58 |
| kfox1111 | jamesbenson: yeah. I think its storing everything in ceph. | 16:59 |
| jamesbenson | ok, cool! That's the plan :-) | 16:59 |
| jamesbenson | keep ceph happy, keep cluster happy :-) | 16:59 |
| jamesbenson | mostly ;-) | 16:59 |
| inc0 | jamesbenson: if you do enable_ceph (or external ceph) it'll use ceph to everything unless explicitly said not to | 16:59 |
| jamesbenson | neat | 17:00 |
| inc0 | everything means nova, glance and cinder | 17:00 |
| jamesbenson | yeah, this is with external ceph | 17:00 |
| inc0 | nova for vm ephemeral disks, glance for images and cinder for volumes | 17:00 |
| inc0 | I'm not sure if swift will use ceph by default | 17:00 |
| jamesbenson | back to my original question, processors are on one node, VM is on another... so detaching compute and storage then? | 17:01 |
| inc0 | processors? | 17:01 |
| inc0 | well vm will run on copute node | 17:02 |
| inc0 | storage will be on different | 17:02 |
| inc0 | it really just mounts rbd storage to / | 17:02 |
| vhosakot | jamesbenson: the entire VM is not on ceph :) I'd say VM's CPU is on compute node, all of the VM's disk are on the storage nodes. | 17:02 |
| inc0 | well, not quite, but logic is the smae | 17:02 |
| jamesbenson | yes, but the instance disk is in ceph... | 17:02 |
| jamesbenson | ah okay | 17:02 |
| inc0 | right | 17:02 |
| jamesbenson | so it mounts the disk locally to bring storage to compute | 17:02 |
| vhosakot | yes | 17:03 |
| inc0 | kinda yeah | 17:03 |
| jamesbenson | that's the logic that I was missing... | 17:03 |
| inc0 | if you do lsblk on compute node you'll see ceph disks mounted | 17:03 |
| jamesbenson | yeah | 17:03 |
| jamesbenson | I noticed that | 17:03 |
| inc0 | this way live migration is much easier as it doesn't need to copy data on disk | 17:04 |
| inc0 | just attaches volume elsewhere | 17:04 |
| jamesbenson | so compute nodes can be in raid 6 for example and have no loss of speed due to the raid level | 17:04 |
| jamesbenson | or raid 5 or whatever | 17:04 |
| inc0 | well you need to ask yourself what data on compute node is worth securing | 17:04 |
| jamesbenson | virtually no data should be on compute node, correct? | 17:05 |
| inc0 | because since vms live on ceph, which is already replicated | 17:05 |
| inc0 | yeah | 17:05 |
| vhosakot | jamesbenson: yes, the disk must be mounted and partioned first and brought into the VM. I wrote a blog about it a few months ago --> https://communities.cisco.com/community/developer/openstack/blog/2017/01/13/how-to-attach-cinderceph-xfs-volume-to-a-nova-instance-in-openstack-horizon | 17:05 |
| jamesbenson | it would just be redundancy for the compute node OS | 17:05 |
| inc0 | very little is left that you'd normally care about | 17:05 |
| inc0 | right | 17:05 |
| jamesbenson | :-D | 17:05 |
| jamesbenson | nice vhosakot | 17:05 |
| vhosakot | jamesbenson: thanks, need to update it with other filesystem types, my customer was happy with xfs at the time I wrote the blog. | 17:06 |
| inc0 | I'd wish I have Vikrams perseverence and write myself :( | 17:06 |
| jamesbenson | in the past we used ceph for everything and live migration was only possible between like nodes, is that still the case. | 17:06 |
| inc0 | that's still the case | 17:06 |
| jamesbenson | I write too, but really to just keep notes to myself... best that an ignorant me can completely understand... | 17:07 |
| vhosakot | inc0: haha, yeah, I write blogs so 1) others can refer in the community 2) I can go myself to the blog and refer/update.. | 17:07 |
| jamesbenson | ditto | 17:07 |
| inc0 | yeah that | 17:07 |
| vhosakot | inc0: if I get something working, I'd die to share and save the steps :) | 17:07 |
| jamesbenson | my userspace is very minimal... | 17:07 |
| inc0 | I need to get into this mindset myself | 17:07 |
| jamesbenson | but documentation! man, it's key | 17:07 |
| jamesbenson | I'm doing one currently on using external ceph with kolla... with all of the commands, etc. | 17:08 |
| jamesbenson | because it took time I don't want to waste again | 17:08 |
| vhosakot | inc0: last time I tried ceph+swift, I failed very well :) | 17:09 |
| jamesbenson | so I'm just trying to understand everything before I write it all down :-) | 17:09 |
| inc0 | there is little value of having ceph+swift imho | 17:09 |
| *** harlowja has joined #openstack-kolla | 17:09 | |
| inc0 | swift is made to do what ceph dowa | 17:09 |
| vhosakot | inc0: right | 17:09 |
| inc0 | does | 17:09 |
| inc0 | and ceph rgw already has S3 api | 17:09 |
| vhosakot | yep | 17:09 |
| jamesbenson | inc0: why is that the case? makes little sense to me. | 17:09 |
| inc0 | jamesbenson: what? that swift does what ceph? | 17:10 |
| jamesbenson | no, like node migration | 17:10 |
| inc0 | I don't understand, node migration?;) | 17:11 |
| * inc0 confusing Monday morning | 17:11 | |
| jamesbenson | sorry, like node VM migration | 17:11 |
| inc0 | are you asking me why ceph helps with vm live migration? | 17:11 |
| jamesbenson | VM's can only live migrate with other similar compute nodes | 17:11 |
| inc0 | right | 17:11 |
| jamesbenson | but why? | 17:12 |
| inc0 | if VM is created with certain set of CPU features | 17:12 |
| inc0 | access to set of CPU features | 17:12 |
| inc0 | unless you migrate it to node with similar CPU | 17:12 |
| jamesbenson | that level of virtualization can't be replicated then? | 17:12 |
| jamesbenson | gotcha | 17:12 |
| jamesbenson | figured it was that... | 17:13 |
| inc0 | well, you can always specify what set of features you want to expose to VM | 17:13 |
| vhosakot | it would be cool if OpenStack can do "node migration" without downtime ;) | 17:13 |
| inc0 | and if you use lowest common denomminator across all cpy types in cluster | 17:13 |
| jamesbenson | I figured openstack would register those features and if compatible, allow the migration | 17:13 |
| inc0 | you can migrate freely | 17:13 |
| jamesbenson | hmm, maybe do that here then... | 17:14 |
| inc0 | vhosakot: evacuate host will try to live migrate all vms from cp | 17:14 |
| kristian__ | inc0: jamesbenson deployed, lets hope it works | 17:14 |
| inc0 | cpu node | 17:14 |
| jamesbenson | k | 17:14 |
| inc0 | jamesbenson: libvirt does | 17:14 |
| vhosakot | inc0: right, that is "compute node migration".. what if we want to migrate a network node? ;) | 17:14 |
| inc0 | I don't think nova scheduler takes that into account, but I haven't been following nova in this space for a while | 17:15 |
| inc0 | ti migh | 17:15 |
| inc0 | t | 17:15 |
| inc0 | vhosakot: technically "migration" would be just shutting it down and letting HA do it's thing;) | 17:15 |
| jamesbenson | hmmm, interesting stuff.... | 17:15 |
| vhosakot | inc0: if neutron HA works all the time ;) | 17:15 |
| *** ramishra has quit IRC | 17:16 | |
| jamesbenson | hey, I'm going to be deploying that! Don't be scaring me now! :-p | 17:16 |
| inc0 | moving router IP from one node to another without downtime is a science field of it's own;) | 17:16 |
| jamesbenson | all of networking stuff is a science of it's own to me... | 17:16 |
| jamesbenson | getting better, but still tricky! | 17:16 |
| vhosakot | all the namespaces must be moved first before we think about moving any neutron IP ;) | 17:17 |
| inc0 | well, yeah, this one detail has probably books written about it; | 17:17 |
| vhosakot | actually neutron HA router is pretty stable.. | 17:17 |
| * jamesbenson whew | 17:17 | |
| vhosakot | HA DHCP was not the best when I last tested | 17:17 |
| kristian__ | inc0: its down :( | 17:17 |
| inc0 | :/ | 17:17 |
| jamesbenson | :-( | 17:17 |
| inc0 | kristian__: soo, here's an idea | 17:17 |
| kristian__ | dunno how long I will be available | 17:17 |
| kristian__ | destroy openstack? | 17:18 |
| inc0 | https://github.com/openstack/kolla-ansible/blob/stable/ocata/ansible/roles/neutron/defaults/main.yml#L28 | 17:18 |
| inc0 | change this to false | 17:18 |
| inc0 | install ovs on host manually | 17:18 |
| inc0 | or in containers and configure it | 17:18 |
| inc0 | rather, better install it with kolla, destroy everything besides ovs | 17:18 |
| inc0 | get into ovs container, reconfigure bridges so it'll have ports bridged properly | 17:19 |
| kristian__ | inc0: so manually destroy containers? or with kolla-ansible destroy | 17:19 |
| inc0 | manually + volumes too | 17:19 |
| kristian__ | ok | 17:19 |
| kristian__ | let me delete it first | 17:19 |
| inc0 | kolla ansible destroy removes containers and volumes | 17:19 |
| inc0 | just keep ovs | 17:19 |
| kristian__ | so * except neutron_openvswitch_agent | 17:20 |
| *** jascott1 has joined #openstack-kolla | 17:20 | |
| inc0 | no | 17:20 |
| inc0 | openvswitch_vswitchd and db | 17:20 |
| kristian__ | ok | 17:20 |
| inc0 | after that http://paste.openstack.org/show/616340/ add this to your globals | 17:22 |
| inc0 | configure switches you have in container manually, add external iface to it, make sure bridges and stuff are named correctly | 17:23 |
| inc0 | add virtual interface + IP for api_network | 17:23 |
| jamesbenson | kfox1111: sorry one last time back to ceph/VMs/etc.... how do the compute nodes talk to ceph? I have the ceph on a 10G and compute nodes without 10G connection. Is it through the public network? | 17:23 |
| inc0 | and try to deploy on top of this. I never tried that but technically if you do it correctly neutron should just use existing ovs containers | 17:24 |
| inc0 | make sure br-ex exists and interface specified in neutron_external_interface is in it | 17:26 |
| kristian__ | inc0: where in globals to add the paste? | 17:26 |
| inc0 | whereever | 17:26 |
| kristian__ | ok | 17:27 |
| inc0 | ordering in globals doesn't matter | 17:27 |
| kristian__ | then also what to do with ovs? Im a noob there | 17:27 |
| kfox1111 | jamesbenson: the ceph public network. | 17:27 |
| inc0 | kristian__: sooo | 17:28 |
| inc0 | you need to add new interface | 17:28 |
| kristian__ | its gonna be long :) | 17:28 |
| inc0 | ok, let's assume you have eth0 on public network | 17:28 |
| kristian__ | also what about the existing br0? and veno0 and veno1? | 17:28 |
| kristian__ | correct | 17:29 |
| inc0 | (guys if I'm mistaken correct me, imagining ifaces in my head is prone to failure) | 17:29 |
| inc0 | you need to create new interface for network interface | 17:29 |
| kristian__ | if neutron works in the end, then it should be good or no? | 17:29 |
| inc0 | and give it IP in same network, as public | 17:30 |
| inc0 | kristian__: we're essentially replicating br0 and veno0+1 in ovs | 17:30 |
| kristian__ | ok, but commands | 17:30 |
| kristian__ | or ssh is better? | 17:30 |
| inc0 | what neutron really needs is name of bridge to which connect flat network | 17:30 |
| inc0 | and tha'ts br-ex | 17:31 |
| jamesbenson | thanks kfox1111 | 17:31 |
| inc0 | kristian__: ovh gives you access to console if you mess up network right? | 17:31 |
| kristian__ | yeah | 17:32 |
| inc0 | cool | 17:32 |
| kristian__ | aten viewer | 17:32 |
| kristian__ | what its called | 17:32 |
| inc0 | so we need to create new virtual interface and add IP of eth0 to it | 17:32 |
| inc0 | and remove this IP from eth0 | 17:32 |
| *** sambetts is now known as sambetts|afk | 17:33 | |
| inc0 | then from inside container | 17:33 |
| kristian__ | so a virtual interface, not the bridge | 17:33 |
| inc0 | ovs-vsctl show - should show you if br-ex exists, if it doesn't run ovs-vsctl --no-wait add-br br-ex | 17:33 |
| inc0 | well yeah, interface should have IP and we'll connect interface to bridge with wth0 | 17:34 |
| inc0 | eth0 | 17:34 |
| kristian__ | br-ex is there | 17:34 |
| inc0 | ok | 17:34 |
| kristian__ | and it doesnt have the public ip of the node | 17:35 |
| inc0 | ovs-vsctl --no-wait add-port br-ex eth0 | 17:35 |
| inc0 | ovs-vsctl --no-wait add-port br-ex veth | 17:35 |
| kristian__ | ok | 17:35 |
| inc0 | then we break our networking | 17:35 |
| inc0 | so careful | 17:36 |
| kristian__ | ok | 17:36 |
| inc0 | ip a flush dev eth0 && ip a add << cidr of eth0 >> dev veth | 17:37 |
| inc0 | we remove address from eth0 and add it to interface bridged to it | 17:38 |
| kristian__ | cidr? | 17:38 |
| kristian__ | or the ip | 17:38 |
| inc0 | cidr | 17:38 |
| kristian__ | ok | 17:38 |
| inc0 | it needs netmask too | 17:38 |
| kristian__ | in that same command? | 17:39 |
| inc0 | well ip a flush dev eth0 will break network | 17:39 |
| inc0 | if you have && it might fix network immediatly | 17:39 |
| inc0 | unless we messed something up then we'll need this console of yours;) | 17:39 |
| kristian__ | ip 137.87.14.22 and netmask 255.255.255.0, so 137.87.14.0/24? | 17:39 |
| inc0 | 137.87.14.22/24 | 17:40 |
| kristian__ | ok | 17:40 |
| kristian__ | going to login to the console first | 17:40 |
| kristian__ | running the command | 17:42 |
| kristian__ | cannt find device veth | 17:43 |
| kristian__ | Cannot find device "veth" | 17:43 |
| kristian__ | inc0: | 17:44 |
| inc0 | well | 17:44 |
| inc0 | we need to create this first | 17:45 |
| kristian__ | but still have ssh :D | 17:45 |
| inc0 | yeah | 17:45 |
| jamesbenson | the commands I sent you before should create veth1 and veth0 | 17:45 |
| inc0 | yup | 17:45 |
| kristian__ | I have that interfaces file jamesbenson | 17:45 |
| jamesbenson | yreah | 17:45 |
| kristian__ | route add thing | 17:46 |
| kristian__ | for br0 | 17:46 |
| kristian__ | jamesbenson: run that and then run the command inc0 gave me? | 17:47 |
| kristian__ | or destroy fully ovs, run the command and deploy openstack? | 17:47 |
| jamesbenson | well my interfaces should create the veth0 and veth1... once created, you should be able to run inc0's commands | 17:47 |
| kristian__ | ok | 17:48 |
| kristian__ | ip link add veno0 ... | 17:48 |
| jamesbenson | I'm not trying to butt in too much... | 17:48 |
| inc0 | it's hacky no matter what we do | 17:48 |
| vhosakot | 137.87.14.22/24 is same as 137.87.14.0/24 :) | 17:49 |
| inc0 | btw jamesbenson in Pike something like this might actually be better solution to single iface problem | 17:49 |
| inc0 | vhosakot: same network | 17:49 |
| kristian__ | File exists for the ip link add | 17:49 |
| jamesbenson | cool | 17:49 |
| inc0 | but when you add ip to iface you need to specify ip and network | 17:49 |
| vhosakot | inc0: right, same block of 256 IPs :) | 17:49 |
| kristian__ | I only have /28 and its on ovh so assign mac to ips | 17:50 |
| inc0 | kristian__: so you already have veno, and you can run my commands with veno instead of verh | 17:50 |
| vhosakot | technically, 254 usable... can't really use 137.87.14.0 and 137.87.14.255 (broadcast IP) for any host. | 17:50 |
| kristian__ | and also Im bombarded with icmps | 17:50 |
| kristian__ | inc0: still cannt find device "veth" | 17:51 |
| kristian__ | jamesbenson: we are going to celebrate in august | 17:52 |
| inc0 | kristian__: replace veth with veno1 | 17:54 |
| kristian__ | ok | 17:54 |
| kristian__ | gotta go | 17:54 |
| inc0 | or whatever name you used with Jameses command | 17:54 |
| kristian__ | will ping you later | 17:54 |
| inc0 | good luch | 17:54 |
| *** kristian__ has quit IRC | 17:54 | |
| inc0 | luck;) | 17:54 |
| *** kristian__ has joined #openstack-kolla | 17:55 | |
| *** omenv has quit IRC | 17:58 | |
| *** kristian__ has quit IRC | 17:59 | |
| *** srnbckr has quit IRC | 18:09 | |
| *** serlex has joined #openstack-kolla | 18:12 | |
| *** prateek has quit IRC | 18:17 | |
| *** emccormick has quit IRC | 18:21 | |
| *** emccormick has joined #openstack-kolla | 18:25 | |
| *** srnbckr has joined #openstack-kolla | 18:27 | |
| *** itlinux has joined #openstack-kolla | 18:53 | |
| *** itlinux_ has joined #openstack-kolla | 18:53 | |
| *** ducttape_ has quit IRC | 19:14 | |
| *** manheim has quit IRC | 19:15 | |
| *** ducttape_ has joined #openstack-kolla | 19:16 | |
| *** ducttape_ has quit IRC | 19:19 | |
| *** ducttape_ has joined #openstack-kolla | 19:19 | |
| *** iniazi has quit IRC | 19:21 | |
| *** ducttape_ has quit IRC | 19:24 | |
| *** ducttape_ has joined #openstack-kolla | 19:35 | |
| *** manheim has joined #openstack-kolla | 19:40 | |
| *** manheim has quit IRC | 19:43 | |
| *** manheim has joined #openstack-kolla | 19:43 | |
| *** awiddersheim has quit IRC | 19:44 | |
| *** awiddersheim has joined #openstack-kolla | 19:44 | |
| sbezverk | kfox1111: ping | 19:52 |
| *** mewald has joined #openstack-kolla | 19:57 | |
| openstackgerrit | Mathias Ewald proposed openstack/kolla master: Add sensu images https://review.openstack.org/486379 | 19:59 |
| openstackgerrit | Mathias Ewald proposed openstack/kolla master: Add sensu images https://review.openstack.org/486379 | 20:10 |
| mewald | For sensu client I need to be able to copy some files into the container here and there. For example, for sensu-plugins-ceph there has to be a keyring file. The mailer plugin takes a parameter to a template file, etc. How to deal with these requirements? Is there a mechanism that allows copying arbitrary files into a container? I am not sure how to deal with this atm | 20:12 |
| openstackgerrit | Mathias Ewald proposed openstack/kolla-ansible master: Add Elasticsearch to Grafana https://review.openstack.org/486747 | 20:40 |
| *** kristian__ has joined #openstack-kolla | 20:40 | |
| *** emccormick has quit IRC | 20:42 | |
| *** ansmith has quit IRC | 20:44 | |
| *** kristian__ has quit IRC | 20:44 | |
| *** serlex has quit IRC | 20:46 | |
| *** ducttap__ has joined #openstack-kolla | 20:46 | |
| *** mewald has quit IRC | 20:47 | |
| *** ducttape_ has quit IRC | 20:50 | |
| *** kristian__ has joined #openstack-kolla | 20:50 | |
| *** robellison has joined #openstack-kolla | 20:53 | |
| robellison | is there any way of securing the internal API with TLS? | 20:55 |
| robellison | docs suggest only the external api | 20:55 |
| *** rhallisey has quit IRC | 21:02 | |
| vhosakot | robellison: kolla has TLS just for the external VIP currently. Why do you want TLS on the internal network? will your internal network be an untrusted segment on the public internet (which should not be the case as internal network is always inside the datacenter behind a firewall) | 21:11 |
| vhosakot | robellison: https://github.com/openstack/kolla-ansible/blob/master/doc/advanced-configuration.rst#tls-configuration | 21:11 |
| *** pc_m has quit IRC | 21:11 | |
| robellison | i'm pretty sure PCI-DSS needs all connections to be encrypted | 21:12 |
| robellison | even if they are just control plane | 21:12 |
| vhosakot | robellison: kolla supports https for the internal network tho --> https://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml#L263 | 21:13 |
| jamesbenson | vhosakot, can admin be https based off that that? line 264? | 21:14 |
| robellison | that is interesting.. same certificate i guess? | 21:15 |
| vhosakot | jamesbenson: yes, in that case, your clients need to auth using https as keystone runs https when admin network has https enabled. | 21:16 |
| vhosakot | jamesbenson: https://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml#L442 | 21:16 |
| jamesbenson | neat | 21:17 |
| vhosakot | cert | 21:17 |
| vhosakot | robellison: you can provide certs for external VIP which will be passed to haproxy --> https://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml#L429-L430 | 21:19 |
| robellison | i cant see where to specify the internal cert | 21:19 |
| robellison | does anyone run this in large prod environments? | 21:19 |
| vhosakot | robellison: yeah, no certs for internal network.. let me check the code | 21:20 |
| vhosakot | robellison: I don't see any certs for the internal network, just https endpoint. | 21:22 |
| robellison | looks like it generates it's own certs and distributes them | 21:23 |
| robellison | in the certificates role | 21:23 |
| *** emccormick has joined #openstack-kolla | 21:24 | |
| vhosakot | robellison: yes, external certs are created using openssl in https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/certificates/tasks/generate.yml#L22-L31 | 21:25 |
| jamesbenson | robellison: I'm using tls for external, but internal is still http.... granted I did not enable the https option. But i'm not seeing it int he code besides the group_vars file... | 21:25 |
| openstackgerrit | Merged openstack/kolla-ansible master: Improve Swift ring setup sample script https://review.openstack.org/482508 | 21:25 |
| robellison | @jamesbenson: do you run Ceph as well? | 21:27 |
| jamesbenson | yes | 21:27 |
| vhosakot | robellison: HTTPs is top of TLS | 21:30 |
| robellison | jamesbenson : i'm strugging to find any decent reference architecutres with Ceph. they all look like they have the Glance/Cinder APIs on the Ceph-client network | 21:32 |
| robellison | which dosn't seem ideal | 21:32 |
| vhosakot | robellison: right, cinder and glance are ceph clients with ceph.conf. why do you think this is not ideal? :) | 21:34 |
| jamesbenson | we use all of the api's with ceph..... | 21:35 |
| *** ansmith has joined #openstack-kolla | 21:38 | |
| robellison | because my understanding is that the containers/servers that host those APIs need to be both public accessible and also on the ceph-client network | 21:38 |
| robellison | so a compromise would give the attacker full access to everything - all hypervisors and all storage nodes | 21:38 |
| robellison | i'm looking for ways of separating this so there is never a single system that is both public facing and on the internal ceph-client network | 21:39 |
| vhosakot | robellison: there must be a secure API network for all the API traffic separate from the public network | 21:40 |
| *** pc_m has joined #openstack-kolla | 21:40 | |
| robellison | any good diagrams? | 21:40 |
| robellison | unless the secure API network is still accessible from the public networks? | 21:41 |
| vhosakot | robellison: https://docs.openstack.org/ocata/networking-guide/ and https://docs.openstack.org/arch-design/ has nice info | 21:41 |
| vhosakot | nothing except the external VIP must be public-facing. clients hit the external frontend VIP that haproxy then sends the traffic on the required private network (API/storage/admin/etc) | 21:43 |
| *** manheim has quit IRC | 21:43 | |
| *** manheim has joined #openstack-kolla | 21:44 | |
| robellison | i've read both of those before, but they dont really clarify | 21:44 |
| robellison | as i understand it... | 21:44 |
| robellison | api user -> haproxy -> cinder -> ceph | 21:44 |
| *** unicell has joined #openstack-kolla | 21:45 | |
| robellison | so a user would be executing code on cinder (wherever that is running) and that would have unfiltered access to ceph | 21:45 |
| SamYaple | robellison: technically a user should only be hitting the loadbalancer | 21:46 |
| robellison | but that will pass them straight though | 21:46 |
| SamYaple | well, no, it will load balance thier request | 21:46 |
| SamYaple | its not like the client can talk to ceph directly | 21:46 |
| SamYaple | if you are suggesting that when cinder gets compromised they can access ceph, the answer is "yes" | 21:46 |
| robellison | any http/s request will end up on the cinder api. regardless of if it is valid or not | 21:47 |
| SamYaple | the *user cannot talk to ceph directly | 21:47 |
| robellison | ok.. any way around it? | 21:47 |
| SamYaple | that is cinder though, that doesnt get to ceph | 21:47 |
| robellison | no, but the server that it is running on is also on the ceph-client network | 21:47 |
| robellison | so has full access to everything? | 21:48 |
| vhosakot | robellison: I see more like: user (using OpenStack CLI) --> hits external frontend VIP of haproxy --> cinder backend API on the active controller --> uses ceph on storage network to do stuff on the remote volume... | 21:48 |
| SamYaple | robellison: no, it has access to whatever the key it has can access | 21:48 |
| *** manheim has quit IRC | 21:48 | |
| robellison | IP wise though it has acess to everything | 21:48 |
| SamYaple | for cinder, that is RX for glance pool, RWX for cinder and RWX for nova i believe | 21:48 |
| robellison | so lets say apache has a vulnerability that means you can run some remote code via an http/s request | 21:49 |
| SamYaple | if an outside user can execute arbitrary code inside your network, there are bigger security risks | 21:49 |
| SamYaple | the biggest i can think of is access to memcache which can be used to yank out valid admin tokens | 21:49 |
| robellison | yeah, thats what we need to protect from. normal would be to terminate those requests in a DMZ and only allow very specific protocols though the firewall | 21:50 |
| SamYaple | you are welcome to only allow users to talk to the _external_ haproxy endpoint | 21:50 |
| robellison | but that will proxy it though | 21:50 |
| SamYaple | but i mean you are worried about ceph when a user has arbitrary code execution inside your network | 21:51 |
| SamYaple | thats silly. if they can do that, they can have full access to whatever the api had access to | 21:51 |
| *** eaguilar has joined #openstack-kolla | 21:51 | |
| robellison | yeah that is my point really.. i want to move all the APIs to a DMZ | 21:51 |
| SamYaple | and the api *needs* access to rabbitmq, the database, memcached, and in some cases ceph | 21:51 |
| robellison | i can see how to do it with most of them, but not cinder/glance when ceph is involved | 21:52 |
| SamYaple | regardless, the api wil lalways have access to the database and rabbitmq | 21:52 |
| SamYaple | so an attacker would too | 21:52 |
| vhosakot | the most vulnerable/insecure nodes are the controllers IMO as they sit on both the external/public network and the internal network | 21:52 |
| SamYaple | they could just inject a message that cinder-volume would do all the stuff to ceph | 21:52 |
| *** jascott1 has quit IRC | 21:52 | |
| *** jascott1 has joined #openstack-kolla | 21:53 | |
| robellison | we can limit to only rabbitmq and the database though | 21:53 |
| SamYaple | ok. so over rabbitmq i can control ceph | 21:53 |
| robellison | so the attacker would have to be someone with knowledge of openstack | 21:53 |
| SamYaple | yes. they would. to be able to compromise the api | 21:54 |
| SamYaple | thats assumed. yes | 21:54 |
| robellison | wheras with the API on the internal secure network, you dont have to know it's openstack | 21:54 |
| vhosakot | yes, totally possible to hack the API endpoint | 21:54 |
| robellison | you just have full access to a large network of high value targets with no firewalls | 21:54 |
| robellison | interesting conversation... | 21:55 |
| robellison | how do people usually try and mitigate this? | 21:55 |
| *** jascott1 has quit IRC | 21:55 | |
| jamesbenson | has anyone tried to impliment openstack-anisble security stig rules? to harden openstack? | 21:56 |
| SamYaple | robellison: i firewall malicious users. you can detect abnormal traffic with a number of methods. but at the end of the day, i trust the api isnt compromised | 21:56 |
| vhosakot | robellison: if API is on the internal network, what would be on the external network then? and what should clinets hit from the public network | 21:56 |
| SamYaple | jamesbenson: well kolla doesnt *need* alot of those rules because we dont run services in the contaienr like... bluetooth *shudders* | 21:56 |
| vhosakot | externally facing endpoints are always risky :) not just in OpenStack :) | 21:56 |
| SamYaple | jamesbenson: but kolla-ansible does lock down things where needed, like memcached | 21:57 |
| jamesbenson | and I thought only the network node, theoretically needs to sit publicly? | 21:57 |
| SamYaple | jamesbenson: not even that needs to sit publically. i do double-natting | 21:57 |
| jamesbenson | the other big security issue with kolla is just the whole, selinux... | 21:57 |
| robellison | vhosakot : i was hoping something like: user -> haproxy -> api (in dmz) -> message queue ->api (internal) -> ceph | 21:57 |
| SamYaple | for other reasons than security, but it works out | 21:57 |
| SamYaple | robellison: thats just getting to ceph in more steps though | 21:58 |
| jamesbenson | from my understanding at least | 21:58 |
| robellison | but the user will never hit it | 21:58 |
| vhosakot | yes, totally possible to have a DMZ with a public message queue allowing just the OpenStack API TCP ports... | 21:58 |
| vhosakot | robellison: ^^ | 21:58 |
| SamYaple | robellison: if you compromize the api in the dmz, you can get to ceph... | 21:58 |
| *** jascott1 has joined #openstack-kolla | 21:58 | |
| robellison | SamYaple : how ? if it is totally firealled off | 21:59 |
| SamYaple | robellison: message queue messages? | 21:59 |
| robellison | vhosakot : i cant see how to do it with ceph involved | 21:59 |
| *** jamesbenson has quit IRC | 21:59 | |
| robellison | SamYaple ; yeah, but they will be correctly formed. the user would not be able to port-scan or try and compromise other hosts | 22:00 |
| SamYaple | robellison, even correctly formed messages can interact with ceph. but its silly to suggest a user could get arbitary command exectuion on the api but the messages wont have vulnerable points | 22:01 |
| *** rwallner has quit IRC | 22:01 | |
| *** jascott1 has quit IRC | 22:01 | |
| robellison | yeah they can interact with ceph.. but nothing else | 22:01 |
| vhosakot | there is _always_ the point of intersection between the public network and the private network, and this is the most insecure/vulnerable point in any cloud | 22:01 |
| robellison | that is a lot better than being able to interact with evertyhing with no restriction and no way of intercepting it | 22:02 |
| SamYaple | ok. youre acting like ceph is an open book | 22:02 |
| SamYaple | it has authentication | 22:02 |
| SamYaple | the only service that is an api and has a ceph key is glance. and thats a known security issue | 22:02 |
| robellison | there are all the kvm hosts on there as well | 22:02 |
| *** Manheim has joined #openstack-kolla | 22:03 | |
| robellison | and unfortunately in my scenario, literally the whole network | 22:03 |
| robellison | SamYaple : not cinder as well? | 22:03 |
| SamYaple | robellison: only cinder-volume which is only interacted with over message queue | 22:04 |
| robellison | ok cool.. whats the impact of not allowing external users access to glance? | 22:04 |
| robellison | i guess they can still upload images via horizon? | 22:04 |
| vhosakot | robellison: are you using OpenStack for a public cloud usecase? | 22:04 |
| *** itlinux has quit IRC | 22:04 | |
| *** itlinux_ has quit IRC | 22:04 | |
| robellison | vhosakot : that is the intention | 22:05 |
| SamYaple | robellison: if they only interact through horizon, literally no other api needs to be exposed | 22:05 |
| vhosakot | robellison: ah..if public cloud, good to throw in the DMZ+rabbit between the fronend public API and the backend private API | 22:05 |
| SamYaple | otherwise you cant really limit glance unfortunately | 22:05 |
| robellison | i guess i could not deploy glance on the DMZ servers | 22:06 |
| *** ducttape_ has joined #openstack-kolla | 22:06 | |
| vhosakot | robellison: yes, I agree that the OpenStack docs do not have great info for public cloud usecases. | 22:06 |
| robellison | and horizon could point to the internal API (through the firewall) | 22:06 |
| SamYaple | look, i feel you are focusing on the wrong issue here. lets talk about the resources the api absolutely must have access to | 22:06 |
| SamYaple | message queue, to which they can do alot | 22:06 |
| SamYaple | database, between those two they can do anything in openstack | 22:06 |
| SamYaple | did you know you can initiate a volume transfer between tenants? | 22:07 |
| vhosakot | DDOS'ing the DMZ can break rabbit :) | 22:07 |
| robellison | as an admin | 22:07 |
| SamYaple | since they have full admin access to openstack they could read any data out of ceph they wanted | 22:07 |
| SamYaple | (apis are full admins in openstack) | 22:08 |
| vhosakot | ^^ | 22:08 |
| SamYaple | by default at least. technically you can lock that down with policy.json | 22:08 |
| SamYaple | but guess what, they compromised the thing that reads the policy.json | 22:08 |
| SamYaple | and they have full db access | 22:08 |
| *** ducttap__ has quit IRC | 22:09 | |
| SamYaple | im not saying you dont hve valid security concerns | 22:09 |
| robellison | you can protect the db with conductors now though.. to some extent | 22:09 |
| SamYaple | libvirt should absolutely be on its own network (or configurable) | 22:09 |
| SamYaple | but the apis are the _last_ line of defense, not the first | 22:09 |
| SamYaple | the pipelines are there for you to modify | 22:10 |
| SamYaple | you can add in whatever security auditing code you want before it hits openstack | 22:10 |
| *** jgriffith has quit IRC | 22:10 | |
| SamYaple | then proper firewalling and an IDS or similiar | 22:10 |
| *** jgriffith has joined #openstack-kolla | 22:10 | |
| robellison | i think it would probably be acceptable for a user to only be able to upload images via horizon | 22:11 |
| robellison | so no access to glance | 22:11 |
| robellison | the rest can go in a DMZ if i can find any documentation | 22:11 |
| robellison | i would feel a lot better having that layer of IPS/etc in between users and internal networks | 22:12 |
| *** jgriffith has quit IRC | 22:12 | |
| robellison | plus, that is the only way it would get through an audit i suspect | 22:12 |
| SamYaple | robellison: its not hta simple | 22:13 |
| vhosakot | robellison: so, block glance port and only allow horizon port so users cannot use glance CLI and are forced to use horizon thru the DMZ/IPS to create an image? | 22:14 |
| SamYaple | so a standard boot command is "openstack server create --flavor 1 --netowkr 123 --image ubuntu server01" | 22:14 |
| SamYaple | but that right there needs access to glance to actually boot | 22:14 |
| SamYaple | the user makes a call to glance to look up info about 'ubuntu' as an image | 22:14 |
| robellison | vhosakot : yeah, if that is possible. that seems to be the API that has the most access | 22:14 |
| vhosakot | robellison: yes, just block _all_ the ports in the DMZ/IPS except horizon... sounds safe | 22:15 |
| robellison | SamYaple : is that not dealt with by Nova? | 22:15 |
| *** jgriffith has joined #openstack-kolla | 22:15 | |
| robellison | nova -> glance -> ceph | 22:15 |
| *** jgriffith has quit IRC | 22:16 | |
| SamYaple | robellison: nope. you can use --debug to see all the api calls the client actually makes | 22:16 |
| robellison | i wonder if that needs full access to ceph to do that or just the glance registry | 22:18 |
| SamYaple | robellison: both the registry and api need access to the database | 22:18 |
| vhosakot | robellison: blocking OpenStack CLIs for the user will make the user not be able to automate anything as GUI automation is boring sometimes :) | 22:18 |
| SamYaple | and i think they both need access to ceph glance bool | 22:18 |
| *** jgriffith has joined #openstack-kolla | 22:19 | |
| SamYaple | i know the api dumps data and pulls data, but i believe the registry audits the pool | 22:19 |
| robellison | complicated | 22:19 |
| SamYaple | either way, the API most definetley needs access tothe ceph pool | 22:19 |
| SamYaple | yea glance is probably the worst service we all still use | 22:19 |
| *** jgriffith has quit IRC | 22:19 | |
| *** awiddersheim has quit IRC | 22:21 | |
| robellison | maybe some pre-authentication at the edge would be the way to go | 22:21 |
| SamYaple | and you can do that! | 22:21 |
| SamYaple | hell you can do that an integrate it into openstack via pipelines | 22:21 |
| *** awiddersheim has joined #openstack-kolla | 22:21 | |
| SamYaple | but i would look at apis as the last layer of defense in an attack and protect them accordingly | 22:22 |
| vhosakot | robellison: https://www.openstack.org/marketplace/public-clouds/ has companies building openstack public cloud | 22:22 |
| *** emccormick has quit IRC | 22:22 | |
| robellison | vhosakot : yeah, there are a lot of big companies that say they have solutions, but they all seem to assume no security | 22:23 |
| robellison | flat networks and no firewalls | 22:23 |
| *** jgriffith has joined #openstack-kolla | 22:23 | |
| robellison | SamYaple : what's pipelines? | 22:24 |
| SamYaple | paste.ini | 22:24 |
| vhosakot | robellison: flat network is fine, no fw is not coot :) | 22:24 |
| SamYaple | all the messages for an api go through that | 22:24 |
| vhosakot | cool* | 22:24 |
| SamYaple | robellison: to be fair, we are dealing with tehcnology that literally has no auth or security. | 22:24 |
| SamYaple | memcache has none | 22:24 |
| SamYaple | qemu's vnc connection.. has none | 22:25 |
| robellison | yeah there seems to be a fair few worrying bits | 22:25 |
| robellison | it would be useful if haproxy routed and validated all the api calls | 22:27 |
| robellison | and did pre-auth | 22:27 |
| SamYaple | it does not, but *you* can via pipelines | 22:27 |
| SamYaple | mind you, these apis are literally hammered on thousands if not millions of times a day | 22:28 |
| SamYaple | automated tests and otherwise | 22:28 |
| robellison | SamYaple : cool, i'll have a look at it | 22:28 |
| SamYaple | to be clear, i do not recommend you trying ot validate the api calls before it gets to the service | 22:28 |
| SamYaple | i think there is a good chance you will introduce a security vulnerability | 22:29 |
| SamYaple | but you *can* do it | 22:29 |
| robellison | i was thinking just checking that it's an http/s request and the routing was something that exists | 22:30 |
| robellison | anyway, lots of thinking | 22:31 |
| SamYaple | haproxy definetely does that | 22:31 |
| SamYaple | mode http vs tcp | 22:32 |
| robellison | i'm pretty sure you can make it check the routes and verbs etc as well | 22:33 |
| vhosakot | robellison: what creds must the user supply to pre-auth at the external haproxy VIP in the DMZ? | 22:34 |
| robellison | vhosakot : hopefully it should be transparrent.. so the same as they would usually | 22:34 |
| robellison | i'm sure it wouldn't be too hard to re-route the /authenticate method - assuming that's what it is | 22:35 |
| vhosakot | robellison: so users must auth twice then... pre-auth and then OpenStack's keystone auth? | 22:35 |
| *** ducttape_ has quit IRC | 22:36 | |
| vhosakot | robellison: you can ship fingerprint recognition pads to all your users ;) | 22:36 |
| robellison | no, more that without the pre-auth they wouldn't get any further than haproxy (or something with no access to anything else) | 22:37 |
| robellison | except keystone i guess | 22:37 |
| robellison | access to keystone api with no pre-auth. everything else validated that is has a valid token | 22:39 |
| SamYaple | robellison: if the token doesnt validate in the pipeline the request doesnt even hit the other apis | 22:40 |
| SamYaple | just saying... | 22:40 |
| robellison | but it would be on the API then | 22:40 |
| robellison | and that API has too much access (glance anyway) | 22:40 |
| robellison | if it was validated before it got to that server, there is no way an unknown user could hit it | 22:41 |
| vhosakot | the horizon devs must implement face recognition ;) | 22:41 |
| *** awiddersheim has quit IRC | 22:42 | |
| SamYaple | robellison: well look, if you really really wanted to, and im not recommending it, you could have a dedicated node that only has the keystone auth validation in the pipeline that doesnt have access to ceph or anything | 22:42 |
| SamYaple | all requests would go through that server (or group of servers) before ferrying onwards | 22:43 |
| robellison | haha yeah maybe i'm being too fussy | 22:43 |
| SamYaple | i really think you are | 22:43 |
| robellison | but i know i will be asked all of these questions | 22:43 |
| SamYaple | fwiw, i appreciate the skeptisms | 22:44 |
| *** awiddersheim has joined #openstack-kolla | 22:44 | |
| SamYaple | but i just think you are focusing on the wrong part | 22:44 |
| vhosakot | robellison: not fussy at all, great discussion about OpenStack public cloud.... I agree public clouds need multiple auths before requests are let in. | 22:44 |
| SamYaple | maliciuos users can still get valid tokens | 22:44 |
| robellison | yeah it's just about reducing the risk as much as possible and reducing the impact should the worst happen | 22:44 |
| SamYaple | an *on* that note, i applaud you | 22:45 |
| SamYaple | make sure to lock down your internal network is the best i can tell you | 22:45 |
| SamYaple | ceph-osds chip traffic around with 0 encryption | 22:45 |
| SamYaple | thats by design, you can't change that | 22:45 |
| SamYaple | secure the network external and have good intrusion detection is th best advice i can give | 22:46 |
| robellison | yeah thats ok, we can deal with that on a protected network | 22:46 |
| robellison | dont worry... many layers of IPS here :) | 22:46 |
| SamYaple | of note, the apis are mostly all moving to WSGI apps | 22:47 |
| SamYaple | so you can run them behind apache2 or nginx with the added security those offer | 22:47 |
| SamYaple | mostly in the form of malformed packets | 22:47 |
| *** ducttape_ has joined #openstack-kolla | 22:47 | |
| SamYaple | but there are a tonne of security options for them | 22:47 |
| vhosakot | brb 10 mins... rebooting | 22:48 |
| robellison | well the good thing is that it's not a closed box | 22:48 |
| robellison | what's the view on kolla anyway.. it seems like the way to go to me | 22:49 |
| robellison | i've looked at a lot of commercial openstack implementations but they all seem too rigid | 22:50 |
| *** vhosakot has quit IRC | 22:50 | |
| *** ducttape_ has quit IRC | 22:51 | |
| *** ducttape_ has joined #openstack-kolla | 22:51 | |
| robellison | right, i've got to go, but thanks for the interesting thoughts SamYaple, vhosakot | 22:56 |
| *** eaguilar is now known as eaguilar|afk | 22:58 | |
| *** eaguilar|afk has quit IRC | 23:00 | |
| *** ducttape_ has quit IRC | 23:02 | |
| *** itlinux_ has joined #openstack-kolla | 23:05 | |
| *** itlinux has joined #openstack-kolla | 23:05 | |
| *** itlinux has quit IRC | 23:10 | |
| *** itlinux_ has quit IRC | 23:10 | |
| *** ducttape_ has joined #openstack-kolla | 23:11 | |
| *** manheim_ has joined #openstack-kolla | 23:12 | |
| *** ducttape_ has quit IRC | 23:12 | |
| *** ducttape_ has joined #openstack-kolla | 23:16 | |
| sdake | robellison view on kolla is generally regarded as good, although IMO documentation is poor - for more details check out the analytics bsaed upon the user survey which shows adoption/interest rates increasing: https://www.openstack.org/analytics 4% in deploy 9% in testing / 28% interest rate - pretty much leading the market (atleast for the user survey data recorded) | 23:58 |
| sdake | robellison if you have further feedback you want to provide to the development team - this is the place to provide it :) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
