| *** lrensing has joined #openstack-kolla | 00:01 | |
| *** manheim has quit IRC | 00:11 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Clear all l3 related namespace before starting neutron-l3-agent https://review.openstack.org/481969 | 00:23 |
|---|---|---|
| *** zhubingbing_ has joined #openstack-kolla | 00:34 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla stable/ocata: Bump OpenStack service version https://review.openstack.org/481970 | 00:35 |
| *** duonghq has joined #openstack-kolla | 00:46 | |
| duonghq | morning guys | 00:46 |
| *** zhurong has joined #openstack-kolla | 00:47 | |
| *** tovin07_ has joined #openstack-kolla | 00:52 | |
| *** caoyuan has joined #openstack-kolla | 00:53 | |
| *** kbaegis has joined #openstack-kolla | 01:02 | |
| *** dixiaoli has joined #openstack-kolla | 01:04 | |
| *** lyang__ has joined #openstack-kolla | 01:06 | |
| duonghq | sup kbaegis | 01:21 |
| *** lrensing has quit IRC | 01:25 | |
| *** dave-mccowan has joined #openstack-kolla | 01:33 | |
| openstackgerrit | fupingxie proposed openstack/kolla master: Replace http with https https://review.openstack.org/481975 | 01:40 |
| *** manheim has joined #openstack-kolla | 01:46 | |
| *** goldyfruit has joined #openstack-kolla | 01:47 | |
| *** manheim has quit IRC | 01:50 | |
| *** ducttape_ has joined #openstack-kolla | 01:54 | |
| *** ducttape_ has quit IRC | 01:59 | |
| *** schwicht has quit IRC | 02:00 | |
| *** kevzha01 has joined #openstack-kolla | 02:10 | |
| *** kevzha01 is now known as kevinz | 02:15 | |
| *** caowei has joined #openstack-kolla | 02:20 | |
| openstackgerrit | fupingxie proposed openstack/kolla master: Replace http with https https://review.openstack.org/481975 | 02:41 |
| *** zhurong has quit IRC | 02:43 | |
| *** schwicht has joined #openstack-kolla | 02:46 | |
| spsurya | morning | 03:17 |
| spsurya | duonghq: morning | 03:18 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add ceph rgw object store support into trove https://review.openstack.org/471553 | 03:30 |
| *** schwicht has quit IRC | 03:34 | |
| *** ducttape_ has joined #openstack-kolla | 03:38 | |
| *** ducttap__ has joined #openstack-kolla | 03:40 | |
| *** ducttape_ has quit IRC | 03:40 | |
| *** goldyfruit has quit IRC | 03:40 | |
| *** ducttap__ has quit IRC | 03:44 | |
| duonghq | morning spsurya | 03:57 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Clear all l3 related namespace before starting neutron-l3-agent https://review.openstack.org/481969 | 03:58 |
| *** dave-mccowan has quit IRC | 04:05 | |
| spsurya | duonghq: https://review.openstack.org/#/c/481975 | 04:17 |
| spsurya | is this https://stackalytics.com/?module=kolla-group&metric=commits working for you | 04:17 |
| spsurya | not in my domain ;) | 04:18 |
| duonghq | hmm, not in mine too | 04:22 |
| duonghq | but I'm not sure about reason | 04:22 |
| duonghq | spsurya, | 04:22 |
| spsurya | duonghq: seems server is not supporting SSL | 04:27 |
| spsurya | so http is fine | 04:27 |
| *** jascott1 has quit IRC | 04:28 | |
| *** mdnadeem has joined #openstack-kolla | 04:28 | |
| *** lrensing has joined #openstack-kolla | 04:38 | |
| *** ducttape_ has joined #openstack-kolla | 04:42 | |
| *** jascott1 has joined #openstack-kolla | 04:43 | |
| *** kbaegis has quit IRC | 04:43 | |
| *** ducttape_ has quit IRC | 04:44 | |
| *** ducttape_ has joined #openstack-kolla | 04:44 | |
| *** zhurong has joined #openstack-kolla | 04:46 | |
| *** ducttape_ has quit IRC | 04:46 | |
| *** ducttape_ has joined #openstack-kolla | 04:49 | |
| *** ducttape_ has quit IRC | 04:53 | |
| openstackgerrit | Yotaro Konishi proposed openstack/kolla-ansible master: Fix an invalid variable name https://review.openstack.org/481999 | 05:05 |
| *** lrensing has quit IRC | 05:07 | |
| *** janki has joined #openstack-kolla | 05:08 | |
| *** skramaja has joined #openstack-kolla | 05:20 | |
| *** unicell has quit IRC | 05:23 | |
| *** unicell has joined #openstack-kolla | 05:23 | |
| *** prateek has joined #openstack-kolla | 05:36 | |
| *** jbadiapa has joined #openstack-kolla | 05:43 | |
| openstackgerrit | fupingxie proposed openstack/kolla master: Replace http with https https://review.openstack.org/481975 | 05:45 |
| *** ducttape_ has joined #openstack-kolla | 05:49 | |
| *** ducttape_ has quit IRC | 05:55 | |
| *** manheim has joined #openstack-kolla | 06:10 | |
| *** manheim has quit IRC | 06:14 | |
| *** dixiaoli has quit IRC | 06:18 | |
| *** dixiaoli has joined #openstack-kolla | 06:18 | |
| *** magicboiz has joined #openstack-kolla | 06:35 | |
| *** lpetrut has joined #openstack-kolla | 06:37 | |
| *** janki has quit IRC | 06:38 | |
| *** asalkeld has joined #openstack-kolla | 06:41 | |
| *** lpetrut has quit IRC | 06:43 | |
| *** coolsvap has joined #openstack-kolla | 06:47 | |
| *** harlowja has quit IRC | 07:11 | |
| *** magicboiz has quit IRC | 07:12 | |
| *** shardy has joined #openstack-kolla | 07:14 | |
| *** jascott1 has quit IRC | 07:14 | |
| *** serlex has joined #openstack-kolla | 07:24 | |
| *** jascott1 has joined #openstack-kolla | 07:24 | |
| *** reidrac has joined #openstack-kolla | 07:32 | |
| reidrac | good morning | 07:32 |
| *** egonzalez has joined #openstack-kolla | 07:38 | |
| *** masber has quit IRC | 07:40 | |
| *** YuYangWang has joined #openstack-kolla | 07:42 | |
| *** reidrac has quit IRC | 07:45 | |
| *** dixiaoli has quit IRC | 07:45 | |
| *** dixiaoli has joined #openstack-kolla | 07:46 | |
| *** sambetts is now known as sambetts|afk | 07:47 | |
| *** YuYangWang has quit IRC | 07:47 | |
| *** masber has joined #openstack-kolla | 07:53 | |
| *** sambetts|afk is now known as sambetts | 07:57 | |
| *** ducttape_ has joined #openstack-kolla | 07:58 | |
| *** reidrac has joined #openstack-kolla | 08:00 | |
| *** ducttape_ has quit IRC | 08:03 | |
| *** shardy has quit IRC | 08:06 | |
| *** shardy has joined #openstack-kolla | 08:07 | |
| openstackgerrit | zhouya proposed openstack/kolla-ansible master: Support assigning HA traffic to dedicated interface https://review.openstack.org/481503 | 08:10 |
| *** manheim has joined #openstack-kolla | 08:18 | |
| *** jistr|off is now known as jistr | 08:25 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: [DNM] Test depends kolla changes https://review.openstack.org/482053 | 08:38 |
| *** hieulq has quit IRC | 08:39 | |
| *** daidv has quit IRC | 08:39 | |
| *** duonghq has quit IRC | 08:40 | |
| *** tovin07_ has quit IRC | 08:40 | |
| *** gfidente has joined #openstack-kolla | 08:49 | |
| *** gfidente has quit IRC | 08:49 | |
| *** gfidente has joined #openstack-kolla | 08:49 | |
| *** tvignaud has quit IRC | 08:58 | |
| *** ducttape_ has joined #openstack-kolla | 08:59 | |
| *** ducttape_ has quit IRC | 09:04 | |
| *** lpetrut has joined #openstack-kolla | 09:07 | |
| *** magicboiz has joined #openstack-kolla | 09:10 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: [DNM] Test depends kolla changes https://review.openstack.org/482053 | 09:11 |
| *** tvignaud has joined #openstack-kolla | 09:11 | |
| openstackgerrit | Merged openstack/kolla stable/ocata: Bump OpenStack service version https://review.openstack.org/481970 | 09:14 |
| *** leni1 has joined #openstack-kolla | 09:15 | |
| *** hieulq has joined #openstack-kolla | 09:15 | |
| *** daidv has joined #openstack-kolla | 09:16 | |
| *** duonghq has joined #openstack-kolla | 09:16 | |
| *** mgoddard_ has joined #openstack-kolla | 09:21 | |
| *** daidv has quit IRC | 09:21 | |
| *** magicboiz has quit IRC | 09:23 | |
| openstackgerrit | fupingxie proposed openstack/kolla master: Replace http with https https://review.openstack.org/481975 | 09:25 |
| *** kbaegis has joined #openstack-kolla | 09:27 | |
| *** magicboiz has joined #openstack-kolla | 09:27 | |
| kbaegis | duonghq: Morning! | 09:28 |
| *** kbaegis1 has joined #openstack-kolla | 09:30 | |
| *** leni1 has quit IRC | 09:32 | |
| *** kbaegis has quit IRC | 09:32 | |
| duonghq | hi kbaegis1 | 09:33 |
| kbaegis1 | Ahh, I got the 1 on the end of my name? | 09:33 |
| kbaegis1 | That's what I get for disconnecting every 5 minutes :) | 09:34 |
| duonghq | :) | 09:34 |
| *** daidv has joined #openstack-kolla | 09:34 | |
| *** kbaegis has joined #openstack-kolla | 09:34 | |
| kbaegis | Better? | 09:35 |
| kbaegis | NIce. | 09:35 |
| duonghq | ya | 09:35 |
| kbaegis | Excited. On Friday I get to start playing around with my new 10G mesh | 09:35 |
| duonghq | nice | 09:36 |
| duonghq | kbaegis, currently, which distro are you using? | 09:37 |
| duonghq | and how many do you have? | 09:37 |
| *** leni1 has joined #openstack-kolla | 09:37 | |
| kbaegis | duonghq: Technically a 4 node cluster w/ Gentoo | 09:37 |
| duonghq | nice | 09:38 |
| *** kbaegis1 has quit IRC | 09:38 | |
| kbaegis | The real trouble I've been having has been setting kolla up on ovs internal ports. | 09:38 |
| kbaegis | Doesn't seem to like clustering in that context | 09:38 |
| leni1 | Hello. Trying to build a kolla image using this guide: https://docs.openstack.org/kolla/latest/image-building.html. I wanted to know if I am to clone the kolla repo, I should do it thus: git clone https://github.com/openstack/kolla.git | 09:44 |
| leni1 | Would that be correct? | 09:44 |
| leni1 | Because I ran into a situation similar to this: https://ask.openstack.org/en/question/93967/kolla-build-command-not-found/ | 09:45 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix depends-on kolla gates https://review.openstack.org/482053 | 09:45 |
| *** YuYangWang has joined #openstack-kolla | 09:45 | |
| leni1 | And that seemed to be the proposed solution to the error: ERROR: toxini file 'tox.ini' not found | 09:46 |
| egonzalez | leni1, have you installed kolla after clone the repo or want to use master without installing? | 09:46 |
| leni1 | egonzalez, I want to install kolla | 09:49 |
| *** rwallner has quit IRC | 09:49 | |
| leni1 | I would like to build an image as per the instructions in the guide and so on. | 09:50 |
| leni1 | I certainly don't think my situation suits just using master without installing, as far as I can tell. | 09:50 |
| egonzalez | leni1, if not want to use master branch, install kolla from pip | 09:51 |
| egonzalez | pip install kolla | 09:51 |
| egonzalez | without cloning git repo and that stuff | 09:51 |
| leni1 | egonzalez, thank you very much for that clarification. Let me go do that. | 09:53 |
| leni1 | egonzalez, ran `pip install kolla`. Still the error of toxini file not found still appears | 09:56 |
| leni1 | I guess that means I should clone the repo, right? | 09:56 |
| egonzalez | leni1, nope | 09:57 |
| egonzalez | change directory out of the repo cloned and install kolla | 09:57 |
| egonzalez | maybe is trying to install local content | 09:57 |
| leni1 | egonzalez, I haven't cloned anything. So far I have been using pip to install what is required. | 09:58 |
| egonzalez | leni1, what steps you followed? what is installed? I just made a "pip install kolla" and succeed | 10:00 |
| *** asalkeld has quit IRC | 10:00 | |
| *** caoyuan has quit IRC | 10:02 | |
| *** ducttape_ has joined #openstack-kolla | 10:02 | |
| leni1 | egonzalez, so far I have run `pip install tox` | 10:02 |
| leni1 | After that I have run `tox -e genconfig` | 10:03 |
| leni1 | And that is giving me the error that tox.ini is not found | 10:03 |
| *** ducttape_ has quit IRC | 10:04 | |
| *** ducttape_ has joined #openstack-kolla | 10:05 | |
| leni1 | egonzalez, as far as I can tell, there are no dependencies that I was supposed to install, according to the guide. | 10:07 |
| egonzalez | leni1, if is installed with pip there is not kolla-build file generated nor tox.ini to create it | 10:07 |
| egonzalez | leni1, I think is a bug, let me dig into it | 10:07 |
| egonzalez | leni1, each option can be used in kolla-build command, what are you custom options want to use? | 10:08 |
| *** caowei has quit IRC | 10:08 | |
| leni1 | egonzalez, I really don't know. I just wanted to build container images for OpenStack Glance and Cinder. | 10:09 |
| leni1 | I was pointed to this guide as a way to do that. | 10:09 |
| *** ducttape_ has quit IRC | 10:10 | |
| egonzalez | leni1, push to any registry, namespace for the images, linux distribution and install type(source/binary)? | 10:10 |
| leni1 | egonzalez, Any registry will work I guess. No idea about namespace. Linux distribution would be CentOS. Install type I think would be binary because I want it to be stable. I could be wrong though ;) | 10:12 |
| egonzalez | ie, this command would build your images for centos, binary under my registry:/kolla namespace with tag 4.0.1 | 10:13 |
| egonzalez | leni1, kolla-ansible --base centos --type binary --namespace 192.168.100.1:4000/kolla --tag 4.0.1 --push cinder | 10:14 |
| egonzalez | *kolla-build rather kolla-ansible | 10:15 |
| *** kevinz has quit IRC | 10:16 | |
| leni1 | egonzalez, okay. Let me give it a shot and see. | 10:17 |
| openstackgerrit | Merged openstack/kolla master: Switch to Cryptography over pycrypto https://review.openstack.org/473702 | 10:19 |
| kbaegis | *sigh* So many of the scripts are hardcoded to assume that modules are available for loading/unloading | 10:21 |
| kbaegis | And if they aren't, they just fail. :/ | 10:21 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: DNM: test master branch https://review.openstack.org/456108 | 10:21 |
| kbaegis | i.e. iscsid starts just fine on my system, but hangs at "created". I have to manually start it. Also, keepalived looks for ip_vs | 10:21 |
| kbaegis | Documenting all those as modules | 10:22 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix depends-on kolla gates https://review.openstack.org/482053 | 10:22 |
| leni1 | egonzalez, by the way, should I do this for each component of OpenStack that I would like a container for? E.g. Glance and Neutron. | 10:29 |
| egonzalez | leni1, nope, if want to build all, not add cinder at the end | 10:29 |
| egonzalez | cinder,glance will build all images matched by a regex with those names | 10:30 |
| *** omenvlad has quit IRC | 10:30 | |
| leni1 | egonzalez, so if I want to build Glance and Neutron, the --push option should look like `--push glance,neutron`? | 10:36 |
| egonzalez | leni1, push is the option to push build images to the registry, image names are the last option | 10:37 |
| leni1 | egonzalez, understood. | 10:45 |
| leni1 | so just `glance,neutron` will work? | 10:45 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: WIP: Implement tacker-conductor service https://review.openstack.org/477517 | 10:48 |
| *** caoyuan has joined #openstack-kolla | 10:53 | |
| *** xinliang has quit IRC | 10:53 | |
| *** kbaegis has quit IRC | 10:54 | |
| *** hieulq has quit IRC | 10:57 | |
| openstackgerrit | Taeha Kim proposed openstack/kolla master: Fixed the invalid link in the kolla/image/build.py https://review.openstack.org/481457 | 10:57 |
| *** pbourke has quit IRC | 10:57 | |
| openstackgerrit | Taeha Kim proposed openstack/kolla master: Fixed the invalid link in the kolla/image/build.py https://review.openstack.org/481457 | 10:58 |
| *** ansmith has quit IRC | 10:58 | |
| *** daidv has quit IRC | 10:58 | |
| *** pbourke has joined #openstack-kolla | 10:59 | |
| *** duonghq has quit IRC | 10:59 | |
| *** duonghq has joined #openstack-kolla | 11:00 | |
| *** dixiaoli has quit IRC | 11:00 | |
| *** xinliang has joined #openstack-kolla | 11:05 | |
| *** ducttape_ has joined #openstack-kolla | 11:06 | |
| *** ducttape_ has quit IRC | 11:10 | |
| *** daidv has joined #openstack-kolla | 11:12 | |
| *** hieulq has joined #openstack-kolla | 11:12 | |
| *** rwallner has joined #openstack-kolla | 11:13 | |
| *** omenv has joined #openstack-kolla | 11:15 | |
| *** rwallner_ has joined #openstack-kolla | 11:15 | |
| *** rwallne__ has joined #openstack-kolla | 11:16 | |
| *** caoyuan has quit IRC | 11:16 | |
| *** rhallisey has joined #openstack-kolla | 11:16 | |
| *** rwallner has quit IRC | 11:17 | |
| *** rwallner_ has quit IRC | 11:20 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix depends-on kolla gates https://review.openstack.org/482053 | 11:28 |
| egonzalez | mind people review this change? https://review.openstack.org/#/c/482053/ fixes ansible gates with a depends-on kolla changes | 11:30 |
| *** kbaegis has joined #openstack-kolla | 11:36 | |
| kbaegis | Huh. So ip_vs is a module | 11:38 |
| kbaegis | And I'm still getting could not insert 'ip_vs': Invalid argument from the keepalived container | 11:38 |
| kbaegis | :/ | 11:38 |
| leni1 | I ran `kolla-ansible --base centos --type binary --namespace 192.168.100.1:4000/kolla --tag 4.0.1 --push cinder` and got this after the build: https://ghostbin.com/paste/f3dwn. Has it failed and if so, what should I do? | 11:39 |
| egonzalez | kbaegis, this sounds like a registry or auth issue "Get https://192.168.100.1:4000/v2/" | 11:40 |
| egonzalez | leni1, ^^ | 11:40 |
| kbaegis | egonzalez: It's a kernel/modprobe issue unfortunately | 11:41 |
| egonzalez | kbaegis, wrong name :) i'm out of the background of your issue, mind explain? | 11:41 |
| egonzalez | leni1, is your local registry bind in 192.168.100.1:4000 | 11:42 |
| kbaegis | egonzalez: keepalived will fail on load if it can't first do modprobe ip_vs. What if ip_vs is builtin? | 11:42 |
| egonzalez | leni1, that sound that you are using my local registry IP address ;) | 11:42 |
| leni1 | egonzalez, I don't know. I haven't had to interact with my local registry. | 11:42 |
| egonzalez | leni1, do you have a local registry running? | 11:43 |
| leni1 | How can I check what my local registry is? I think it is still set to the defaults though. | 11:43 |
| egonzalez | guess no have a registry if you didnt configured it | 11:44 |
| leni1 | I have never run a registry, so unless it is running in the background, I don't think so. | 11:44 |
| leni1 | Yes. | 11:44 |
| kbaegis | leni1: For what it's worth, I can't think of anything less intuitive (for beginners) than what the docker community has chosen to do with their registries. Once you get it set up it works flawlessly though. | 11:44 |
| egonzalez | leni1, is an allinone or multinode? | 11:44 |
| egonzalez | if is multinode will need one for share images between all nodes | 11:44 |
| leni1 | I think it is an all-in-one. I am going to be using the containers on one machine | 11:45 |
| egonzalez | leni1, so no need a registry at all, remove --push from the command, also, the other images how are named? docker images | grep nova | 11:46 |
| egonzalez | kbaegis, modprobe is done in extend_start of keepalived, in which cases ip_vs is builtin? | 11:48 |
| kbaegis | egonzalez: Sure. What if it's built directly into the kernel, and modprobe won't work? | 11:49 |
| leni1 | egonzalez, the other images are named thus: 192.168.100.1:4000/kolla/centos-binary-cinder-api | 11:49 |
| kbaegis | egonzalez: There's no handling whatsoever for that case. Just fails | 11:49 |
| egonzalez | leni1, i mean, the other images not built, you only build cinder,glance and neutron right? but need a bunch more of images | 11:50 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: [wip] Implement tacker-conductor image, rework all tacker https://review.openstack.org/477494 | 11:51 |
| *** caoyuan has joined #openstack-kolla | 11:52 | |
| leni1 | egonzalez, yes. | 11:52 |
| leni1 | Only cinder has built. I am going to do glance and neutron next | 11:52 |
| leni1 | I hope the `192.168.100.1:4000/` part in the image name won't cause me problems, right? | 11:53 |
| egonzalez | leni1, all images need to be under the same namespace <in your case 192.168.109.1:4000/lokolla> | 11:54 |
| egonzalez | but if some images are from dockerhub are only kolla/<image> | 11:54 |
| leni1 | okay. Understood. Thank you. | 11:54 |
| egonzalez | leni1, what i try to say is, all need to share the same namespace, adjust --namespace in kolla-build command to match your other images | 11:55 |
| *** dave-mccowan has joined #openstack-kolla | 11:55 | |
| leni1 | egonzalez, understood. So my namespace should be kolla/<image>, right? Since all my other images are from Docker Hub | 11:56 |
| *** janki has joined #openstack-kolla | 11:56 | |
| egonzalez | leni1, yep only --namespace kolla (is default iirc) | 11:58 |
| leni1 | I think that also means I will have to build each image for cinder, glance, and neutron individually, not all at once. | 11:58 |
| leni1 | Oh... Yes. So --namespace kolla will work for all of them at once? | 11:58 |
| egonzalez | leni1, you can use cinder neutron glance at the end of the kolla-build command to build those images | 12:02 |
| *** pradk has joined #openstack-kolla | 12:02 | |
| egonzalez | note the spaces between image names | 12:02 |
| *** hieulq has quit IRC | 12:04 | |
| *** daidv has quit IRC | 12:04 | |
| *** duonghq has quit IRC | 12:04 | |
| *** kevinz has joined #openstack-kolla | 12:05 | |
| *** schwicht has joined #openstack-kolla | 12:05 | |
| *** kevzha01_ has joined #openstack-kolla | 12:08 | |
| *** kevinz has quit IRC | 12:12 | |
| openstackgerrit | Michel Rode proposed openstack/kolla-ansible master: Added open-iscsi to default removals https://review.openstack.org/479226 | 12:15 |
| *** brad[] has quit IRC | 12:16 | |
| leni1 | egonzalez, thank you very much | 12:17 |
| *** brad[] has joined #openstack-kolla | 12:18 | |
| *** daidv has joined #openstack-kolla | 12:19 | |
| *** krtaylor_ has joined #openstack-kolla | 12:19 | |
| *** hieulq has joined #openstack-kolla | 12:19 | |
| *** duonghq has joined #openstack-kolla | 12:19 | |
| *** caoyuan has quit IRC | 12:20 | |
| *** krtaylor has quit IRC | 12:20 | |
| *** eaguilar has joined #openstack-kolla | 12:22 | |
| *** caoyuan has joined #openstack-kolla | 12:26 | |
| *** hrw has quit IRC | 12:31 | |
| *** hrw has joined #openstack-kolla | 12:32 | |
| *** zhurong has quit IRC | 12:36 | |
| *** kbaegis has quit IRC | 12:36 | |
| *** sambetts has quit IRC | 12:38 | |
| *** schwicht has quit IRC | 12:42 | |
| *** sambetts has joined #openstack-kolla | 12:42 | |
| manheim | hi | 12:42 |
| *** ansmith has joined #openstack-kolla | 12:44 | |
| *** leni1 has quit IRC | 12:44 | |
| serlex | hi | 12:44 |
| *** awiddersheim has joined #openstack-kolla | 12:45 | |
| manheim | i have some questions, they may seem conceptual | 12:47 |
| manheim | why put the libvirt process in a container? | 12:47 |
| manheim | why not let it on the host | 12:47 |
| *** awidders_ has quit IRC | 12:49 | |
| *** rwallne__ has quit IRC | 12:50 | |
| *** rwallner_ has joined #openstack-kolla | 12:53 | |
| openstackgerrit | Merged openstack/kolla master: Fixed the invalid link in the kolla/image/build.py https://review.openstack.org/481457 | 12:53 |
| *** goldyfruit has joined #openstack-kolla | 12:56 | |
| *** mdnadeem has quit IRC | 12:59 | |
| *** schwicht has joined #openstack-kolla | 13:00 | |
| *** wahnsinn has joined #openstack-kolla | 13:02 | |
| *** masber has quit IRC | 13:06 | |
| *** goldyfruit1 has joined #openstack-kolla | 13:06 | |
| *** caoyuan has quit IRC | 13:07 | |
| *** goldyfruit has quit IRC | 13:08 | |
| wahnsinn | Hey, I have a question regarding the network configuration after a kolla install: | 13:08 |
| manheim | hi, shoot | 13:08 |
| wahnsinn | on my controller node neither br-ex nor br-tun are UP and the vm's have no internet connection | 13:09 |
| *** kristian__ has joined #openstack-kolla | 13:09 | |
| wahnsinn | if i tcpdump vxlan_sys_4789 I can see the ping requests but no replys | 13:09 |
| wahnsinn | is it normal that the bridge devices are down? | 13:10 |
| wahnsinn | (my environment consists of one controller node and one compute with two NICs each, eth0 is connected to the outside) | 13:12 |
| egonzalez | wahnsinn, is normal to be down in the host, but if you use docker exec openvswitch_vswitchd ovs-vsctl show should be there up | 13:13 |
| wahnsinn | true | 13:15 |
| egonzalez | wahnsinn, is ping going outside the q-router qvo* interface? | 13:15 |
| wahnsinn | do I have to check that inside the container? | 13:16 |
| *** hieulq has quit IRC | 13:16 | |
| egonzalez | wahnsinn, in the namespaces, do ip netns in network node | 13:16 |
| *** daidv has quit IRC | 13:17 | |
| *** trozet has joined #openstack-kolla | 13:17 | |
| egonzalez | there you can find a qrouter namespace, do ip netns exec qrouter<id> ip a to find the qvo interface | 13:17 |
| *** duonghq has quit IRC | 13:17 | |
| *** omenvl has joined #openstack-kolla | 13:20 | |
| *** mmehan has joined #openstack-kolla | 13:20 | |
| wahnsinn | I don't have a qvo interface there, only qr-* and qg-* | 13:21 |
| egonzalez | wahnsinn, sorry, qg is the one | 13:22 |
| *** omenv has quit IRC | 13:23 | |
| wahnsinn | and how can I run tcpdump on that interface? | 13:24 |
| *** ducttape_ has joined #openstack-kolla | 13:24 | |
| egonzalez | wahnsinn, the same as a physical interface, you can use every command present in the host in the namespaces | 13:25 |
| *** karimb has joined #openstack-kolla | 13:25 | |
| karimb | hello buddies, how does kolla handle missing bridge namespace in network related containers ? | 13:26 |
| *** ducttap__ has joined #openstack-kolla | 13:26 | |
| egonzalez | karimb, im not getting the question, refer to how bridges are created? | 13:27 |
| karimb | egonzalez actually, i m referfing to the logical happening when launching an instance | 13:27 |
| karimb | as part of the process, sysctl -N net.bridge seems to be launched , which fails on containers as bridge namespace isnt exposed | 13:28 |
| karimb | egonzalez maybe you re running those containers with host network | 13:28 |
| karimb | ? | 13:28 |
| egonzalez | karimb, yep, all containers runs with host networking | 13:29 |
| *** ducttape_ has quit IRC | 13:29 | |
| karimb | egonzalez then in this case, the question is how do you handle dns for those containers | 13:29 |
| karimb | cause using host network means you miss kubernetes service discovery features | 13:29 |
| karimb | ( when running on kubernetes) | 13:30 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix an invalid variable name https://review.openstack.org/481999 | 13:30 |
| sdake | morning peeps | 13:30 |
| egonzalez | karimb, you talking about k8s, not idea about that sorry | 13:30 |
| openstackgerrit | Merged openstack/kolla-ansible master: Use latest docker engine in gate https://review.openstack.org/480212 | 13:30 |
| karimb | egonzalez that s fair :) | 13:30 |
| karimb | egonzalez no one using kolla containers on k8s? | 13:31 |
| egonzalez | morning sdake | 13:31 |
| sdake | karimb yes indeeed http:/github.com/openstack/kolla-kubernete | 13:31 |
| sdake | s | 13:31 |
| *** hieulq has joined #openstack-kolla | 13:36 | |
| *** duonghq has joined #openstack-kolla | 13:36 | |
| *** daidv has joined #openstack-kolla | 13:36 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Avoid launching "local" actions too many times https://review.openstack.org/480256 | 13:40 |
| openstackgerrit | Merged openstack/kolla-ansible master: Set neutron RPC workers and state report workers https://review.openstack.org/444151 | 13:44 |
| *** schwicht has quit IRC | 13:44 | |
| wahnsinn | egonzales, I can see a ping request from <vm-private-ip> to 8.8.8.8 and a unreachable reply from <controllernode> to <vm-private-ip> | 13:46 |
| wahnsinn | so the ping is going outside | 13:46 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Update cinder sanity check to use built in shade https://review.openstack.org/437920 | 13:46 |
| *** caoyuan has joined #openstack-kolla | 13:47 | |
| *** iniazi_ has joined #openstack-kolla | 13:48 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Make glance filesystem_store_datadir configurable. https://review.openstack.org/452094 | 13:48 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Only start one glance_api when using file backend https://review.openstack.org/448654 | 13:48 |
| pbourke | berendt: zhubingbing_: hi, can you please revisit https://review.openstack.org/#/c/448654/ | 13:48 |
| *** rwallner_ has quit IRC | 13:50 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add dev mode for Murano https://review.openstack.org/467608 | 13:50 |
| *** omenvl has quit IRC | 13:50 | |
| *** iniazi has quit IRC | 13:50 | |
| openstackgerrit | Noboru Iwamatsu proposed openstack/kolla-ansible master: delete delimiter from init fernet condition check https://review.openstack.org/480853 | 13:51 |
| *** schwicht has joined #openstack-kolla | 13:51 | |
| wahnsinn | egonzales, I'm starting to belive that I have a made mistake in the initial config: assuming that I have to NIC's, eth0/<public ip to internet> and eth1/<private ip between nodes>: is network_interface: eth0 and neutron_external_interface: eth1 right? | 13:51 |
| *** omenv has joined #openstack-kolla | 13:51 | |
| *** rwallner has joined #openstack-kolla | 13:52 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix zun-api logging and state_path https://review.openstack.org/480963 | 13:52 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix tools/cleanup-images script error https://review.openstack.org/480628 | 13:53 |
| *** rwallner has quit IRC | 13:53 | |
| egonzalez | wahnsinn, exacly the other way, eth0 neutron_external_interface and eth1 network_interface | 13:53 |
| *** rwallner_ has joined #openstack-kolla | 13:54 | |
| egonzalez | wahnsinn, note that you connection to the hosts with ansible should be through eth1 | 13:54 |
| wahnsinn | okay, but will the controller node than still be availabe at eth0 ip? | 13:54 |
| *** tonyjun01 has joined #openstack-kolla | 13:54 | |
| egonzalez | wahnsinn, eth0 will be managed by ovs, and lost connectivity once is added to br-ex | 13:55 |
| wahnsinn | and horizon will be available than at eth1 ip? | 13:55 |
| egonzalez | wahnsinn, all services will be available at haproxy VIP bind at eth1 | 13:57 |
| *** omenv has quit IRC | 13:59 | |
| *** rmichel has joined #openstack-kolla | 13:59 | |
| *** pbourke has quit IRC | 14:01 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix bandit tests directory https://review.openstack.org/481016 | 14:01 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Add heat_stack_owner role to admin project https://review.openstack.org/474173 | 14:01 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Add warning while upgrading with dockerhub images N->O https://review.openstack.org/475802 | 14:02 |
| *** skramaja has quit IRC | 14:02 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Implement blazar ansible role https://review.openstack.org/454752 | 14:02 |
| *** tonyjun01 has quit IRC | 14:06 | |
| openstackgerrit | sean mooney proposed openstack/kolla-ansible master: introduce contrib playbook for ovs-dpdk https://review.openstack.org/408872 | 14:08 |
| *** janki has quit IRC | 14:13 | |
| *** ducttap__ has quit IRC | 14:13 | |
| *** hongbin has joined #openstack-kolla | 14:14 | |
| *** ducttape_ has joined #openstack-kolla | 14:14 | |
| *** pbourke has joined #openstack-kolla | 14:18 | |
| *** [tony] has joined #openstack-kolla | 14:19 | |
| openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible master: Upgrade from docker-py to docker https://review.openstack.org/475941 | 14:19 |
| *** ducttape_ has quit IRC | 14:25 | |
| *** ducttape_ has joined #openstack-kolla | 14:27 | |
| *** rmichel has quit IRC | 14:29 | |
| *** omenv has joined #openstack-kolla | 14:29 | |
| *** zhurong has joined #openstack-kolla | 14:30 | |
| openstackgerrit | Juan Badia Payno proposed openstack/kolla master: Changed fluentd repository for CentOS/RHEL https://review.openstack.org/467219 | 14:30 |
| *** rmichel has joined #openstack-kolla | 14:30 | |
| *** omenvl has joined #openstack-kolla | 14:30 | |
| openstackgerrit | Juan Badia Payno proposed openstack/kolla master: Changed fluentd repository for CentOS/RHEL https://review.openstack.org/467219 | 14:32 |
| *** dixiaoli has joined #openstack-kolla | 14:33 | |
| *** dixiaoli has quit IRC | 14:33 | |
| *** omenv has quit IRC | 14:34 | |
| *** caoyuan_ has joined #openstack-kolla | 14:35 | |
| *** karimb has quit IRC | 14:36 | |
| *** caoyuan has quit IRC | 14:37 | |
| *** [tony] has quit IRC | 14:38 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Fix gnocchi in version_check script https://review.openstack.org/482162 | 14:42 |
| *** jamesbenson has joined #openstack-kolla | 14:44 | |
| *** hieulq has quit IRC | 14:51 | |
| *** daidv has quit IRC | 14:51 | |
| *** duonghq has quit IRC | 14:52 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Remove ignore tempest project in version-check https://review.openstack.org/482171 | 14:54 |
| openstackgerrit | Andy Smith proposed openstack/kolla-ansible master: Add support for hybrid messaging backends https://review.openstack.org/468966 | 14:55 |
| *** jtriley has quit IRC | 14:56 | |
| *** zhurong has quit IRC | 14:56 | |
| *** manheim has quit IRC | 14:57 | |
| *** manheim has joined #openstack-kolla | 14:57 | |
| *** manheim has quit IRC | 14:58 | |
| *** manheim has joined #openstack-kolla | 14:58 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Clean some Boolean condition checks https://review.openstack.org/480225 | 15:00 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Remove empty line in config.py https://review.openstack.org/482174 | 15:01 |
| *** kevzha01__ has joined #openstack-kolla | 15:01 | |
| *** klindgren has joined #openstack-kolla | 15:01 | |
| *** klindgren_ has joined #openstack-kolla | 15:02 | |
| *** rmichel has quit IRC | 15:03 | |
| *** tony_ has joined #openstack-kolla | 15:03 | |
| *** klindgren has quit IRC | 15:04 | |
| openstackgerrit | Andy Smith proposed openstack/kolla-ansible master: Add qdrouterd role for messaging infrastructure component https://review.openstack.org/469462 | 15:04 |
| *** kevzha01_ has quit IRC | 15:05 | |
| *** kevzha01__ has quit IRC | 15:06 | |
| openstackgerrit | Rich Wellum proposed openstack/kolla master: Add a section on building custom kolla images from source. https://review.openstack.org/481127 | 15:07 |
| *** omenvl has quit IRC | 15:08 | |
| *** daidv has joined #openstack-kolla | 15:08 | |
| *** duonghq has joined #openstack-kolla | 15:08 | |
| *** hieulq has joined #openstack-kolla | 15:09 | |
| *** tony_ has quit IRC | 15:11 | |
| sbezverk | kfox1111: ping | 15:13 |
| *** manheim has quit IRC | 15:14 | |
| *** karimb has joined #openstack-kolla | 15:18 | |
| kolla-slack | <kfox1111> At the gas station. Driving in. What's up? | 15:18 |
| sbezverk | kfox1111: :) need to discuss one solution, would you have 10-15 minutes sometime today? | 15:19 |
| kolla-slack | <kfox1111> Yeah. Will be at work in about 15min | 15:19 |
| *** ducttape_ has quit IRC | 15:19 | |
| kolla-slack | <kfox1111> Will ping you when I get in | 15:19 |
| *** vhosakot has joined #openstack-kolla | 15:20 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add dev mode for Horizon https://review.openstack.org/474595 | 15:20 |
| *** unicell1 has joined #openstack-kolla | 15:21 | |
| *** unicell has quit IRC | 15:23 | |
| inc0 | good morning | 15:24 |
| egonzalez | morning inc0 | 15:25 |
| vhosakot | mornin all! | 15:25 |
| egonzalez | mind checking https://review.openstack.org/#/c/482053/ is a gate fix | 15:25 |
| rwellum | vhosakot: inc0 - deadline for submissions in the 14th correct? | 15:26 |
| vhosakot | rwellum: yes, correct | 15:27 |
| inc0 | yup | 15:27 |
| inc0 | we need to do it asap, today?;) | 15:27 |
| *** hieulq has quit IRC | 15:28 | |
| rwellum | Yup - can you guys look at the etherpad and sketch out changes according to our discussion last week? | 15:28 |
| vhosakot | rwellum: cool, will do | 15:28 |
| *** daidv has quit IRC | 15:28 | |
| *** klindgren__ has joined #openstack-kolla | 15:29 | |
| *** duonghq has quit IRC | 15:29 | |
| rwellum | QQ for anyone. When we change code, are we supposed to pull a branch - or work directly off Master? | 15:30 |
| *** ducttape_ has joined #openstack-kolla | 15:30 | |
| sbezverk | rwellum : I prefer to work on the branch and then rebase on the master when master gets changed | 15:31 |
| *** klindgren_ has quit IRC | 15:33 | |
| inc0 | guys fyi, our gates will be broken for a while | 15:33 |
| vhosakot | rwellum: upstream has "fix master first, backport later to an older/stable branch if needed" policy :) | 15:33 |
| inc0 | (short hopefully) | 15:33 |
| inc0 | our tarballs overloaded infra static server | 15:33 |
| vhosakot | inc0: yeah, I'm seeing messages about it in the infra channel. | 15:34 |
| vhosakot | inc0: what overload? disk or network traffic? | 15:34 |
| *** klindgren_ has joined #openstack-kolla | 15:34 | |
| rwellum | vhosakot: well I meant a local branch - like sbezverk is suggesting I think. Because when you commit with git review - it leaves your workspace behind master I was just trying to figure out if my workflow was wrong. | 15:34 |
| vhosakot | rwellum: ah, yeah, local branch is always good (which shows up as a topic in gerrit UI). | 15:35 |
| *** klindgren__ has quit IRC | 15:36 | |
| rwellum | ok noted - cool thanks guys. | 15:36 |
| vhosakot | rwellum: our worksapce is always behind, cannot help that :) sometimes, we may need to rebase multiple times a day depending on which latest merged fix is needed. | 15:36 |
| rwellum | vhosakot: 'our workspace' - meaning you have forked or branched kolla - and make changes from there? | 15:37 |
| inc0 | vhosakot: network | 15:37 |
| vhosakot | rwellum: our workspace = the workspace we've cloned in which we're branched to fix something. | 15:37 |
| egonzalez | rwellum, I always use git review -f to cleanup, or will end up with thousand local branchs | 15:37 |
| vhosakot | rwellum: no need to fork (thankfully) as openstack upstream does not need "pull requests in GitHub". Gerrit does not need forks/pull requests. | 15:38 |
| vhosakot | I'm not a fan ot forking because we may end up carry fat forks that diverge from upstream very fast. | 15:39 |
| vhosakot | inc0: ah I see, network is always the issue ;) | 15:39 |
| kfox1111 | sbezverk: ping | 15:40 |
| *** wahnsinn has quit IRC | 15:40 | |
| vhosakot | egonzalez: you do "git review -f" _after_ the new branch is merged right :) | 15:40 |
| sbezverk | kfox1111: I was working on adding lioadm as iscsi helper, some people wants to get rid of tgtd | 15:40 |
| sbezverk | and I got most of things figured out and working, but | 15:41 |
| egonzalez | vhosakot, nope, if have to change something i fetch the change. if I wait until is merged i'll forget to clean up ;) | 15:41 |
| sbezverk | tgtd used username and password authentication and lioadm/target uses username/password and iqn | 15:41 |
| sbezverk | in our model nova-compute uses iscsd container to carry out requests for attaching | 15:42 |
| vhosakot | egonzalez: ah, if you forget to cleanup.. cool.. yeah, I clean up regularly as .ansible and .tox all take up a lot of space... neutron's .tox is like ~500 M or something :O | 15:42 |
| sbezverk | iscsi targets and these two containers happen to have different iqn | 15:42 |
| sbezverk | so I see authentication failer | 15:42 |
| kfox1111 | hmm... | 15:42 |
| sbezverk | now I see two solutions | 15:42 |
| sbezverk | 1: make iscsd as a sidfe car container to nova-compute | 15:43 |
| sbezverk | in this case they can share /etc/iscsi/initiator name | 15:43 |
| *** daidv has joined #openstack-kolla | 15:43 | |
| *** duonghq has joined #openstack-kolla | 15:43 | |
| *** hieulq has joined #openstack-kolla | 15:43 | |
| sbezverk | 2. use hosts /etc/iscsi/initiator name | 15:43 |
| sbezverk | what do you think would work better for ops folks? | 15:44 |
| kfox1111 | I kind of like #2 a little better. | 15:44 |
| kfox1111 | by coupling iscsid and nova-compute, if you want to update nova-compute you have to do iscsid too. | 15:44 |
| kfox1111 | I think nova-compute is purely controlplane, but iscid is dataplane too? | 15:45 |
| sbezverk | kfox1111: iscsid here is control plane too, since it is just a vehicle for nova-compute but 2 works for me too | 15:46 |
| kfox1111 | k | 15:46 |
| *** mgoddard_ has quit IRC | 15:46 | |
| rwellum | egonzalez: ah -f is fetch not force :) | 15:47 |
| kfox1111 | yeah, either probably works then. #2's probably simpler too. | 15:47 |
| sbezverk | kfox1111: so the logic would be nova-compute, cinder-volume and iscsid will use host maps for /etc/iscsi | 15:47 |
| kfox1111 | hmm... what else is in there? | 15:47 |
| kfox1111 | should we do a /var/lib/kolla/xxxxx thing instead? | 15:47 |
| egonzalez | rwellum, nope -f means if the change is pushed to gerrit, remove local branch and checkout to origin branch (most of times master) | 15:47 |
| jamesbenson | morning all :-) | 15:48 |
| sbezverk | if initiator file does not exist, which is possible if initiator tools are not installed on the host, then one of this will generate one | 15:48 |
| sbezverk | and the other will start using it | 15:48 |
| rwellum | egonzalez: that's really a neat idea thanks. | 15:48 |
| jamesbenson | inc0: I think I know what's up with ceph | 15:49 |
| sbezverk | kfox1111: /etc/iscsi is default location, I am not sure if everything can be customized to use /var/lib/kolla/xxx | 15:49 |
| kfox1111 | sbezverk: its a container. can host mount it there to /var/lib/kolla/xxx in the container. | 15:49 |
| kfox1111 | /etc/iscsi in the container I mean. | 15:50 |
| kfox1111 | I'm a little leary of letting stuff on the host leach into the containers. | 15:50 |
| inc0 | jamesbenson: do tell please | 15:50 |
| kfox1111 | if the host has a different iscsi version or something then the containers, it might cause problems. | 15:50 |
| kfox1111 | maybe the iscsi container does the creating of the files, and the other containers have an init container that waits till the file exists? | 15:51 |
| sbezverk | kofx1111: ok you mean we keep file in var/lib/kolla/iscsi on the host but mount it to /etc/iscsi in the containers? | 15:51 |
| jamesbenson | I did a manual deploy and having issues as well, but I eventually got it working. The solution was how I had my network configured in ceph (and that's something that changed here as well when I was deploying kolla). | 15:51 |
| kfox1111 | sbezverk: yeah. | 15:51 |
| inc0 | jamesbenson: what was the change? | 15:52 |
| jamesbenson | Ceph default options failed for me. I had to explicitly state my public and cluster network in my .conf file along with mon_host was set to internal by default and needed to be external. | 15:52 |
| inc0 | right...well, you can do this with kolla-ansible too | 15:53 |
| jamesbenson | I still need to play a little bit more to see exactly what failed, but with those 3 things it work | 15:53 |
| inc0 | but since you have ceph working | 15:53 |
| *** jascott1 has quit IRC | 15:53 | |
| inc0 | just keep it;) | 15:53 |
| sbezverk | kfox1111: ok, sounds good.. will try to check first in common initiator name will actually resolve the authentication failure I see. | 15:53 |
| kfox1111 | ah. good plan. :) | 15:54 |
| jamesbenson | is public and cluster in kolla the storage_interface & cluster_interface specifically for ceph? | 15:54 |
| *** harbie has quit IRC | 15:55 | |
| sbezverk | kfox1111: cool thanks for discussion! | 15:55 |
| inc0 | jamesbenson: yes | 15:56 |
| inc0 | cluster = mon to mon (and osd to mon) communication | 15:57 |
| *** jtriley has joined #openstack-kolla | 15:57 | |
| inc0 | storage is for vm to rbd communication | 15:57 |
| *** unicell1 has quit IRC | 15:57 | |
| inc0 | well storage iface is also used with other storage backends (so not techincally ceph specific;))) | 15:57 |
| *** kristian__ has quit IRC | 15:57 | |
| *** kristian__ has joined #openstack-kolla | 15:58 | |
| *** harbie has joined #openstack-kolla | 15:58 | |
| egonzalez | jamesbenson, https://github.com/openstack/kolla-ansible/blob/master/doc/production-architecture-guide.rst#interface-configuration | 15:58 |
| *** rhallisey is now known as rhallisey-biab | 15:58 | |
| jamesbenson | thanks egonzalez :-) | 15:59 |
| *** rhallisey-biab has quit IRC | 15:59 | |
| *** egonzalez has quit IRC | 16:03 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Allow tmp path to download registry https://review.openstack.org/475940 | 16:05 |
| vhosakot | jamesbenson: storage_interface defaults to network_interface (which is the public interface or kolla_external_vip_interface) as the default behavior in kolla --> https://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml#L127 and https://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml#L125. But, in production, yes, storage is a separate network/segment. | 16:06 |
| openstackgerrit | Bradley Jones proposed openstack/kolla-kubernetes master: (WIP) Deploy playbook using microservices https://review.openstack.org/473588 | 16:08 |
| vhosakot | jamesbenson: the official openstack docs also mention that storage network must be a separate non-public segment that just connects the computes to the storage nodes - https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html | 16:08 |
| vhosakot | jamesbenson: I observed a lot less traffic drop when I enabled jumbo frames with ceph -- https://communities.cisco.com/community/developer/openstack/blog/2017/01/13/jumbo-mumbo-in-openstack | 16:10 |
| *** serlex has quit IRC | 16:10 | |
| vhosakot | ceph_mon kills TCP on the storage nodes when parallel writes into volumes start :) | 16:11 |
| jamesbenson | thanks for the info vhosakot | 16:12 |
| vhosakot | jamesbenson: cool, I'm about write another blog about OpenStack and ceph (hopefull in the next 2-3 weeks). | 16:13 |
| openstackgerrit | Bradley Jones proposed openstack/kolla-kubernetes master: (WIP) Deploy playbook using microservices https://review.openstack.org/473588 | 16:13 |
| openstackgerrit | Bradley Jones proposed openstack/kolla-kubernetes master: (WIP) Deploy playbook testing with nova PV https://review.openstack.org/479826 | 16:14 |
| jamesbenson | here we've never done anything special for ceph, just sort of let it do it's thing.... :-/ We will have to investigate more. Currently our machines are only wired for 2 ports, public and internal (all of them have 4 or 8 ports though) | 16:14 |
| inc0 | jamesbenson: but you had same issues with ceph-deploy right? | 16:16 |
| inc0 | so it doesn't seem to be kolla specific? | 16:16 |
| *** coolsvap has quit IRC | 16:16 | |
| jamesbenson | well once I got the networking correct, ceph is healthy with 3 osd and 1 mon (http://paste.openstack.org/show/614949/ on our dev cluster) | 16:18 |
| jamesbenson | I haven't finished a kolla deploy with external ceph though | 16:18 |
| jamesbenson | right now looking at the kolla details | 16:18 |
| *** jascott1 has joined #openstack-kolla | 16:19 | |
| openstackgerrit | Merged openstack/kolla master: Fix mock order https://review.openstack.org/480434 | 16:19 |
| jamesbenson | vhosakot: how do you set the jumbo frames, is that solely in the configs in kolla/ceph, or do you need to modify switch settings as well? (Just trying to look ahead once I get things sorted out) | 16:20 |
| *** caoyuan_ has quit IRC | 16:20 | |
| openstackgerrit | Bradley Jones proposed openstack/kolla-kubernetes master: (WIP) Deploy playbook using microservices https://review.openstack.org/473588 | 16:20 |
| vhosakot | jamesbenson: no need to do anything on the switch/TOR (as they will/should not be in the path of the storage network between the compute node and the storgae/rbd node). Jumbo on all OpenStack nodes can be configured by tweaking configs in neutron.conf/ml2_conf and OVS config. | 16:22 |
| openstackgerrit | Bradley Jones proposed openstack/kolla-kubernetes master: (WIP) Deploy playbook testing with nova PV https://review.openstack.org/479826 | 16:22 |
| *** reidrac has quit IRC | 16:22 | |
| vhosakot | jamesbenson: in my case, I had Cisco FEX (Fabric Extenders) and FI's (Fabric Interconnects) between computes nodes and ceph nodes, so I just configured jumbo frames on them manually using NXOS CLIs. | 16:23 |
| *** mgoddard_ has joined #openstack-kolla | 16:23 | |
| vhosakot | jamesbenson: keeping the ceph/storage/rbd nodes as nearest as possible to the computes nodes gave me best perf (have minimum or zero L2/L3 hops between compute node and storage nodes). | 16:24 |
| vhosakot | every hop must have jumbo, else PMTU descovery (if turned on) will default to non-jumbo (1,500 bytes) :) | 16:25 |
| sbezverk | kfox1111: that was it, sharing iqn between nova-compute and iscsd, now lioadm is working :) | 16:25 |
| jamesbenson | vhosakot, always interested in what others think, but we plan on (and let me know if this is crazy because we always do it this way - but are willing to change), 3 controller/network/storage combo; 3 storage/compute combo; 25 compute nodes. (Total nodes are 32) | 16:32 |
| *** jascott1 has quit IRC | 16:33 | |
| jamesbenson | inc0: in regards to the ll kernel, once I know that I can get consistent deploys here, and I'll try a few deploys back to back... I'll try the only change is the kernel and let you know. | 16:34 |
| SamYaple | jamesbenson: dont run ceph-mon on the same nodes as the ceph-osds | 16:34 |
| SamYaple | but otherwise, that looks good | 16:34 |
| SamYaple | hyperconveraged storage/compute is all the rage these days | 16:34 |
| vhosakot | jamesbenson: also, dont run ceph osd on compute nodes (I know a team that does that :)) | 16:35 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla master: Adding new iscsi-target image to support lioadm https://review.openstack.org/481858 | 16:35 |
| SamYaple | vhosakot: its defintely fine to run ceph-osds on compute nodes. that has been the majority of my deploys | 16:35 |
| vhosakot | jamesbenson: the number of ceph nodes depends on how much data the customer wants to write in the volumes... | 16:35 |
| sbezverk | inc0: wrt lioadm I will need ^^^ these changes, could you please check when you have time.. | 16:35 |
| vhosakot | SamYaple: in that case, if the compute node crashes, we lose a storage node, plus, computes need a lot of disks in that case. | 16:36 |
| *** hieulq has quit IRC | 16:36 | |
| *** daidv has quit IRC | 16:36 | |
| *** duonghq has quit IRC | 16:36 | |
| SamYaple | vhosakot: yea. thats what would happen. not sure how seperating computes and storage is going to fix that | 16:36 |
| *** shardy is now known as shardy_afk | 16:37 | |
| SamYaple | given the goal is to have as many nodes as possible, running storage on all the computes is really useful | 16:37 |
| vhosakot | SamYaple: separating computes nodes and storages, atleast we do not lose customer data in vloume when a compute node crashes. | 16:37 |
| *** harlowja has joined #openstack-kolla | 16:37 | |
| SamYaple | you should _never_ be losing data when you lose a node | 16:38 |
| SamYaple | if you are, you have setup your cluster wrong | 16:38 |
| jamesbenson | and our storage nodes have a ton of space, compute has less. | 16:38 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding alternative iscsi_helper support https://review.openstack.org/481690 | 16:38 |
| Reepicheep | I have taken kinda a hybrid for compute nodes and osds | 16:38 |
| vhosakot | SamYaple: what if a compute node dies before ceph has replicated/rebalanced? | 16:38 |
| Reepicheep | I have a few osds on each compute node but they are fast disks that are writeback cache | 16:39 |
| SamYaple | vhosakot: thats not how ceph works. it is "strongly consistent" which means before the client gets an ack on the write, it has written it to the entire cluster | 16:39 |
| jamesbenson | we use raid on everything (old hard drives but when a drive goes, we replace quick) | 16:39 |
| SamYaple | vhosakot: also, remember, you may not be writing any data to the node the compute is on | 16:39 |
| jamesbenson | performance isn't super critical for us, we try our best of course to make it fast, but staying up and not dying is important, hence the raid. | 16:42 |
| Reepicheep | I took this advice "Do enable RBD caching on Nova compute nodes" from: | 16:42 |
| Reepicheep | https://www.hastexo.com/resources/hints-and-kinks/dos-donts-ceph-openstack/ | 16:42 |
| Reepicheep | hopefully it's not to crazy.. it seems to work | 16:43 |
| vhosakot | SamYaple: yeah, in our case, osd was on a compute node, compute node died due to disk failure, ceph hadn't rebalanced fully, any many volumes were lost, could not be recovered. | 16:44 |
| SamYaple | vhosakot: like i said, if you lost data with ceph it was not because the node was also a compute node | 16:46 |
| SamYaple | to lose data with a proper ceph cluster you need to have 3 complete disk failures without having rebalanced | 16:46 |
| vhosakot | SamYaple: ah I see | 16:47 |
| jamesbenson | SamYaple, and to chime in with the little knowledge that I know, 3 complete disk failures where the data happens to be on those three nodes.... if the data is still scattered it might not be a complete loss. | 16:48 |
| rwellum | Reepicheep: good article that | 16:49 |
| SamYaple | jamesbenson: "properly" setup ceph would not allow the data to exist on one node, so yes. it would have to be three disk failures on different nodes at the same time | 16:49 |
| Reepicheep | rwellum: yeah.. the one advice I haven't followed is "put all-flash OSDs in a separete CRUSH root" | 16:50 |
| kfox1111 | SamYaple: and even then, depending on number of disks per host, it still may not end up loosing data. | 16:50 |
| Reepicheep | it's not so easy with kolla-ansible short of using external ceph | 16:50 |
| jamesbenson | yes, I've recovered where we were dismantling and realized we did something wrong, recovered some, not all, but we were lucky and got what we needed. | 16:50 |
| kfox1111 | but you need a good sized ceph at that point. :) | 16:50 |
| Reepicheep | which I haven't taken that jump yet | 16:50 |
| jamesbenson | no flash in our datacentre | 16:51 |
| SamYaple | kfox1111: to get to that range is something like 1500 disks though. but yea | 16:51 |
| rwellum | Are there any viable distributed storage alternatives to Ceph? Like Gluster etc. | 16:51 |
| *** daidv has joined #openstack-kolla | 16:51 | |
| *** hieulq has joined #openstack-kolla | 16:51 | |
| *** duonghq has joined #openstack-kolla | 16:51 | |
| rwellum | Just curious as I know several large companies that refuse to use Ceph. | 16:51 |
| SamYaple | rwellum: the real "win" for ceph is the distributed block storage. Gluster is all file based | 16:52 |
| rwellum | Ah | 16:52 |
| rwellum | The negatives I head are 'it's a development project', 'not supported', 'too complicated' etc. | 16:53 |
| rwellum | heard :) | 16:53 |
| kfox1111 | rwellum: perception issue. :/ | 16:54 |
| Reepicheep | years ago I tried to use file based block devices stored in gluster.. it didn't go well | 16:54 |
| kfox1111 | its a developemnt project is false. lots of production usage. | 16:54 |
| kfox1111 | not supported. false. you can pay redhat or others to support it. | 16:54 |
| SamYaple | rwellum: ceph is older that openstack | 16:54 |
| kfox1111 | too complicated? perhaps. less complicated then other ha storage systems I've played with though. | 16:54 |
| SamYaple | thats my go to when people talk about ceph being young | 16:55 |
| rwellum | bradjones: regarding your review 473588 - were you looking at the other ansible pb going through review to see if they can be merged at all? | 16:55 |
| rwellum | SamYaple: that I did not know :) | 16:55 |
| kfox1111 | yeah. I remember a guy here kicking the tires on ceph/gluster I wana say 2 years before we started playing with openstack. | 16:55 |
| *** jascott1 has joined #openstack-kolla | 16:55 | |
| kfox1111 | they really were green then though. | 16:56 |
| rwellum | jascott1: hi | 16:56 |
| jascott1 | rwellum hey! hows it going? | 16:56 |
| rwellum | Not bad :) - I am curious - have you looked at bradjones ansible solution? | 16:57 |
| bradjones | rwellum, which other review? | 16:57 |
| jascott1 | I have, actually working on cramming it into a container with "all the things" to automate kk8s setup | 16:59 |
| *** daidv has quit IRC | 16:59 | |
| *** hieulq has quit IRC | 16:59 | |
| SamYaple | jascott1: a container starting a container orchestration to start containers? | 16:59 |
| *** duonghq has quit IRC | 16:59 | |
| jascott1 | how perverse | 17:00 |
| jascott1 | whatup SamYaple! | 17:00 |
| SamYaple | o/ | 17:00 |
| rwellum | bradjones: I don't think it's in review yet, jascott1 correct me if I'm wrong. | 17:01 |
| *** dciabrin has quit IRC | 17:01 | |
| kfox1111 | a container at its core really is just a software distribution mechanism. | 17:02 |
| *** dciabrin has joined #openstack-kolla | 17:02 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable proxy cache for tarballs download https://review.openstack.org/482217 | 17:02 |
| kfox1111 | with that definition, the above isn't as perverse as it seems. | 17:02 |
| jamesbenson | btw, thank you vhosakot, SamYaple, Reepicheep, inc0. Your knowledge is impressive and took many pearls from that convo. :-) | 17:02 |
| SamYaple | jamesbenson: i made it all up. trust no one | 17:03 |
| jamesbenson | ha! | 17:03 |
| vhosakot | jamesbenson: cool, I still need a PhD in ceph :) | 17:03 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable proxy cache for tarballs download https://review.openstack.org/482217 | 17:03 |
| vhosakot | jamesbenson: ceph is addictive :) | 17:03 |
| jamesbenson | hey, I started a phd, not worth the hype ;-) | 17:03 |
| vhosakot | jamesbenson: :) | 17:03 |
| jamesbenson | 2 MS's instead | 17:04 |
| vhosakot | I got 1 MS | 17:04 |
| vhosakot | still no use ;) | 17:04 |
| jascott1 | kfox1111 i was quoting C3PO | 17:04 |
| inc0 | vhosakot: as in, when you start to use it it's hard to get out? You watch your life wasted and opportunities missed, you really really want to get out but it's just too hard... | 17:04 |
| jamesbenson | (I could have had a third, but no point in having 2 both in physics) | 17:04 |
| inc0 | one...more...osd... | 17:04 |
| kfox1111 | jascott1: ah. I guess that means I need to go watch again... my quote capability is slipping. :) | 17:04 |
| jascott1 | haha dont worry it was in a prequel :) | 17:05 |
| SamYaple | inc0: have you seen in 17.06 docker you can do a FROM {{ buildarg }} ? | 17:05 |
| kfox1111 | jascott1: ah. that might explain it. ;) | 17:05 |
| inc0 | wow that's awesome | 17:05 |
| SamYaple | So you *could* do a kolla base image with set_config and the like, then build LOCI images | 17:05 |
| inc0 | did they enable "templating"? | 17:05 |
| SamYaple | inc0: no | 17:06 |
| jascott1 | rwellum are you talking about this? https://review.openstack.org/#/c/457317/ | 17:06 |
| SamYaple | its the standard build-arg system | 17:06 |
| SamYaple | this might make merging the two projects back together more... possible though | 17:06 |
| jascott1 | rwellum oh you are prob talking about the conainer thing I was talking about... i am working off inc0's PS | 17:06 |
| inc0 | SamYaple: yeah, possible today with overrides but I hear ya, if we'd add one more image layer for *just tini/set_config* we could do that... | 17:07 |
| inc0 | well still would need to include all the extended starts | 17:07 |
| rwellum | Yeah jascott1 there seem to be about 3 very similar efforts. | 17:07 |
| SamYaple | inc0: *possible* sure, but i think we can both agree injecting the scripts with overrides isnt that best | 17:08 |
| SamYaple | we just included that as a stop gap until 17.06 and this feature | 17:08 |
| rwellum | But maybe they should be separate - not sure. | 17:08 |
| inc0 | worth thinking about for sure | 17:08 |
| jascott1 | rwellum we should probably coordinate at some point :/ | 17:09 |
| SamYaple | $ cat Dockerfile | 17:10 |
| SamYaple | ARG FROM=gcr.io/google_containers/ubuntu-slim:0.12 | 17:10 |
| SamYaple | FROM $FROM | 17:10 |
| SamYaple | SHELL ["/bin/bash", "-c"] | 17:10 |
| SamYaple | $ docker build --build-arg FROM=ubuntu . | 17:10 |
| SamYaple | oops | 17:10 |
| SamYaple | meant to pastebin that | 17:10 |
| SamYaple | inc0: http://paste.openstack.org/show/614952/ | 17:11 |
| bradjones | rwellum the main difference between 457317 and my patch is that mine isn't trying to setup kubernetes for you it's just about deploying kolla, the other main differences is using microcharts vs service charts to make upgrading in the future easier & also to remove the dependency on entrypoing | 17:11 |
| bradjones | so they are different approaches | 17:11 |
| bradjones | but both valid, 457317 is a great all in one deployment playbook | 17:12 |
| kfox1111 | should reduce duplication if we can and share as much as possible. | 17:12 |
| inc0 | SamYaple: if we'd move tini/set_config to end-containers | 17:12 |
| kfox1111 | its hard to keep lots of implmentations up to date. | 17:12 |
| inc0 | as nova-api | 17:12 |
| inc0 | and do FROM loci-nova | 17:12 |
| inc0 | that would work too | 17:12 |
| rwellum | bradjones: ah thanks -good to know | 17:13 |
| inc0 | and we wouldn't need to wait for this docker change | 17:13 |
| SamYaple | inc0: but then you have to add a layer to _all_ the images and rebuild everything ontop of things | 17:13 |
| SamYaple | this way its one layer | 17:13 |
| inc0 | yeah, but extended-starts are also needed | 17:13 |
| rwellum | bradjones: I also agree with kfox1111 - good to share where possible. Or at very least document the heck out of each tool so the user can see and understand the differences. | 17:15 |
| sbezverk | inc0: please review https://review.openstack.org/481858 | 17:17 |
| bradjones | yeah I agree on both points, especially around docs, that's something i'll add shortly | 17:17 |
| inc0 | sbezverk: why new container? | 17:19 |
| rwellum | bradjones: thanks | 17:19 |
| sbezverk | inc0: we need a container hosting targets, it activates target kernel module and restores saved targets on container restarts | 17:21 |
| *** manheim has joined #openstack-kolla | 17:22 | |
| vhosakot | SamYaple: so, from the time a nova VM writes data to a volume, to the time this data is replicated/balanced on all the osds, how much is this time window? like micro seconds/milli seconds/seconds? also, does this time window increase if ceph is busy due to a lot of nova VMs simultaneuosly writing to volumes? | 17:22 |
| sbezverk | inc0: I got PoC wokring with lioadm so I think I am on the right track | 17:24 |
| inc0 | cool, thanks! | 17:25 |
| *** manheim has quit IRC | 17:26 | |
| *** manheim has joined #openstack-kolla | 17:26 | |
| *** lpetrut has quit IRC | 17:26 | |
| *** Mannheim has joined #openstack-kolla | 17:27 | |
| *** ducttape_ has quit IRC | 17:28 | |
| sean-k-mooney | SamYaple: is thier a specific reason not to run ceph mons on same nodes as osd if you have the cpus free? | 17:29 |
| sean-k-mooney | SamYaple: just reading back over ceph convo | 17:30 |
| vhosakot | SamYaple: let me know what you think when you get a chance about the time window question. | 17:31 |
| vhosakot | sean-k-mooney: yeah, ceph_mon does not consume a lot of CPU/mem/disk I've seen, you can fit them on the osd boxes if you have enough resources. | 17:32 |
| vhosakot | actually, ceph_mon gets busy when too many writes happen. | 17:32 |
| sean-k-mooney | vhosakot: i run my 3 controllers as allinone nodes (openstack contorler+ ceph mon+ ceph osd+ nova compute) seams to work fine | 17:33 |
| kfox1111 | it depends more on the size of your boxes and size of the overall cluster. | 17:34 |
| kfox1111 | if you only have 3 nodes, shared mon is usually fine. | 17:34 |
| sean-k-mooney | then again they are 48 core servers so they are not exactly resource starved and i only have 12 osd in my cluster for it to manage. | 17:34 |
| kfox1111 | unless they have a huge amount of storage. in which case, you can afford a few more thin nodes. | 17:35 |
| *** manheim has quit IRC | 17:35 | |
| vhosakot | sean-k-mooney: I'm kinda those people who think compute node should not have anything other than hypervisor :) hahaha, it's just a arch design I think. we keep storage (osd's) nodes separate | 17:35 |
| kfox1111 | vhosakot: yeah. we did shared ceph/compute for a while. | 17:35 |
| kfox1111 | it was fine right up until it wasn't. :) | 17:35 |
| vhosakot | yeah, our computes and controllers are almost diskless (very minimum disks as needed) | 17:36 |
| sean-k-mooney | vhosakot: i started my cluster with 3 nodes all in one, then adde 3 as storage + compute and add another 3 later as just compute. i have another 30 or so to add soon that will just be compute | 17:36 |
| vhosakot | kfox1111: yeah, many people do share osd on compute design, and it works for them in prod. | 17:36 |
| kfox1111 | had good luck with that until recovery kicked in. then it was really disruptive to services. :/ | 17:38 |
| *** rhallisey has joined #openstack-kolla | 17:38 | |
| sean-k-mooney | kfox1111: vhosakot i guess the main think if you do share is to make sure you have a dedicated nic for cpehs replication and that you addjust the vcpu_pin_set in nova to ensure that you have enough coures reserved for the system to handel the ceph recovery | 17:39 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable proxy cache for tarballs download https://review.openstack.org/482217 | 17:39 |
| *** daidv has joined #openstack-kolla | 17:39 | |
| vhosakot | kfox1111: ceph's vulnerability is == if the usecase increases the time window from the time a nova VM writes something into a volume, to the time ceph_mon replicates/rebalances that data onto all the osds, and this time window is a function of how busy ceph_mon is and the number of osds ceph_mon needs to replicate the data onto. | 17:39 |
| *** duonghq has joined #openstack-kolla | 17:40 | |
| *** hieulq has joined #openstack-kolla | 17:40 | |
| kfox1111 | yeah. | 17:40 |
| sean-k-mooney | vhosakot: provided your not doing asycn io in the gust i whould have assumed the rbd device would not have completed the right until ceph had singneled that it had been completed which it should not do until it has atleast been written to the journal | 17:42 |
| sean-k-mooney | vhosakot: there may be a delay betwenn getting to your dsired level of replication but that would likely be load dependent and hevily influnced by your ceph config | 17:43 |
| *** hieulq has quit IRC | 17:43 | |
| vhosakot | sean-k-mooney: yeah, I have not looked into ceph config's tuning much... tried to just make it lose volume in my scale testing :) | 17:44 |
| *** duonghq has quit IRC | 17:44 | |
| vhosakot | sean-k-mooney: yes, faster dedicated NIC for storage fastens replication. | 17:45 |
| inc0 | vhosakot rwellum let's make summit proposals today, we can always edit stuff afterwards | 17:45 |
| rwellum | WFM inc0 | 17:45 |
| rwellum | I've gotta rack some switches in 15m for a couple of hours - but will be back | 17:46 |
| vhosakot | we're like customer data, customer data, customer data.. so ended up having separate osd nodes. | 17:46 |
| vhosakot | inc0: rwellum cool, ready when you are :) | 17:46 |
| inc0 | https://etherpad.openstack.org/p/kolla-sydney-custom-repo-talk | 17:47 |
| sean-k-mooney | inc0: is https://review.openstack.org/#/c/466132/ relevent to ^ | 17:49 |
| inc0 | yeah | 17:49 |
| sean-k-mooney | inc0: ill fix the typo and reupload that shortly | 17:49 |
| inc0 | cool | 17:49 |
| inc0 | sean-k-mooney: can you review odl patch? | 17:50 |
| sean-k-mooney | sure which one. i notice it was not building correctly | 17:50 |
| inc0 | https://review.openstack.org/#/c/416367/ | 17:50 |
| SamYaple | vhosakot: re: time window question | 17:51 |
| SamYaple | when the write is ack'd by the cluster it is replicated everywhere. so if you do a 'sync' in the vm, when that returns everything is sync'd | 17:52 |
| SamYaple | when something is actually commited to the backing storage and not _only_ in ram is up to the kernel | 17:52 |
| *** kristia__ has joined #openstack-kolla | 17:52 | |
| rwellum | sbezverk: have you seen any oddities with openvswitch lately - in my setup - everything comes up but 'neutron-openvswitch-agent-network'. | 17:53 |
| vhosakot | SamYaple: cool, thanks for the info | 17:53 |
| sean-k-mooney | inc0: oh the ansible role, cool do you know if the guy working on ovn are planing to submit a role for that also? | 17:54 |
| inc0 | no idea | 17:54 |
| sbezverk | rwellum nope, I am using latest kolla-k8s, seems to be working fine | 17:54 |
| sbezverk | rwellum : what is it complaining about? bad weather ;) ?? | 17:54 |
| SamYaple | sean-k-mooney: if you run the osds on the same nod as the ceph mon and that node _crashes_ the cluster locks up for ~30 seconds | 17:55 |
| rwellum | sbezverk - maybe! :) https://www.irccloud.com/pastebin/dqgvyekd/ | 17:56 |
| SamYaple | sean-k-mooney: when an osd dies, its a topology change that the mons have to agree on. when a mon dies a new election is called (to ensure quorum). when both die at the same time the cluster freezes to call an election and then update the topology | 17:56 |
| *** kristian__ has quit IRC | 17:56 | |
| sean-k-mooney | SamYaple: ah ok that makes sense | 17:56 |
| *** wahnsinn has joined #openstack-kolla | 17:57 | |
| sean-k-mooney | SamYaple: so colocating compute nodes and ceph_osd should be ok but not ceph_mons and ceph_osd to avoid the quorum sync latency on a node failure | 17:57 |
| sbezverk | rwellum : you need to check docker log for that container, last paste only shows that it was crashing.. not very useful.. | 17:58 |
| *** duonghq has joined #openstack-kolla | 17:58 | |
| *** hieulq has joined #openstack-kolla | 17:58 | |
| *** ducttape_ has joined #openstack-kolla | 18:00 | |
| *** dmellado has quit IRC | 18:02 | |
| *** dmellado has joined #openstack-kolla | 18:02 | |
| *** schwicht has quit IRC | 18:02 | |
| rwellum | docker logs <name of container> sbezverk ? | 18:03 |
| jamesbenson | vhosakot & sean-k-mooney: I'm reading Learning Ceph by Singh right now, they have some interesting info in there... | 18:03 |
| *** kristia__ has quit IRC | 18:03 | |
| vhosakot | SamYaple: when the primary ceph_mon died, we saw the nova vm was not able to read data too, as reading also first contacts ceph_mon to find out which osd to read from. for a long time, I thought reading does not involve ceph_mon at all and that reading is just compute<-->osd interaction without involving ceph_mon... | 18:04 |
| vhosakot | jamesbenson: link please :) | 18:05 |
| *** sambetts is now known as sambetts|afk | 18:05 | |
| *** schwicht has joined #openstack-kolla | 18:07 | |
| jamesbenson | vhosakot: https://www.packtpub.com/virtualization-and-cloud/learning-ceph. (but google the title, you might find it else where ;-) ) | 18:07 |
| sbezverk | rwellum docker ps -a | grep neutron-openvswitch-agent | 18:07 |
| jamesbenson | and keep an eye out on packt.... They have a free book every day https://www.packtpub.com/packt/offers/free-learning | 18:08 |
| vhosakot | jamesbenson: cool, thanks | 18:08 |
| inc0 | vhosakot rwellum I wrote some abstract in etherpad, feel free to comment/fix language/repeal and replace it alltogether | 18:08 |
| vhosakot | inc0: don't worry, we won't repeal-and-replace bigly ;) | 18:09 |
| rwellum | sbezverk https://www.irccloud.com/pastebin/DgYamAOY/ | 18:11 |
| vhosakot | sean-k-mooney: what about orchestration for the out-of-tree non-kolla external images built --> https://review.openstack.org/#/c/466132/ the user mus have his/her own orchestration mechanism right using ansible/k8s/whatever? | 18:12 |
| sean-k-mooney | vhosakot: basically you can replicate https://review.openstack.org/#/c/408872/ for the kolla-ansible case | 18:13 |
| sean-k-mooney | vhosakot: for k8s i guess you would jsut have a chart | 18:14 |
| jamesbenson | vhosakot: http://paste.openstack.org/show/614956/ | 18:14 |
| vhosakot | sean-k-mooney: I see, cool, thanks for the info | 18:14 |
| SamYaple | sean-k-mooney: correct. but in that scenario it is just a delay, you dont lose data. so in some situations (if you know the risks) its ok to run the mons and osds | 18:15 |
| vhosakot | jamesbenson: yes, separate mon nodes is the recommendation I've read.. coo | 18:15 |
| *** kristian__ has joined #openstack-kolla | 18:15 | |
| vhosakot | cool* | 18:15 |
| *** tonanhngo has joined #openstack-kolla | 18:15 | |
| *** kristian__ has quit IRC | 18:15 | |
| *** lpetrut has joined #openstack-kolla | 18:15 | |
| sbezverk | rwellum : I do not see it on the list | 18:16 |
| SamYaple | vhosakot: it gets a copy of the cluster map from the ceph mon and just needs to verify that is up to date | 18:17 |
| sbezverk | rwellum : but do you see it in kubectl get pods in kolla? | 18:17 |
| sbezverk | as crashing right? | 18:17 |
| rwellum | Yes | 18:18 |
| sean-k-mooney | was there an issue with tarballs.openstack.org today http://logs.openstack.org/72/408872/12/check/gate-kolla-ansible-dsvm-deploy-centos-binary-centos-7-nv/1cf0d29/console.html#_2017-07-10_14_21_21_864217 | 18:18 |
| rwellum | Sorry sbezverk - I got pulled off for an hour or so - will be back to debug | 18:18 |
| vhosakot | SamYaple: right, so compute<-->ceph_mon is pure ceph control plane, right, not data plane (in which VM's app's data written to volume travels)... we saw it, and hence ended up having slower NICs for compute<-->ceph_mon links as they need not be super fast like compute<-->osd links | 18:19 |
| openstackgerrit | sean mooney proposed openstack/kolla master: enable the kolla build system to use out of tree docker files https://review.openstack.org/466132 | 18:20 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding alternative iscsi_helper support https://review.openstack.org/481690 | 18:21 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix designate-api listen interface https://review.openstack.org/482238 | 18:22 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla master: Adding new iscsi-target image to support lioadm https://review.openstack.org/481858 | 18:26 |
| *** unicell has joined #openstack-kolla | 18:29 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding alternative iscsi_helper support https://review.openstack.org/481690 | 18:31 |
| *** lpetrut has quit IRC | 18:35 | |
| *** karimb has quit IRC | 18:39 | |
| *** lpetrut has joined #openstack-kolla | 18:42 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix searchlight nova resources indexation https://review.openstack.org/482243 | 18:47 |
| *** mgoddard_ has quit IRC | 18:49 | |
| *** rwallner_ has quit IRC | 18:52 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Searchlight: activate Ironic resources https://review.openstack.org/482248 | 18:54 |
| SamYaple | vhosakot: im not entirely sure what you are saying there | 18:54 |
| vhosakot | SamYaple: I meant, does ceph_mon see any data? :) | 18:55 |
| SamYaple | vhosakot: the "public_network" is where all the clients talk ot mons _and_ read data over | 18:58 |
| SamYaple | but no data from a vm will exist in the mons | 18:58 |
| vhosakot | SamYaple: not "exist". does ceph_mon forward any data? receive data from nova VM and send it to the right osd? | 18:59 |
| *** shardy_afk is now known as shardy | 18:59 | |
| *** rwallner has joined #openstack-kolla | 18:59 | |
| SamYaple | vhosakot: the clients talk directly to the osds, there is never any data forwarding | 19:00 |
| vhosakot | SamYaple: right, so, compute<-->ceph_mon can have slow links vs compute<-->osd must be super fast links, is what I meant/asked :) | 19:01 |
| jamesbenson | SamYaple, I thought compute/storage wasn't recommended due to the high CPU/RAM used when rebalancing OSD, backfill or recovery... | 19:02 |
| SamYaple | vhosakot: if you mean 1gb and 10gb, yea that mihgt be ok | 19:02 |
| SamYaple | jamesbenson: thats why most people have a bad _experince_ with hyperconveraged compute/storage. but if you are aware of it you can avoid it causing issues | 19:02 |
| vhosakot | SamYaple: yep, few of our kits have 10G compute<-->ceph_mon and 40G compute<-->osd. | 19:03 |
| SamYaple | 1gb of ram per TB of storage in reserve | 19:03 |
| SamYaple | vhosakot: perfect. that sounds great | 19:03 |
| *** rwallner has quit IRC | 19:04 | |
| jamesbenson | our computes have 1200gb, (4x300gb), is it even worth to put 1-2 OSD's there? | 19:05 |
| vhosakot | jamesbenson: well, don't "hyperconveraged" does not necessarily mean compute+storage combined into one box, is it? I thought hyperconveraged means all network, compute, storage all combined into a nice-looking, easy-installable "rack" like flexpod/vblock. I've seen separate storage nodes at least in vblock. | 19:06 |
| sdake | hey folks - ptl election is about a month out | 19:06 |
| SamYaple | jamesbenson: so the deal with that is if you have dedicated storage boxes that will not be running computes then you should _not_ run the osds on your compute (probably) | 19:07 |
| sdake | elections | 19:07 |
| jamesbenson | gah, just trying to figure out a good model for our production that is still being designed... | 19:07 |
| SamYaple | but each situation is differnet | 19:07 |
| vhosakot | sdake: thanks for the info | 19:07 |
| SamYaple | jamesbenson: do you have a list of hardware and specs? | 19:08 |
| vhosakot | jamesbenson: 2 OSDs with 1.2TB == 600 GB per OSD. not bad for a few volumes IMO. really depends on how much data the VMS want to write into volumes. | 19:10 |
| *** manheim has joined #openstack-kolla | 19:10 | |
| *** manheim has quit IRC | 19:10 | |
| *** manheim has joined #openstack-kolla | 19:11 | |
| jamesbenson | we have 28, dell r410's (4HD, 300GB); 11 dell r610's (4HD, 300GB); 7 Dell r710's (6HD, 2TB); we also have a SAN that we want to connect, but not sure if it is possible Dell equalogic PS6510. The servers all have 4 nics minimum, but only 2 wired, all 1G. | 19:11 |
| jamesbenson | We typically put the 410 | 19:12 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix designate-api listen interface https://review.openstack.org/482238 | 19:12 |
| jamesbenson | 's as compute, 710's at controller/storage, and 610's as compute | 19:12 |
| jamesbenson | sorry! Typo! 14 R410's, not 28.... | 19:14 |
| SamYaple | jamesbenson: youre going to get hurt pretty bad with the 1GB nics. minimum you need to have a cluster_network for all the ceph replication traffic | 19:14 |
| vhosakot | yeah, I'd cards faster than 1G | 19:14 |
| jamesbenson | the 410's cant have a 10g card in them... but we could wire up a 3rd port for ceph | 19:14 |
| SamYaple | with that collection of different disks and networking as it is, i would recommend running as many osds as possible | 19:14 |
| SamYaple | on as many different boxes | 19:15 |
| *** blallau has joined #openstack-kolla | 19:15 | |
| jamesbenson | we have 10g cards, just can't insert them in... no space.. | 19:15 |
| SamYaple | jamesbenson: thats _really_ going to hurt you | 19:15 |
| vhosakot | jamesbenson: you can google "Walmart OpenStack architecture"... I remember they showed great topo's in one of their talks at OpenStack summits. | 19:16 |
| *** blallau has quit IRC | 19:16 | |
| *** rhallisey is now known as rhallisey-biab | 19:16 | |
| *** Mannheim has quit IRC | 19:17 | |
| vhosakot | frankly, OpenSTack has a very simple topology requirement, and that is what I love about it :) | 19:17 |
| jamesbenson | yeah, I/we know the lack of 10g isn't good... hoping to eventually move the 410's to a different area but budgets are no capable of that atm | 19:17 |
| jamesbenson | definitely vhosakot, just trying to find optimal :-). | 19:18 |
| vhosakot | jamesbenson: the most I"ve seen is like api network, management network, neutron network (provider network, tenant network), storage network, some OOB network, all boxes uplinked to redundant TOR box in VPC. | 19:19 |
| jamesbenson | and we always used mirantis before, so it was a simple, select your controller, compute, storage and go.... | 19:19 |
| vhosakot | you can get experimental near/in the TOR area like adding FEX, Fabric Interconnects, Cisco VTS/ACI, etc etc | 19:20 |
| jamesbenson | we've got juniper switches here... I'll have to check if they have anything like that but I've never heard of this before :-/ | 19:22 |
| *** ducttape_ has quit IRC | 19:23 | |
| *** ducttape_ has joined #openstack-kolla | 19:23 | |
| vhosakot | jamesbenson: actually, simple dual TOR'ing works great for OpenSTack IMO. | 19:24 |
| manheim | http://paste.openstack.org/show/614958/ | 19:28 |
| manheim | my neutron metadata agent is in error | 19:29 |
| manheim | everything else works | 19:29 |
| manheim | have you seen this before? | 19:29 |
| *** omenvl has joined #openstack-kolla | 19:33 | |
| *** kristian__ has joined #openstack-kolla | 19:34 | |
| *** gfidente is now known as gfidente|afk | 19:36 | |
| SamYaple | manheim: yea | 19:36 |
| SamYaple | keystone issues *maybe*, more likely is misconfiguration of neutron | 19:37 |
| SamYaple | or the vip | 19:37 |
| *** prateek has quit IRC | 19:38 | |
| *** daidv has quit IRC | 19:38 | |
| *** hieulq has quit IRC | 19:38 | |
| *** duonghq has quit IRC | 19:38 | |
| *** kristian__ has quit IRC | 19:39 | |
| manheim | ok | 19:40 |
| *** kristian__ has joined #openstack-kolla | 19:45 | |
| *** trozet has quit IRC | 19:51 | |
| *** kristian__ has quit IRC | 19:52 | |
| *** kristian__ has joined #openstack-kolla | 20:01 | |
| *** trozet has joined #openstack-kolla | 20:04 | |
| *** jtriley has quit IRC | 20:07 | |
| *** goldyfruit1 has quit IRC | 20:07 | |
| vhosakot | I'll tell you an openstack joke, but you'll have to build all the missing parts yourself to make it funny. | 20:13 |
| kfox1111 | indeed. | 20:15 |
| *** lpetrut has quit IRC | 20:17 | |
| rwellum | vhosakot: inc0 - made small corrections to the abstract | 20:19 |
| vhosakot | rwellum: cool | 20:19 |
| rwellum | sbezverk - you can see the neutron-openvswitch container crashing. https://www.irccloud.com/pastebin/GqFo8JZR/ | 20:20 |
| *** tonanhngo has quit IRC | 20:27 | |
| *** manheim has quit IRC | 20:28 | |
| *** manheim has joined #openstack-kolla | 20:28 | |
| *** rhallisey has joined #openstack-kolla | 20:30 | |
| *** ducttape_ has quit IRC | 20:31 | |
| *** tonanhngo has joined #openstack-kolla | 20:31 | |
| *** manheim has quit IRC | 20:33 | |
| *** ducttape_ has joined #openstack-kolla | 20:34 | |
| *** tonanhngo has quit IRC | 20:35 | |
| *** tonanhngo has joined #openstack-kolla | 20:35 | |
| *** vhosakot has quit IRC | 20:37 | |
| *** vhosakot has joined #openstack-kolla | 20:37 | |
| *** ansmith has quit IRC | 20:38 | |
| *** goldyfruit has joined #openstack-kolla | 20:40 | |
| *** tonanhngo has quit IRC | 20:40 | |
| vhosakot | inc0: I'm creating an etherpad for the upgrade talk. Will send it once it is done. will need feedback from you and duong. | 20:40 |
| *** lpetrut has joined #openstack-kolla | 20:40 | |
| *** tonanhngo has joined #openstack-kolla | 20:42 | |
| *** manheim has joined #openstack-kolla | 20:43 | |
| *** Mannheim has joined #openstack-kolla | 20:44 | |
| *** tonanhngo has quit IRC | 20:46 | |
| Mannheim | is there a known issue with the metadata service in stable/ocata? 4.0.0 ubuntu source | 20:46 |
| *** lpetrut has quit IRC | 20:47 | |
| *** tonanhngo has joined #openstack-kolla | 20:54 | |
| *** lpetrut has joined #openstack-kolla | 20:54 | |
| *** jtriley has joined #openstack-kolla | 20:55 | |
| vhosakot | Mannheim: I see the the ocata gate is broken https://review.openstack.org/#/c/474173/ | 20:56 |
| vhosakot | inc0: is this due to the ZUUL network traffic issue? | 20:57 |
| *** tonanhngo has quit IRC | 20:58 | |
| vhosakot | no jobs running in http://zuul.openstack.org | 20:58 |
| sbezverk | rwellum : try to get kubectl logs neutron-openvswitch-agent-network-5jjpv -n kolla | 20:59 |
| *** tonanhngo has joined #openstack-kolla | 21:00 | |
| jascott1 | sbezverk have you run into the 'helm delete hangs' issue? | 21:01 |
| sbezverk | jascott1 I noticed when daemonset's pod is crashing sometimes helm delete takes very very long time but it kills it finally | 21:02 |
| sbezverk | never seen forever hang scenario | 21:02 |
| kfox1111 | I personally haven't, but I have a co-worker who has. | 21:02 |
| kfox1111 | I have seen daemonsets fail to delete nicely when a node's down though. | 21:02 |
| sbezverk | with earlier k8s version I hit kubectl issue when it was hanging forever deleting something | 21:03 |
| jascott1 | earlier==? | 21:03 |
| sbezverk | the only workaround was to ctrlC and then manually delete from etcd | 21:03 |
| sbezverk | 1.4.X | 21:04 |
| jascott1 | well for helm, restarting tiller pod fixes it temporarily (so it seems) | 21:04 |
| jascott1 | k thanks | 21:04 |
| *** tonanhngo has quit IRC | 21:04 | |
| sbezverk | cool, will keep an eye on it | 21:04 |
| kfox1111 | the tiller hang my co-worker reported was with 1.6.x | 21:04 |
| jascott1 | ya ive still got 1.6 | 21:05 |
| jascott1 | and helm 2.5 | 21:05 |
| jascott1 | its getting to be a problem https://github.com/kubernetes/helm/issues/2560 | 21:05 |
| inc0 | yeah, but I think if you hit recheck it should be better now | 21:05 |
| sbezverk | kfox1111: was he deleting something special ;) ?? | 21:05 |
| inc0 | or not | 21:05 |
| *** jtriley has quit IRC | 21:06 | |
| sbezverk | inc0: FYI without new target image I will not be able to complete lioadm thing. please keep that in mind ;) | 21:07 |
| jascott1 | i was running tiller local in IDE with debug and eventually it just stops responding. initially I do a `helm delete` and i hit my breakpoint normally but after a few it just stops all together. wont even hit main.init breakpoint | 21:07 |
| *** jtriley has joined #openstack-kolla | 21:08 | |
| sbezverk | jascott1 but tiller should have been still alive, otherwise k8s controller probably killed it,no? | 21:09 |
| vhosakot | zuul is coming back up | 21:09 |
| inc0 | sbezverk: no worries, I trust you with this one | 21:10 |
| sbezverk | inc0: do not need trust ;), need review :) | 21:10 |
| jascott1 | so i killed all the tiller things in the cluser and was just running local. it was still creating pods on my cluster so I know it was working | 21:11 |
| Mannheim | unfortunately I cannot debug the metadata issue atm | 21:11 |
| *** bmace has quit IRC | 21:11 | |
| *** bmace has joined #openstack-kolla | 21:12 | |
| Mannheim | could it be be a dhcp service issue? | 21:12 |
| *** jtriley has quit IRC | 21:14 | |
| vhosakot | Mannheim: docker logs <metadata_container> | 21:15 |
| Mannheim | the container was ok from docker pov | 21:16 |
| Mannheim | it was running | 21:16 |
| Mannheim | it's true, i did not run that | 21:16 |
| Mannheim | do you think i could have seen something there? | 21:17 |
| *** dklyle has joined #openstack-kolla | 21:18 | |
| vhosakot | Mannheim: you should see neutron-metadataagent's stdout there. could have errors? | 21:18 |
| *** david-lyle has quit IRC | 21:19 | |
| *** Teck72 has joined #openstack-kolla | 21:21 | |
| Mannheim | unfortunately I do not have access to the env anymore | 21:22 |
| *** rfxn has joined #openstack-kolla | 21:23 | |
| *** tonanhngo has joined #openstack-kolla | 21:24 | |
| *** Teck7 has quit IRC | 21:24 | |
| *** Teck7 has joined #openstack-kolla | 21:25 | |
| *** Teck72 has quit IRC | 21:26 | |
| *** jtriley has joined #openstack-kolla | 21:26 | |
| *** oanson has quit IRC | 21:28 | |
| *** tonanhngo has quit IRC | 21:28 | |
| *** oanson has joined #openstack-kolla | 21:29 | |
| *** rfxn has quit IRC | 21:29 | |
| *** tonanhngo has joined #openstack-kolla | 21:32 | |
| *** jtriley has quit IRC | 21:33 | |
| *** trozet has quit IRC | 21:33 | |
| *** jtriley has joined #openstack-kolla | 21:34 | |
| jamesbenson | to use external ceph in kolla, you need to follow these directions: https://docs.openstack.org/kolla-ansible/latest/external-ceph-guide.html; question is, those are on the deploy node correct? The glance-api.conf, ceph.conf, ceph.client.glance.keyring? | 21:35 |
| *** pc_m has quit IRC | 21:35 | |
| *** mmehan has quit IRC | 21:35 | |
| inc0 | jamesbenson: these are config overrides | 21:36 |
| inc0 | so all on deploy nodes | 21:36 |
| jamesbenson | inc0; so on all of the deployed nodes in the cluster, not the kolla deploy node, right? | 21:37 |
| *** kristian__ has quit IRC | 21:38 | |
| *** tonanhngo has quit IRC | 21:38 | |
| inc0 | kolla deploy node | 21:38 |
| inc0 | node you run kolla-ansible from | 21:38 |
| jamesbenson | okay, | 21:38 |
| *** tonanhngo has joined #openstack-kolla | 21:39 | |
| jamesbenson | okay, just wanted to make sure, because the config dir doesn't exist.... so I'll have to make this pathway then. | 21:39 |
| *** tonanhngo has quit IRC | 21:43 | |
| jamesbenson | when I made the ceph cluster, it never spit out a glance keyring, where do you get the key from? | 21:45 |
| *** pc_m has joined #openstack-kolla | 21:48 | |
| *** jamesbenson has quit IRC | 21:51 | |
| *** iniazi has joined #openstack-kolla | 21:51 | |
| *** jtriley has quit IRC | 21:52 | |
| *** jamesbenson has joined #openstack-kolla | 21:53 | |
| *** daidv has joined #openstack-kolla | 21:53 | |
| Reepicheep | jamesbenson: did you use a tool to create your ceph cluster? | 21:54 |
| *** hieulq has joined #openstack-kolla | 21:54 | |
| SamYaple | jamesbenson: you have to create the glance keyring | 21:54 |
| *** duonghq has joined #openstack-kolla | 21:55 | |
| SamYaple | jamesbenson: or, more acurately, a key with access to the glance pool | 21:55 |
| *** iniazi_ has quit IRC | 21:55 | |
| SamYaple | jamesbenson: technically you can just use the admin key if you update the client name and such | 21:55 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding alternative iscsi_helper support https://review.openstack.org/481690 | 21:56 |
| *** Teck72 has joined #openstack-kolla | 21:56 | |
| Reepicheep | jamesbenson: yeah the point is that more then likely an external tool is not responsible to connect it to glance | 21:56 |
| *** jascott1 has quit IRC | 21:56 | |
| *** jascott1 has joined #openstack-kolla | 21:56 | |
| *** tonanhngo has joined #openstack-kolla | 21:57 | |
| *** jamesbenson has quit IRC | 21:58 | |
| *** duonghq has quit IRC | 21:58 | |
| *** jascott1 has quit IRC | 21:58 | |
| *** lpetrut has quit IRC | 21:58 | |
| *** jascott1 has joined #openstack-kolla | 21:58 | |
| *** Teck7 has quit IRC | 21:59 | |
| *** jascott1 has quit IRC | 22:00 | |
| *** tonanhngo has quit IRC | 22:01 | |
| *** jascott1 has joined #openstack-kolla | 22:01 | |
| *** tonanhngo has joined #openstack-kolla | 22:02 | |
| *** jascott1 has quit IRC | 22:03 | |
| *** jascott1_ has joined #openstack-kolla | 22:03 | |
| Mannheim | i was looking on how the volumes are mounted for the nova services and i found this "/run/:/run/:shared" | 22:06 |
| Mannheim | what does share mean exactly? i mean it's not shared by default? | 22:06 |
| kfox1111 | its a mount namespace flag. | 22:07 |
| kfox1111 | by default a mount copied in only affects the one volume and submounts/events are not propigated. | 22:07 |
| *** tonanhngo has quit IRC | 22:07 | |
| kfox1111 | so, /run on the host would be /run in the container normally, but | 22:08 |
| *** jascott1_ has quit IRC | 22:08 | |
| kfox1111 | if the user mounted /dev/cdrom to /run/mycd, it would be on the host, but not in the container. | 22:08 |
| *** jtriley has joined #openstack-kolla | 22:08 | |
| kfox1111 | shared syncs mount events from the host <->container both ways. | 22:08 |
| kfox1111 | so the container would see /run/mycd/*. | 22:08 |
| kfox1111 | and could further change it like unmount it, and the host would see it unmounted too. | 22:08 |
| kfox1111 | slave is a different argment that is host -> container only. | 22:09 |
| kfox1111 | so the container would see mounts, but the container unmounting something wouldn't show up on the host. | 22:09 |
| *** daidv has quit IRC | 22:12 | |
| *** hieulq has quit IRC | 22:12 | |
| *** schwicht has quit IRC | 22:12 | |
| *** jtriley has quit IRC | 22:13 | |
| *** Teck7 has joined #openstack-kolla | 22:14 | |
| *** Teck72 has quit IRC | 22:15 | |
| Mannheim | makes perfect sense, thank you | 22:17 |
| *** duonghq has joined #openstack-kolla | 22:18 | |
| *** gfidente|afk has quit IRC | 22:19 | |
| *** deep-book-gk_ has joined #openstack-kolla | 22:20 | |
| *** deep-book-gk_ has left #openstack-kolla | 22:20 | |
| *** tonanhngo has joined #openstack-kolla | 22:20 | |
| *** shardy has quit IRC | 22:22 | |
| *** Teck72 has joined #openstack-kolla | 22:23 | |
| *** tonanhngo has quit IRC | 22:25 | |
| *** Teck7 has quit IRC | 22:26 | |
| *** jtriley has joined #openstack-kolla | 22:26 | |
| *** Teck7 has joined #openstack-kolla | 22:27 | |
| *** daidv has joined #openstack-kolla | 22:27 | |
| *** hieulq has joined #openstack-kolla | 22:27 | |
| SamYaple | kfox1111: container unmounting sometihng in a shared mount namespace will absolutely affect the host | 22:28 |
| *** Teck72 has quit IRC | 22:28 | |
| *** schwicht has joined #openstack-kolla | 22:28 | |
| SamYaple | kfox1111: otherwise the host would have lots of neutron network namespaces that never unmount | 22:29 |
| *** manheim has quit IRC | 22:29 | |
| SamYaple | kfox1111: oh sorry. my buffer is all messed up. i see you said just that | 22:31 |
| *** Mannheim has quit IRC | 22:32 | |
| *** tonanhngo has joined #openstack-kolla | 22:32 | |
| *** manheim has joined #openstack-kolla | 22:33 | |
| *** ducttape_ has quit IRC | 22:37 | |
| *** tonanhngo has quit IRC | 22:37 | |
| *** manheim has quit IRC | 22:37 | |
| *** jascott1 has joined #openstack-kolla | 22:38 | |
| jascott1 | i messed my buffer up in '95 | 22:38 |
| *** manheim has joined #openstack-kolla | 22:39 | |
| *** ducttape_ has joined #openstack-kolla | 22:45 | |
| *** ducttape_ has quit IRC | 22:50 | |
| *** schwicht has quit IRC | 23:07 | |
| *** ducttape_ has joined #openstack-kolla | 23:12 | |
| *** Teck72 has joined #openstack-kolla | 23:16 | |
| *** Teck7 has quit IRC | 23:19 | |
| *** rhallisey has quit IRC | 23:20 | |
| *** ducttape_ has quit IRC | 23:20 | |
| *** manheim has quit IRC | 23:26 | |
| *** hongbin has quit IRC | 23:42 | |
| *** goldyfruit has quit IRC | 23:45 | |
| *** caowei has joined #openstack-kolla | 23:55 | |
| *** oanson has quit IRC | 23:59 | |
| *** jtriley has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!