| *** zhurong has quit IRC | 00:00 | |
| sbezverk | kfox1111 ping | 00:03 |
|---|---|---|
| kfox1111 | ping | 00:03 |
| sbezverk | kfox1111 if you have time please check this: https://review.openstack.org/#/c/346215/ | 00:04 |
| patchbot | sbezverk: patch 346215 - kolla-kubernetes - Glance bootstrap to create service/project/role | 00:04 |
| kfox1111 | k... let me look. | 00:05 |
| kfox1111 | reviewed | 00:11 |
| sbezverk | kfox1111 thank you | 00:13 |
| kfox1111 | np | 00:13 |
| sbezverk | kfox1111 can you point to a specific place with regards to credentials passing? | 00:15 |
| sbezverk | I mean we used the same approach even in kolla, but I would like to better understand your concern | 00:15 |
| kfox1111 | kolla-kubernetes.yaml line 59 | 00:16 |
| kfox1111 | and various places in the job where it references them in the command. | 00:16 |
| kfox1111 | I guess my concern is with the level of secretness. | 00:16 |
| kfox1111 | any pod you can just kubectl get pod name -o yaml and get stuff back. | 00:16 |
| kfox1111 | where as you might have to work harder at retrieving a secret. | 00:17 |
| kfox1111 | kolla-ansible gets around the issue as stuff is always pushed from ansible host to individual hosts in one direction. | 00:17 |
| kfox1111 | k8s has a central server(s) that you can pull back stuff from. | 00:18 |
| sbezverk | I see what you mean. it is going to be very hard as we pass this to kolla-toolbox ansible module | 00:18 |
| kfox1111 | I think if we just put the vars used there now in a bash env var, and loaded that as a secret, | 00:18 |
| sbezverk | it would require major rearchitecture and this approach will become kube specific | 00:19 |
| kfox1111 | in the command you can just source it first, then reference the vars inline? | 00:19 |
| kfox1111 | here... let me mark up a little bit and maybe I can show you what I'm thinking... | 00:20 |
| sbezverk | would be better :-) | 00:21 |
| *** daneyon_ has joined #openstack-kolla | 00:22 | |
| kfox1111 | replied. | 00:26 |
| *** daneyon_ has quit IRC | 00:26 | |
| kfox1111 | not sure how readable that is or not. | 00:27 |
| openstackgerrit | Kabir Sial proposed openstack/kolla: Customizations for dind https://review.openstack.org/353202 | 00:27 |
| openstackgerrit | Kabir Sial proposed openstack/kolla: Customizations for dind https://review.openstack.org/353203 | 00:27 |
| sbezverk | I see what you mean.. currently kolla and kolla-kube stores all password in the file | 00:30 |
| kfox1111 | yeah. but it ends up storing it in the pod descriptions too, which I think's a really bad idea. :/ | 00:31 |
| sbezverk | which is used during rendering by kolla and kolla-kube | 00:31 |
| kfox1111 | you can still template out a secret. | 00:31 |
| kfox1111 | added an example of what the secret would look like I think. | 00:33 |
| sbezverk | we will always hit one problem, consistency with kolla | 00:34 |
| kfox1111 | there can not be perfect consistancy as they have totally different security models. | 00:34 |
| kfox1111 | ansible assumes you have one central node, you secure the heck out of, and push from there to other nodes. | 00:35 |
| sbezverk | kube has bunch of features to go wild, but using them would mean kube specific images | 00:35 |
| *** duonghq has joined #openstack-kolla | 00:35 | |
| kfox1111 | kube's plan is to secure secrets different then everything else and only make them available as needed. | 00:35 |
| kfox1111 | the proposed solution requires nothing kubernetes specific in the image. | 00:36 |
| kfox1111 | it just passes the passwords via a shell environment file loaded via k8s secret, instead of hard coding them into the generated pod description. | 00:36 |
| sbezverk | who will create kube secrets? | 00:37 |
| kfox1111 | in the end, the same command line's are executed in the containers. | 00:37 |
| kfox1111 | the bootstrap process. | 00:37 |
| *** dwalsh has joined #openstack-kolla | 00:38 | |
| kfox1111 | I've got to head out. talk more tomorrow? | 00:40 |
| sbezverk | absolutely, I will read up more on secrets and how they are consumed, have a good one | 00:40 |
| kfox1111 | kk. | 00:41 |
| kfox1111 | basically, they function exactly like configmaps. | 00:41 |
| kfox1111 | they are just a seperate resource so you can rbac them differently/more aggressively. | 00:41 |
| kfox1111 | the plan is to make it such that all kubelets can download all configmaps, but kubelets can only download a secret if they have a pod registered to them that references it. | 00:42 |
| kfox1111 | so if you break into a node, its harder to steal a secret that you don't already have on the node. | 00:42 |
| kfox1111 | ok. really gota go now. l8r. | 00:43 |
| *** hieulq_ has joined #openstack-kolla | 00:43 | |
| *** sdake has joined #openstack-kolla | 00:59 | |
| sdake | wirehead_ any latest updates from sv? | 01:02 |
| *** zhurong has joined #openstack-kolla | 01:07 | |
| *** fragatin_ has quit IRC | 01:12 | |
| *** fragatina has joined #openstack-kolla | 01:12 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Add Ceph support for Mariadb https://review.openstack.org/350312 | 01:17 |
| *** dwalsh has quit IRC | 01:19 | |
| *** unicell has quit IRC | 01:26 | |
| duonghq | Jeffrey41: I pushed new patch sets according your reply, can you review it? | 01:31 |
| duonghq | https://review.openstack.org/#/c/352275/ | 01:31 |
| patchbot | duonghq: patch 352275 - kolla - Port with_ loop variable syntax to Ansbile 2 | 01:31 |
| duonghq | Jeffrey4l: can you review my updated patch set | 01:32 |
| Jeffrey4l | duonghq, np | 01:35 |
| *** v1k0d3n has quit IRC | 01:40 | |
| *** v1k0d3n has joined #openstack-kolla | 01:40 | |
| *** v1k0d3n has quit IRC | 01:43 | |
| *** v1k0d3n has joined #openstack-kolla | 01:44 | |
| *** mbound has joined #openstack-kolla | 01:51 | |
| *** v1k0d3n has quit IRC | 01:51 | |
| *** mbound has quit IRC | 01:52 | |
| duonghq | Jeffrey4l: thanks | 01:52 |
| Jeffrey4l | my pleasure :) | 01:53 |
| *** hieulq has joined #openstack-kolla | 02:11 | |
| *** ayoung has quit IRC | 02:11 | |
| openstackgerrit | Hiroki Ito proposed openstack/kolla: Fix prechecks of a vip address https://review.openstack.org/353214 | 02:13 |
| sdake | Jeffrey4l my man whats up | 02:13 |
| Jeffrey4l | not much. But need leave for a while, kid is ill :( | 02:15 |
| openstackgerrit | Merged openstack/kolla: Introduce a script to launch ovsdb-server process https://review.openstack.org/351784 | 02:17 |
| openstackgerrit | Hiroki Ito proposed openstack/kolla: Fix prechecks of a vip address https://review.openstack.org/353221 | 02:22 |
| *** senk_ has joined #openstack-kolla | 02:28 | |
| *** zhurong has quit IRC | 02:42 | |
| *** zhurong has joined #openstack-kolla | 02:44 | |
| *** zhurong has quit IRC | 02:48 | |
| *** coolsvap has joined #openstack-kolla | 02:49 | |
| *** zhurong has joined #openstack-kolla | 02:49 | |
| *** diogogmt has quit IRC | 02:51 | |
| *** zhurong has quit IRC | 02:58 | |
| *** zhurong has joined #openstack-kolla | 02:59 | |
| Jeffrey4l | sbezverk, could see the comments https://review.openstack.org/#/c/351784/16 | 03:04 |
| patchbot | Jeffrey4l: patch 351784 - kolla - Introduce a script to launch ovsdb-server process (MERGED) | 03:04 |
| *** zhubingbing has joined #openstack-kolla | 03:06 | |
| Jeffrey4l | Daviey, https://review.openstack.org/349764 https://review.openstack.org/349197 need your comments :) | 03:07 |
| Jeffrey4l | sdake, https://review.openstack.org/352089 ready for review | 03:08 |
| sdake | you will go to the degabah system | 03:08 |
| sdake | and learn from yoda, the jedi master that instructed me | 03:08 |
| sdake | Jeffrey4l thats french for taking a break :) | 03:09 |
| Jeffrey4l | haha new language | 03:10 |
| sdake | its from star wars | 03:10 |
| sdake | empire strikes back | 03:11 |
| sdake | general veers, prepare your men! | 03:16 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 | 03:16 |
| *** sdake has quit IRC | 03:17 | |
| *** britthouser has joined #openstack-kolla | 03:20 | |
| *** britthouser has quit IRC | 03:25 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 | 03:31 |
| *** huikang has joined #openstack-kolla | 03:31 | |
| *** phuongnh has joined #openstack-kolla | 03:39 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 | 03:55 |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair aodh image dockerfile https://review.openstack.org/350715 | 03:58 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Vagrant doc updates https://review.openstack.org/353240 | 04:08 |
| *** rkrum has joined #openstack-kolla | 04:13 | |
| duonghq | coolsvap: can you review my change at https://review.openstack.org/#/c/352275/ | 04:27 |
| patchbot | duonghq: patch 352275 - kolla - Port with_ loop variable syntax to Ansbile 2 | 04:27 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 | 04:27 |
| * coolsvap checks | 04:28 | |
| *** zhubingbing_ has joined #openstack-kolla | 04:30 | |
| *** zhubingbing has quit IRC | 04:32 | |
| *** senk_ has quit IRC | 04:33 | |
| *** senk_ has joined #openstack-kolla | 04:33 | |
| *** fragatin_ has joined #openstack-kolla | 04:36 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 | 04:38 |
| *** zhubingbing has joined #openstack-kolla | 04:39 | |
| *** fragatina has quit IRC | 04:39 | |
| zhubingbing | hi，Duong Ha-Quang | 04:39 |
| duonghq | hi zhubingbing | 04:39 |
| zhubingbing | I think it makes sense to you that I think it is reasonable in the new version, you can teach me how to modify it. | 04:40 |
| *** fragatin_ has quit IRC | 04:40 | |
| duonghq | ah, for the varible syntax: as Ansible doc point out there: https://docs.ansible.com/ansible/porting_guide_2.0.html#deprecated | 04:41 |
| duonghq | the bare variable syntax is deprecated for Ansible 2.0 | 04:41 |
| duonghq | for ex: the old style is | 04:41 |
| duonghq | - var_a | 04:42 |
| duonghq | and it is recommended to write as: | 04:42 |
| duonghq | - {{ var_a }} | 04:42 |
| duonghq | ah | 04:42 |
| duonghq | - "{{ var_a }}" | 04:42 |
| duonghq | the full variable syntax is {{ var_a }} but it must be quoted in " " | 04:43 |
| zhubingbing | - {{ container_envs.results | 04:43 |
| zhubingbing | }} - check_results.results | 04:43 |
| duonghq | it must be quoted in " " | 04:43 |
| zhubingbing | What is the definition of this variable in my place | 04:43 |
| coolsvap | duonghq: looks good rechecking to see if I can get green in oraclelinux deploy gate | 04:44 |
| coolsvap | does not seem to be related to fix | 04:44 |
| coolsvap | just to see before I approve | 04:44 |
| duonghq | coolsvap: thanks | 04:45 |
| zhubingbing | - "container_envs.results" | 04:45 |
| zhubingbing | - "check_results.results" | 04:45 |
| zhubingbing | Like this？ | 04:45 |
| duonghq | - "{{ container_envs.results }}" | 04:46 |
| zhubingbing | duonghq: | 04:46 |
| duonghq | like this | 04:46 |
| zhubingbing | This variable does not need to be defined, is he a ansible default variable? | 04:47 |
| zhubingbing | sure | 04:47 |
| duonghq | zhubingbing: this variable is defined at line 33 | 04:48 |
| duonghq | register: .... | 04:48 |
| zhubingbing | i fixed now,thank you very much,Your advice is really good for me | 04:48 |
| duonghq | my pleasure | 04:48 |
| *** coolsvap is now known as _coolsvap_ | 04:49 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 | 04:53 |
| *** daneyon_ has joined #openstack-kolla | 04:53 | |
| zhubingbing | duonghq: Can you help us to see, just look at this. | 04:53 |
| zhubingbing | https://review.openstack.org/352515 | 04:53 |
| duonghq | zhubingbing: I just +1 on this | 04:54 |
| zhubingbing | OK thanks,I'm going to deal with the other role. | 04:54 |
| zhubingbing | Your advice is really good. | 04:55 |
| duonghq | you're welcome | 04:55 |
| *** daneyon_ has quit IRC | 04:58 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 | 04:58 |
| *** unicell has joined #openstack-kolla | 05:00 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 | 05:01 |
| zhubingbing | All of my role have been modified. | 05:01 |
| *** unicell has quit IRC | 05:06 | |
| *** unicell has joined #openstack-kolla | 05:06 | |
| *** senk_ has quit IRC | 05:09 | |
| *** phuongnh has quit IRC | 05:09 | |
| *** unicell has quit IRC | 05:09 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add Barbican container dockerfile https://review.openstack.org/351822 | 05:10 |
| zhubingbing | hi,Christian Berendt | 05:10 |
| zhubingbing | Christian Berendt: | 05:10 |
| *** unicell has joined #openstack-kolla | 05:13 | |
| *** _coolsvap_ is now known as coolsvap | 05:16 | |
| huikang | Hi, Jeffrey4l, are you around? | 05:17 |
| Jeffrey4l | huikang, yup | 05:18 |
| huikang | when I deploy all-in-one, the mariadb bootstrap always failed | 05:18 |
| huikang | Jeffrey4l, http://paste.openstack.org/show/553320/ here is the log message | 05:19 |
| Jeffrey4l | huikang, i meet this once. no chance to debug it. :( | 05:19 |
| Jeffrey4l | how many node are u deploying ? huikang | 05:20 |
| huikang | all-in-one | 05:20 |
| duonghq | huikang: you can try execute deploy twice in a round | 05:21 |
| huikang | let me try now | 05:21 |
| duonghq | zhubingbing: are you still there? IMO, you should add one line for your module in globals.conf | 05:22 |
| zhubingbing | yes | 05:22 |
| duonghq | ah globals.yml | 05:22 |
| duonghq | like other modules | 05:22 |
| zhubingbing | add globals.conf? | 05:22 |
| duonghq | zhubingbing: etc/kolla/globals.yml | 05:22 |
| zhubingbing | Do you need to add this file? | 05:23 |
| duonghq | zhubingbing: this file is already in source code | 05:23 |
| huikang | duonghq, it bypasses the bootstrap error, but the mariadb datatable is not functioning | 05:23 |
| Jeffrey4l | huikang, could i reach that server? | 05:23 |
| duonghq | huikang: it seem that you get error same mine: https://bugs.launchpad.net/kolla/+bug/1609502 | 05:24 |
| openstack | Launchpad bug 1609502 in kolla "Container bootstrap_mariadb exits due to Galera failed" [Undecided,New] | 05:24 |
| -openstackstatus- NOTICE: zuul is being restarted to reload configuration. Jobs should be re-enqueued but if you're missing anything (and it's not on http://status.openstack.org/zuul/) please issue a recheck in 30min. | 05:24 | |
| huikang | Jeffrey4l, no, the server is behind firewall. | 05:24 |
| Jeffrey4l | huikang, try to add `set -x` to the docker/mariadb/extend_start file and build , then deploy. it will print more useful info. | 05:25 |
| *** phuongnh has joined #openstack-kolla | 05:25 | |
| huikang | will do | 05:25 |
| huikang | duonghq, yes, I confirm your bug report | 05:26 |
| duonghq | huikang: which base os and deploy type are you using? | 05:26 |
| huikang | I tried both centos binary and ubuntu source; both failed | 05:26 |
| duonghq | do you use local docker registry? | 05:27 |
| huikang | duonghq, yes; does that matter? | 05:28 |
| duonghq | not really, when I setup my local docker registry, this bug completely go away. | 05:28 |
| zhubingbing | I didn't see the gloable.yaml in the other modules | 05:29 |
| duonghq | I cannot reproduce this bug up to now. | 05:29 |
| duonghq | zhubingbing: it is in root of kolla project | 05:29 |
| huikang | do you mean this file etc/kolla/globals.yml | 05:30 |
| duonghq | yes, | 05:30 |
| huikang | duonghq what are the base OS and type you are using now? | 05:30 |
| duonghq | huikang: I'm using ubuntu/source | 05:31 |
| zhubingbing | globals.yml file not nned | 05:33 |
| zhubingbing | What's wrong？http://logs.openstack.org/15/350715/9/check/gate-kolla-dsvm-build-centos-source/27113df/console.html | 05:33 |
| *** huikang has quit IRC | 05:36 | |
| *** duonghq has quit IRC | 05:39 | |
| openstackgerrit | Merged openstack/kolla: Remove unused project_yaml parameter from role metadata files https://review.openstack.org/351928 | 05:50 |
| openstackgerrit | Ross Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate https://review.openstack.org/353261 | 05:51 |
| *** unicell has quit IRC | 05:55 | |
| *** unicell has joined #openstack-kolla | 05:55 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair aodh image dockerfile https://review.openstack.org/350715 | 05:55 |
| *** duonghq has joined #openstack-kolla | 06:01 | |
| *** hieulq_ has quit IRC | 06:04 | |
| *** egonzalez90 has joined #openstack-kolla | 06:12 | |
| *** unicell has quit IRC | 06:22 | |
| *** unicell has joined #openstack-kolla | 06:22 | |
| *** senk_ has joined #openstack-kolla | 06:28 | |
| *** bootsha has joined #openstack-kolla | 06:29 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 | 06:31 |
| *** dmueller-evoila has joined #openstack-kolla | 06:31 | |
| duonghq | coolsvap: Finally, all build and deploy test is green | 06:33 |
| coolsvap | duonghq: ack! | 06:35 |
| duonghq | coolsvap: thanks | 06:35 |
| *** fragatina has joined #openstack-kolla | 06:37 | |
| zhubingbing | hi coolsvap | 06:39 |
| coolsvap | zhubingbing: hello | 06:39 |
| zhubingbing | Can you help me reivew my code | 06:39 |
| *** inc0 has joined #openstack-kolla | 06:39 | |
| zhubingbing | --)) | 06:40 |
| *** fragatina has quit IRC | 06:42 | |
| openstackgerrit | Merged openstack/kolla: Port with_ loop variable syntax to Ansbile 2 https://review.openstack.org/352275 | 06:44 |
| coolsvap | zhubingbing: will do | 06:44 |
| coolsvap | but cant tell you the time to be honest | 06:44 |
| zhubingbing | - -thanks you | 06:44 |
| coolsvap | but will do | 06:44 |
| *** inc0 has quit IRC | 06:45 | |
| zhubingbing | http://logs.openstack.org/88/348388/23/check/gate-kolla-dsvm-deploy-centos-binary/e4d4203/console.html | 06:48 |
| zhubingbing | who can help me look it | 06:48 |
| duonghq | zhubingbing which change? | 06:51 |
| duonghq | i.e patch set | 06:52 |
| zhubingbing_ | http://logs.openstack.org/88/348388/23/check/gate-kolla-dsvm-deploy-centos-binary/e4d4203/console.html | 06:58 |
| duonghq | Did you recheck? | 06:59 |
| zhubingbing_ | duonghq: https://review.openstack.org/#/c/351320/ | 07:00 |
| patchbot | zhubingbing_: patch 351320 - kolla - fix sahara dockerfile | 07:00 |
| *** duonghq_ has joined #openstack-kolla | 07:04 | |
| *** duonghq has quit IRC | 07:05 | |
| *** rkrum has quit IRC | 07:06 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair aodh image dockerfile https://review.openstack.org/350715 | 07:10 |
| zhubingbing_ | ok | 07:13 |
| zhubingbing_ | thankins | 07:13 |
| *** b_bezak has joined #openstack-kolla | 07:17 | |
| *** athomas has joined #openstack-kolla | 07:21 | |
| *** ad_rien_ has joined #openstack-kolla | 07:22 | |
| *** ad_rien_ has quit IRC | 07:25 | |
| *** ad_rien_ has joined #openstack-kolla | 07:27 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 | 07:28 |
| *** ad_rien_ has quit IRC | 07:29 | |
| *** ad_rien_ has joined #openstack-kolla | 07:32 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 | 07:32 |
| *** matrohon has joined #openstack-kolla | 07:33 | |
| *** bootsha has quit IRC | 07:35 | |
| *** shardy has joined #openstack-kolla | 07:35 | |
| *** daneyon_ has joined #openstack-kolla | 07:36 | |
| *** bootsha has joined #openstack-kolla | 07:38 | |
| *** daneyon_ has quit IRC | 07:40 | |
| *** duonghq has joined #openstack-kolla | 07:41 | |
| *** matrohon has quit IRC | 07:41 | |
| *** duonghq_ has quit IRC | 07:43 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 | 07:43 |
| *** duonghq has quit IRC | 07:44 | |
| *** Serlex has joined #openstack-kolla | 07:45 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 | 07:46 |
| *** godleon has quit IRC | 07:50 | |
| *** kong has quit IRC | 07:50 | |
| *** matrohon has joined #openstack-kolla | 07:57 | |
| *** bootsha has quit IRC | 07:59 | |
| *** bootsha has joined #openstack-kolla | 08:05 | |
| *** duonghq has joined #openstack-kolla | 08:09 | |
| *** matrohon has quit IRC | 08:15 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** openstackgerrit has joined #openstack-kolla | 08:18 | |
| *** zhubingbing has quit IRC | 08:29 | |
| *** zhubingbing_ has quit IRC | 08:29 | |
| *** matrohon has joined #openstack-kolla | 08:31 | |
| *** shardy has quit IRC | 08:44 | |
| *** shardy has joined #openstack-kolla | 08:45 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Vagrant doc updates https://review.openstack.org/353240 | 08:48 |
| *** gfidente|afk is now known as gfidente | 08:50 | |
| *** ntpttr has quit IRC | 08:51 | |
| *** ntpttr has joined #openstack-kolla | 08:58 | |
| *** shardy is now known as shardy_mtg | 09:00 | |
| duonghq | anybody try swift deploy in master branch? | 09:09 |
| *** ad_rien_ has quit IRC | 09:15 | |
| *** matrohon has quit IRC | 09:22 | |
| *** bootsha has quit IRC | 09:26 | |
| *** ad_rien_ has joined #openstack-kolla | 09:34 | |
| *** ad_rien_ has quit IRC | 09:36 | |
| *** matrohon has joined #openstack-kolla | 09:38 | |
| *** mnasiadka has joined #openstack-kolla | 09:40 | |
| openstackgerrit | Kirill Shileev proposed openstack/kolla: Customizations for Glance https://review.openstack.org/352977 | 09:41 |
| *** mbound has joined #openstack-kolla | 09:44 | |
| *** sdake has joined #openstack-kolla | 09:47 | |
| sdake | morning folks | 09:53 |
| sdake | Jeffrey4l around? | 10:00 |
| sdake | coolsvap around? | 10:00 |
| coolsvap | sdake: yes | 10:00 |
| sdake | anything happening with the osic gear? | 10:00 |
| *** zhurong has quit IRC | 10:02 | |
| coolsvap | sdake: I did the tempest verification today morning | 10:03 |
| sdake | on the ceph based scenario? | 10:03 |
| coolsvap | I was waiting for updates from rally team related to the issues we are having related to benchmark tests | 10:03 |
| coolsvap | yes on the ceph based scenario | 10:04 |
| sdake | dont block on rally | 10:04 |
| sdake | what is the probelm there | 10:04 |
| *** matrohon has quit IRC | 10:04 | |
| coolsvap | the image we add has private visibility | 10:04 |
| coolsvap | so the benchmark tests cannot recognize the cirros image | 10:04 |
| coolsvap | and fail the validation of scenario | 10:05 |
| sdake | add it with public visbility | 10:05 |
| coolsvap | yes | 10:05 |
| coolsvap | i was talking on rally channel when you pinged :) | 10:05 |
| sdake | oh | 10:05 |
| sdake | ok carry on then :) | 10:05 |
| coolsvap | I am planning to do some benchmark tests in next couple of hours and then will free the environment | 10:06 |
| sdake | does rally benchmark vm performance? | 10:14 |
| sdake | coolsvap ^^ | 10:14 |
| coolsvap | not vm the openstack cloud performance | 10:15 |
| sdake | throw in a vm test | 10:16 |
| sdake | of dd performance from inside the vm pls | 10:16 |
| sdake | or bonnie would work too | 10:17 |
| coolsvap | sdake: i have not used bonnie before | 10:20 |
| sdake | in the docs i started | 10:20 |
| sdake | there is a bonnie command run | 10:20 |
| sdake | instead of 520G | 10:20 |
| *** matrohon has joined #openstack-kolla | 10:21 | |
| sdake | change it to 2xmemoryofvm+10 | 10:21 |
| *** rkrum has joined #openstack-kolla | 10:23 | |
| sdake | does rally output to json? | 10:25 |
| *** shardy_mtg is now known as shardy | 10:25 | |
| *** bootsha has joined #openstack-kolla | 10:26 | |
| openstackgerrit | Kirill Shileev proposed openstack/kolla: Customizations for Glance https://review.openstack.org/352977 | 10:28 |
| *** matrohon has quit IRC | 10:29 | |
| *** mbound has quit IRC | 10:33 | |
| *** bootsha has quit IRC | 10:33 | |
| coolsvap | yes it gives output in json | 10:35 |
| sdake | we can use pandoc to convert json to rst | 10:36 |
| sdake | coolsvap did that election wrap up for requirements? | 10:36 |
| coolsvap | i think it will today | 10:37 |
| coolsvap | tomorrow | 10:41 |
| coolsvap | sdake: in 2xmemoryofvm+10 | 10:41 |
| *** ad_rien_ has joined #openstack-kolla | 10:42 | |
| coolsvap | I am keeping memory of vm 2G | 10:42 |
| *** mkoderer_ has joined #openstack-kolla | 10:42 | |
| *** mkoderer_ has quit IRC | 10:42 | |
| sdake | bonnie++ -G 6G should do | 10:43 |
| *** matrohon has joined #openstack-kolla | 10:45 | |
| openstackgerrit | Ross Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate https://review.openstack.org/353261 | 10:45 |
| sdake | every time i see those reviews it makes me cry :( | 10:47 |
| sdake | 2 hrs of reviewwork | 10:47 |
| sdake | incoming | 10:47 |
| *** duonghq has left #openstack-kolla | 10:47 | |
| pbourke | same | 10:49 |
| coolsvap | umm i tested with 520G again, how much time it will take? | 10:49 |
| *** bootsha has joined #openstack-kolla | 10:49 | |
| *** jtriley has joined #openstack-kolla | 10:50 | |
| openstackgerrit | Ross Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate https://review.openstack.org/353261 | 10:51 |
| pbourke | coolsvap: tested what? | 10:53 |
| *** matrohon has quit IRC | 10:53 | |
| coolsvap | pbourke: bonnie | 10:53 |
| pbourke | ah | 10:53 |
| coolsvap | pbourke: I did the tempest run and a vm create delete benchmark test | 10:53 |
| openstackgerrit | Ross Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate https://review.openstack.org/353261 | 10:54 |
| coolsvap | adding it to the review | 10:55 |
| coolsvap | i hope we can integrate rally reports in the doc somehow | 10:56 |
| *** vbel has quit IRC | 10:59 | |
| *** vbel has joined #openstack-kolla | 10:59 | |
| *** JoseMello has joined #openstack-kolla | 11:01 | |
| *** b_bezak has quit IRC | 11:01 | |
| *** b_bezak has joined #openstack-kolla | 11:01 | |
| *** b_bezak has quit IRC | 11:06 | |
| *** matrohon has joined #openstack-kolla | 11:09 | |
| *** daneyon_ has joined #openstack-kolla | 11:12 | |
| *** matrohon has quit IRC | 11:12 | |
| *** kong has joined #openstack-kolla | 11:16 | |
| *** daneyon_ has quit IRC | 11:16 | |
| sdake | get rst output | 11:18 |
| sdake | i'll take care of the rest | 11:19 |
| sdake | 520g in a vm will take forever | 11:19 |
| sdake | try with 6G | 11:19 |
| sdake | coolsvap ^^ | 11:19 |
| sdake | rather get json output | 11:20 |
| coolsvap | sdake: I will kill the bonnie then | 11:20 |
| sdake | yup | 11:20 |
| coolsvap | just a min | 11:20 |
| *** ppowell has joined #openstack-kolla | 11:21 | |
| sdake | hey ppowell long time no chat | 11:21 |
| coolsvap | sdake: done! | 11:21 |
| sdake | how long did it take to run? | 11:22 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet https://review.openstack.org/349366 | 11:22 |
| coolsvap | i killed the 520G one, it was running from the time i asked you before | 11:23 |
| sdake | 30 seconds then? | 11:23 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Add OSIC Scale Testing Documentation https://review.openstack.org/352101 | 11:23 |
| coolsvap | no, it was running for 35 mins | 11:24 |
| openstackgerrit | Ross Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate https://review.openstack.org/353261 | 11:24 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 | 11:26 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet https://review.openstack.org/349366 | 11:27 |
| *** matrohon has joined #openstack-kolla | 11:28 | |
| sdake | results of bonnie++ were what? | 11:29 |
| sdake | does rally only test instance start/stoptime? | 11:30 |
| *** b_bezak has joined #openstack-kolla | 11:31 | |
| *** zhubingbing has joined #openstack-kolla | 11:31 | |
| *** zhubingbing_ has joined #openstack-kolla | 11:32 | |
| *** matrohon has quit IRC | 11:36 | |
| *** fragatina has joined #openstack-kolla | 11:36 | |
| *** sean-k-mooney-aw is now known as sean-k-mooney | 11:37 | |
| sdake | morning sean-k-mooney | 11:37 |
| sean-k-mooney | sdake: morning :) | 11:37 |
| *** zhubingbing_ has quit IRC | 11:38 | |
| sdake | get any reviews on your work? | 11:38 |
| coolsvap | rally tests inside vm statistics with dd but currently its not able to get cinder | 11:38 |
| *** zhubingbing has quit IRC | 11:38 | |
| sdake | cinder wont deploy? | 11:39 |
| sean-k-mooney | sdake: just looking at them now i dont see any new comments | 11:39 |
| *** dwalsh has joined #openstack-kolla | 11:39 | |
| * sdake sadfaces | 11:39 | |
| Daviey | i'm not sure how interesting stats are with all SSD deployment.. nobody is surely doing that? | 11:39 |
| sdake | Daviey the ceph 10gig network is the limiting factor here | 11:40 |
| Daviey | if we could artificially throttle the mons to spinning speed, that becomes more interesting.. Then testing with and without ssd journal | 11:40 |
| sean-k-mooney | sdake: i do have some on the kolla-host though which i will take a look at now | 11:40 |
| Daviey | sdake: interesting | 11:40 |
| Daviey | sdake: are we maxing out 10G? | 11:40 |
| sdake | i do in my ceph deployment at home | 11:41 |
| *** fragatina has quit IRC | 11:41 | |
| sdake | i wnted to see what happens on a large scaledeploy | 11:41 |
| Daviey | sdake: is this ceph rebalancing network IO or general usage? | 11:41 |
| sdake | no idea | 11:41 |
| Daviey | sdake: i mean, are you able to max out the network once the thing is 'stable'? | 11:42 |
| Daviey | ie, a day old deployment | 11:42 |
| sdake | with ceph + dd yes on 3 nodes | 11:43 |
| sdake | with ceph + dd on 20 storage nodes = ?? | 11:43 |
| *** rkrum has quit IRC | 11:43 | |
| Daviey | sdake: i never imagined network would be the limiting factor on spinning disks | 11:44 |
| sdake | i have ssds in my 3 node cluster - intel 750 nvme | 11:44 |
| Daviey | ah | 11:44 |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Customizations for Cinder https://review.openstack.org/350587 | 11:47 |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Revert "Remove files from /var/lib/apt/lists when cleaning up on Ubuntu/Debian" https://review.openstack.org/353408 | 11:47 |
| *** rmart04 has joined #openstack-kolla | 11:49 | |
| *** v1k0d3n has joined #openstack-kolla | 11:50 | |
| coolsvap | sdake: so there is not cinder service in openstack service list | 11:50 |
| *** sdake has quit IRC | 11:51 | |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Customizations for Nova https://review.openstack.org/351676 | 11:51 |
| *** sdake has joined #openstack-kolla | 11:51 | |
| *** athomas has quit IRC | 11:52 | |
| *** matrohon has joined #openstack-kolla | 11:52 | |
| *** v1k0d3n has quit IRC | 11:54 | |
| Daviey | coolsvap: no surprising, cinder isn't enabled | 11:55 |
| coolsvap | Daviey: yeah but thats not known to rally | 11:56 |
| *** sdake_ has joined #openstack-kolla | 11:56 | |
| *** athomas has joined #openstack-kolla | 11:58 | |
| Daviey | coolsvap: we want cinder tho, right? | 11:58 |
| *** sdake has quit IRC | 11:59 | |
| coolsvap | Daviey: I have not deployed ceph as of now either with kolla or without | 11:59 |
| sdake_ | cinder and nova use similar drivers | 11:59 |
| *** msimonin has joined #openstack-kolla | 11:59 | |
| coolsvap | sdake_: we do need cinder for ceph deployment right? | 12:00 |
| sdake_ | ceph is already deplyed in our current scenario | 12:00 |
| Daviey | coolsvap: I mean, currently we have ceph deployed.. but we are not using either the object or block integration with openstack | 12:00 |
| Daviey | so currently the ceph is useless to us? | 12:01 |
| sdake_ | nova uses ceph | 12:01 |
| sdake_ | for its ephemeral storage | 12:01 |
| Daviey | sdake_: Oh, the test case wants to use nova block storage rather than cinder? | 12:01 |
| sdake_ | the test case i want is bonnie on nova ephemeral storage | 12:02 |
| sdake_ | (onceph) | 12:02 |
| Daviey | but not via cinder? | 12:02 |
| sdake_ | that is a different scenario iirc | 12:02 |
| *** rhallisey has joined #openstack-kolla | 12:04 | |
| sdake_ | morning rhallisey | 12:06 |
| rhallisey | sdake_, hey | 12:06 |
| ad_rien_ | Hi Guys, in the context of the performance working group, we are working on a 1000 compute node experiments (see http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-08-09-16.00.log.html for more inforamtions). Right now, we developed a set of scripts to deploy kolla on top of Grid'5000 (i.e. a testbed in France dedidacted to research on distributed systems) using the nova fake driver. However, we would like to know wh | 12:11 |
| *** mliima has quit IRC | 12:12 | |
| Daviey | ad_rien_: your message was too long and truncated... ended with "However, we would like to know wh" | 12:15 |
| ad_rien_ | Daviey: sorry, "However, we would like to know whether it is possible to deploy several 'real/containerized' computes nodes on the same physical server. I asked the questions few days ago but I didn't get any answer. Is there any core member of Kolla online ? Can someone tell me whether posting such a question on the dev mailing list makes sense ? Thanks" | 12:16 |
| rhallisey | ad_rien_, https://github.com/openstack/kolla/blob/master/doc/nova-fake-driver.rst | 12:17 |
| ad_rien_ | rhallisey: yes we are using the nova fake driver right now but we would like to deploy a real compute node container. | 12:18 |
| ad_rien_ | I mean several compute node containers per physical server. | 12:18 |
| rhallisey | ad_rien_, I'm not sure it will work | 12:19 |
| ad_rien_ | Right now, the solution is to use virtualization (for instance two VMs per physical servers) and then deploy kolla as if those VMs where standard servers. But this means : physical server / Virtualization layer / docker which is rather strange from our point of view | 12:20 |
| rhallisey | ad_rien_, the problem you may run into is with net=host, where the compute nodes will use the hosts network namespace | 12:20 |
| sdake_ | lets not forget pid=host | 12:21 |
| *** phuongnh has quit IRC | 12:21 | |
| *** rcherrueau has joined #openstack-kolla | 12:21 | |
| *** matrohon has quit IRC | 12:21 | |
| sdake_ | ad_rien_ the short answer is - if you want to simulate more gear then you have available, you will have to simulate that using some other tech besides docker | 12:22 |
| sdake_ | such as vms | 12:22 |
| ad_rien_ | rhallisey: I agree and that 's why we would like to know whether someone already plays with such a use-case or if Kolla key persons already know it will not run | 12:22 |
| sdake_ | if you want to use kolla, deploy 2 vms on your 500 nodes, then deploy kolla on top of that | 12:22 |
| ad_rien_ | sdake_: so should I understand that conceptually speaking (i.e. from the docker view point), it will not be possible | 12:23 |
| *** magnus-loof has joined #openstack-kolla | 12:23 | |
| sdake_ | it is not a docker issue | 12:23 |
| rhallisey | ad_rien_, I think it would require removing net=host. You may have to go with kolla on kjolla | 12:23 |
| sdake_ | its a libvirt issue | 12:23 |
| rhallisey | ya and a libvirt issue | 12:23 |
| sdake_ | only one copy of libvirt may be running on a system at a time | 12:23 |
| rhallisey | ad_rien_, this would be a problem regardless of docker or not | 12:23 |
| Daviey | ad_rien_: I think the only way you can reasonably do it, is by running VM's on the bare metal and expose them as bridged networking | 12:24 |
| sdake_ | rhallisey right - has nothign to do with docker | 12:24 |
| rhallisey | sdake_, ya I'm agreeing with you | 12:24 |
| Daviey | ad_rien_: so you have the appearance of more physical servers. Using qemu nested virt, it should work just fine | 12:24 |
| ad_rien_ | s/docker/containers | 12:24 |
| magnus-loof | Hi! I am trying to deploy OS using kolla. I want to use the published 2.0.2 for the time being. However, when running kolla-ansible deploy with stable/mitaka or 2.0.2 tag, I get the following errors: https://gist.github.com/maglo/be4a0c039dfd5d02fdc4e45c41f31c36 | 12:25 |
| * ad_rien_ is looking on the net to understand why several libvirt cannot run on a system at the same time or why container technologies does not enable to separate such daemons. | 12:25 | |
| *** zhurong has joined #openstack-kolla | 12:26 | |
| Daviey | ad_rien_: I'd probably say.. On a kolla compute node, the key container is the libvirt container.. And kolla doesn't support generating separate libvirt socket files for each one. | 12:27 |
| sdake_ | Daviey and that isn't a use case that makes any sense, except for profiling, which is why libvirt doesn't support it | 12:28 |
| magnus-loof | I managed to deploy OS using Kolla and master when I downgrade to 2.0.2 in /etc/kolla/globals.yml. Howerver, this was bricked today, probably due to https://github.com/openstack/kolla/commit/41ec75b56734604ad548b354bc929105babbb538 | 12:28 |
| sdake_ | magnus-loof that is a master only patch | 12:28 |
| Daviey | sdake_: well.. i'd be suprised if livirt couldn't itself work.. you can specify the socket file to use at runtime.. but the libvirt container we have doesn't | 12:28 |
| sdake_ | 2.0.2 is stable/mitaka | 12:29 |
| Daviey | sdake_: but equally, i agree it doesn't make sense unless you are benchmarking | 12:29 |
| Daviey | sdake_: and then.. i'd say it gets pretty messy fast.. and usig nested virt makes much more sense to have multiple kolla hosts per baremetal | 12:29 |
| sbezverk | magnus-loof: the change you are refering to applied only to master | 12:29 |
| sbezverk | magnus-loof: you should not see it in stable mitaka | 12:30 |
| magnus-loof | sdake_ yes I know. To clarify, yesterday when deploying from kolla/master with 2.0.2 i globals.yml, it worked. Today it was bricked (could not find the file that was changed in the commit mentioned). | 12:31 |
| rhallisey | ad_rien_, pid=host is why | 12:31 |
| Daviey | magnus-loof: that probably means you were using master with previously built mitaka images in your registry... this is a configuration that shouldn't be done | 12:31 |
| ad_rien_ | sdake_: daviey: I agree that such a use-case makes sense for benchmarking purposes… that's our goal actually and adding the virtualization layer (i.e. the nested one) may lead to some OS noises that we would like to prevent/mitigate | 12:32 |
| magnus-loof | So, I want to deploy from stable/mitaka. However, that fails with message in https://gist.github.com/maglo/be4a0c039dfd5d02fdc4e45c41f31c36 | 12:32 |
| Daviey | ad_rien_: what is the scope of the test? | 12:32 |
| sdake_ | magnus-loof please run pip show kolla -> paste | 12:32 |
| magnus-loof | Daviey Yes I know - my initial try was to install from stable/mitaka. But that fails. | 12:32 |
| ad_rien_ | Daviey: 1./ compare that results obtained with the fake driver are the same ones that with real containerized compute nodes | 12:33 |
| Daviey | ad_rien_: I mean, what is the limiting factor you are trying to find? I mean, 1 compute stack per baremetal should be able to run the same amount of fake vm's as 10 x fake nova vm's. Equally, i'd be suprised if running 10 VM's on the baremetal would cause much noise tot he actual test. | 12:33 |
| Daviey | ad_rien_: why not try kolla on kolla.. Deploy an undercloud, boot 10 VM's per bare metal.. and deploy kolla on that? | 12:35 |
| magnus-loof | currently running: https://gist.github.com/maglo/59c40924b01c965a86e02fab1ac02696 That is from master -> https://github.com/openstack/kolla/commit/559631815afed3a9bf51f89b2d121b0d1a34de12 | 12:35 |
| Daviey | magnus-loof: What are you trying to achieve ? | 12:36 |
| magnus-loof | I will abort and downgrade to what? 2.0.2 or stable/mitaka? | 12:36 |
| ad_rien_ | Daviey: that can be a way to go on, indeed | 12:36 |
| Daviey | magnus-loof: when do you want your cloud in prod? | 12:36 |
| sdake_ | magnus-loof - can you run pip show kolla pls | 12:37 |
| magnus-loof | Daviey sorry for being unclear. I want to deploy OS using kolla. For now we are just kicking the tires and rebuilding every once in a while. I want to avoid building docker 3.0.x images, due to time contraints. | 12:37 |
| Daviey | magnus-loof: Generally, i'm of the opinion that using stable/* for your first prod deployment is a bad idea.. I think tracking Master whilst you are in dev/test and submitting bugs/patches to make sure it does what you need (and help shape kolla) is better. | 12:38 |
| *** matrohon has joined #openstack-kolla | 12:38 | |
| Daviey | magnus-loof: And then, post release, you can go prod and track stable/newton which was the Master you knew worked | 12:38 |
| magnus-loof | So therefore I set out to install OS using stable/mitaka -> Fail 2.0.2 -> Fail. However, master with openstack_release: \"2.0.2\" worked fine until yesterday. | 12:38 |
| sdake_ | Daviey generall8yt the message we give to people evaluating kolla is to eval on stable branches | 12:38 |
| Daviey | magnus-loof: rebuilding all the images should normally take under an hour | 12:39 |
| magnus-loof | sdake_ output from pip show kolla https://gist.github.com/maglo/59c40924b01c965a86e02fab1ac02696 | 12:39 |
| sdake_ | magnus-loof - asking for third time - please run pip show kolla -> paste | 12:39 |
| Daviey | sdake_: well, sure - kicking the tires.. but if people are commited to going into prod with it.. tracking Master during implementation phase makes more sense IMO | 12:39 |
| sdake_ | Version: 3.0.0.0b3.dev170 | 12:39 |
| magnus-loof | sdake_ I put the output in the gist | 12:40 |
| sdake_ | magnus-loof if you want to run mitaka, you need a 2.0.0 version | 12:40 |
| sdake_ | so lets get you going | 12:40 |
| sdake_ | pip remove kolla | 12:40 |
| sdake_ | or pip uninstall kolla | 12:40 |
| Daviey | sdake_: The moment you need to do something a little left-field, you are stuffed if using stable/*.. but Master, you can help make it happen | 12:40 |
| sdake_ | can never remember which one works | 12:40 |
| ad_rien_ | Daviey: sdake_ thanks for the information, we will check on our side whether using the nested virtualisation is really an issue or not. Thanks for your answer | 12:40 |
| sdake_ | ad_rien_ ack | 12:41 |
| magnus-loof | sdake_ ok. I am aborting the deploy using kolla 3.0.0 with release 2.0.2 in globals.yml | 12:41 |
| Daviey | ad_rien_: Nested virt on ubuntu has been the default since 12.04... it is pretty solid, espeically on intel | 12:41 |
| magnus-loof | sdake_ running kolla-ansible destroy -i <inventory> | 12:41 |
| sdake_ | magnus-loof - the plan is as follows - use 2.0.2 images from docker hub with stable/mitaka | 12:41 |
| sdake_ | magnus-loof i'll wlak you through it | 12:41 |
| ad_rien_ | Daviey: my doubts is related to operations such as booting VMs for instance if we want to perform scheduling performance analysis etc... | 12:41 |
| *** srwilkers has joined #openstack-kolla | 12:42 | |
| Daviey | ad_rien_: Then why not test 1 kolla stack per bare metal? I think you need to tailor your test harness for what you are testing | 12:42 |
| ad_rien_ | our goal is to try to setup a framework that can enable researchers/engineers to evaluate particular services/components of OpenStack and see whether there are some side-effect at large scale | 12:43 |
| sdake_ | magnus-loof pip uninstall kolla or pip remove kolla - whichever comand works proplery | 12:43 |
| Daviey | ad_rien_: If you want to test control plane scalability, then the nested stuff works great. | 12:43 |
| ad_rien_ | Daviey: I need more time to give relevant remarks ;) | 12:43 |
| Daviey | ad_rien_: but if you want to test VM performance, then just launch X concurrent VM's on the same baremetal (without the nested virt) | 12:43 |
| *** bootsha has quit IRC | 12:43 | |
| magnus-loof | Successfully uninstalled kolla-3.0.0.0b3.dev170 | 12:44 |
| sdake_ | magnus-loof if you have a git checkout of kolla, you can git checkout stable/mitaka | 12:44 |
| sdake_ | magnus-loof if yu dont', ou can pip isntall kolla==2.0.2 | 12:44 |
| magnus-loof | sdake git checkout stable/mitaka | 12:44 |
| sdake_ | now in kolla home dir, run pip install -e . | 12:45 |
| sdake_ | then pip show kolla -> paste | 12:45 |
| ad_rien_ | Daviey: but as a distributed computing researcher, we are used to discover corner cases only at large scale…. our goal is to try to be able to detect such corner case situations. If you are using the fake driver you do not really start VMs and so can miss the corner cases we are looking for. | 12:45 |
| Daviey | sdake_: Hmm, as root? or --user ? | 12:46 |
| magnus-loof | sdake_ sudo pip install -e . Version: 2.0.3.dev11 | 12:46 |
| sdake_ | nice | 12:46 |
| *** bootsha has joined #openstack-kolla | 12:46 | |
| sdake_ | magnus-loof in the tools directory there are 3 cleanup tools | 12:46 |
| sdake_ | run all 3 | 12:46 |
| ad_rien_ | Daviey: anyway, your replies were helpful, we need to braimstorm on our side to investigate whether nested VM would it the right way to go. | 12:46 |
| Daviey | ad_rien_: get back to us, and importantly share your results :) | 12:47 |
| sdake_ | magnus-loof after that docker ps -a should show nothing , docker images should show nothing | 12:48 |
| magnus-loof | sdake_ now I might have a clue as to why deploy does not work on 2.0.2: I do not have docker on the provisioning host. I will fix. | 12:48 |
| magnus-loof | sdake_ cleanup-host failed with User root can't seem to run Docker commands. | 12:49 |
| *** matrohon has quit IRC | 12:49 | |
| sdake_ | magnus-loof sounds like docker isn't installed | 12:49 |
| ad_rien_ | Daviey: sure (BTW we propose a presentation to the next summit ''Chasing 1000 nodes scale'': let' see whether it will selected or not ;)) | 12:50 |
| ad_rien_ | (s/it will selected/it will be selected) | 12:50 |
| Daviey | ad_rien_: I'll keep an eye out to attend it, thanks | 12:50 |
| magnus-loof | sdake_ yes, docker was not installed on provisioning host. It is installed on network / host / controller though | 12:50 |
| magnus-loof | sdake_ but now I am installing docker on provisioning host | 12:50 |
| sdake_ | magnus-loof - my instructions wer ein error | 12:51 |
| sdake_ | run kolla-ansible cleanup -i your-inventory-file | 12:51 |
| sdake_ | you dont need docker on the provision host at this time | 12:51 |
| sdake_ | hmm, cleanup is only on master | 12:51 |
| magnus-loof | sdake_ without it I could not run clean-host | 12:51 |
| sdake_ | you will hae to log into each of your nodes | 12:52 |
| magnus-loof | [centos@operator tools]$ git describe 2.0.2-11-ge0a49d8 | 12:52 |
| sdake_ | and run the 3 cleanup scripts manually in 2.0.2 | 12:52 |
| magnus-loof | sdake_ ok | 12:52 |
| sdake_ | and for future reference, cleanup has been renamed destroy :) | 12:52 |
| sdake_ | magnus-loof how many nodes do you have | 12:53 |
| *** jtriley has quit IRC | 12:54 | |
| magnus-loof | sdake_ I have only one control, one network and one host for now | 12:55 |
| *** zhubingbing_ has joined #openstack-kolla | 12:56 | |
| *** zhubingbing has joined #openstack-kolla | 12:56 | |
| sdake_ | cool so not too onerous to cleanup then | 12:56 |
| sdake_ | this is also my testbed at home | 12:56 |
| *** egonzalez90 has quit IRC | 12:58 | |
| sbezverk | sdake_ qq the review from last night for rabbitmq, woudl you accept introducing rabbitmq_version variable instead of hardcoding version? | 12:58 |
| magnus-loof | sdake_ I think we are clean | 12:59 |
| sdake_ | magnus-loof cool | 12:59 |
| sdake_ | magnus-loof if not - deploy will fail and we will knwo about it | 12:59 |
| sdake_ | sbezverk i had considered that during reiewing | 12:59 |
| sdake_ | let me get back to you after helping magnus-loof | 12:59 |
| sbezverk | sdake_ ok | 12:59 |
| magnus-loof | sdake_ ok so clear /etc/kolla on provisioning host? | 13:00 |
| sdake_ | no | 13:00 |
| sdake_ | but run cleanup-host on provision host | 13:00 |
| magnus-loof | sdake_ Creating a fstab backup... Removing ceph references from fstab... Getting folders name... Skipping: /etc/kolla/config Skipping: /etc/kolla/globals.yml Skipping: /etc/kolla/kolla-build.conf Skipping: /etc/kolla/passwords.yml | 13:00 |
| *** v1k0d3n has joined #openstack-kolla | 13:02 | |
| magnus-loof | [centos@operator kolla]$ diff etc/kolla/globals.yml /etc/kolla/globals.yml 20c20 < #openstack_release: "2.0.3" --- > openstack_release: "2.0.2" 26c26 < kolla_internal_vip_address: "10.10.10.254" --- > kolla_internal_vip_address: "192.168.1.200" 108c108 < #enable_central_logging: "no" --- > #enable_central_logging: "yes" | 13:02 |
| magnus-loof | so I am a little worried about 2.0.3 as it was not published on hub.docker.com this morning | 13:02 |
| sdake_ | magnus-loof yup we are going to fix that now | 13:03 |
| sdake_ | 2.0.3 is unreleased | 13:03 |
| sdake_ | Daviey has some ideas to clarify that - wiating on him and Jeffrey4l to powwow on that topic and come to a conclusion | 13:03 |
| sdake_ | magnus-loof would you mind pasting your globals.yml pls | 13:03 |
| srwilkers | good morning everyone :D | 13:04 |
| rhallisey | srwilkers, morning! | 13:04 |
| magnus-loof | sdake_ globals.yml: https://gist.github.com/maglo/4b880f00d5cce70d29361bc3537fca5d | 13:04 |
| sdake_ | Daviey this version pbr stuff is becoming an emerging problem - can you drive it to conclusion for the stable branches? | 13:05 |
| sdake_ | sup srwilkers | 13:05 |
| sdake_ | magnus-loof are you on centos? | 13:06 |
| *** matrohon has joined #openstack-kolla | 13:06 | |
| magnus-loof | sdake_ yes | 13:06 |
| sdake_ | for your host os | 13:06 |
| sdake_ | magnus-loof eth0/eth1 are not your ethernet interfaces | 13:06 |
| sdake_ | atleast those are not the default ethernet interfaces on centos | 13:06 |
| magnus-loof | they are on my cloud (Ravello) | 13:06 |
| sdake_ | oh your deploying on vms? | 13:06 |
| magnus-loof | sdake_ yes, and on vagrant | 13:07 |
| sdake_ | magnus-loof have to warn ahead of time - that may not work - kolla uses multicast to determine the VIP - and some clouds don't support multicast | 13:07 |
| sdake_ | magnus-loof but lets try anyway | 13:07 |
| sdake_ | let me finish reviewing your globals.yml | 13:07 |
| sdake_ | magnus-loof what is located at the .200 address? | 13:08 |
| magnus-loof | sdake_ I think Ravello will work for that: https://www.ravellosystems.com/solutions/openstack/kvm-on-aws-technology | 13:09 |
| magnus-loof | sdake_ there is nothing on .200 now. It will be claimed by haproxy | 13:09 |
| sdake_ | nice | 13:09 |
| sdake_ | ok pull the trigger on the deploy | 13:09 |
| sdake_ | time for a cigar | 13:09 |
| *** berendt has joined #openstack-kolla | 13:09 | |
| *** diogogmt has joined #openstack-kolla | 13:10 | |
| magnus-loof | sdake_ I think we cleaned a little too much: ERROR! the playbook: /usr/share/kolla/ansible/site.yml could not be found | 13:10 |
| berendt | Please fast forward https://review.openstack.org/#/c/353408 | 13:10 |
| patchbot | berendt: patch 353408 - kolla - Revert "Remove files from /var/lib/apt/lists when ... | 13:10 |
| sdake_ | magnus-loof moment to think | 13:11 |
| sdake_ | magnus-loof there is some pip command that lists files in a package | 13:11 |
| sdake_ | magnus-loof probably pip list kolla or soething similiar -> paste | 13:11 |
| *** jtriley has joined #openstack-kolla | 13:12 | |
| sdake_ | magnus-loof - I think my instructions were in error | 13:12 |
| sdake_ | pip remove kolla | 13:12 |
| sdake_ | or pip unisntall kolla | 13:12 |
| sdake_ | then sudo pip install . from the kolla tlD | 13:13 |
| sdake_ | TLD | 13:13 |
| sdake_ | we don't want the -e in centos apparently :) | 13:13 |
| sdake_ | apologies for the churn | 13:13 |
| magnus-loof | sdake_ Aargh. Ravello stopped my application ( I had set it to run for 4 hrs). A few minutes and I will be back! | 13:15 |
| sdake_ | roger | 13:15 |
| sdake_ | ping me when your ready to roll | 13:15 |
| sdake_ | sbezverk - version # hardcode in ansible scripts is better then version # hardcode in docker scripts | 13:16 |
| sdake_ | sbezverk where would you hardcode it ? all.yml? | 13:16 |
| magnus-loof | sdake_ OK we are back | 13:17 |
| sdake_ | sbezverk lets put that convo on hold while you think about that question | 13:17 |
| *** matrohon has quit IRC | 13:17 | |
| sdake_ | magnus-loof - ok were you able to get the pip install . in the kolla TLD? | 13:17 |
| sdake_ | after remvoing kolla | 13:18 |
| *** sdake_ is now known as sdake | 13:18 | |
| magnus-loof | sdake_ it is running now... | 13:18 |
| sdake | deploy or pip? | 13:18 |
| magnus-loof | sdake_ Successfully installed kolla-2.0.3.dev11 | 13:19 |
| sdake | ok pull trigger on the deploy now | 13:19 |
| * sdake just learned something new | 13:20 | |
| magnus-loof | sdake_ running... | 13:20 |
| sbezverk | sdake_ yes | 13:20 |
| sdake | sbezverk this hardcode would not go in globals.yml? | 13:20 |
| sbezverk | sdake it is not up to the user to change this version | 13:21 |
| sdake | pbourke coolsvap are u guys off for the evening? | 13:21 |
| magnus-loof | sdake OK so now I am expecting to see https://gist.github.com/maglo/be4a0c039dfd5d02fdc4e45c41f31c36 | 13:21 |
| sbezverk | so exposing it in globals.yml is not the best approach imho | 13:21 |
| magnus-loof | unless the installation of docker on the provisioning host rectified the problem. | 13:21 |
| pbourke | sdake: im around but dont have a huge amount of time to do osic stuff | 13:21 |
| sdake | magnus-loof i would expect it to work | 13:21 |
| pbourke | for today | 13:21 |
| sdake | pbourke roger | 13:22 |
| magnus-loof | sdake me too! but BOOM we are stuck at the same place | 13:22 |
| sdake | pbourke can you give a quick status update for today | 13:22 |
| magnus-loof | NoneType | 13:22 |
| sdake | magnus-loof can i get a paste o the log | 13:22 |
| sdake | of the log | 13:22 |
| sdake | magnus-loof ansible --version | 13:23 |
| magnus-loof | sure, what logfile? | 13:23 |
| *** harlowja has quit IRC | 13:23 | |
| magnus-loof | [centos@operator kolla]$ ansible --version ansible 2.1.0.0 config file = /home/centos/.ansible.cfg configured module search path = Default w/o overrides | 13:23 |
| sdake | pbourke related to osic | 13:23 |
| sdake | magnus-loof one thing the documetnation doesn't mentoin is ansible 1.9.4 is required for stable branches | 13:23 |
| magnus-loof | ah | 13:23 |
| sdake | pip install ansible==1.9.4 | 13:24 |
| pbourke | sdake: from what I can see we are ready for scenario 3 | 13:24 |
| sdake | pbourke did we get the disk io benchmarks for ceph? | 13:24 |
| sdake | we being you or coolsvap | 13:24 |
| pbourke | sdake: I dont see them on the doc, I'll let coolsvap respond on that one | 13:25 |
| sdake | pbourke roger | 13:25 |
| magnus-loof | sdake I had ansible installed from repo. which one is preferred? repo or pip? | 13:25 |
| sdake | pbourke - i will gather them if you can give me a host to log in to with a vm rolling in it | 13:25 |
| sdake | magnus-loof sudo yum remove ansible | 13:26 |
| pbourke | sdake: 172.21.8.253 is the deploy host | 13:26 |
| magnus-loof | ansible 1.9.4 configured module search path = None | 13:26 |
| sdake | magnus-loof obviously packaged versions from distros are better, but this is just a limitation people have to live with because ansible doesn't support 1.9.z | 13:26 |
| pbourke | sdake: 'tmux attach' on that, no VMs running at the moment but you could spawn one | 13:26 |
| sdake | pbourke i can look in inventory from there | 13:26 |
| sdake | pbourke thanks again for all your super hard work on this effort | 13:27 |
| sdake | pbourke this work is super critical | 13:27 |
| sdake | as important as the bifrost work | 13:27 |
| magnus-loof | sdake: ok there is a difference ansible 1.9.4 does not recognize .ansible.cfg as 2.1 was configured to... | 13:27 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Fix interface address for RabbitMQ and disable cluster for Kubernetes https://review.openstack.org/336751 | 13:27 |
| sdake | magnus-loof save a copy of that file and move it out of the way | 13:28 |
| sdake | ansible will create a new one | 13:28 |
| sbezverk | sdake even with variable is not ideal, becauase we hardcode a specific plugin. | 13:28 |
| sdake | magnus-loof thanks for sticking with us :) | 13:28 |
| magnus-loof | sdake thank YOU for helping me. | 13:29 |
| sdake | the specific plugin is hardcoded ina docker file | 13:29 |
| magnus-loof | run kolla-ansible deploy | 13:29 |
| magnus-loof | ? | 13:29 |
| sdake | magnus-loof your welcome for the support :) | 13:29 |
| sdake | magnus-loof yes deploy | 13:29 |
| coolsvap | sdake: I'm afk for hour or so | 13:29 |
| sdake | sbezverk thats an orthognoal problem | 13:29 |
| sdake | if people complain in the review, i'll say as much | 13:30 |
| sdake | coolsvap ack - did you gather the io benchmarks from a vm? | 13:30 |
| sdake | coolsvap and havne't publislhed? | 13:30 |
| sdake | or need me to do so | 13:30 |
| sbezverk | sdake: how about assigning rabbitmq_version: the whole thing: rabbitmq_server-3.5.7/plugins/rabbitmq_clusterer-3.5.x-189b3a81.ez/rabbitmq_clusterer-3.5.x-189b3a81/ebin | 13:30 |
| sbezverk | in this case it will be more flexible | 13:31 |
| *** srwilkers has quit IRC | 13:31 | |
| sdake | sbezverk bounce that idea off the review qeue and see what the cr's have to think about it | 13:31 |
| coolsvap | I haven't done | 13:31 |
| sdake | coolsvap ok i've got it up next after wrapping up with magnus | 13:31 |
| sdake | sbezverk recommend rabbitmq_plugin_path | 13:32 |
| openstackgerrit | Dave Walker proposed openstack/kolla: Add --user-domain when adding heat role admin https://review.openstack.org/353471 | 13:33 |
| *** mntdevops has joined #openstack-kolla | 13:33 | |
| sdake | sbezverk the issue is the plugin path is needed, not hte version # is needed | 13:34 |
| magnus-loof | sdake: so far so good | 13:34 |
| *** matrohon has joined #openstack-kolla | 13:34 | |
| sdake | magnus-loof yup i'd expect unicorns and rainbows from here on out :) | 13:34 |
| sdake | magnus-loof just to recap what we did: | 13:35 |
| sdake | 1. we are using the stable 2..0.2 images on docker hub | 13:35 |
| sdake | 2. we are using the stable/mitaka ansible code and extra other stuff required by kolla | 13:36 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Fix interface address for RabbitMQ and disable cluster for Kubernetes https://review.openstack.org/336751 | 13:36 |
| sdake | stable/mitaka is pretty close to 2.0.2 | 13:36 |
| magnus-loof | but i still need to sed 2.0.3 -> 2.0.2 until they are released, right? | 13:36 |
| sdake | until we release our mitaka z stream whcih happens every 45 days | 13:36 |
| magnus-loof | sweet | 13:37 |
| sdake | rhallisey can you rebase your docs patch please | 13:37 |
| *** bootsha has quit IRC | 13:37 | |
| sdake | rhallisey so we can get this recurring problem fixed | 13:37 |
| rhallisey | ya | 13:38 |
| *** bootsha has joined #openstack-kolla | 13:38 | |
| sdake | you know when you get an answer back "its complicated" from someone of high authority, its complicated :) | 13:38 |
| sdake | magnus-loof just to be clear, then you will be seding 2.0.4 to 2.0.3 until Daviey and Jeffrey4l come to conclusion on how to fix this pbr problem | 13:39 |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla: iscsi initiator names must be unique among hosts https://review.openstack.org/353483 | 13:40 |
| *** matrohon has quit IRC | 13:41 | |
| *** rcherrueau has quit IRC | 13:41 | |
| sdake | mandre ping | 13:41 |
| mandre | hey sdake | 13:42 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Fix intf address for RabbitMQ and disable cluster for Kube https://review.openstack.org/336751 | 13:42 |
| *** srwilkers has joined #openstack-kolla | 13:42 | |
| Daviey | sdake: you know... we should probably recommend people install from pip on pypi rather than git... your morning might have been easier :) | 13:46 |
| magnus-loof | sdake all work and no play makes jack a dull boy | 13:46 |
| sdake | Daviey roger that - hence the request for rhallisey to rebae his docs change | 13:47 |
| sdake | if nobody else wll fix this problem i guess I'll fix it | 13:47 |
| Daviey | chop chop rhallisey | 13:47 |
| *** diogogmt has quit IRC | 13:47 | |
| *** zhubingbing has quit IRC | 13:47 | |
| *** zhubingbing_ has quit IRC | 13:47 | |
| rhallisey | sdake, this was a confusing rebase | 13:47 |
| *** rkrum has joined #openstack-kolla | 13:47 | |
| *** zhubingbing__ has joined #openstack-kolla | 13:47 | |
| *** zhubingbing has joined #openstack-kolla | 13:47 | |
| rhallisey | let's merge before I have to do more of these | 13:48 |
| * rhallisey almost done | 13:48 | |
| sdake | rhallisey make sure it renders correctly | 13:49 |
| rhallisey | are there any tools to test in rendered? | 13:50 |
| Daviey | rhallisey: sphinx ? | 13:50 |
| rhallisey | k | 13:51 |
| sdake | tox -e doc | 13:52 |
| sdake | or tox -e docs rather | 13:52 |
| *** ayoung has joined #openstack-kolla | 13:53 | |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla: iscsi initiator names must be unique among hosts https://review.openstack.org/353483 | 13:54 |
| sdake | battery at 2% - if I drop off i'll be back | 13:55 |
| openstackgerrit | Dave Walker proposed openstack/kolla: Add --user-domain when adding heat role admin https://review.openstack.org/353471 | 13:57 |
| *** matrohon has joined #openstack-kolla | 13:58 | |
| magnus-loof | sdake TY so much for your help. kolla 2.0.2 with ansible 1.9.4 works :-D | 13:59 |
| *** JoseMello has quit IRC | 14:00 | |
| wirehead_ | sdake: So, chatted with the SAP people. There seems to be some cultural baggage and hard feelings. | 14:02 |
| sdake | hard feelings about what? | 14:02 |
| rhallisey | what does that mean O>o | 14:03 |
| sdake | we did them no wrong | 14:03 |
| sdake | magnus-loof happy to help support you in your ODM journey :) - enjoy | 14:03 |
| wirehead_ | They felt like they treid to get their patches merged but got nowhere, sometime last year. | 14:03 |
| sdake | wirehead_ i dont recall any sap patches at all | 14:04 |
| rhallisey | I don't think they ever submitted any | 14:04 |
| sdake | we included sap in our design sessions and have reached out numerous times to coordinate efforts | 14:04 |
| sdake | but thats in the past, anything we can do to repair those hard feelings? | 14:05 |
| Daviey | sdake: Is there a spec/bp for SAP? | 14:07 |
| sdake | Daviey we included sap engineers in the original kolla-kubernetes review | 14:08 |
| wirehead_ | Not sure yet. Will talk to the various people some more. | 14:08 |
| sdake | and asked them to review the spec and provide their unique insight | 14:08 |
| rhallisey | that's stange | 14:08 |
| Daviey | sdake: would be nice to understand the scope of potential SAP integration | 14:08 |
| wirehead_ | Yeah, like I got an earfull out of the guy. | 14:08 |
| sdake | anyone that reviewed the spec got an automatic cr spot on the kolla-kubernetes team | 14:08 |
| wirehead_ | The concern was mostly Kolla patches, not Kolla-Kubernetes patches. | 14:09 |
| *** zhubingbing_ has joined #openstack-kolla | 14:09 | |
| sdake | i see | 14:09 |
| *** zhurong has quit IRC | 14:09 | |
| sdake | our review constraints have become more relaxed since last year | 14:09 |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla: iscsi initiator names must be unique among hosts https://review.openstack.org/353483 | 14:10 |
| wirehead_ | Oh, and I’m going to talk to one of the folks from The New Stack about Kolla-Kubernetes. | 14:10 |
| *** bootsha has quit IRC | 14:10 | |
| *** kong has quit IRC | 14:10 | |
| sdake | wirehead_ if you get a chance to speak with the sap dude that gave you the earful, let him know themakeup of the community has changed, and their patches are welcome | 14:11 |
| sdake | what is the new stack | 14:11 |
| wirehead_ | http://thenewstack.io/ | 14:12 |
| *** zhubingbing has quit IRC | 14:12 | |
| *** zhurong has joined #openstack-kolla | 14:12 | |
| *** matrohon has quit IRC | 14:13 | |
| sdake | wirehead_ its better if you dont linke me those sorts of things | 14:14 |
| sdake | thanks ;) | 14:14 |
| *** zhubingbing__ has quit IRC | 14:14 | |
| *** rkrum has quit IRC | 14:15 | |
| *** zhubingbing__ has joined #openstack-kolla | 14:16 | |
| *** magnus-loof has quit IRC | 14:19 | |
| *** srwilkers has quit IRC | 14:19 | |
| *** matrohon has joined #openstack-kolla | 14:30 | |
| *** athomas has quit IRC | 14:32 | |
| sdake | future leaders -> http://www.cnn.com/2016/08/09/africa/orange-drought-kiara-nirghin/index.html | 14:32 |
| *** mnasiadka has quit IRC | 14:32 | |
| *** ad_rien_ has quit IRC | 14:35 | |
| *** srwilkers has joined #openstack-kolla | 14:36 | |
| openstackgerrit | Kirill Shileev proposed openstack/kolla: Customizations for Glance https://review.openstack.org/352977 | 14:38 |
| berendt | team meeting today at 16 UTC? | 14:39 |
| *** ppowell has quit IRC | 14:39 | |
| sdake | berendt roger | 14:41 |
| *** duonghq has joined #openstack-kolla | 14:42 | |
| *** athomas has joined #openstack-kolla | 14:43 | |
| sdake | evening athomas - how ya been | 14:44 |
| athomas | sdake, Hi. I'm pretty good, all in all. | 14:44 |
| athomas | sdake, You? | 14:44 |
| *** matrohon has quit IRC | 14:44 | |
| sdake | good - changing the world is hard but rewarding ;) | 14:44 |
| *** daneyon_ has joined #openstack-kolla | 14:49 | |
| *** zhurong has quit IRC | 14:49 | |
| *** senk_ has quit IRC | 14:51 | |
| *** daneyon_ has quit IRC | 14:53 | |
| *** ppowell has joined #openstack-kolla | 14:56 | |
| *** matrohon has joined #openstack-kolla | 14:59 | |
| *** ayoung has quit IRC | 15:01 | |
| *** b_bezak has quit IRC | 15:03 | |
| *** b_bezak has joined #openstack-kolla | 15:04 | |
| *** vhosakot has joined #openstack-kolla | 15:04 | |
| *** sdake_ has joined #openstack-kolla | 15:04 | |
| *** Serlex has quit IRC | 15:04 | |
| mntdevops | wanted to say "thank you" again to all of you for this project. I wrote up my efforts on my blog: http://mntdevops.com/2016/08/08/iaas-2/ | 15:05 |
| rhallisey | sdake_, doc patches are up | 15:05 |
| mntdevops | I have my all-in-one setup going -- next steps are to make it multi-node | 15:05 |
| sdake_ | mntdevops cool - thanks for the (rare in the tech industry) positive feedback | 15:06 |
| rhallisey | mntdevops, neat! | 15:06 |
| mntdevops | I've watched all your summit talks maybe 3 times | 15:06 |
| mntdevops | and I keep waiting for the next ones | 15:07 |
| *** sdake has quit IRC | 15:07 | |
| sdake_ | mntdevops we depend on the track chairs to accept our talks | 15:07 |
| sdake_ | no kolla talks were accepted in austin | 15:07 |
| mntdevops | kinda hoping for a how-to webinar or workshop things | 15:07 |
| mntdevops | bummer | 15:07 |
| mntdevops | for me | 15:08 |
| mntdevops | I think the hardest part | 15:08 |
| sdake_ | who knows what will happen with barcelona - results not out yet | 15:08 |
| mntdevops | was getting the "prep" stuff in the right order | 15:08 |
| *** b_bezak has quit IRC | 15:08 | |
| mntdevops | and a working globals.yml | 15:08 |
| mntdevops | so I rolled them into an ansible role | 15:09 |
| mntdevops | well, I hope a kolla talk is accepted at some point, because while Openstack Ansible is good | 15:09 |
| mntdevops | I prefer docker to lxc | 15:09 |
| *** vhosakot_ has joined #openstack-kolla | 15:11 | |
| mntdevops | oh, two questions: #1 Have you considered ansible-container for creating the containers (instead of kolla-build)? and #2 What are the reasons for not having as many pre-built releases on docker hub? | 15:12 |
| *** david-lyle has quit IRC | 15:12 | |
| *** vhosakot has quit IRC | 15:12 | |
| *** ayoung has joined #openstack-kolla | 15:12 | |
| *** david-lyle has joined #openstack-kolla | 15:12 | |
| *** senk_ has joined #openstack-kolla | 15:14 | |
| mntdevops | I know ansible-container is very new, but just wanted to say that it's promising in my usage | 15:15 |
| mntdevops | and #2 isnt' a dig | 15:15 |
| mntdevops | it's just that building the images slows down the initial getting started quite a bit | 15:16 |
| rhallisey | mntdevops, it was considered. I think it requires some major changed on kolla's end | 15:16 |
| *** fragatina has joined #openstack-kolla | 15:16 | |
| *** fragatina has quit IRC | 15:16 | |
| rhallisey | mntdevops, #2 kolla pushes per release | 15:16 |
| mntdevops | rhallisey, yeah, I looked at all the nested loops in the jinja templates for the Dockerfiles | 15:16 |
| sdake_ | sbezverk can you lend me a hand with neutron | 15:16 |
| rhallisey | mntdevops, it's possible not all releases are there atm | 15:17 |
| mntdevops | and thought, "man, this would be great with roles/tags" | 15:17 |
| *** fragatina has joined #openstack-kolla | 15:17 | |
| mntdevops | right | 15:17 |
| mntdevops | for instance, centos binary 2.0.3 | 15:17 |
| mntdevops | stable/mitaka references 2.0.3 IIRC in the example globals.yml | 15:18 |
| sdake_ | 2..0.3 is not yet releaed, hence we hven't pushed it | 15:18 |
| sdake_ | we odnt push per commit | 15:18 |
| *** senk_ has quit IRC | 15:19 | |
| mntdevops | ah | 15:19 |
| sdake_ | re ansiblec-ontiner, jeffrey4l ha a spec in the queue for that concept | 15:19 |
| sdake_ | but atm its too late to do any sort of that rework for mitaka | 15:19 |
| mntdevops | certainly understand that. The whole build process would get overhauled there | 15:20 |
| sdake_ | the fact that stablle/mitaka refrences 2.0.3 is to deal with a bug in pbr | 15:20 |
| sdake_ | or a bug in how we use pbr | 15:20 |
| sdake_ | not sure which | 15:20 |
| Jeffrey4l | it is not a bug. it is how the pbr works. | 15:21 |
| Jeffrey4l | we are using it in a wrong. because the ansible playbook require that. | 15:21 |
| *** jrist has quit IRC | 15:22 | |
| sdake_ | ya - what daviey had proposed i think was to use the full naem inclding dev-xx | 15:22 |
| sdake_ | then we could make it work properly in all conditions | 15:22 |
| Jeffrey4l | yep. using devxx is fine. | 15:22 |
| *** jrist has joined #openstack-kolla | 15:23 | |
| *** srwilkers has quit IRC | 15:24 | |
| sdake_ | mntdevops read throug hyur blog looks pretty good | 15:28 |
| sdake_ | note we are working on a kolla-ost playbook which is jut about done 9sean-k-money actually) which instals the necessary host dependencies | 15:28 |
| *** haplo37__ has joined #openstack-kolla | 15:29 | |
| *** mark-casey has joined #openstack-kolla | 15:32 | |
| *** senk_ has joined #openstack-kolla | 15:32 | |
| *** srwilkers has joined #openstack-kolla | 15:34 | |
| dims | sdake_ : rhallisey : inc0 : seen this? https://github.com/sapcc/openstack-kube | 15:37 |
| sdake_ | dims yes | 15:38 |
| dims | cool | 15:38 |
| kfox1111 | so many parallel implementations. :/ | 15:38 |
| *** senk_ has quit IRC | 15:39 | |
| kfox1111 | interesting they use kolla containers though. | 15:39 |
| kfox1111 | has anyone talked with them about joining forces? | 15:40 |
| kfox1111 | seems like they implemented it for their production system, then just released it. | 15:40 |
| kfox1111 | they may not be so interesting in carrying it forever if thats the case. | 15:40 |
| sdake_ | need to switch networks brb | 15:43 |
| *** sdake has joined #openstack-kolla | 15:45 | |
| *** b_bezak has joined #openstack-kolla | 15:45 | |
| *** zhubingbing__ has quit IRC | 15:46 | |
| *** zhubingbing_ has quit IRC | 15:46 | |
| *** sdake_ has quit IRC | 15:48 | |
| *** matrohon has quit IRC | 15:50 | |
| berendt | sdake, dims : how takes care of the SAP fork? wirehead_ is already in contact with the guys responsible for it. I think this should be coordinated. SAP has an OpenStack evangelist reponsible for such topics. | 15:51 |
| *** b_bezak has quit IRC | 15:51 | |
| berendt | The fork is initiated from a project on a SAP site in Berlin. | 15:52 |
| *** b_bezak has joined #openstack-kolla | 15:52 | |
| berendt | There was also a talk proposal for the summit in Barcelona presenting this environment. | 15:52 |
| berendt | At the moment they do violate the used license. sapcc/openstack-kube includes Kolla code and they do not use the Apache2 license there. This really sucks. | 15:53 |
| kfox1111 | fascinating.... one of the kube folks recommended using a side car container to do configmap templating instead of doing it in kube itself. | 15:53 |
| rhallisey | dims, interesting | 15:54 |
| kfox1111 | that could totally work to assembe a config map and a secret into a config a kolla container could accept. | 15:54 |
| rhallisey | kfox1111, they have a summit talk | 15:54 |
| kfox1111 | we just need a jinja container now... :) | 15:55 |
| berendt | The talk is "Baremetal to Openstack on Kubernetes from 0 to 100", proposed by Michael Schmidt | 15:55 |
| *** b_bezak has quit IRC | 15:56 | |
| *** egonzalez has joined #openstack-kolla | 15:59 | |
| *** inc0 has joined #openstack-kolla | 16:00 | |
| berendt | meeting time? | 16:01 |
| inc0 | yup | 16:01 |
| *** vhosakot_ has quit IRC | 16:01 | |
| *** vhosakot has joined #openstack-kolla | 16:03 | |
| Jeffrey4l | berendt, any feedback? https://review.openstack.org/348432 | 16:03 |
| berendt | Jeffrey4l will check after the meeting | 16:04 |
| Jeffrey4l | np thanks. | 16:04 |
| *** diogogmt has joined #openstack-kolla | 16:07 | |
| *** britthouser has joined #openstack-kolla | 16:08 | |
| *** egonzalez has quit IRC | 16:08 | |
| *** egonzalez90 has joined #openstack-kolla | 16:11 | |
| *** ppowell has quit IRC | 16:12 | |
| *** janki has joined #openstack-kolla | 16:15 | |
| dims | berendt : rhallisey : no clue who takes care of that repo. i saw it float by (openstack SV?) | 16:19 |
| berendt | I want to discuss this in the meeting | 16:19 |
| *** rmart04 has quit IRC | 16:19 | |
| *** unicell1 has joined #openstack-kolla | 16:21 | |
| *** unicell has quit IRC | 16:23 | |
| *** vhosakot has quit IRC | 16:26 | |
| *** vhosakot has joined #openstack-kolla | 16:26 | |
| *** dave-mccowan has joined #openstack-kolla | 16:30 | |
| pbourke | inc0: ceph is there so cinder should... just work? | 16:31 |
| inc0 | yeah | 16:31 |
| inc0 | in fact if we just run deploy -t cinder now | 16:31 |
| inc0 | it should just work | 16:31 |
| inc0 | without repave | 16:31 |
| pbourke | doing | 16:31 |
| *** janki has quit IRC | 16:34 | |
| openstackgerrit | sean mooney proposed openstack/kolla: add baremetal role to install kolla deps. https://review.openstack.org/325631 | 16:34 |
| openstackgerrit | sean mooney proposed openstack/kolla: This change allow setting kernel commandline args https://review.openstack.org/351412 | 16:34 |
| *** janki has joined #openstack-kolla | 16:35 | |
| *** daneyon_ has joined #openstack-kolla | 16:37 | |
| pbourke | inc0: 20 mins may be great deploy time but sucks waiting for one service | 16:39 |
| pbourke | inc0: want to kill and rerun with --tags? | 16:39 |
| inc0 | pbourke, let's just let it go | 16:40 |
| pbourke | kk | 16:40 |
| inc0 | it's on nova alreadt | 16:41 |
| *** daneyon_ has quit IRC | 16:41 | |
| *** unicell1 has quit IRC | 16:42 | |
| berendt | Jeffrey4l done | 16:46 |
| Jeffrey4l | thanks. | 16:47 |
| inc0 | lol nyancurses-cat ;) | 16:51 |
| *** rmart04 has joined #openstack-kolla | 16:51 | |
| pbourke | look at him go | 16:53 |
| *** msimonin has quit IRC | 16:59 | |
| inc0 | pbourke, kibana is up | 16:59 |
| pbourke | http://172.21.8.200:5601/ ? | 17:00 |
| coolsvap | pbourke: I have got a list of scenarios prepared I will add the steps for rally | 17:00 |
| pbourke | coolsvap: thanks! | 17:00 |
| pbourke | inc0: above url not working for me | 17:00 |
| inc0 | duh | 17:00 |
| Jeffrey4l | inc0, could u allocate some time for the osic? | 17:01 |
| *** ppowell has joined #openstack-kolla | 17:01 | |
| inc0 | Jeffrey4l, you mean work on it?:) | 17:01 |
| Jeffrey4l | inc0, correct. | 17:01 |
| inc0 | I'm doing it as we speak:) | 17:01 |
| inc0 | but will have more time next week | 17:01 |
| Jeffrey4l | i am in Asia. When the osic is free, we can use based on the geo. | 17:02 |
| coolsvap | guys I need to give some rest to my (wounded) hand, or I will not be able to work tomorrow :/ | 17:02 |
| coolsvap | Jeffrey4l: we can work together | 17:03 |
| pbourke | inc0: should reconfigure respect --tags ? | 17:03 |
| inc0 | coolsvap, Jeffrey4l sure guys, take a rest | 17:03 |
| Jeffrey4l | np. | 17:03 |
| inc0 | pbourke, yes | 17:03 |
| Daviey | Jeffrey4l: can you tak another look at my standalone example of what i was talking about? | 17:03 |
| pbourke | seems its not | 17:03 |
| Jeffrey4l | pbourke, it respect. | 17:03 |
| Jeffrey4l | Daviey, Ok | 17:03 |
| Daviey | Jeffrey4l: I think the standalone example makes it pretty clear what i am intending | 17:03 |
| inc0 | pbourke, maybe it's good thing tbh, as lots might have to change after what we did:) | 17:04 |
| Jeffrey4l | Daviey, could you paste the link again? I lost it. | 17:04 |
| Jeffrey4l | pbourke, what happen? logs? | 17:04 |
| Jeffrey4l | it should. if it not, it is buggy. | 17:04 |
| pbourke | inc0: have to run now, will catch up tomorrow. Might be good to throw whatever kibana queries you end up using to diagnose cinder into the review doc. All info helps | 17:05 |
| Daviey | Jeffrey4l: see the two scenarios in README, https://github.com/Daviey/tmp-kolla-version | 17:05 |
| inc0 | pbourke, I'm on it, thanks for help | 17:05 |
| Daviey | Jeffrey4l: one scenario is "trust upstream and use the latest version number", the other is defining what version number people want hardcoded) | 17:05 |
| pbourke | inc0: good luck! | 17:05 |
| berendt | 2 +2 on https://review.openstack.org/#/c/306928/.. lets merge? | 17:05 |
| patchbot | berendt: patch 306928 - kolla - Replace horizon default config with custom config | 17:05 |
| *** ayoung has quit IRC | 17:06 | |
| Jeffrey4l | Daviey, i think it is fine. please push a PS. | 17:08 |
| *** dmueller-evoila has quit IRC | 17:08 | |
| Jeffrey4l | and use 'latest' as the openstack_release value. | 17:08 |
| Jeffrey4l | in all the branch. | 17:08 |
| Jeffrey4l | also need change the ./tools/build.py --version logical, to use the pbr.release_version rather than the cached_version | 17:09 |
| Jeffrey4l | Daviey, ^ | 17:09 |
| duonghq | Jeffrey4l: iirc somebody mention 'current' as the openstack_release value | 17:10 |
| *** rmart04 has quit IRC | 17:10 | |
| duonghq | so 'latest' and 'current' is same or 2 meanings? | 17:10 |
| *** berendt has quit IRC | 17:11 | |
| Jeffrey4l | duonghq, hmm. no clue. nova api microversion use the word ` latest`, and i like this word. | 17:11 |
| Jeffrey4l | anyway, it is not a big deal. | 17:11 |
| duonghq | okay, just need final decision | 17:11 |
| Jeffrey4l | duonghq, Daviey please push a PS to fix this based on your idea. :) | 17:12 |
| duonghq | ah, anybody deploy kolla successful with Swift enable? | 17:12 |
| sean-k-mooney | Jeffrey4l: regarding https://review.openstack.org/#/c/352089 is there anyone working on enabling ubuntu 16.04 for the contaier os currently | 17:13 |
| patchbot | sean-k-mooney: patch 352089 - openstack-infra/project-config - Enable ubuntu-xenial for Kolla | 17:13 |
| Jeffrey4l | duonghq, I never deployed that. :( | 17:13 |
| Jeffrey4l | sean-k-mooney, i am. | 17:13 |
| Jeffrey4l | sean-k-mooney, https://review.openstack.org/329454 | 17:13 |
| Jeffrey4l | but it is block by the host os kernel version. | 17:13 |
| sean-k-mooney | Jeffrey4l: oh ok cool. | 17:13 |
| Daviey | Jeffrey4l: great, i'll try and do it tomorrow | 17:13 |
| Jeffrey4l | Daviey, thanks. | 17:13 |
| janki | hey, I could like to help with any bugs | 17:14 |
| Jeffrey4l | duonghq, do u find anything about the swift? | 17:14 |
| Daviey | Jeffrey4l: Incidentally, we probably need an early task to check that all images are available for required projects at the start of each run.. currently pretty crappy error message if we dont | 17:14 |
| Jeffrey4l | Daviey, pull action should work. | 17:15 |
| duonghq | Jeffrey4l: I got trouble in deploy config task, I'll try to fix it tomorrrow, just wonder if it only happen to me. | 17:15 |
| Jeffrey4l | janki, pick one from the bug list https://launchpad.net/kolla/+milestone/newton-3 | 17:15 |
| Daviey | Jeffrey4l: hmm | 17:15 |
| Jeffrey4l | we have lost of bugs. | 17:15 |
| Jeffrey4l | duonghq, cool | 17:16 |
| duonghq | It seem that automation testing for Ansible is really hard | 17:16 |
| janki | Jeffrey4l: sure, will do. Thanks | 17:17 |
| Jeffrey4l | duonghq, it is. official do not have such solution. | 17:17 |
| duonghq | yeah | 17:18 |
| *** inc0 has quit IRC | 17:19 | |
| *** ayoung has joined #openstack-kolla | 17:20 | |
| *** unicell has joined #openstack-kolla | 17:22 | |
| *** lrensing_ has joined #openstack-kolla | 17:26 | |
| *** harlowja has joined #openstack-kolla | 17:27 | |
| *** dwalsh_ has joined #openstack-kolla | 17:29 | |
| *** duonghq has quit IRC | 17:30 | |
| *** dwalsh has quit IRC | 17:31 | |
| *** dwalsh_ has quit IRC | 17:35 | |
| sbezverk | Jeffrey4l: ping | 17:35 |
| Jeffrey4l | sbezverk, pong | 17:35 |
| Jeffrey4l | sbezverk, need talk about this https://review.openstack.org/351784 | 17:35 |
| Jeffrey4l | sorry for the delay. | 17:36 |
| *** janki has quit IRC | 17:36 | |
| sbezverk | Jeffrey4l: sure when I have rabbitmq and neutron up I will come back to it | 17:37 |
| Jeffrey4l | ok. | 17:37 |
| sbezverk | Jeffrey4l: I wanted to check with you about vcpus thing which keeps breaking rendering on kubernetes side | 17:37 |
| sbezverk | I pinged you about that few days back | 17:38 |
| Jeffrey4l | sorry. not always online. | 17:38 |
| Jeffrey4l | so what' the detail? | 17:38 |
| Jeffrey4l | i saw you fix patch. | 17:39 |
| sbezverk | this line causes issues on kube side | 17:39 |
| sbezverk | openstack_service_workers: "{{ [ansible_processor_vcpus, 5]|min if orchestration_engine == 'ANSIBLE' else '1'}}" | 17:39 |
| Jeffrey4l | yes. But your fix should fix it, right? sbezverk | 17:40 |
| sbezverk | it is in all.yml, we have to comment it out to be able to use kolla-kubernetes cli | 17:40 |
| sbezverk | nope, I do not have fix for this | 17:40 |
| Jeffrey4l | does this break the k8s now? | 17:41 |
| *** egonzalez90 has quit IRC | 17:41 | |
| sbezverk | Jeffrey4l: yes | 17:41 |
| sbezverk | the workaround is easy but annoying | 17:41 |
| Jeffrey4l | what's the workaround? | 17:42 |
| mntdevops | sdake, thanks for looking over my blog. Hope to continue working on it as time permits | 17:42 |
| Jeffrey4l | i thought this PS fix it https://review.openstack.org/#/c/350356/ | 17:42 |
| patchbot | Jeffrey4l: patch 350356 - kolla - Change workers to be static when using kubernates (MERGED) | 17:42 |
| sbezverk | Jeffrey4l: Commenting this line | 17:43 |
| Jeffrey4l | OK. | 17:43 |
| *** inc0 has joined #openstack-kolla | 17:43 | |
| sbezverk | hmm, really strange I hit that issue yesterday | 17:43 |
| sbezverk | let me run more tests | 17:44 |
| *** senk_ has joined #openstack-kolla | 17:44 | |
| *** senk_ has quit IRC | 17:45 | |
| Jeffrey4l | sbezverk, OK. | 17:45 |
| *** dwalsh_ has joined #openstack-kolla | 17:46 | |
| *** egonzalez90 has joined #openstack-kolla | 17:47 | |
| inc0 | ok, I know what's wrong with cinder | 17:50 |
| Mech422 | Anyone know what I should be googling if I want to control Openstack 'regions'(?) in multiple DCs from a single control plain? Is that cells ? availability zones ? | 17:51 |
| inc0 | volume_type cannot be None | 17:51 |
| inc0 | didn't we fix this one? | 17:51 |
| Mech422 | inc0: it used to be defined to a non-existant volume type | 17:51 |
| kfox1111 | regions are a totally seperate thing from the rest of those. | 17:51 |
| inc0 | Mech422, so regions is single keystone, multiple novas and such | 17:51 |
| kfox1111 | regions assume a seperate control plane. | 17:51 |
| inc0 | cells are single everything besides nova, but they don't work very well as of now | 17:51 |
| Mech422 | inc0: sounds like they the definition of it got removed, but not the usage ? | 17:51 |
| jrich523 | where do the horizon logs end up? from what i can tell all the logs should be /var/log/kolla, but that folder isnt even there | 17:51 |
| kfox1111 | you probably want either host aggregates, az's or cells. | 17:52 |
| kfox1111 | though, cells really are a seperate control plane... | 17:52 |
| inc0 | jrich523, try /var/lib/docker/volumes/kolla_logs/_data | 17:52 |
| jrich523 | thanks | 17:52 |
| inc0 | kfox1111, do you know anyone running cells with neutron tho? | 17:52 |
| kfox1111 | neutron doesn't support them. you have to share a neutron with cells as far as I know. | 17:53 |
| Mech422 | kfox1111: how about storage? can I put a local (small) ceph cluster in each DC and have the DC use that one to prevent long net traversals for i/o ? | 17:53 |
| kfox1111 | cells are just a way to scale out nova a bit further. | 17:53 |
| kfox1111 | as far as I know, no. | 17:53 |
| Mech422 | I think I want something like the eucalyptus 'cluster' idiom... | 17:53 |
| kfox1111 | regions really were designed around datacenters. | 17:54 |
| kfox1111 | each datacenter getting a region. | 17:54 |
| kfox1111 | totally seperate. | 17:54 |
| kfox1111 | cause often you loose a datacenter at a time. | 17:54 |
| Mech422 | kfox1111: yeah - that what I thought, but boss wants to know if there's any 'single mgmt' options.. | 17:54 |
| inc0 | "often you lose datacenter at a time"...uhh | 17:54 |
| Mech422 | kfox1111, inc0 : thanks - I'll keep googling | 17:55 |
| inc0 | how often is "often"?;) | 17:55 |
| kfox1111 | sure. just depends on how much pain you want to take apon yourself. :) | 17:55 |
| inc0 | Mech422, so regions are kinda single mgmt | 17:55 |
| inc0 | tho | 17:55 |
| kfox1111 | inc0: had an eagle drop a rabbit on a power transformer the other week. ;) | 17:55 |
| inc0 | as you have single ks | 17:55 |
| inc0 | and ks is managing endpoints well | 17:55 |
| kfox1111 | you'd think the power folks would learn by now to put up some plastic shields... :) | 17:55 |
| inc0 | so for end user it's pretty convenient | 17:55 |
| inc0 | lol | 17:56 |
| kfox1111 | sharding keystone in such a way that its multidata center resilient is fairly easy. | 17:56 |
| wirehead_ | As far as I can tell, based on swapping stories with my mom who programmed mainframes, when they made the first mainframe, they also invented the truck-magnet. Because data centers have been taken out by trucks since there were datacenters. | 17:56 |
| kfox1111 | its harder for the other services. | 17:56 |
| kfox1111 | just use fernet tokens and make sure you have one mariadb replica at each of 3+ dc's. | 17:56 |
| wirehead_ | Um, ask the rackers about the great data center truck incident, inc0. :D | 17:56 |
| inc0 | will do wirehead_ | 17:57 |
| * kfox1111 chuckles | 17:57 | |
| kfox1111 | what can go wrong, will go wrong. :) | 17:57 |
| *** berendt has joined #openstack-kolla | 17:57 | |
| inc0 | I have one story about the fastest DC migration in Polish history | 17:57 |
| inc0 | less than 2 weeks | 17:57 |
| Mech422 | sounds like regions is the place to start then...I'll dig into them more | 17:57 |
| kfox1111 | so, yeah. you can span a region across multiple data centers.... even across the world. I just don't recommend it. | 17:57 |
| inc0 | flood in old dc makes you move quicker | 17:57 |
| wirehead_ | There’s ALWAYS a t-junction adjacent to a datacenter and trucks ALWAYS manage to miss that it’s a T-junction. And then you put up bollards and the trucks knock the bollards over. | 17:57 |
| *** daneyon_ has joined #openstack-kolla | 17:57 | |
| Mech422 | couple of years ago, when the gpu compute craze hit - we actually lost an air handler and setoff the sprinklers in a DC with all the gpu's we had packed in there | 17:58 |
| inc0 | and yes, servers were prioritized based on rack position - lower ones went first;) | 17:58 |
| wirehead_ | And, frankly, infra makes that kinda easy. If the control plane for data center A lives in data center A and it all goes down, no harm no fowl. | 17:59 |
| kfox1111 | we had an incident where some rear door chillers had a pipe bust. | 17:59 |
| kfox1111 | hosed down the datacenter. :/ | 17:59 |
| kfox1111 | was on the top of the rack, so it flew good, and was a water/glycol mix, so not only wet, but sticky. :/ | 18:00 |
| *** ad_rien_ has joined #openstack-kolla | 18:01 | |
| *** daneyon has quit IRC | 18:01 | |
| Mech422 | kfox1111: yeah - I'm actually surprised we had water based fire suppression...but the story goes we had rusty standpipe water flying out the sprinklers and down all the racks | 18:02 |
| *** senk_ has joined #openstack-kolla | 18:03 | |
| kfox1111 | us too. they value employees over machines, which is good. :) | 18:04 |
| Mech422 | kfox1111: water + elect doesn't sound like they value much of anything to me.... | 18:05 |
| Mech422 | except maybe BBQ ? | 18:05 |
| *** berendt has quit IRC | 18:05 | |
| sean-k-mooney | halon is not that friendly to machines either but they react much better then humans | 18:05 |
| kfox1111 | yeah, but you can breath long enough to escape more often. :) | 18:05 |
| *** rmart04 has joined #openstack-kolla | 18:05 | |
| Mech422 | kfox1111: "cake or death" ? "suffocate or electrocute" ? | 18:06 |
| *** huikang has joined #openstack-kolla | 18:07 | |
| kfox1111 | either way your having a bad day. :) | 18:07 |
| *** fragatina has quit IRC | 18:07 | |
| Mech422 | sean-k-mooney: is there much halon left? I thought they only had a 5 year grace to replace them when they were banned ? | 18:07 |
| sean-k-mooney | i would guess most halon has been replaced with co2 or other gas based suppresion systmes. the ozone layer like halon even less then humans | 18:08 |
| *** ad_rien_ has quit IRC | 18:09 | |
| Mech422 | sean-k-mooney: yeah - I know it got banned back in the 90s ... I worked on wall st. in a halon 'protected' facility back then | 18:09 |
| *** jemcevoy has joined #openstack-kolla | 18:10 | |
| Mech422 | same place had motion sensors inside the DC to unlock crash bars... | 18:10 |
| sean-k-mooney | i mean if it wasnt for the increased uv form the impact to the ozone layer we would probably still be using it. it worked very well | 18:10 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fix bad merge of ceph mariadb https://review.openstack.org/353669 | 18:10 |
| Mech422 | so you could get in the DC by putting a piece of paper under the door, and flicking it with a finger so it tripped the motion sensor | 18:11 |
| *** daneyon has joined #openstack-kolla | 18:11 | |
| sean-k-mooney | Mech422: feature or bug :) | 18:11 |
| Mech422 | sean-k-mooney: heh - depends on who you ask...the boss wasn't very happy about his 'secure' DC being breached with a piece of photo copy paper | 18:12 |
| *** berendt has joined #openstack-kolla | 18:12 | |
| *** berendt has quit IRC | 18:13 | |
| jemcevoy | sdake, I am still stuck when ceph creates osd's on stable/mitaka. Is there anyone who can help me past this? I errors I am seeing are here: http://paste.openstack.org/show/553622/ | 18:13 |
| *** daneyon_ has quit IRC | 18:14 | |
| *** rmart04 has quit IRC | 18:14 | |
| *** senk_ has quit IRC | 18:14 | |
| *** ad_rien_ has joined #openstack-kolla | 18:14 | |
| Mech422 | jemcevoy: your using the same device for both a full disk OSD data partition AND its journal ? | 18:15 |
| *** ad_rien_ has quit IRC | 18:15 | |
| Mech422 | jemcevoy: if your doing full disk ceph setup, IIRC kolla does the journal setup for you | 18:15 |
| Mech422 | jemcevoy: (also, I'm not sure stable/mitaka supports partition based osd data for ceph - it _think_ its full disk only?) | 18:17 |
| sean-k-mooney | mitaka should support external journals | 18:19 |
| sean-k-mooney | sams blog that doumented it was deploying mitaka | 18:19 |
| Mech422 | sean-k-mooney: oh? cool ... | 18:20 |
| Mech422 | sean-k-mooney: ahh - I should did that up for jemcevoy - might help him | 18:20 |
| sean-k-mooney | jemcevoy: this might help https://yaple.net/2016/03/20/deploying-openstack-mitaka-with-kolla-docker-and-ansible/ | 18:21 |
| dcwangmit01 | rhallisey: Hey yo ryan | 18:22 |
| rhallisey | dcwangmit01, hey! | 18:22 |
| dcwangmit01 | Are the ubuntu-xenial gate jobs new? gate-kolla-kubernetes-docs-ubuntu-xenial | 18:22 |
| dcwangmit01 | http://status.openstack.org/zuul/ | 18:22 |
| sbezverk | Jeffrey4l: jinja2.exceptions.UndefinedError: 'ansible_processor_vcpus' is undefined and it is on latest kolla | 18:23 |
| dcwangmit01 | Do we have control over those, or are they created by the openstack infra team? | 18:23 |
| sean-k-mooney | dcwangmit01: the upstream gate in other projects started swaping over in june | 18:23 |
| rhallisey | dcwangmit01, they've been there for a little. I have to fix them | 18:23 |
| sean-k-mooney | dcwangmit01: Jeffrey4l has a review open to enable them for kolla | 18:23 |
| jemcevoy | I do not have any SDDs. Just some old lab systems with 4 - 1TB disks... I want to use three disks for ceph and the other for docker... | 18:24 |
| rhallisey | dcwangmit01, I got them from infra, but they script they run is wip | 18:24 |
| Mech422 | pbourke: that reminds me - we need to decide what to do about the ceph-partiton/sgdisk patch - right now its hard-coded to use /usr/sbin/sgdisk, but ubuntu puts it in /sbin/sgdisk - sdace doesn't want to use shell path to locate the bin, and asked me to talk to you about it when you got back from PTO | 18:24 |
| sean-k-mooney | dcwangmit01: https://review.openstack.org/#/c/352089/ | 18:24 |
| patchbot | sean-k-mooney: patch 352089 - openstack-infra/project-config - Enable ubuntu-xenial for Kolla | 18:24 |
| dcwangmit01 | sean-k-mooney: rhallisey: So I queried the openstack-infra team yesterday about why the ubuntu-xenial jobs are queuing for 6 hours, as opposed to the jobs for the other platforms. Apparently, there are limited nodes ofr ubuntu-xenial. I don't remember them ever being there before. Is there a way to run those jobs on another platform like they used to be? | 18:25 |
| *** daneyon_ has joined #openstack-kolla | 18:25 | |
| sean-k-mooney | dcwangmit01: oh wait sorry i missed the kubernetes in the change i linked was for kolla-ansible | 18:25 |
| dcwangmit01 | sean-k-mooney: I'm not sure you want to enable ubuntu-xenial. It'll add 6 hours to your test pipeline | 18:26 |
| sean-k-mooney | they enabled them by default for some projects | 18:26 |
| rhallisey | hm | 18:26 |
| sean-k-mooney | networking-ovs-dpdk swaped over without me changing it too | 18:26 |
| dcwangmit01 | All I know is that once ubuntu-xenial appeared, our kolla-kube test pipeline is suddently taking hours and hours. | 18:27 |
| dcwangmit01 | I suppose this is fine, since it all eventually goes through | 18:28 |
| dcwangmit01 | But it is a huge pain to wait hours, only to find out there is a merge conflict | 18:28 |
| rhallisey | dcwangmit01, weird. I didn't ask for those specifcally | 18:28 |
| *** egonzalez90 has quit IRC | 18:28 | |
| rhallisey | jsut a subset of what kolla has | 18:28 |
| *** egonzalez90 has joined #openstack-kolla | 18:29 | |
| *** egonzalez90 has quit IRC | 18:29 | |
| dcwangmit01 | rhallisey: Just check out any kolla or kolla-kube job in http://status.openstack.org/zuul/ | 18:29 |
| *** daneyon_ has quit IRC | 18:30 | |
| *** ayoung has quit IRC | 18:31 | |
| rhallisey | the pep8 ubuntu-xenial | 18:31 |
| sean-k-mooney | rhallisey: i think the swapped over all the python-jobs to use xenial on master | 18:32 |
| rhallisey | I think so too | 18:32 |
| jemcevoy | sean-k-mooney, Did I have to configure one of the disks for journals and cache? I do not understand how to label the disks so they will do an OSD per disk? I tried not creating a single partition and that did not work either... | 18:32 |
| rhallisey | dcwangmit01, I'm not sure what we can do on our end | 18:32 |
| dcwangmit01 | rhallisey: That's okay. I know they are working on adding more nodes in openstac-infra | 18:33 |
| rhallisey | ya we'll just have to be patient | 18:33 |
| jemcevoy | Mech422 what does parted <dev> print look like for an OSD and Journal to be co resident on a disk? | 18:33 |
| *** lrensing_ has quit IRC | 18:33 | |
| sean-k-mooney | jemcevoy: if you dont want to use external journals just run parted $DISK -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1 | 18:34 |
| sean-k-mooney | where disk is /dev/sdX | 18:34 |
| *** shardy is now known as shardy_afk | 18:35 | |
| jemcevoy | sean-k-mooney, I was using the label KOLLA_CEPH_OSD_BOOTSTRAP_1 ,2,3... Do I just label all 9 disks the same KOLLA_CEPH_OSD_BOOTSTRAP | 18:36 |
| sean-k-mooney | jemcevoy: yes if you are not using external journals | 18:36 |
| *** ad_rien_ has joined #openstack-kolla | 18:36 | |
| sean-k-mooney | kolla will enumerate the disks when its bootstrapping. | 18:37 |
| Mech422 | jemcevoy: I run master with ceph co-located on my boot disks... I'm a very bad man | 18:37 |
| sean-k-mooney | Mech422: :) i see no issue with this | 18:38 |
| jrich523 | sorry... silly question.. how do i login to horizon? is there a default username/password? | 18:39 |
| jemcevoy | I am looking into getting some nvme sdd's for the production servers for the expansion to pod.penguincomputing.com | 18:39 |
| sean-k-mooney | Mech422: honestly though i thinkits valid to at lease share os and journals on the same ssd | 18:39 |
| Mech422 | sean-k-mooney: :-) We're just Rebels :-) | 18:39 |
| sean-k-mooney | jrich523: yes you can find it in the passwords.tml | 18:39 |
| jemcevoy | btw. the expansion is falsom to mitaka or newton... Adding like 4000 cores | 18:40 |
| sean-k-mooney | passwords.yml | 18:40 |
| jrich523 | ok thats what i thought, but those are encrypted, so how do i use them? | 18:40 |
| Mech422 | jrich523: actually - they're not - the passwords ARE uuids | 18:40 |
| Mech422 | jrich523: just copy and past them | 18:40 |
| sean-k-mooney | they are not encrypted they are actuly that long string of charaters | 18:40 |
| jrich523 | ... oh... lol | 18:40 |
| jrich523 | super | 18:40 |
| *** ad_rien_ has quit IRC | 18:42 | |
| jrich523 | what are the default username and domain? | 18:42 |
| sean-k-mooney | domain i think is default and username is admin | 18:42 |
| jrich523 | hmm nope, that didnt fly | 18:43 |
| *** fragatina has joined #openstack-kolla | 18:44 | |
| sean-k-mooney | did you run the post-deploy playbook | 18:44 |
| sean-k-mooney | that will give you an openrc file with the correct values | 18:45 |
| jrich523 | i believe i did that | 18:45 |
| sean-k-mooney | if you look in /etc/kolla that should have created an admin-openrc.sh | 18:46 |
| jrich523 | however based on the docs, it says the openrc file should end up in /etc/kolla, which i do not see, so, maybe i did that on a different attempt | 18:46 |
| jrich523 | yeah | 18:46 |
| jrich523 | dont see that, let me run the post deploy | 18:46 |
| sean-k-mooney | this is what mine looks like | 18:48 |
| sean-k-mooney | export OS_PROJECT_DOMAIN_ID=default | 18:48 |
| sean-k-mooney | export OS_USER_DOMAIN_ID=default | 18:48 |
| sean-k-mooney | export OS_PROJECT_NAME=admin | 18:48 |
| sean-k-mooney | export OS_TENANT_NAME=admin | 18:48 |
| sean-k-mooney | export OS_USERNAME=admin | 18:48 |
| sean-k-mooney | export OS_PASSWORD=<removed> | 18:48 |
| sean-k-mooney | export OS_AUTH_URL=http://10.10.10.254:35357/v3 | 18:48 |
| sean-k-mooney | export OS_IDENTITY_API_VERSION=3 | 18:48 |
| sean-k-mooney | so by default your domain should be "default" and user "admin" | 18:49 |
| jrich523 | :D | 18:50 |
| jrich523 | i actually got it up! | 18:50 |
| jrich523 | and by "i" i mean that zhugaoxiao, sdake, and sean-k-mooney got it up! :) | 18:50 |
| jrich523 | really appreciate the hand holding guys | 18:51 |
| *** inc0 has quit IRC | 18:51 | |
| sean-k-mooney | jrich523: http://i.imgur.com/yOCM63a.png | 18:52 |
| sean-k-mooney | jrich523: glad we could help | 18:52 |
| *** ad_rien_ has joined #openstack-kolla | 18:52 | |
| *** ad_rien_ has quit IRC | 18:55 | |
| *** sean-k-mooney is now known as sean-k-mooneyAFK | 18:59 | |
| *** ad_rien_ has joined #openstack-kolla | 19:05 | |
| sean-k-mooneyAFK | sdake: im going to try and rebase the bifrost work this evening and add the rel note/custimisation. | 19:05 |
| sean-k-mooneyAFK | sdake: at that point that series should be ready to merge but i might make some changes later if required based on more testing. | 19:07 |
| sean-k-mooneyAFK | sdake: but first dinner. good evening everyone. | 19:09 |
| *** lrensing_ has joined #openstack-kolla | 19:12 | |
| *** ad_rien_ has quit IRC | 19:16 | |
| Daviey | How are you chaps using kolla with non-root logins? I've tried setting ansible_become=true in the inventory for each host.. no joy. | 19:26 |
| *** senk_ has joined #openstack-kolla | 19:29 | |
| *** mark-casey has quit IRC | 19:30 | |
| *** ppowell has quit IRC | 19:37 | |
| *** ayoung has joined #openstack-kolla | 19:44 | |
| *** JoseMello has joined #openstack-kolla | 19:49 | |
| jrich523 | sean-k-mooneyAFK lol i just got back from lunch, and i was watching archer while i ate lol | 19:50 |
| sdake | yo | 20:01 |
| sdake | folks using the osic cluster I dont think anyone ever answered my quetions how about to ssh into th vms | 20:02 |
| *** senk_ has quit IRC | 20:02 | |
| *** mark-casey has joined #openstack-kolla | 20:06 | |
| *** haplo37__ has quit IRC | 20:07 | |
| *** srwilkers has quit IRC | 20:15 | |
| sdake | Daviey i use sudo | 20:22 |
| sdake | Daviey but i know it can be done without | 20:22 |
| sdake | Daviey but is undocumtend | 20:22 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet https://review.openstack.org/349366 | 20:24 |
| *** inc0 has joined #openstack-kolla | 20:24 | |
| *** kbaegis has joined #openstack-kolla | 20:28 | |
| kbaegis | Hi all | 20:28 |
| kbaegis | Having issues setting kolla up on gentoo | 20:28 |
| kbaegis | I know it's probably not a "supported" distro | 20:29 |
| mark-casey | kbaegis: https://i.imgflip.com/2/gjx1b.jpg | 20:29 |
| mark-casey | no I'm kidding | 20:29 |
| mark-casey | I'm not actually sure if anyone has tried that | 20:30 |
| *** msimonin has joined #openstack-kolla | 20:30 | |
| kbaegis | :) | 20:30 |
| kbaegis | Well I'll be the first then | 20:30 |
| inc0 | kbaegis, sorry about that:P | 20:30 |
| kbaegis | Keep getting "ERROR! Unexpected Exception: name 'unicode' is not defined" | 20:30 |
| inc0 | what's the issue> | 20:30 |
| inc0 | eee? | 20:30 |
| inc0 | python --version plz | 20:30 |
| *** shardy_afk has quit IRC | 20:31 | |
| mark-casey | kbaegis: I assume this is using it as the OS running kolla containers? not the OS in the container images? | 20:31 |
| kbaegis | Python 3.4.3 | 20:31 |
| sdake | inc0 i can't ssh into vms | 20:31 |
| kbaegis | Correct | 20:31 |
| kbaegis | :) | 20:31 |
| sdake | i want o ssh into vms to run bonnie within them | 20:31 |
| sdake | inc0 any ideas? | 20:31 |
| inc0 | sdake, you tried via namespace? | 20:31 |
| *** coolsvap has quit IRC | 20:31 | |
| sdake | no i tiried via the deploy box | 20:32 |
| kbaegis | So it's "NameError: name 'unicode' is not defined" from ansible/inventory/__init__.py", line 736 | 20:32 |
| inc0 | ok, jon me on tmux | 20:32 |
| sdake | i have meeting shortly | 20:32 |
| inc0 | you wont be able to ssh from deployment | 20:32 |
| sdake | i see | 20:32 |
| kbaegis | And that'd be ansible/cli/playbook.py", line 154 | 20:32 |
| sdake | i cant join you on tmux atm - have meeting shorty | 20:33 |
| sdake | when i gt wrapped up i'll ping ou | 20:33 |
| inc0 | I'll leave ssh open in tmux | 20:33 |
| inc0 | I'll have more meetings myself so just look at history what I did | 20:34 |
| kbaegis | traceback: http://pastebin.com/k8RKTPLV | 20:34 |
| *** Jeffrey4l has quit IRC | 20:34 | |
| inc0 | sdake, on tmux 3 you'll be able to ssh | 20:35 |
| *** Jeffrey4l has joined #openstack-kolla | 20:35 | |
| kbaegis | I definitely have a config at /etc/kolla/globals.yml | 20:36 |
| mark-casey | kbaegis: what version of ansible you have installed? | 20:36 |
| inc0 | kbaegis, so let me put it this way....unicode is a buildin function | 20:36 |
| inc0 | to python | 20:36 |
| kbaegis | the latest from pip2/pip3 | 20:36 |
| inc0 | which makes me think that something's really wrong with your python | 20:36 |
| kbaegis | ansible 2.1.1.0 | 20:36 |
| mark-casey | inc0: kbaegis: this maybe? http://stackoverflow.com/a/36110612/6595787 | 20:37 |
| inc0 | oh? | 20:37 |
| mark-casey | I see ansible/cli stuff in the stacktrace. Ansible cli was a standalone/community thing recently added ansible official so perhaps that call to unicode needs to be renamed? | 20:38 |
| inc0 | so I guess ansible is no-bueno on python 3.4 | 20:38 |
| kbaegis | I tried python2, but it wasn't working either | 20:38 |
| sbezverk | Kolla cores, please check this https://review.openstack.org/#/c/336751/ The gate is super green :-) | 20:38 |
| patchbot | sbezverk: patch 336751 - kolla - Fix intf address for RabbitMQ and disable cluster ... | 20:38 |
| inc0 | sbezverk, ooo even ubnutu-source | 20:38 |
| inc0 | we managed to get thins libvirt thing fixed? | 20:38 |
| inc0 | anyone can tell me what was the issue? | 20:39 |
| *** michauds has joined #openstack-kolla | 20:40 | |
| kbaegis | Alright, thanks guys. I'll look through my packages and whatnot | 20:41 |
| kbaegis | See if I can roll back or update | 20:41 |
| mark-casey | inc0: kbaegis: https://github.com/ansible/ansible/issues/1409#issuecomment-94266101 | 20:42 |
| inc0 | kbaegis, I suggest to use virtualen | 20:42 |
| inc0 | v | 20:42 |
| inc0 | it helps control version of python and python libs | 20:43 |
| kbaegis | Thanks | 20:43 |
| mark-casey | oh I didn't realize it could do version. Yea. perhaps a little learning curve there but better than hacking up system python | 20:43 |
| kbaegis | So I just need to roll back to python 3.3? | 20:47 |
| *** shardy_afk has joined #openstack-kolla | 20:48 | |
| kbaegis | I think I'll go that route | 20:48 |
| mark-casey | 2.6/2.7 I think. My read of that was no Python 3 support at all yet, but generally yes... roll back to any version they list as supported | 20:48 |
| kbaegis | Oh, I have python2.7 slotted | 20:48 |
| kbaegis | Yeah, that was having other issues | 20:48 |
| *** imcsk8 is now known as imcsk8|lunch | 20:50 | |
| inc0 | kbyrne, 2.7.x is the way to go | 20:50 |
| sdake | any python nerds out there | 20:50 |
| inc0 | I'm ex python nerd | 20:50 |
| inc0 | I moved to deployment of openstack | 20:50 |
| sdake | what is it giong to take to get us to python 3.5 exclusively | 20:50 |
| inc0 | why would we ever go to 3.5 exclusively? | 20:51 |
| inc0 | especially if, as you see in log above, ansible is no bueno in 3+ | 20:51 |
| sdake | https://review.openstack.org/#/c/349069/ | 20:51 |
| inc0 | so you'd need different python for kolla and ansible | 20:51 |
| kbaegis | Well, that's exactly the problem | 20:51 |
| patchbot | sdake: patch 349069 - governance - add ocata goal "support python 3.5" | 20:51 |
| inc0 | which is kinda...duh | 20:51 |
| inc0 | we can *support* it | 20:51 |
| inc0 | but not exclusively | 20:52 |
| kbaegis | python 2.7 and 3.4 are treated as separate languages by devs | 20:52 |
| inc0 | we can be compatible with both 2 and 3 | 20:52 |
| inc0 | yes and no | 20:52 |
| kbaegis | Yeah, but devs aren't | 20:52 |
| inc0 | there are libs, like six | 20:52 |
| sdake | inc0 read the proposal | 20:52 |
| inc0 | that allows you to write cross-compatible code | 20:52 |
| sdake | rather then just the title | 20:52 |
| *** ravig has joined #openstack-kolla | 20:53 | |
| inc0 | sdake, thing is...we'd say good bye to kolla-ansible | 20:53 |
| sdake | goals are a new thing for projects coming in cocata | 20:53 |
| sdake | you mean the anible implementation? | 20:53 |
| inc0 | I mean command | 20:54 |
| sdake | so we could rewrite that, no? | 20:54 |
| inc0 | no, we'd need to rewrite ansible | 20:54 |
| inc0 | ansible will not run on 3.5 | 20:54 |
| inc0 | and kolla-ansible calls ansible | 20:54 |
| sdake | ok thanks | 20:55 |
| kbaegis | I mean, the version split from 2-3 is a community wide python problem | 20:55 |
| kbaegis | Not just ansible | 20:55 |
| kbaegis | But that's OT | 20:55 |
| kbaegis | :) | 20:55 |
| inc0 | yeah, but for our particular case | 20:55 |
| inc0 | it's fubar | 20:55 |
| inc0 | and ansible folk is not a community that will be bound by these goals | 20:56 |
| inc0 | sdake, unless you ask your wonderful wife to make it happen | 20:56 |
| inc0 | I mean support for python3 in ansible | 20:56 |
| mark-casey | inc0: it looks like they see the writing on the wall, even if they don't like python 3 | 20:56 |
| mark-casey | I mean it may be a year and a half but it looks like it's coming | 20:57 |
| inc0 | yeah, if it comes, it's cool | 20:57 |
| inc0 | in fact, I think our code is python3 compliant even now | 20:57 |
| inc0 | but if it's not, we don't have THAT much of it | 20:57 |
| inc0 | it will be pretty fast move for us | 20:57 |
| inc0 | but...ansible. | 20:57 |
| inc0 | brb | 20:57 |
| *** shardy_afk has quit IRC | 20:58 | |
| kbaegis | Well, for the record I'd have greatly appreciated a note on http://docs.openstack.org/developer/kolla/quickstart.html stating that python 3.x is not necessarily compatible | 20:59 |
| kbaegis | Easy enough to switch over :) | 21:00 |
| sbezverk | kfox1111 | 21:00 |
| sbezverk | ping | 21:00 |
| mark-casey | kbaegis: I think we've just learned that too :D afaik most of the supported kolla platforms default to v2 | 21:01 |
| *** dwalsh_ has quit IRC | 21:02 | |
| inc0 | kbaegis, you can make this note:) you'll get free admission to summit for it;) | 21:02 |
| kbaegis | I guess it's either that or booth duty | 21:03 |
| inc0 | I don't think you'll get full pass on booth duty really | 21:03 |
| inc0 | ATC gives you full access to summit, not just markeplace | 21:03 |
| mark-casey | inc0: atc? | 21:03 |
| sdake | mark-casey - fwiw, project-wide goals are to be prioritized above all else, and to be finished prior to m1. | 21:03 |
| sdake | (of ocata) | 21:04 |
| inc0 | active technical contributor | 21:04 |
| inc0 | type of summit pass | 21:04 |
| inc0 | sdake, not much we can do about that | 21:05 |
| inc0 | we might want to talk with dhellman about it | 21:05 |
| inc0 | I'll ping him | 21:06 |
| sdake | i am taking cre of it relax | 21:07 |
| sdake | feel free to read my response to the review | 21:08 |
| *** daneyon_ has joined #openstack-kolla | 21:08 | |
| *** fragatin_ has joined #openstack-kolla | 21:08 | |
| *** msimonin has quit IRC | 21:09 | |
| mark-casey | sdake: for sure. I was only suggesting a reason that a python 3 note was not already in our docs | 21:10 |
| inc0 | sdake, btw ssh is ready for you | 21:10 |
| sdake | still waking up | 21:11 |
| sdake | give me 15 mins | 21:11 |
| sdake | pls | 21:11 |
| sdake | they had to call the crash cart in about 30 minutes ago | 21:11 |
| *** fragatina has quit IRC | 21:12 | |
| *** daneyon_ has quit IRC | 21:12 | |
| *** rmart04 has joined #openstack-kolla | 21:12 | |
| *** msimonin has joined #openstack-kolla | 21:15 | |
| kbaegis | Well, I've repaired my system. :) Looks like 2.7 is working for me | 21:16 |
| mark-casey | kbaegis: great. have you already built images? | 21:19 |
| kbaegis | My first kolla deployment. A beautiful thing | 21:19 |
| kbaegis | Yes | 21:19 |
| sdake | kbaegis grats! | 21:19 |
| *** rmart04 has quit IRC | 21:20 | |
| *** huikang has quit IRC | 21:21 | |
| kbaegis | I was always unhappy with devstack | 21:21 |
| kbaegis | Nice | 21:21 |
| sdake | wirehead_ any report from sv days? | 21:21 |
| wirehead_ | I have concentrated rage from devstack. | 21:22 |
| wirehead_ | Nothing too much. Didn’t run into the guy from SAP yet. | 21:22 |
| kbaegis | :) | 21:22 |
| kbaegis | It's the most hideously coded bash I've ever read | 21:22 |
| sdake | concetrated rage? | 21:23 |
| sdake | could you expand? | 21:23 |
| wirehead_ | I like Kubernetes because kube-up.sh is also hideously coded bash, but it works most of the time. | 21:23 |
| mark-casey | kbaegis: my brain isn't working right now... need more coffee... but sometimes there have been issues building for example centos images while running in ubuntu | 21:24 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Neutron bootstrap to create service/project/role https://review.openstack.org/353740 | 21:24 |
| mark-casey | just something to keep in mind if you hit weird stuff | 21:24 |
| sdake | mark-casey sucks when your brain doens't cooperate right ;) | 21:24 |
| kbaegis | I had issues with haproxy. I'm rebuilding that image now | 21:24 |
| mark-casey | sdake: indeed | 21:24 |
| kbaegis | "INFO:kolla.image.build.base:Cannot open: http://repo.percona.com/release/7/RPMS/x86_64/percona-release-0.1-3.noarch.rpm. Skipping' | 21:24 |
| sdake | kbaegis that link wfm, maybe your firewall or proxy is getting in the wya | 21:25 |
| wirehead_ | I spent too much time while at Rackspace futzing with Devstack not working so I never got anything committed to OpenStack. | 21:25 |
| wirehead_ | Concentrated rage is overstating my feelings. | 21:25 |
| wirehead_ | But I can be a drama queen. | 21:26 |
| sdake | can't we all :) | 21:26 |
| wirehead_ | Well, a considerable pool of software engineers claim to be totally rational, all the time. | 21:26 |
| wirehead_ | Also, frequently “just making a joke, man" | 21:26 |
| wirehead_ | :P | 21:26 |
| wirehead_ | Like, for an OpenStack conference, Kubernetes has been discussed a LOT. | 21:28 |
| wirehead_ | But most of the running OpenStack atop Kubernetes talks were yesterday. | 21:28 |
| *** jtriley has quit IRC | 21:28 | |
| *** inc0 has quit IRC | 21:30 | |
| sean-k-mooneyAFK | the one thing ill say in defece of devstack for whtat it does( provide a dev enviorment for developing openstack) it is a good solution. i would like to find a way to use kolla the same way but even source build from git are not quite as simple to develop with | 21:31 |
| *** ad_rien_ has joined #openstack-kolla | 21:31 | |
| kbaegis | sean-k Easy enough to do git clone from a container | 21:32 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Introducing DaemonSet for Neutron Openvswitch components https://review.openstack.org/352433 | 21:33 |
| sean-k-mooneyAFK | kbaegis: i have often built kolla containers from git then exec in and modify the code and restart the container to pick up the chagned. | 21:34 |
| sean-k-mooneyAFK | kbaegis: its still simpler to boot a vm on my kolla cluster and run devstack in it normally | 21:34 |
| sean-k-mooneyAFK | kbaegis: the main issue for me is how to debuge with a ide when the code is runing in a docker container | 21:35 |
| kbaegis | Just rebuilt the image from another docker host on other side | 21:36 |
| *** lrensing_ has quit IRC | 21:37 | |
| kbaegis | Glad I put up a image repo | 21:37 |
| kbaegis | Anyone know how much memory is taken by the controller playbook? | 21:38 |
| kbaegis | I have 2 nodes with 3-4 gigs I'll use if I can | 21:38 |
| kbaegis | They have docker but they're atom. Nice for failover, terrible for vms | 21:38 |
| *** shardy has joined #openstack-kolla | 21:40 | |
| sean-k-mooneyAFK | rabbitmq is a little ram hungrey but you should be able to get a controler deploy with 4GB but that is pushing it a little. 8GB would run fine. not sure about 4G | 21:40 |
| *** inc0 has joined #openstack-kolla | 21:42 | |
| *** gfidente has quit IRC | 21:44 | |
| kbaegis | I'm getting timeouts from my docker registry | 21:47 |
| kbaegis | Anyone know what the curl is to check the api for | 21:47 |
| kbaegis | like an ls | 21:47 |
| kbaegis | Ping works | 21:47 |
| sdake | kbaegis are you deploying multinode? | 21:47 |
| openstackgerrit | Merged openstack/kolla: Fix intf address for RabbitMQ and disable cluster for Kube https://review.openstack.org/336751 | 21:48 |
| kbaegis | sdake I'm not actually. Still using a registry | 21:48 |
| sdake | or doing an aio eval? | 21:48 |
| kbaegis | Necessary here, as I can't get haproxy w/o percona on centos evidently | 21:48 |
| kbaegis | It's my home lab. I'm learning how kolla works :) | 21:48 |
| kbaegis | Sort of | 21:49 |
| sdake | kbaegis roger - reecommend just using docker hub in that case | 21:49 |
| kbaegis | Eww. I prefer self-hosted :) | 21:49 |
| sdake | ok | 21:49 |
| sdake | docker ps -ef | grep docker -> pate | 21:49 |
| sdake | paste | 21:49 |
| kbaegis | registry:2 "/entrypoint.sh /etc/" 2 hours ago Up 2 hours 192.168.254.3:4000->5000/tcp registry | 21:50 |
| sdake | sorry tired | 21:50 |
| sdake | ps -e gerp docker -> paste | 21:50 |
| kbaegis | grep right? :) | 21:51 |
| sdake | gerp/grep | 21:51 |
| kbaegis | s/er/re/g | 21:51 |
| sdake | but you gotta hang tought when your all out there on your own | 21:51 |
| kbaegis | wait, is there a password generated for the | 21:52 |
| kbaegis | Hmm | 21:52 |
| kbaegis | https://192.168.254.3:4000/v1/_ping returns error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol | 21:52 |
| kbaegis | So the registry is locked to the host that spawned it? | 21:52 |
| kbaegis | I'll figure this out | 21:53 |
| kbaegis | I think this is the "secure registry" | 21:53 |
| kbaegis | I don't have insecure set on the host uploading | 21:54 |
| kbaegis | Yet another argument for my poor poor docker host | 21:55 |
| *** inc0 has quit IRC | 21:56 | |
| sean-k-mooneyAFK | kbaegis: i havent got around to impentat the bootstrap-build-host part of the kolla-host playbook but i should have that coded up by the end of the week. so hopefully setup the build host including the registry will be automate as a single command soon | 21:58 |
| *** msimonin has quit IRC | 21:58 | |
| *** inc0 has joined #openstack-kolla | 21:59 | |
| *** vhosakot has quit IRC | 21:59 | |
| sdake | kbaegis my speculation is if you run ps -ef | grep docker you will see it lacking the --insecure-registry flag | 22:00 |
| *** ravig has quit IRC | 22:00 | |
| sdake | kbaegis you need that to run your own registry - unless yougo throug hthe trouble of setting up tls for your registry | 22:00 |
| *** vhosakot has joined #openstack-kolla | 22:01 | |
| kbaegis | I've already set one up :) | 22:02 |
| kbaegis | It's timing out | 22:02 |
| *** ravig has joined #openstack-kolla | 22:02 | |
| kbaegis | Already figured that one out earlier :) | 22:02 |
| kbaegis | You're right, it was missing | 22:02 |
| kbaegis | I put it in and restarted. It didn't help unfortunately | 22:03 |
| kbaegis | So I'm just going to docker save, scp, | 22:03 |
| *** kong has joined #openstack-kolla | 22:04 | |
| *** ravig has quit IRC | 22:07 | |
| sean-k-mooneyAFK | kbaegis: what os are you using and where did you add the insecure-regestry setting | 22:07 |
| sdake | kbaegis are you sure insecure-registry is set- you can check with the ps command | 22:07 |
| *** shardy has quit IRC | 22:07 | |
| kbaegis | docker ps -e doesn't give me anything | 22:09 |
| kbaegis | docker ps --help | 22:09 |
| sean-k-mooneyAFK | kbaegis: ps -ef | grep docker | 22:10 |
| sean-k-mooneyAFK | kbaegis: not docker ps | 22:10 |
| kbaegis | '/usr/bin/docker daemon -p /run/docker.pid --insecure-registry 192.168.254.3:4000' | 22:11 |
| kbaegis | '/usr/bin/docke daemon -p /run/docker.pid --bip=172.22.1.1/24 --fixed-cidr=172.22.1.0/24 --dns=172.21.5.254 --ip-forward=true --iptables=true --icc=true --insecure-registry 192.168.254.3:4000' | 22:11 |
| kbaegis | the r got garbled by the terminal evidently | 22:12 |
| sean-k-mooneyAFK | kbaegis: that looks fine | 22:13 |
| kbaegis | Yup | 22:13 |
| *** michauds has quit IRC | 22:13 | |
| kbaegis | stil somehow docker registry times out | 22:13 |
| sean-k-mooneyAFK | if you did not have the insecure-registrey set when you built the images they will not have been pushed | 22:13 |
| mark-casey | kbaegis: pushing during build or pulling during deploy? | 22:13 |
| kbaegis | pushing after build | 22:13 |
| sdake | sean-k-mooneyAFK n++ | 22:13 |
| mark-casey | was the push flag set on build.py (and also what sean said could be it) | 22:14 |
| kbaegis | Host 2: python2 /usr/bin/kolla-build haproxy --registry 192.168.254.3:4000 --push | 22:14 |
| kbaegis | different kolla git sync | 22:14 |
| sean-k-mooneyAFK | you can push with bash after the fact. we probably should script that. | 22:15 |
| sean-k-mooneyAFK | let me see if i still have the command in my bash history | 22:15 |
| kbaegis | docker push --help wasn't very helpful | 22:15 |
| kbaegis | Bad documentation by docker | 22:15 |
| kbaegis | Ahh | 22:16 |
| kbaegis | The manpage is very enlightening | 22:16 |
| sean-k-mooneyAFK | kbaegis: i belive this will push the images for you on the host that built them | 22:18 |
| sean-k-mooneyAFK | for img in $(docker images | grep "192.168.254.3:4000" | awk '{ print $1 }'); do docker push $img; done | 22:18 |
| *** JoseMello has quit IRC | 22:20 | |
| sean-k-mooneyAFK | i should add that as a command to kolla-ansible as the counterpart to pull | 22:21 |
| kbaegis | tcp 192.168.254.3:4000: i/o timeout | 22:21 |
| kbaegis | Hmm | 22:21 |
| ntpttr___ | inc0: ping | 22:22 |
| kbaegis | Since when did hard drives just start getting fatter rather than more densely packed? | 22:23 |
| kbaegis | *sigh* | 22:23 |
| *** vhosakot has quit IRC | 22:23 | |
| kbaegis | What's the "offline" way to do it? | 22:24 |
| kbaegis | Pretend I had a sneakernet instead | 22:24 |
| kbaegis | How do I export the image to a file, scp (which works fine), and upload it locally to the registry? | 22:25 |
| sean-k-mooneyAFK | that or build the images on every host | 22:25 |
| *** inc0 has quit IRC | 22:25 | |
| kbaegis | OH! | 22:25 |
| kbaegis | Confident that this is a networking issue | 22:26 |
| *** fragatin_ has quit IRC | 22:26 | |
| sean-k-mooneyAFK | kbaegis: do you have a proxy? | 22:26 |
| *** fragatina has joined #openstack-kolla | 22:27 | |
| kbaegis | No, I have multiple route tables per host | 22:27 |
| kbaegis | ip route add 172.22.1.0/24 dev docker0 src 172.22.1.1 table os | 22:27 |
| kbaegis | I'll let ya know if that works :) | 22:27 |
| kbaegis | I keep my lab very isolated | 22:28 |
| kbaegis | Nope | 22:28 |
| kbaegis | That wasn't it | 22:28 |
| kbaegis | :/ | 22:28 |
| *** imcsk8|lunch is now known as imcsk8 | 22:28 | |
| *** rstarmer has joined #openstack-kolla | 22:29 | |
| kbaegis | curl http://192.168.254.3:4000/v2/_ping doesn't work from host2 | 22:30 |
| kbaegis | from the global route table | 22:30 |
| kbaegis | icmp to that address works fine | 22:30 |
| sean-k-mooneyAFK | in by working registry that curl command gives a 404 not found | 22:33 |
| kbaegis | Yeah | 22:33 |
| kbaegis | I get no response whatsoever | 22:33 |
| kbaegis | Something networking or iptables related I'm guessing | 22:33 |
| *** stevemcquaid has joined #openstack-kolla | 22:33 | |
| sean-k-mooneyAFK | well i hate iptables so beyond iptables -F iptables -X and try again i cant help much on that front | 22:35 |
| stevemcquaid | @kfox1111 - @harlowja said you might be the person with the answers! | 22:35 |
| harlowja | obviously! | 22:35 |
| harlowja | ha | 22:35 |
| jrich523 | actually i had to modify iptables to allow access to horizon, which should probably be part of the deployment scripts :) | 22:36 |
| sean-k-mooneyAFK | jrich523: really i have never had to do that. | 22:36 |
| jrich523 | it threw me for a loop | 22:36 |
| jrich523 | its a basic install of CentOS 7 | 22:36 |
| jrich523 | firewall-cmd --zone=public --add-port=80/tcp --permanent; firewall-cmd --reload | 22:37 |
| stevemcquaid | WRT: kubernetes, have you ever used ABAC access control before? We are running V1.2 and have not been able to do get a policy set to deny all, but allow svc accounts for ingress controller yet | 22:38 |
| kfox1111 | stevemcquaid: hi. | 22:38 |
| sean-k-mooneyAFK | oh i think sdake said someting about disableing firewalld on centos was assumed to be done when deploying kolla | 22:38 |
| *** sdake has quit IRC | 22:38 | |
| jrich523 | interesting assumption :) | 22:38 |
| stevemcquaid | hi @kfox1111 :) | 22:38 |
| jrich523 | notice he leaves when you mention him! | 22:39 |
| kfox1111 | whats up? | 22:40 |
| jrich523 | im going to blow away what i have (or make another) so see if i can rebuild it again... and maybe... create a pull request for that 'quick start' doc :) | 22:40 |
| harlowja | reposting from a little older stevemcquaid message | 22:41 |
| harlowja | kfox1111 'kubernetes, have you ever used ABAC access control before? We are running V1.2 and have not been able to do get a policy set to deny all, but allow svc accounts for ingress controller yet' | 22:41 |
| *** inc0 has joined #openstack-kolla | 22:41 | |
| * harlowja thought u might have an idea (or if u don't thats ok to) | 22:41 | |
| kbaegis | Well, this is definitely my fault :) | 22:46 |
| *** rstarmer has quit IRC | 22:47 | |
| *** zhubingbing has joined #openstack-kolla | 22:48 | |
| *** zhubingbing_ has joined #openstack-kolla | 22:48 | |
| *** inc0 has quit IRC | 22:49 | |
| kbaegis | So my dns wasn't enabled on my new lab server. I think that's why it wasn't finding percona. | 22:50 |
| kbaegis | Forgetting that it doesn't use the "hypervisor" dns | 22:50 |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 | 22:51 |
| kbaegis | Well hopefully knowing idiot admins like me are out there making mistakes like this is helpful to you too! :) | 22:52 |
| *** sould has joined #openstack-kolla | 22:52 | |
| *** ad_rien_ has quit IRC | 22:54 | |
| *** ad_rien_ has joined #openstack-kolla | 22:54 | |
| sould | good evening everyone :) i had a question about kolla-kubernetes. im following the quickstart guide, and im having an issue with the section regarding whether skydns is responding | 22:54 |
| sould | when i run 'nslookup kubernetes.default.svc.openstack.local 127.0.0.1', i get ';; connection timed out; no servers could be reached'. is there some additional configuration required outside of the quickstart guide i need to poke around and be aware of? | 22:55 |
| kbaegis | Would be wayyyy to convenient if it was the problem I just solved, huh? | 22:56 |
| kbaegis | DOCKER_OPTS="--dns=<dns server>"? | 22:56 |
| rhallisey | sbezverk, ^^^ | 22:56 |
| *** daneyon_ has joined #openstack-kolla | 22:56 | |
| *** ad_rien_1 has joined #openstack-kolla | 22:56 | |
| kbaegis | Confirmed, that was my problem guys | 22:56 |
| *** inc0 has joined #openstack-kolla | 22:57 | |
| *** ad_rien_ has quit IRC | 22:57 | |
| *** inc0 has quit IRC | 22:57 | |
| *** daneyon_ has quit IRC | 23:00 | |
| *** inc0 has joined #openstack-kolla | 23:01 | |
| mark-casey | kbaegis: ahhh, makes sense. nice catch | 23:01 |
| rhallisey | sould, could be the skydns container failing | 23:01 |
| rhallisey | docker logs <skydns_contianer> | 23:01 |
| *** ad_rien_1 has quit IRC | 23:02 | |
| *** ad_rien_ has joined #openstack-kolla | 23:02 | |
| kbaegis | deploy still failed. I'm rebuilding everything | 23:03 |
| kbaegis | should I be concerned about: 587 packages excluded due to repository priority protections? | 23:04 |
| kbaegis | Kind of like, "please go away, you're too troublesome? | 23:05 |
| mark-casey | kbaegis: how failed? have you run prechecks? | 23:09 |
| kbaegis | haproxy was unable to pass a config | 23:09 |
| kbaegis | It fails the cinder check, but I have cinder disable in globals | 23:10 |
| kbaegis | Specifically, this is what failed: "TASK [haproxy : Copying over haproxy.cfg] ************************************** | 23:14 |
| kbaegis | failed: [localhost] (item=haproxy) => {"failed": true, "item": "haproxy", "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'ipv4'"}" | 23:14 |
| kbaegis | ls | 23:14 |
| *** inc0 has quit IRC | 23:17 | |
| mark-casey | kbaegis: k. worth rebuilding once, but image build is more or less deterministic so you'll often get the same result. | 23:25 |
| mark-casey | At some point it becomes more about fixing interactions like between ansible on the deploy host and sshd or docker daemon on the target hosts, and you don't necessarily have to rebuild images unless you've made changes to, for example, the default dockerfiles | 23:25 |
| mark-casey | in this case the haproxy.cfg can't be templated by ansible because a variable doesn't exist, which (shot in the dark) could be something like a missing python lib on target hosts | 23:25 |
| mark-casey | so you'd install that and redeploy w/out rebuilding images | 23:26 |
| *** sdake has joined #openstack-kolla | 23:26 | |
| kbaegis | Hmm | 23:27 |
| mark-casey | two things to keep in mind next little bit...: one is you have to run clean-containers and clean-images on the target hosts after making changes but before redeploying, because the nodes don't know you've started over; last time's containers still running, and two if you haven't already you should run kolla-ansible prechecks instead of going straight to deploy | 23:28 |
| kbaegis | Alright, I'll follow the guide :) | 23:30 |
| *** ayoung has quit IRC | 23:30 | |
| kbaegis | This is really slick, and doesn't require 20 nodes like fuel | 23:31 |
| kbaegis | Liking it a lot | 23:31 |
| openstackgerrit | Dave Walker proposed openstack/kolla: Add --user-domain when adding heat role admin https://review.openstack.org/353471 | 23:31 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Fix bad merge of ceph mariadb https://review.openstack.org/353669 | 23:32 |
| sdake | sean-k-mooneyAFK ya we need to disable firewalld | 23:34 |
| sean-k-mooneyAFK | sdake: ok cool i dont have that in kolla-host currently but ill add it in another commit | 23:35 |
| *** zhurong has joined #openstack-kolla | 23:38 | |
| mark-casey | kbaegis: cool. I *should* be off and on tonight so if you're still at it in a few hours maybe we can see if there's a new roadbloack | 23:44 |
| kbaegis | thanks! | 23:46 |
| *** rkrum has joined #openstack-kolla | 23:50 | |
| openstackgerrit | Christine Hicks proposed openstack/kolla: Following rst guidelines fixing the documentation for kolla https://review.openstack.org/353786 | 23:52 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
