Wednesday, 2016-08-10

*** zhurong has quit IRC00:00
sbezverkkfox1111 ping00:03
kfox1111ping00:03
sbezverkkfox1111 if you have time please check this: https://review.openstack.org/#/c/346215/00:04
patchbotsbezverk: patch 346215 - kolla-kubernetes - Glance bootstrap to create service/project/role00:04
kfox1111k... let me look.00:05
kfox1111reviewed00:11
sbezverkkfox1111 thank you00:13
kfox1111np00:13
sbezverkkfox1111 can you point to a specific place with regards to credentials passing?00:15
sbezverkI mean we used the same approach even in kolla, but I would like to better understand your concern00:15
kfox1111kolla-kubernetes.yaml line 5900:16
kfox1111and various places in the job where it references them in the command.00:16
kfox1111I guess my concern is with the level of secretness.00:16
kfox1111any pod you can just kubectl get pod name -o yaml and get stuff back.00:16
kfox1111where as you might have to work harder at retrieving a secret.00:17
kfox1111kolla-ansible gets around the issue as stuff is always pushed from ansible host to individual hosts in one direction.00:17
kfox1111k8s has a central server(s) that you can pull back stuff from.00:18
sbezverkI see what you mean. it is going to be very hard as we pass this  to kolla-toolbox ansible module00:18
kfox1111I think if we just put the vars used there now in a bash env var, and loaded that as a secret,00:18
sbezverkit would require major rearchitecture and this approach will become kube specific00:19
kfox1111in the command you can just source it first, then reference the vars inline?00:19
kfox1111here... let me mark up a little bit and maybe I can show you what I'm thinking...00:20
sbezverkwould be better :-)00:21
*** daneyon_ has joined #openstack-kolla00:22
kfox1111replied.00:26
*** daneyon_ has quit IRC00:26
kfox1111not sure how readable that is or not.00:27
openstackgerritKabir Sial proposed openstack/kolla: Customizations for dind  https://review.openstack.org/35320200:27
openstackgerritKabir Sial proposed openstack/kolla: Customizations for dind  https://review.openstack.org/35320300:27
sbezverkI see what you mean.. currently kolla and kolla-kube stores all password in the file00:30
kfox1111yeah. but it ends up storing it in the pod descriptions too, which I think's a really bad idea. :/00:31
sbezverkwhich is used during rendering by kolla and kolla-kube00:31
kfox1111you can still template out a secret.00:31
kfox1111added an example of what the secret would look like I think.00:33
sbezverkwe will always hit one problem, consistency with kolla00:34
kfox1111there can not be perfect consistancy as they have totally different security models.00:34
kfox1111ansible assumes you have one central node, you secure the heck out of, and push from there to other nodes.00:35
sbezverkkube has bunch of features to go wild, but using them would mean kube specific images00:35
*** duonghq has joined #openstack-kolla00:35
kfox1111kube's plan is to secure secrets different then everything else and only make them available as needed.00:35
kfox1111the proposed solution requires nothing kubernetes specific in the image.00:36
kfox1111it just passes the passwords via a shell environment file loaded via k8s secret, instead of hard coding them into the generated pod description.00:36
sbezverkwho will create kube secrets?00:37
kfox1111in the end, the same command line's are executed in the containers.00:37
kfox1111the bootstrap process.00:37
*** dwalsh has joined #openstack-kolla00:38
kfox1111I've got to head out. talk more tomorrow?00:40
sbezverkabsolutely, I will read up more on secrets and how they are consumed, have a good one00:40
kfox1111kk.00:41
kfox1111basically, they function exactly like configmaps.00:41
kfox1111they are just a seperate resource so you can rbac them differently/more aggressively.00:41
kfox1111the plan is to make it such that all kubelets can download all configmaps, but kubelets can only download a secret if they have a pod registered to them that references it.00:42
kfox1111so if you break into a node, its harder to steal a secret that you don't already have on the node.00:42
kfox1111ok. really gota go now. l8r.00:43
*** hieulq_ has joined #openstack-kolla00:43
*** sdake has joined #openstack-kolla00:59
sdakewirehead_ any latest updates from sv?01:02
*** zhurong has joined #openstack-kolla01:07
*** fragatin_ has quit IRC01:12
*** fragatina has joined #openstack-kolla01:12
openstackgerritMerged openstack/kolla-kubernetes: Add Ceph support for Mariadb  https://review.openstack.org/35031201:17
*** dwalsh has quit IRC01:19
*** unicell has quit IRC01:26
duonghqJeffrey41: I pushed new patch sets according your reply, can you review it?01:31
duonghqhttps://review.openstack.org/#/c/352275/01:31
patchbotduonghq: patch 352275 - kolla - Port with_ loop variable syntax  to Ansbile 201:31
duonghqJeffrey4l: can you review my updated patch set01:32
Jeffrey4lduonghq, np01:35
*** v1k0d3n has quit IRC01:40
*** v1k0d3n has joined #openstack-kolla01:40
*** v1k0d3n has quit IRC01:43
*** v1k0d3n has joined #openstack-kolla01:44
*** mbound has joined #openstack-kolla01:51
*** v1k0d3n has quit IRC01:51
*** mbound has quit IRC01:52
duonghqJeffrey4l: thanks01:52
Jeffrey4lmy pleasure :)01:53
*** hieulq has joined #openstack-kolla02:11
*** ayoung has quit IRC02:11
openstackgerritHiroki Ito proposed openstack/kolla: Fix prechecks of a vip address  https://review.openstack.org/35321402:13
sdakeJeffrey4l my man whats up02:13
Jeffrey4lnot much. But need leave for a while, kid is ill :(02:15
openstackgerritMerged openstack/kolla: Introduce a script to launch ovsdb-server process  https://review.openstack.org/35178402:17
openstackgerritHiroki Ito proposed openstack/kolla: Fix prechecks of a vip address  https://review.openstack.org/35322102:22
*** senk_ has joined #openstack-kolla02:28
*** zhurong has quit IRC02:42
*** zhurong has joined #openstack-kolla02:44
*** zhurong has quit IRC02:48
*** coolsvap has joined #openstack-kolla02:49
*** zhurong has joined #openstack-kolla02:49
*** diogogmt has quit IRC02:51
*** zhurong has quit IRC02:58
*** zhurong has joined #openstack-kolla02:59
Jeffrey4lsbezverk, could see the comments https://review.openstack.org/#/c/351784/1603:04
patchbotJeffrey4l: patch 351784 - kolla - Introduce a script to launch ovsdb-server process (MERGED)03:04
*** zhubingbing has joined #openstack-kolla03:06
Jeffrey4lDaviey, https://review.openstack.org/349764   https://review.openstack.org/349197   need your comments :)03:07
Jeffrey4lsdake, https://review.openstack.org/352089   ready for review03:08
sdakeyou will go to the degabah system03:08
sdakeand learn from yoda, the jedi master that instructed me03:08
sdakeJeffrey4l thats french for taking a break :)03:09
Jeffrey4lhaha new language03:10
sdakeits from star wars03:10
sdakeempire strikes back03:11
sdakegeneral veers, prepare your men!03:16
openstackgerritzhubingbing proposed openstack/kolla: Add Barbican ansible role  https://review.openstack.org/35251503:16
*** sdake has quit IRC03:17
*** britthouser has joined #openstack-kolla03:20
*** britthouser has quit IRC03:25
openstackgerritzhubingbing proposed openstack/kolla: Add gnocchi ansible role  https://review.openstack.org/34935103:31
*** huikang has joined #openstack-kolla03:31
*** phuongnh has joined #openstack-kolla03:39
openstackgerritzhubingbing proposed openstack/kolla: Add sahara ansible role  https://review.openstack.org/35129403:55
openstackgerritzhubingbing proposed openstack/kolla: repair aodh image dockerfile  https://review.openstack.org/35071503:58
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Vagrant doc updates  https://review.openstack.org/35324004:08
*** rkrum has joined #openstack-kolla04:13
duonghqcoolsvap: can you review my change at https://review.openstack.org/#/c/352275/04:27
patchbotduonghq: patch 352275 - kolla - Port with_ loop variable syntax  to Ansbile 204:27
openstackgerritzhubingbing proposed openstack/kolla: Add sahara ansible role  https://review.openstack.org/35129404:27
* coolsvap checks04:28
*** zhubingbing_ has joined #openstack-kolla04:30
*** zhubingbing has quit IRC04:32
*** senk_ has quit IRC04:33
*** senk_ has joined #openstack-kolla04:33
*** fragatin_ has joined #openstack-kolla04:36
openstackgerritzhubingbing proposed openstack/kolla: repair gnocchi  dockerfile  https://review.openstack.org/34838804:38
*** zhubingbing has joined #openstack-kolla04:39
*** fragatina has quit IRC04:39
zhubingbinghi,Duong Ha-Quang04:39
duonghqhi zhubingbing04:39
zhubingbingI think it makes sense to you that I think it is reasonable in the new version, you can teach me how to modify it.04:40
*** fragatin_ has quit IRC04:40
duonghqah, for the varible syntax: as Ansible doc point out there: https://docs.ansible.com/ansible/porting_guide_2.0.html#deprecated04:41
duonghqthe bare variable syntax is deprecated for Ansible 2.004:41
duonghqfor ex: the old style is04:41
duonghq- var_a04:42
duonghqand it is recommended to write as:04:42
duonghq- {{ var_a }}04:42
duonghqah04:42
duonghq- "{{ var_a }}"04:42
duonghqthe full variable syntax is {{ var_a }} but it must be quoted in " "04:43
zhubingbing- {{ container_envs.results04:43
zhubingbing }}   - check_results.results04:43
duonghqit must be quoted in " "04:43
zhubingbingWhat is the definition of this variable in my place04:43
coolsvapduonghq: looks good rechecking to see if I can get green in oraclelinux deploy gate04:44
coolsvapdoes not seem to be related to fix04:44
coolsvapjust to see before I approve04:44
duonghqcoolsvap: thanks04:45
zhubingbing  - "container_envs.results"04:45
zhubingbing  - "check_results.results"04:45
zhubingbingLike this?04:45
duonghq- "{{ container_envs.results }}"04:46
zhubingbingduonghq:04:46
duonghqlike  this04:46
zhubingbingThis variable does not need to be defined, is he a ansible default variable?04:47
zhubingbingsure04:47
duonghqzhubingbing: this variable is defined at line 3304:48
duonghqregister: ....04:48
zhubingbingi fixed now,thank you very much,Your advice is really good for me04:48
duonghqmy pleasure04:48
*** coolsvap is now known as _coolsvap_04:49
openstackgerritzhubingbing proposed openstack/kolla: Add Barbican ansible role  https://review.openstack.org/35251504:53
*** daneyon_ has joined #openstack-kolla04:53
zhubingbingduonghq: Can you help us to see, just look at this.04:53
zhubingbinghttps://review.openstack.org/35251504:53
duonghqzhubingbing: I just +1 on this04:54
zhubingbingOK thanks,I'm going to deal with the other role.04:54
zhubingbingYour advice is really good.04:55
duonghqyou're welcome04:55
*** daneyon_ has quit IRC04:58
openstackgerritzhubingbing proposed openstack/kolla: Add sahara ansible role  https://review.openstack.org/35129404:58
*** unicell has joined #openstack-kolla05:00
openstackgerritzhubingbing proposed openstack/kolla: Add gnocchi ansible role  https://review.openstack.org/34935105:01
zhubingbingAll of my role have been modified.05:01
*** unicell has quit IRC05:06
*** unicell has joined #openstack-kolla05:06
*** senk_ has quit IRC05:09
*** phuongnh has quit IRC05:09
*** unicell has quit IRC05:09
openstackgerritzhubingbing proposed openstack/kolla: Add Barbican container dockerfile  https://review.openstack.org/35182205:10
zhubingbinghi,Christian Berendt05:10
zhubingbingChristian Berendt:05:10
*** unicell has joined #openstack-kolla05:13
*** _coolsvap_ is now known as coolsvap05:16
huikangHi, Jeffrey4l, are you around?05:17
Jeffrey4lhuikang, yup05:18
huikangwhen I deploy all-in-one, the mariadb bootstrap always failed05:18
huikangJeffrey4l, http://paste.openstack.org/show/553320/  here is the log message05:19
Jeffrey4lhuikang, i meet this once. no chance to debug it. :(05:19
Jeffrey4lhow many node are u deploying ? huikang05:20
huikangall-in-one05:20
duonghqhuikang: you can try execute deploy twice in a round05:21
huikanglet me try now05:21
duonghqzhubingbing: are you still there? IMO, you should add one line for your module in globals.conf05:22
zhubingbingyes05:22
duonghqah globals.yml05:22
duonghqlike other modules05:22
zhubingbingadd globals.conf?05:22
duonghqzhubingbing: etc/kolla/globals.yml05:22
zhubingbingDo you need to add this file?05:23
duonghqzhubingbing: this file is already in source code05:23
huikangduonghq, it bypasses the bootstrap error, but the mariadb datatable is not functioning05:23
Jeffrey4lhuikang, could i reach that server?05:23
duonghqhuikang: it seem that you get error same mine: https://bugs.launchpad.net/kolla/+bug/160950205:24
openstackLaunchpad bug 1609502 in kolla "Container bootstrap_mariadb exits due to Galera failed" [Undecided,New]05:24
-openstackstatus- NOTICE: zuul is being restarted to reload configuration. Jobs should be re-enqueued but if you're missing anything (and it's not on http://status.openstack.org/zuul/) please issue a recheck in 30min.05:24
huikangJeffrey4l, no, the server is behind firewall.05:24
Jeffrey4lhuikang, try to add `set -x` to the docker/mariadb/extend_start file and build , then deploy. it will print more useful info.05:25
*** phuongnh has joined #openstack-kolla05:25
huikangwill do05:25
huikangduonghq, yes, I confirm your bug report05:26
duonghqhuikang: which base os and deploy type are you using?05:26
huikangI tried both centos binary and ubuntu source; both failed05:26
duonghqdo you use local docker registry?05:27
huikangduonghq, yes; does that matter?05:28
duonghqnot really, when I setup my local docker registry, this bug completely go away.05:28
zhubingbingI didn't see the gloable.yaml in the other modules05:29
duonghqI cannot reproduce this bug up to now.05:29
duonghqzhubingbing: it is in root of kolla project05:29
huikangdo you mean this file etc/kolla/globals.yml05:30
duonghqyes,05:30
huikangduonghq what are the base OS and type you are using now?05:30
duonghqhuikang: I'm using ubuntu/source05:31
zhubingbingglobals.yml file not nned05:33
zhubingbingWhat's wrong?http://logs.openstack.org/15/350715/9/check/gate-kolla-dsvm-build-centos-source/27113df/console.html05:33
*** huikang has quit IRC05:36
*** duonghq has quit IRC05:39
openstackgerritMerged openstack/kolla: Remove unused project_yaml parameter from role metadata files  https://review.openstack.org/35192805:50
openstackgerritRoss Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate  https://review.openstack.org/35326105:51
*** unicell has quit IRC05:55
*** unicell has joined #openstack-kolla05:55
openstackgerritzhubingbing proposed openstack/kolla: repair aodh image dockerfile  https://review.openstack.org/35071505:55
*** duonghq has joined #openstack-kolla06:01
*** hieulq_ has quit IRC06:04
*** egonzalez90 has joined #openstack-kolla06:12
*** unicell has quit IRC06:22
*** unicell has joined #openstack-kolla06:22
*** senk_ has joined #openstack-kolla06:28
*** bootsha has joined #openstack-kolla06:29
openstackgerritzhubingbing proposed openstack/kolla: Add gnocchi ansible role  https://review.openstack.org/34935106:31
*** dmueller-evoila has joined #openstack-kolla06:31
duonghqcoolsvap: Finally, all build and deploy test is green06:33
coolsvapduonghq: ack!06:35
duonghqcoolsvap: thanks06:35
*** fragatina has joined #openstack-kolla06:37
zhubingbinghi coolsvap06:39
coolsvapzhubingbing: hello06:39
zhubingbingCan you help me  reivew my code06:39
*** inc0 has joined #openstack-kolla06:39
zhubingbing--))06:40
*** fragatina has quit IRC06:42
openstackgerritMerged openstack/kolla: Port with_ loop variable syntax  to Ansbile 2  https://review.openstack.org/35227506:44
coolsvapzhubingbing: will do06:44
coolsvapbut cant tell you the time to be honest06:44
zhubingbing- -thanks you06:44
coolsvapbut will do06:44
*** inc0 has quit IRC06:45
zhubingbinghttp://logs.openstack.org/88/348388/23/check/gate-kolla-dsvm-deploy-centos-binary/e4d4203/console.html06:48
zhubingbingwho can help me look it06:48
duonghqzhubingbing which change?06:51
duonghqi.e patch set06:52
zhubingbing_http://logs.openstack.org/88/348388/23/check/gate-kolla-dsvm-deploy-centos-binary/e4d4203/console.html06:58
duonghqDid you recheck?06:59
zhubingbing_duonghq: https://review.openstack.org/#/c/351320/07:00
patchbotzhubingbing_: patch 351320 - kolla - fix sahara dockerfile07:00
*** duonghq_ has joined #openstack-kolla07:04
*** duonghq has quit IRC07:05
*** rkrum has quit IRC07:06
openstackgerritzhubingbing proposed openstack/kolla: repair aodh image dockerfile  https://review.openstack.org/35071507:10
zhubingbing_ok07:13
zhubingbing_thankins07:13
*** b_bezak has joined #openstack-kolla07:17
*** athomas has joined #openstack-kolla07:21
*** ad_rien_ has joined #openstack-kolla07:22
*** ad_rien_ has quit IRC07:25
*** ad_rien_ has joined #openstack-kolla07:27
openstackgerritzhubingbing proposed openstack/kolla: Add sahara ansible role  https://review.openstack.org/35129407:28
*** ad_rien_ has quit IRC07:29
*** ad_rien_ has joined #openstack-kolla07:32
openstackgerritzhubingbing proposed openstack/kolla: Add Barbican ansible role  https://review.openstack.org/35251507:32
*** matrohon has joined #openstack-kolla07:33
*** bootsha has quit IRC07:35
*** shardy has joined #openstack-kolla07:35
*** daneyon_ has joined #openstack-kolla07:36
*** bootsha has joined #openstack-kolla07:38
*** daneyon_ has quit IRC07:40
*** duonghq has joined #openstack-kolla07:41
*** matrohon has quit IRC07:41
*** duonghq_ has quit IRC07:43
openstackgerritzhubingbing proposed openstack/kolla: repair gnocchi  dockerfile  https://review.openstack.org/34838807:43
*** duonghq has quit IRC07:44
*** Serlex has joined #openstack-kolla07:45
openstackgerritzhubingbing proposed openstack/kolla: repair gnocchi  dockerfile  https://review.openstack.org/34838807:46
*** godleon has quit IRC07:50
*** kong has quit IRC07:50
*** matrohon has joined #openstack-kolla07:57
*** bootsha has quit IRC07:59
*** bootsha has joined #openstack-kolla08:05
*** duonghq has joined #openstack-kolla08:09
*** matrohon has quit IRC08:15
*** openstackgerrit has quit IRC08:18
*** openstackgerrit has joined #openstack-kolla08:18
*** zhubingbing has quit IRC08:29
*** zhubingbing_ has quit IRC08:29
*** matrohon has joined #openstack-kolla08:31
*** shardy has quit IRC08:44
*** shardy has joined #openstack-kolla08:45
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Vagrant doc updates  https://review.openstack.org/35324008:48
*** gfidente|afk is now known as gfidente08:50
*** ntpttr has quit IRC08:51
*** ntpttr has joined #openstack-kolla08:58
*** shardy is now known as shardy_mtg09:00
duonghqanybody try swift deploy in master branch?09:09
*** ad_rien_ has quit IRC09:15
*** matrohon has quit IRC09:22
*** bootsha has quit IRC09:26
*** ad_rien_ has joined #openstack-kolla09:34
*** ad_rien_ has quit IRC09:36
*** matrohon has joined #openstack-kolla09:38
*** mnasiadka has joined #openstack-kolla09:40
openstackgerritKirill Shileev proposed openstack/kolla: Customizations for Glance  https://review.openstack.org/35297709:41
*** mbound has joined #openstack-kolla09:44
*** sdake has joined #openstack-kolla09:47
sdakemorning folks09:53
sdakeJeffrey4l around?10:00
sdakecoolsvap around?10:00
coolsvapsdake: yes10:00
sdakeanything happening with the osic gear?10:00
*** zhurong has quit IRC10:02
coolsvapsdake: I did the tempest verification today morning10:03
sdakeon the ceph based scenario?10:03
coolsvapI was waiting for updates from rally team related to the issues we are having related to benchmark tests10:03
coolsvapyes on the ceph based scenario10:04
sdakedont block on rally10:04
sdakewhat is the probelm there10:04
*** matrohon has quit IRC10:04
coolsvapthe image we add has private visibility10:04
coolsvapso the benchmark tests cannot recognize the cirros image10:04
coolsvapand fail the validation of scenario10:05
sdakeadd it with public visbility10:05
coolsvapyes10:05
coolsvapi was talking on rally channel when you pinged :)10:05
sdakeoh10:05
sdakeok carry on then :)10:05
coolsvapI am planning to do some benchmark tests in next couple of hours and then will free the environment10:06
sdakedoes rally benchmark vm performance?10:14
sdakecoolsvap ^^10:14
coolsvapnot vm the openstack cloud performance10:15
sdakethrow in a vm test10:16
sdakeof dd performance from inside the vm pls10:16
sdakeor bonnie would work too10:17
coolsvapsdake: i have not used bonnie before10:20
sdakein the docs i started10:20
sdakethere is a bonnie command run10:20
sdakeinstead of 520G10:20
*** matrohon has joined #openstack-kolla10:21
sdakechange it to 2xmemoryofvm+1010:21
*** rkrum has joined #openstack-kolla10:23
sdakedoes rally output to json?10:25
*** shardy_mtg is now known as shardy10:25
*** bootsha has joined #openstack-kolla10:26
openstackgerritKirill Shileev proposed openstack/kolla: Customizations for Glance  https://review.openstack.org/35297710:28
*** matrohon has quit IRC10:29
*** mbound has quit IRC10:33
*** bootsha has quit IRC10:33
coolsvapyes it gives output in json10:35
sdakewe can use pandoc to convert json to rst10:36
sdakecoolsvap did that election wrap up for requirements?10:36
coolsvapi think it will today10:37
coolsvaptomorrow10:41
coolsvapsdake: in 2xmemoryofvm+1010:41
*** ad_rien_ has joined #openstack-kolla10:42
coolsvapI am keeping memory of vm 2G10:42
*** mkoderer_ has joined #openstack-kolla10:42
*** mkoderer_ has quit IRC10:42
sdakebonnie++ -G 6G should do10:43
*** matrohon has joined #openstack-kolla10:45
openstackgerritRoss Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate  https://review.openstack.org/35326110:45
sdakeevery time i see those reviews it makes me cry :(10:47
sdake2 hrs of reviewwork10:47
sdakeincoming10:47
*** duonghq has left #openstack-kolla10:47
pbourkesame10:49
coolsvapumm i tested with 520G again, how much time it will take?10:49
*** bootsha has joined #openstack-kolla10:49
*** jtriley has joined #openstack-kolla10:50
openstackgerritRoss Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate  https://review.openstack.org/35326110:51
pbourkecoolsvap: tested what?10:53
*** matrohon has quit IRC10:53
coolsvappbourke: bonnie10:53
pbourkeah10:53
coolsvappbourke: I did the tempest run and a vm create delete benchmark test10:53
openstackgerritRoss Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate  https://review.openstack.org/35326110:54
coolsvapadding it to the review10:55
coolsvapi hope we can integrate rally reports in the doc somehow10:56
*** vbel has quit IRC10:59
*** vbel has joined #openstack-kolla10:59
*** JoseMello has joined #openstack-kolla11:01
*** b_bezak has quit IRC11:01
*** b_bezak has joined #openstack-kolla11:01
*** b_bezak has quit IRC11:06
*** matrohon has joined #openstack-kolla11:09
*** daneyon_ has joined #openstack-kolla11:12
*** matrohon has quit IRC11:12
*** kong has joined #openstack-kolla11:16
*** daneyon_ has quit IRC11:16
sdakeget rst output11:18
sdakei'll take care of the rest11:19
sdake520g in a vm will take forever11:19
sdaketry with 6G11:19
sdakecoolsvap ^^11:19
sdakerather get json output11:20
coolsvapsdake: I will kill the bonnie then11:20
sdakeyup11:20
coolsvapjust a min11:20
*** ppowell has joined #openstack-kolla11:21
sdakehey ppowell long time no chat11:21
coolsvapsdake: done!11:21
sdakehow long did it take to run?11:22
openstackgerritShaun Smekel proposed openstack/kolla: Add full support for fernet  https://review.openstack.org/34936611:22
coolsvapi killed the 520G one, it was running from the time i asked you before11:23
sdake30 seconds then?11:23
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Add OSIC Scale Testing Documentation  https://review.openstack.org/35210111:23
coolsvapno, it was running for 35 mins11:24
openstackgerritRoss Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate  https://review.openstack.org/35326111:24
openstackgerritShaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet  https://review.openstack.org/35113911:26
openstackgerritShaun Smekel proposed openstack/kolla: Add full support for fernet  https://review.openstack.org/34936611:27
*** matrohon has joined #openstack-kolla11:28
sdakeresults of bonnie++ were what?11:29
sdakedoes rally only test instance start/stoptime?11:30
*** b_bezak has joined #openstack-kolla11:31
*** zhubingbing has joined #openstack-kolla11:31
*** zhubingbing_ has joined #openstack-kolla11:32
*** matrohon has quit IRC11:36
*** fragatina has joined #openstack-kolla11:36
*** sean-k-mooney-aw is now known as sean-k-mooney11:37
sdakemorning sean-k-mooney11:37
sean-k-mooneysdake: morning :)11:37
*** zhubingbing_ has quit IRC11:38
sdakeget any reviews on your work?11:38
coolsvaprally tests inside vm statistics with dd but currently its not able to get cinder11:38
*** zhubingbing has quit IRC11:38
sdakecinder wont deploy?11:39
sean-k-mooneysdake: just looking at them now i dont see any new comments11:39
*** dwalsh has joined #openstack-kolla11:39
* sdake sadfaces11:39
Davieyi'm not sure how interesting stats are with all SSD deployment.. nobody is surely doing that?11:39
sdakeDaviey the ceph 10gig network is the limiting factor here11:40
Davieyif we could artificially throttle the mons to spinning speed, that becomes more interesting.. Then testing with and without ssd journal11:40
sean-k-mooneysdake: i do have some on the kolla-host though which i will take a look at now11:40
Davieysdake: interesting11:40
Davieysdake: are we maxing out 10G?11:40
sdakei do in my ceph deployment at home11:41
*** fragatina has quit IRC11:41
sdakei wnted to see what happens on a large scaledeploy11:41
Davieysdake: is this ceph rebalancing network IO or general usage?11:41
sdakeno idea11:41
Davieysdake: i mean, are you able to max out the network once the thing is 'stable'?11:42
Davieyie, a day old deployment11:42
sdakewith ceph + dd yes on 3 nodes11:43
sdakewith ceph + dd on 20 storage nodes = ??11:43
*** rkrum has quit IRC11:43
Davieysdake: i never imagined network would be the limiting factor on spinning disks11:44
sdakei have ssds in my 3 node cluster - intel 750 nvme11:44
Davieyah11:44
openstackgerritPaul Bourke proposed openstack/kolla: Customizations for Cinder  https://review.openstack.org/35058711:47
openstackgerritPaul Bourke proposed openstack/kolla: Revert "Remove files from /var/lib/apt/lists when cleaning up on Ubuntu/Debian"  https://review.openstack.org/35340811:47
*** rmart04 has joined #openstack-kolla11:49
*** v1k0d3n has joined #openstack-kolla11:50
coolsvapsdake: so there is not cinder service in openstack service list11:50
*** sdake has quit IRC11:51
openstackgerritPaul Bourke proposed openstack/kolla: Customizations for Nova  https://review.openstack.org/35167611:51
*** sdake has joined #openstack-kolla11:51
*** athomas has quit IRC11:52
*** matrohon has joined #openstack-kolla11:52
*** v1k0d3n has quit IRC11:54
Davieycoolsvap: no surprising, cinder isn't enabled11:55
coolsvapDaviey: yeah but thats not known to rally11:56
*** sdake_ has joined #openstack-kolla11:56
*** athomas has joined #openstack-kolla11:58
Davieycoolsvap: we want cinder tho, right?11:58
*** sdake has quit IRC11:59
coolsvapDaviey: I have not deployed ceph as of now either with kolla or without11:59
sdake_cinder and nova use similar drivers11:59
*** msimonin has joined #openstack-kolla11:59
coolsvapsdake_: we do need cinder for ceph deployment right?12:00
sdake_ceph is already deplyed in our current scenario12:00
Davieycoolsvap: I mean,  currently we have ceph deployed.. but we are not using either the object or block integration with openstack12:00
Davieyso currently the ceph is useless to us?12:01
sdake_nova uses ceph12:01
sdake_for its ephemeral storage12:01
Davieysdake_: Oh, the test case wants to use nova block storage rather than cinder?12:01
sdake_the test case i want is bonnie on nova ephemeral storage12:02
sdake_(onceph)12:02
Davieybut not via cinder?12:02
sdake_that is a different scenario iirc12:02
*** rhallisey has joined #openstack-kolla12:04
sdake_morning rhallisey12:06
rhalliseysdake_, hey12:06
ad_rien_Hi Guys, in the context of the performance working group, we are working on a 1000 compute node experiments (see http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-08-09-16.00.log.html for more inforamtions). Right now, we developed a set of scripts to deploy kolla on top of Grid'5000 (i.e. a testbed in France dedidacted to research on distributed systems) using the nova fake driver. However, we would like to know wh12:11
*** mliima has quit IRC12:12
Davieyad_rien_: your message was too long and truncated... ended with "However, we would like to know wh"12:15
ad_rien_Daviey:  sorry, "However, we would like to know whether it is possible to deploy several 'real/containerized' computes nodes on the same physical server.  I asked the questions few days ago but I didn't get any answer. Is there any core member of Kolla online  ? Can someone tell me whether posting such a question on the dev mailing list makes sense ?  Thanks"12:16
rhalliseyad_rien_, https://github.com/openstack/kolla/blob/master/doc/nova-fake-driver.rst12:17
ad_rien_rhallisey: yes we are using the nova fake driver right now but we would like to deploy a real compute node container.12:18
ad_rien_I mean several compute node containers per physical server.12:18
rhalliseyad_rien_, I'm not sure it will work12:19
ad_rien_Right now, the solution is to use virtualization (for instance two VMs per physical servers) and then deploy kolla as if those VMs where standard servers.  But this means : physical server / Virtualization layer /  docker  which is rather strange from our point of view12:20
rhalliseyad_rien_, the problem you may run into is with net=host, where the compute nodes will use the hosts network namespace12:20
sdake_lets not forget pid=host12:21
*** phuongnh has quit IRC12:21
*** rcherrueau has joined #openstack-kolla12:21
*** matrohon has quit IRC12:21
sdake_ad_rien_ the short answer is - if you want to simulate more gear then you have available, you will have to simulate that using some other tech besides docker12:22
sdake_such as vms12:22
ad_rien_rhallisey:  I agree and that 's why we would like to know whether someone already plays with such a use-case or if Kolla key persons already know it will not run12:22
sdake_if you want  to use kolla, deploy 2 vms on your 500 nodes, then deploy kolla on top of that12:22
ad_rien_sdake_:  so should I understand that conceptually speaking (i.e. from the docker view point), it will not be possible12:23
*** magnus-loof has joined #openstack-kolla12:23
sdake_it is not a docker issue12:23
rhalliseyad_rien_, I think it would require removing net=host.  You may have to go with kolla on kjolla12:23
sdake_its a libvirt issue12:23
rhalliseyya and a libvirt issue12:23
sdake_only one copy of libvirt may be running on a system at a time12:23
rhalliseyad_rien_, this would be a problem regardless of docker or not12:23
Davieyad_rien_: I think the only way you can reasonably do it, is by running VM's on the bare metal and expose them as bridged networking12:24
sdake_rhallisey right - has nothign to do with docker12:24
rhalliseysdake_, ya I'm agreeing with you12:24
Davieyad_rien_: so you have the appearance of more physical servers.  Using qemu nested virt, it should work just fine12:24
ad_rien_s/docker/containers12:24
magnus-loofHi! I am trying to deploy OS using kolla. I want to use the published 2.0.2 for the time being. However, when running kolla-ansible deploy with stable/mitaka or 2.0.2 tag, I get the following errors: https://gist.github.com/maglo/be4a0c039dfd5d02fdc4e45c41f31c3612:25
* ad_rien_ is looking on the net to understand why several libvirt cannot run on a system at the same time or why container technologies does not enable to separate such daemons. 12:25
*** zhurong has joined #openstack-kolla12:26
Davieyad_rien_: I'd probably say.. On a kolla compute node, the key container is the libvirt container.. And kolla doesn't support generating separate libvirt socket files for each one.12:27
sdake_Daviey and that isn't a use case that makes any sense, except for profiling, which is why libvirt doesn't support it12:28
magnus-loofI managed to deploy OS using Kolla and master when I downgrade to 2.0.2 in /etc/kolla/globals.yml. Howerver, this was bricked today, probably due to https://github.com/openstack/kolla/commit/41ec75b56734604ad548b354bc929105babbb53812:28
sdake_magnus-loof that is a master only patch12:28
Davieysdake_: well.. i'd be suprised if livirt couldn't itself work.. you can specify the socket file to use at runtime.. but the libvirt container we have doesn't12:28
sdake_2.0.2 is stable/mitaka12:29
Davieysdake_: but equally, i agree it doesn't make sense unless you are benchmarking12:29
Davieysdake_: and then.. i'd say it gets pretty messy fast.. and usig nested virt makes much more sense to have multiple kolla hosts per baremetal12:29
sbezverkmagnus-loof: the change you are refering to applied only to master12:29
sbezverkmagnus-loof: you should not see it in stable mitaka12:30
magnus-loofsdake_ yes I know. To clarify, yesterday when deploying from kolla/master with 2.0.2 i globals.yml, it worked. Today it was bricked (could not find the file that was changed in the commit mentioned).12:31
rhalliseyad_rien_, pid=host is why12:31
Davieymagnus-loof: that probably means you were using master with previously built mitaka images in your registry... this is a configuration that shouldn't be done12:31
ad_rien_sdake_:  daviey: I agree that such a use-case makes sense for benchmarking purposes… that's our goal actually and adding the virtualization layer (i.e. the nested one) may lead to some OS noises that we would like to prevent/mitigate12:32
magnus-loofSo, I want to deploy from stable/mitaka. However, that fails with message in https://gist.github.com/maglo/be4a0c039dfd5d02fdc4e45c41f31c3612:32
Davieyad_rien_: what is the scope of the test?12:32
sdake_magnus-loof please run pip show kolla -> paste12:32
magnus-loofDaviey Yes I know - my initial try was to install from stable/mitaka. But that fails.12:32
ad_rien_Daviey:  1./ compare that results obtained with the fake driver are the same ones that with real containerized compute nodes12:33
Davieyad_rien_: I mean, what is the limiting factor you are trying to find?  I mean, 1 compute stack per baremetal should be able to run the same amount of fake vm's as 10 x fake nova vm's.  Equally, i'd be suprised if running 10 VM's on the baremetal would cause much noise tot he actual test.12:33
Davieyad_rien_: why not try kolla on kolla.. Deploy an undercloud, boot 10 VM's per bare metal.. and deploy kolla on that?12:35
magnus-loofcurrently running: https://gist.github.com/maglo/59c40924b01c965a86e02fab1ac02696 That is from master -> https://github.com/openstack/kolla/commit/559631815afed3a9bf51f89b2d121b0d1a34de1212:35
Davieymagnus-loof: What are you trying to achieve ?12:36
magnus-loofI will abort and downgrade to what? 2.0.2 or stable/mitaka?12:36
ad_rien_Daviey: that can be a way to go on, indeed12:36
Davieymagnus-loof: when do you want your cloud in prod?12:36
sdake_magnus-loof - can you run pip show kolla pls12:37
magnus-loofDaviey sorry for being unclear. I want to deploy OS using kolla. For now we are just kicking the tires and rebuilding every once in a while. I want to avoid building docker 3.0.x images, due to time contraints.12:37
Davieymagnus-loof: Generally, i'm of the opinion that using stable/* for your first prod deployment is a bad idea.. I think tracking Master whilst you are in dev/test and submitting bugs/patches to make sure it does what you need (and help shape kolla) is better.12:38
*** matrohon has joined #openstack-kolla12:38
Davieymagnus-loof: And then, post release, you can go prod and track stable/newton which was the Master you knew worked12:38
magnus-loofSo therefore I set out to install OS using stable/mitaka -> Fail 2.0.2 -> Fail. However, master with openstack_release: \"2.0.2\" worked fine until yesterday.12:38
sdake_Daviey generall8yt the message we give to people evaluating kolla is to eval on stable branches12:38
Davieymagnus-loof: rebuilding all the images should normally take under an hour12:39
magnus-loofsdake_ output from pip show kolla https://gist.github.com/maglo/59c40924b01c965a86e02fab1ac0269612:39
sdake_magnus-loof - asking for third time - please run pip show kolla -> paste12:39
Davieysdake_: well, sure - kicking the tires.. but if people are commited to going into prod with it.. tracking Master during implementation phase makes more sense IMO12:39
sdake_Version: 3.0.0.0b3.dev17012:39
magnus-loofsdake_ I put the output in the gist12:40
sdake_magnus-loof if you want to run mitaka, you need a 2.0.0 version12:40
sdake_so lets get you going12:40
sdake_pip remove kolla12:40
sdake_or pip uninstall kolla12:40
Davieysdake_: The moment you need to do something a little left-field, you are stuffed if using stable/*.. but Master, you can help make it happen12:40
sdake_can never remember which one works12:40
ad_rien_Daviey: sdake_ thanks for the information, we will check on our side whether using the nested virtualisation is really an issue or not.  Thanks for your answer12:40
sdake_ad_rien_ ack12:41
magnus-loofsdake_ ok. I am aborting the deploy using kolla 3.0.0 with release 2.0.2 in globals.yml12:41
Davieyad_rien_: Nested virt on ubuntu has been the default since 12.04... it is pretty solid, espeically on intel12:41
magnus-loofsdake_ running kolla-ansible destroy -i <inventory>12:41
sdake_magnus-loof - the plan is as follows - use 2.0.2 images from docker hub with stable/mitaka12:41
sdake_magnus-loof i'll wlak you through it12:41
ad_rien_Daviey:  my doubts is related to operations such as booting VMs for instance if we want to perform scheduling performance analysis etc...12:41
*** srwilkers has joined #openstack-kolla12:42
Davieyad_rien_: Then why not test 1 kolla stack per bare metal?  I think you need to tailor your test harness for what you are testing12:42
ad_rien_our goal is to try to setup a framework that can enable researchers/engineers to evaluate particular services/components of OpenStack and see whether there are some side-effect at large scale12:43
sdake_magnus-loof pip uninstall kolla or pip remove kolla - whichever comand works proplery12:43
Davieyad_rien_: If you want to test control plane scalability, then the nested stuff works great.12:43
ad_rien_Daviey: I need more time to give relevant remarks ;)12:43
Davieyad_rien_: but if you want to test VM performance, then just launch X concurrent VM's on the same baremetal (without the nested virt)12:43
*** bootsha has quit IRC12:43
magnus-loofSuccessfully uninstalled kolla-3.0.0.0b3.dev17012:44
sdake_magnus-loof if you have a git checkout of kolla, you can git checkout stable/mitaka12:44
sdake_magnus-loof if yu dont', ou can pip isntall kolla==2.0.212:44
magnus-loofsdake git checkout stable/mitaka12:44
sdake_now in kolla home dir, run pip install -e .12:45
sdake_then pip show kolla -> paste12:45
ad_rien_Daviey: but as a distributed computing researcher, we are used to discover corner cases only at large scale…. our goal is to try to be able to detect such corner case situations. If you are using the fake driver you do not really start VMs and so can miss the corner cases we are looking for.12:45
Davieysdake_: Hmm, as root?  or --user ?12:46
magnus-loofsdake_ sudo pip install -e . Version: 2.0.3.dev1112:46
sdake_nice12:46
*** bootsha has joined #openstack-kolla12:46
sdake_magnus-loof in the tools directory there are 3 cleanup tools12:46
sdake_run all 312:46
ad_rien_Daviey:  anyway, your replies were helpful,  we need to braimstorm on our side to investigate whether nested VM would it the right way to go.12:46
Davieyad_rien_: get back to us, and importantly share your results :)12:47
sdake_magnus-loof after that docker ps -a should show nothing , docker images should show nothing12:48
magnus-loofsdake_ now I might have a clue as to why deploy does not work on 2.0.2: I do not have docker on the provisioning host. I will fix.12:48
magnus-loofsdake_ cleanup-host failed with User root can't seem to run Docker commands.12:49
*** matrohon has quit IRC12:49
sdake_magnus-loof sounds like docker isn't installed12:49
ad_rien_Daviey: sure (BTW we propose a presentation to the next summit ''Chasing 1000 nodes scale'':  let' see whether it will selected or not ;))12:50
ad_rien_(s/it will selected/it will be selected)12:50
Davieyad_rien_: I'll keep an eye out to attend it, thanks12:50
magnus-loofsdake_ yes, docker was not installed on provisioning host. It is installed on network / host / controller though12:50
magnus-loofsdake_ but now I am installing docker on provisioning host12:50
sdake_magnus-loof - my instructions wer ein error12:51
sdake_run kolla-ansible cleanup -i your-inventory-file12:51
sdake_you dont need docker on the provision host at this time12:51
sdake_hmm, cleanup is only on master12:51
magnus-loofsdake_ without it I could not run clean-host12:51
sdake_you will hae to log into each of your nodes12:52
magnus-loof[centos@operator tools]$ git describe 2.0.2-11-ge0a49d812:52
sdake_and run the 3 cleanup scripts manually in 2.0.212:52
magnus-loofsdake_ ok12:52
sdake_and for future reference, cleanup has been renamed destroy :)12:52
sdake_magnus-loof how many nodes do you have12:53
*** jtriley has quit IRC12:54
magnus-loofsdake_ I have only one control, one network and one host for now12:55
*** zhubingbing_ has joined #openstack-kolla12:56
*** zhubingbing has joined #openstack-kolla12:56
sdake_cool so not too onerous to cleanup then12:56
sdake_this is also my testbed at home12:56
*** egonzalez90 has quit IRC12:58
sbezverksdake_ qq the review from last night for rabbitmq, woudl you accept introducing rabbitmq_version variable instead of hardcoding version?12:58
magnus-loofsdake_ I think we are clean12:59
sdake_magnus-loof cool12:59
sdake_magnus-loof if not - deploy will fail and we will knwo about it12:59
sdake_sbezverk i had considered that during reiewing12:59
sdake_let me get back to you after helping magnus-loof12:59
sbezverksdake_ ok12:59
magnus-loofsdake_ ok so clear /etc/kolla on provisioning host?13:00
sdake_no13:00
sdake_but run cleanup-host on provision host13:00
magnus-loofsdake_ Creating a fstab backup... Removing ceph references from fstab... Getting folders name... Skipping: /etc/kolla/config Skipping: /etc/kolla/globals.yml Skipping: /etc/kolla/kolla-build.conf Skipping: /etc/kolla/passwords.yml13:00
*** v1k0d3n has joined #openstack-kolla13:02
magnus-loof[centos@operator kolla]$ diff etc/kolla/globals.yml /etc/kolla/globals.yml  20c20 < #openstack_release: "2.0.3" --- > openstack_release: "2.0.2" 26c26 < kolla_internal_vip_address: "10.10.10.254" --- > kolla_internal_vip_address: "192.168.1.200" 108c108 < #enable_central_logging: "no" --- > #enable_central_logging: "yes"13:02
magnus-loofso I am a little worried about 2.0.3 as it was not published on hub.docker.com this morning13:02
sdake_magnus-loof yup we are going to fix that now13:03
sdake_2.0.3 is unreleased13:03
sdake_Daviey has some ideas to clarify that - wiating on him and Jeffrey4l to powwow on that topic and come to a conclusion13:03
sdake_magnus-loof would you mind pasting your globals.yml pls13:03
srwilkersgood morning everyone :D13:04
rhalliseysrwilkers, morning!13:04
magnus-loofsdake_ globals.yml: https://gist.github.com/maglo/4b880f00d5cce70d29361bc3537fca5d13:04
sdake_Daviey this version pbr stuff is becoming an emerging problem - can you drive it to conclusion for the stable branches?13:05
sdake_sup srwilkers13:05
sdake_magnus-loof are you on centos?13:06
*** matrohon has joined #openstack-kolla13:06
magnus-loofsdake_ yes13:06
sdake_for your host os13:06
sdake_magnus-loof eth0/eth1 are not your ethernet interfaces13:06
sdake_atleast those are not the default ethernet interfaces on centos13:06
magnus-loofthey are on my cloud (Ravello)13:06
sdake_oh your deploying on vms?13:06
magnus-loofsdake_ yes, and on vagrant13:07
sdake_magnus-loof have to warn ahead of time - that may not work - kolla uses multicast to determine the VIP - and some clouds don't support multicast13:07
sdake_magnus-loof but lets try anyway13:07
sdake_let me finish reviewing your globals.yml13:07
sdake_magnus-loof what is located at the .200 address?13:08
magnus-loofsdake_ I think Ravello will work for that: https://www.ravellosystems.com/solutions/openstack/kvm-on-aws-technology13:09
magnus-loofsdake_ there is nothing on .200 now. It will be claimed by haproxy13:09
sdake_nice13:09
sdake_ok pull the trigger on the deploy13:09
sdake_time for a cigar13:09
*** berendt has joined #openstack-kolla13:09
*** diogogmt has joined #openstack-kolla13:10
magnus-loofsdake_ I think we cleaned a little too much: ERROR! the playbook: /usr/share/kolla/ansible/site.yml could not be found13:10
berendtPlease fast forward https://review.openstack.org/#/c/35340813:10
patchbotberendt: patch 353408 - kolla - Revert "Remove files from /var/lib/apt/lists when ...13:10
sdake_magnus-loof moment to think13:11
sdake_magnus-loof there is some pip command that lists files in a package13:11
sdake_magnus-loof probably pip list kolla or soething similiar -> paste13:11
*** jtriley has joined #openstack-kolla13:12
sdake_magnus-loof - I think my instructions were in error13:12
sdake_pip remove kolla13:12
sdake_or pip unisntall kolla13:12
sdake_then sudo pip install . from the kolla tlD13:13
sdake_TLD13:13
sdake_we don't want the -e in centos apparently :)13:13
sdake_apologies for the churn13:13
magnus-loofsdake_ Aargh. Ravello stopped my application ( I had set it to run for 4 hrs). A few minutes and I will be back!13:15
sdake_roger13:15
sdake_ping me when your ready to roll13:15
sdake_sbezverk - version # hardcode in ansible scripts is better then version # hardcode in docker scripts13:16
sdake_sbezverk where would you hardcode it ?  all.yml?13:16
magnus-loofsdake_ OK we are back13:17
sdake_sbezverk lets put that convo on hold while you think about that question13:17
*** matrohon has quit IRC13:17
sdake_magnus-loof - ok were you able to get the pip install . in the kolla TLD?13:17
sdake_after remvoing kolla13:18
*** sdake_ is now known as sdake13:18
magnus-loofsdake_ it is running now...13:18
sdakedeploy or pip?13:18
magnus-loofsdake_ Successfully installed kolla-2.0.3.dev1113:19
sdakeok pull trigger on the deploy now13:19
* sdake just learned something new13:20
magnus-loofsdake_ running...13:20
sbezverksdake_ yes13:20
sdakesbezverk this hardcode would not go in globals.yml?13:20
sbezverksdake it is not up to the user to change this version13:21
sdakepbourke coolsvap are u guys off for the evening?13:21
magnus-loofsdake OK so now I am expecting to see https://gist.github.com/maglo/be4a0c039dfd5d02fdc4e45c41f31c3613:21
sbezverkso exposing it in globals.yml is not the best approach imho13:21
magnus-loofunless the installation of docker on the provisioning host rectified the problem.13:21
pbourkesdake: im around but dont have a huge amount of time to do osic stuff13:21
sdakemagnus-loof i would expect it to work13:21
pbourkefor today13:21
sdakepbourke roger13:22
magnus-loofsdake me too! but BOOM we are stuck at the same place13:22
sdakepbourke can you give a quick status update for today13:22
magnus-loofNoneType13:22
sdakemagnus-loof can i get a paste o the log13:22
sdakeof the log13:22
sdakemagnus-loof ansible --version13:23
magnus-loofsure, what logfile?13:23
*** harlowja has quit IRC13:23
magnus-loof[centos@operator kolla]$ ansible --version ansible 2.1.0.0   config file = /home/centos/.ansible.cfg   configured module search path = Default w/o overrides13:23
sdakepbourke related to osic13:23
sdakemagnus-loof one thing the documetnation doesn't mentoin is ansible 1.9.4 is required for stable branches13:23
magnus-loofah13:23
sdakepip install ansible==1.9.413:24
pbourkesdake: from what I can see we are ready for scenario 313:24
sdakepbourke did we get the disk io benchmarks for ceph?13:24
sdakewe being you or coolsvap13:24
pbourkesdake: I dont see them on the doc, I'll let coolsvap respond on that one13:25
sdakepbourke roger13:25
magnus-loofsdake I had ansible installed from repo. which one is preferred? repo or pip?13:25
sdakepbourke - i will gather them if you can give me a host to log in to with a vm rolling in it13:25
sdakemagnus-loof sudo yum remove ansible13:26
pbourkesdake: 172.21.8.253 is the deploy host13:26
magnus-loofansible 1.9.4   configured module search path = None13:26
sdakemagnus-loof obviously packaged versions from distros are better, but this is just a limitation people have to live with because ansible doesn't support 1.9.z13:26
pbourkesdake: 'tmux attach' on that, no VMs running at the moment but you could spawn one13:26
sdakepbourke i can look in inventory from there13:26
sdakepbourke thanks again for all your super hard work on this effort13:27
sdakepbourke this work is super critical13:27
sdakeas important as the bifrost work13:27
magnus-loofsdake: ok there is a difference ansible 1.9.4 does not recognize .ansible.cfg as 2.1 was configured to...13:27
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Fix interface address for RabbitMQ and disable cluster for Kubernetes  https://review.openstack.org/33675113:27
sdakemagnus-loof save a copy of that file and move it out of the way13:28
sdakeansible will create a new one13:28
sbezverksdake even with variable is not ideal, becauase we hardcode a specific plugin.13:28
sdakemagnus-loof thanks for  sticking with us :)13:28
magnus-loofsdake thank YOU for helping me.13:29
sdakethe specific plugin is hardcoded ina docker file13:29
magnus-loofrun kolla-ansible deploy13:29
magnus-loof?13:29
sdakemagnus-loof your welcome for the support :)13:29
sdakemagnus-loof yes deploy13:29
coolsvapsdake: I'm afk for hour or so13:29
sdakesbezverk thats an orthognoal problem13:29
sdakeif people complain in the review, i'll say as much13:30
sdakecoolsvap ack - did you gather the io benchmarks from a vm?13:30
sdakecoolsvap and havne't publislhed?13:30
sdakeor need me to do so13:30
sbezverksdake: how about assigning rabbitmq_version: the whole thing: rabbitmq_server-3.5.7/plugins/rabbitmq_clusterer-3.5.x-189b3a81.ez/rabbitmq_clusterer-3.5.x-189b3a81/ebin13:30
sbezverkin this case it will be more flexible13:31
*** srwilkers has quit IRC13:31
sdakesbezverk bounce that idea off the review qeue and see what the cr's have to think about it13:31
coolsvapI haven't done13:31
sdakecoolsvap ok i've got it up next after wrapping up with magnus13:31
sdakesbezverk recommend rabbitmq_plugin_path13:32
openstackgerritDave Walker proposed openstack/kolla: Add --user-domain when adding heat role admin  https://review.openstack.org/35347113:33
*** mntdevops has joined #openstack-kolla13:33
sdakesbezverk the issue is the plugin path is needed, not hte version # is needed13:34
magnus-loofsdake: so far so good13:34
*** matrohon has joined #openstack-kolla13:34
sdakemagnus-loof yup i'd expect unicorns and rainbows from here on out :)13:34
sdakemagnus-loof just to recap what we did:13:35
sdake1. we are using the stable 2..0.2 images on docker hub13:35
sdake2. we are using the stable/mitaka ansible code and extra other stuff required by kolla13:36
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Fix interface address for RabbitMQ and disable cluster for Kubernetes  https://review.openstack.org/33675113:36
sdakestable/mitaka is pretty close to 2.0.213:36
magnus-loofbut i still need to sed 2.0.3 -> 2.0.2 until they are released, right?13:36
sdakeuntil we release our mitaka z stream whcih happens every 45 days13:36
magnus-loofsweet13:37
sdakerhallisey can you rebase your docs patch please13:37
*** bootsha has quit IRC13:37
sdakerhallisey so we can get this recurring problem fixed13:37
rhalliseyya13:38
*** bootsha has joined #openstack-kolla13:38
sdakeyou know when you get an answer back "its complicated" from someone of high authority, its complicated :)13:38
sdakemagnus-loof just to be clear, then you will be seding 2.0.4 to 2.0.3 until Daviey and Jeffrey4l come to conclusion on how to fix this pbr problem13:39
openstackgerritVladislav Belogrudov proposed openstack/kolla: iscsi initiator names must be unique among hosts  https://review.openstack.org/35348313:40
*** matrohon has quit IRC13:41
*** rcherrueau has quit IRC13:41
sdakemandre ping13:41
mandrehey sdake13:42
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Fix intf address for RabbitMQ and disable cluster for Kube  https://review.openstack.org/33675113:42
*** srwilkers has joined #openstack-kolla13:42
Davieysdake: you know... we should probably recommend people install from pip on pypi rather than git... your morning might have been easier :)13:46
magnus-loofsdake all work and no play makes jack a dull boy13:46
sdakeDaviey roger that - hence the request for rhallisey to rebae his docs change13:47
sdakeif nobody else wll fix this problem i guess I'll fix it13:47
Davieychop chop rhallisey13:47
*** diogogmt has quit IRC13:47
*** zhubingbing has quit IRC13:47
*** zhubingbing_ has quit IRC13:47
rhalliseysdake, this was a confusing rebase13:47
*** rkrum has joined #openstack-kolla13:47
*** zhubingbing__ has joined #openstack-kolla13:47
*** zhubingbing has joined #openstack-kolla13:47
rhalliseylet's merge before I have to do more of these13:48
* rhallisey almost done13:48
sdakerhallisey make sure it renders correctly13:49
rhalliseyare there any tools to test in rendered?13:50
Davieyrhallisey: sphinx ?13:50
rhalliseyk13:51
sdaketox -e doc13:52
sdakeor tox -e docs rather13:52
*** ayoung has joined #openstack-kolla13:53
openstackgerritVladislav Belogrudov proposed openstack/kolla: iscsi initiator names must be unique among hosts  https://review.openstack.org/35348313:54
sdakebattery at 2% - if I drop off i'll be back13:55
openstackgerritDave Walker proposed openstack/kolla: Add --user-domain when adding heat role admin  https://review.openstack.org/35347113:57
*** matrohon has joined #openstack-kolla13:58
magnus-loofsdake TY so much for your help. kolla 2.0.2 with ansible 1.9.4 works :-D13:59
*** JoseMello has quit IRC14:00
wirehead_sdake: So, chatted with the SAP people.  There seems to be some cultural baggage and hard feelings.14:02
sdakehard feelings about what?14:02
rhalliseywhat does that mean O>o14:03
sdakewe did them no wrong14:03
sdakemagnus-loof happy to help support you in your ODM journey :) - enjoy14:03
wirehead_They felt like they treid to get their patches merged but got nowhere, sometime last year.14:03
sdakewirehead_ i dont recall any sap patches at all14:04
rhalliseyI don't think they ever submitted any14:04
sdakewe included sap in our design sessions and have reached out numerous times to coordinate efforts14:04
sdakebut thats in the past, anything we can do to repair those hard feelings?14:05
Davieysdake: Is there a spec/bp for SAP?14:07
sdakeDaviey we included sap engineers in the original kolla-kubernetes review14:08
wirehead_Not sure yet.  Will talk to the various people some more.14:08
sdakeand asked them to review the spec and provide their unique insight14:08
rhalliseythat's stange14:08
Davieysdake: would be nice to understand the scope of potential SAP integration14:08
wirehead_Yeah, like I got an earfull out of the guy.14:08
sdakeanyone that reviewed the spec got an automatic cr spot on the kolla-kubernetes team14:08
wirehead_The concern was mostly Kolla patches, not Kolla-Kubernetes patches.14:09
*** zhubingbing_ has joined #openstack-kolla14:09
sdakei see14:09
*** zhurong has quit IRC14:09
sdakeour review constraints have become more relaxed since last year14:09
openstackgerritVladislav Belogrudov proposed openstack/kolla: iscsi initiator names must be unique among hosts  https://review.openstack.org/35348314:10
wirehead_Oh, and I’m going to talk to one of the folks from The New Stack about Kolla-Kubernetes.14:10
*** bootsha has quit IRC14:10
*** kong has quit IRC14:10
sdakewirehead_ if you get a chance to speak with the sap dude that gave you the earful, let him know themakeup of the community has changed, and their patches are welcome14:11
sdakewhat is the new stack14:11
wirehead_http://thenewstack.io/14:12
*** zhubingbing has quit IRC14:12
*** zhurong has joined #openstack-kolla14:12
*** matrohon has quit IRC14:13
sdakewirehead_ its better if you dont linke me those sorts of things14:14
sdakethanks ;)14:14
*** zhubingbing__ has quit IRC14:14
*** rkrum has quit IRC14:15
*** zhubingbing__ has joined #openstack-kolla14:16
*** magnus-loof has quit IRC14:19
*** srwilkers has quit IRC14:19
*** matrohon has joined #openstack-kolla14:30
*** athomas has quit IRC14:32
sdakefuture leaders -> http://www.cnn.com/2016/08/09/africa/orange-drought-kiara-nirghin/index.html14:32
*** mnasiadka has quit IRC14:32
*** ad_rien_ has quit IRC14:35
*** srwilkers has joined #openstack-kolla14:36
openstackgerritKirill Shileev proposed openstack/kolla: Customizations for Glance  https://review.openstack.org/35297714:38
berendtteam meeting today at 16 UTC?14:39
*** ppowell has quit IRC14:39
sdakeberendt roger14:41
*** duonghq has joined #openstack-kolla14:42
*** athomas has joined #openstack-kolla14:43
sdakeevening athomas - how ya been14:44
athomassdake, Hi. I'm pretty good, all in all.14:44
athomassdake, You?14:44
*** matrohon has quit IRC14:44
sdakegood - changing the world is hard but rewarding ;)14:44
*** daneyon_ has joined #openstack-kolla14:49
*** zhurong has quit IRC14:49
*** senk_ has quit IRC14:51
*** daneyon_ has quit IRC14:53
*** ppowell has joined #openstack-kolla14:56
*** matrohon has joined #openstack-kolla14:59
*** ayoung has quit IRC15:01
*** b_bezak has quit IRC15:03
*** b_bezak has joined #openstack-kolla15:04
*** vhosakot has joined #openstack-kolla15:04
*** sdake_ has joined #openstack-kolla15:04
*** Serlex has quit IRC15:04
mntdevopswanted to say "thank you" again to all of you for this project.  I wrote up my efforts on my blog: http://mntdevops.com/2016/08/08/iaas-2/15:05
rhalliseysdake_, doc patches are up15:05
mntdevopsI have my all-in-one setup going -- next steps are to make it multi-node15:05
sdake_mntdevops cool - thanks for the (rare in the tech industry) positive feedback15:06
rhalliseymntdevops, neat!15:06
mntdevopsI've watched all your summit talks maybe 3 times15:06
mntdevopsand I keep waiting for the next ones15:07
*** sdake has quit IRC15:07
sdake_mntdevops we depend on the track chairs to accept our talks15:07
sdake_no kolla talks were accepted in austin15:07
mntdevopskinda hoping for a how-to webinar or workshop things15:07
mntdevopsbummer15:07
mntdevopsfor me15:08
mntdevopsI think the hardest part15:08
sdake_who knows what will happen with barcelona - results not out yet15:08
mntdevopswas getting the "prep" stuff in the right order15:08
*** b_bezak has quit IRC15:08
mntdevopsand a working globals.yml15:08
mntdevopsso I rolled them into an ansible role15:09
mntdevopswell, I hope a kolla talk is accepted at some point, because while Openstack Ansible is good15:09
mntdevopsI prefer docker to lxc15:09
*** vhosakot_ has joined #openstack-kolla15:11
mntdevopsoh, two questions:  #1 Have you considered ansible-container for creating the containers (instead of kolla-build)?  and #2 What are the reasons for not having as many pre-built releases on docker hub?15:12
*** david-lyle has quit IRC15:12
*** vhosakot has quit IRC15:12
*** ayoung has joined #openstack-kolla15:12
*** david-lyle has joined #openstack-kolla15:12
*** senk_ has joined #openstack-kolla15:14
mntdevopsI know ansible-container is very new, but just wanted to say that it's promising in my usage15:15
mntdevopsand #2 isnt' a dig15:15
mntdevopsit's just that building the images slows down the initial getting started quite a bit15:16
rhalliseymntdevops, it was considered.  I think it requires some major changed on kolla's end15:16
*** fragatina has joined #openstack-kolla15:16
*** fragatina has quit IRC15:16
rhalliseymntdevops, #2 kolla pushes per release15:16
mntdevopsrhallisey, yeah, I looked at all the nested loops in the jinja templates for the Dockerfiles15:16
sdake_sbezverk can you lend me a hand with neutron15:16
rhalliseymntdevops, it's possible not all releases are there atm15:17
mntdevopsand thought, "man, this would be great with roles/tags"15:17
*** fragatina has joined #openstack-kolla15:17
mntdevopsright15:17
mntdevopsfor instance, centos binary 2.0.315:17
mntdevopsstable/mitaka references 2.0.3 IIRC in the example globals.yml15:18
sdake_2..0.3 is not yet releaed, hence we hven't pushed it15:18
sdake_we odnt push per commit15:18
*** senk_ has quit IRC15:19
mntdevopsah15:19
sdake_re ansiblec-ontiner, jeffrey4l ha a spec  in the queue for that concept15:19
sdake_but atm its too late to do any sort of that rework for mitaka15:19
mntdevopscertainly understand that.  The whole build process would get overhauled there15:20
sdake_the fact that stablle/mitaka refrences 2.0.3 is to deal with a bug in pbr15:20
sdake_or a bug in how we use pbr15:20
sdake_not sure which15:20
Jeffrey4lit is not a bug. it is how the pbr works.15:21
Jeffrey4lwe are using it in a wrong.   because the ansible playbook require that.15:21
*** jrist has quit IRC15:22
sdake_ya - what daviey had proposed i think was to use the full naem inclding dev-xx15:22
sdake_then we could make it work properly in all conditions15:22
Jeffrey4lyep. using devxx is fine.15:22
*** jrist has joined #openstack-kolla15:23
*** srwilkers has quit IRC15:24
sdake_mntdevops read throug hyur blog looks pretty good15:28
sdake_note we are working on a kolla-ost playbook which is jut about done 9sean-k-money actually) which instals the necessary host dependencies15:28
*** haplo37__ has joined #openstack-kolla15:29
*** mark-casey has joined #openstack-kolla15:32
*** senk_ has joined #openstack-kolla15:32
*** srwilkers has joined #openstack-kolla15:34
dimssdake_ : rhallisey : inc0 : seen this? https://github.com/sapcc/openstack-kube15:37
sdake_dims yes15:38
dimscool15:38
kfox1111so many parallel implementations. :/15:38
*** senk_ has quit IRC15:39
kfox1111interesting they use kolla containers though.15:39
kfox1111has anyone talked with them about joining forces?15:40
kfox1111seems like they implemented it for their production system, then just released it.15:40
kfox1111they may not be so interesting in carrying it forever if thats the case.15:40
sdake_need to switch networks brb15:43
*** sdake has joined #openstack-kolla15:45
*** b_bezak has joined #openstack-kolla15:45
*** zhubingbing__ has quit IRC15:46
*** zhubingbing_ has quit IRC15:46
*** sdake_ has quit IRC15:48
*** matrohon has quit IRC15:50
berendtsdake, dims : how takes care of the SAP fork? wirehead_ is already in contact with the guys responsible for it. I think this should be coordinated. SAP has an OpenStack evangelist reponsible for such topics.15:51
*** b_bezak has quit IRC15:51
berendtThe fork is initiated from a project on a SAP site in Berlin.15:52
*** b_bezak has joined #openstack-kolla15:52
berendtThere was also a talk proposal for the summit in Barcelona presenting this environment.15:52
berendtAt the moment they do violate the used license. sapcc/openstack-kube includes Kolla code and they do not use the Apache2 license there. This really sucks.15:53
kfox1111fascinating.... one of the kube folks recommended using a side car container to do configmap templating instead of doing it in kube itself.15:53
rhalliseydims, interesting15:54
kfox1111that could totally work to assembe a config map and a secret into a config a kolla container could accept.15:54
rhalliseykfox1111, they have a summit talk15:54
kfox1111we just need a jinja container now... :)15:55
berendtThe talk is "Baremetal to Openstack on Kubernetes from 0 to 100", proposed by Michael Schmidt15:55
*** b_bezak has quit IRC15:56
*** egonzalez has joined #openstack-kolla15:59
*** inc0 has joined #openstack-kolla16:00
berendtmeeting time?16:01
inc0yup16:01
*** vhosakot_ has quit IRC16:01
*** vhosakot has joined #openstack-kolla16:03
Jeffrey4lberendt, any feedback? https://review.openstack.org/34843216:03
berendtJeffrey4l will check after the meeting16:04
Jeffrey4lnp thanks.16:04
*** diogogmt has joined #openstack-kolla16:07
*** britthouser has joined #openstack-kolla16:08
*** egonzalez has quit IRC16:08
*** egonzalez90 has joined #openstack-kolla16:11
*** ppowell has quit IRC16:12
*** janki has joined #openstack-kolla16:15
dimsberendt : rhallisey : no clue who takes care of that repo. i saw it float by (openstack SV?)16:19
berendtI want to discuss this in the meeting16:19
*** rmart04 has quit IRC16:19
*** unicell1 has joined #openstack-kolla16:21
*** unicell has quit IRC16:23
*** vhosakot has quit IRC16:26
*** vhosakot has joined #openstack-kolla16:26
*** dave-mccowan has joined #openstack-kolla16:30
pbourkeinc0: ceph is there so cinder should... just work?16:31
inc0yeah16:31
inc0in fact if we just run deploy -t cinder now16:31
inc0it should just work16:31
inc0without repave16:31
pbourkedoing16:31
*** janki has quit IRC16:34
openstackgerritsean mooney proposed openstack/kolla: add baremetal role to install kolla deps.  https://review.openstack.org/32563116:34
openstackgerritsean mooney proposed openstack/kolla: This change allow setting kernel commandline args  https://review.openstack.org/35141216:34
*** janki has joined #openstack-kolla16:35
*** daneyon_ has joined #openstack-kolla16:37
pbourkeinc0: 20 mins may be great deploy time but sucks waiting for one service16:39
pbourkeinc0: want to kill and rerun with --tags?16:39
inc0pbourke, let's just let it go16:40
pbourkekk16:40
inc0it's on nova alreadt16:41
*** daneyon_ has quit IRC16:41
*** unicell1 has quit IRC16:42
berendtJeffrey4l done16:46
Jeffrey4l thanks.16:47
inc0lol nyancurses-cat ;)16:51
*** rmart04 has joined #openstack-kolla16:51
pbourkelook at him go16:53
*** msimonin has quit IRC16:59
inc0pbourke, kibana is up16:59
pbourkehttp://172.21.8.200:5601/ ?17:00
coolsvappbourke: I have got a list of scenarios prepared I will add the steps for rally17:00
pbourkecoolsvap: thanks!17:00
pbourkeinc0: above url not working for me17:00
inc0duh17:00
Jeffrey4linc0, could u allocate some time for the osic?17:01
*** ppowell has joined #openstack-kolla17:01
inc0Jeffrey4l, you mean work on it?:)17:01
Jeffrey4linc0, correct.17:01
inc0I'm doing it as we speak:)17:01
inc0but will have more time next week17:01
Jeffrey4li am in Asia. When the osic is free, we can use based on the geo.17:02
coolsvapguys I need to give some rest to my (wounded) hand, or I will not be able to work tomorrow :/17:02
coolsvapJeffrey4l: we can work together17:03
pbourkeinc0: should reconfigure respect --tags ?17:03
inc0coolsvap, Jeffrey4l sure guys, take a rest17:03
Jeffrey4lnp.17:03
inc0pbourke, yes17:03
DavieyJeffrey4l: can you tak another look at my standalone example of what i was talking about?17:03
pbourkeseems its not17:03
Jeffrey4lpbourke, it respect.17:03
Jeffrey4lDaviey, Ok17:03
DavieyJeffrey4l: I think the standalone example makes it pretty clear what i am intending17:03
inc0pbourke, maybe it's good thing tbh, as lots might have to change after what we did:)17:04
Jeffrey4lDaviey, could you paste the link again? I lost it.17:04
Jeffrey4lpbourke, what happen? logs?17:04
Jeffrey4lit should. if it not, it is buggy.17:04
pbourkeinc0: have to run now, will catch up tomorrow. Might be good to throw whatever kibana queries you end up using to diagnose cinder into the review doc. All info helps17:05
DavieyJeffrey4l: see the two scenarios in README, https://github.com/Daviey/tmp-kolla-version17:05
inc0pbourke, I'm on it, thanks for help17:05
DavieyJeffrey4l: one scenario is "trust upstream and use the latest version number", the other is defining what version number people want hardcoded)17:05
pbourkeinc0: good luck!17:05
berendt2 +2 on https://review.openstack.org/#/c/306928/.. lets merge?17:05
patchbotberendt: patch 306928 - kolla - Replace horizon default config with custom config17:05
*** ayoung has quit IRC17:06
Jeffrey4lDaviey, i think it is fine. please push a PS.17:08
*** dmueller-evoila has quit IRC17:08
Jeffrey4land use 'latest' as the openstack_release value.17:08
Jeffrey4lin all the branch.17:08
Jeffrey4lalso need change the ./tools/build.py --version logical, to use the pbr.release_version rather than the cached_version17:09
Jeffrey4lDaviey, ^17:09
duonghqJeffrey4l: iirc somebody mention 'current' as the openstack_release value17:10
*** rmart04 has quit IRC17:10
duonghqso 'latest' and 'current' is same or 2 meanings?17:10
*** berendt has quit IRC17:11
Jeffrey4lduonghq, hmm. no clue.   nova api microversion use the word ` latest`, and i like this word.17:11
Jeffrey4lanyway, it is not a big deal.17:11
duonghqokay, just need final decision17:11
Jeffrey4lduonghq, Daviey please push a PS to fix this based on your idea. :)17:12
duonghqah, anybody deploy kolla successful with Swift enable?17:12
sean-k-mooneyJeffrey4l: regarding https://review.openstack.org/#/c/352089 is there anyone working on enabling ubuntu 16.04 for the contaier os currently17:13
patchbotsean-k-mooney: patch 352089 - openstack-infra/project-config - Enable ubuntu-xenial for Kolla17:13
Jeffrey4lduonghq, I never deployed that. :(17:13
Jeffrey4lsean-k-mooney, i am.17:13
Jeffrey4lsean-k-mooney, https://review.openstack.org/32945417:13
Jeffrey4lbut it is block by the host os kernel version.17:13
sean-k-mooneyJeffrey4l: oh ok cool.17:13
DavieyJeffrey4l: great, i'll try and do it tomorrow17:13
Jeffrey4lDaviey, thanks.17:13
jankihey, I could like to help with any bugs17:14
Jeffrey4lduonghq, do u find anything about the swift?17:14
DavieyJeffrey4l: Incidentally, we probably need an early task to check that all images are available for required projects at the start of each run.. currently pretty crappy error message if we dont17:14
Jeffrey4lDaviey, pull action should work.17:15
duonghqJeffrey4l: I got trouble in deploy config task, I'll try to fix it tomorrrow, just wonder if it only happen to me.17:15
Jeffrey4ljanki, pick one from the bug list https://launchpad.net/kolla/+milestone/newton-317:15
DavieyJeffrey4l: hmm17:15
Jeffrey4lwe have lost of bugs.17:15
Jeffrey4lduonghq, cool17:16
duonghqIt seem that automation testing for Ansible is really hard17:16
jankiJeffrey4l: sure, will do. Thanks17:17
Jeffrey4lduonghq, it is. official do not have such solution.17:17
duonghqyeah17:18
*** inc0 has quit IRC17:19
*** ayoung has joined #openstack-kolla17:20
*** unicell has joined #openstack-kolla17:22
*** lrensing_ has joined #openstack-kolla17:26
*** harlowja has joined #openstack-kolla17:27
*** dwalsh_ has joined #openstack-kolla17:29
*** duonghq has quit IRC17:30
*** dwalsh has quit IRC17:31
*** dwalsh_ has quit IRC17:35
sbezverkJeffrey4l: ping17:35
Jeffrey4lsbezverk, pong17:35
Jeffrey4lsbezverk, need talk about this https://review.openstack.org/35178417:35
Jeffrey4lsorry for the delay.17:36
*** janki has quit IRC17:36
sbezverkJeffrey4l: sure when I have rabbitmq and neutron up I will come back to it17:37
Jeffrey4lok.17:37
sbezverkJeffrey4l: I wanted to check with you about vcpus thing which keeps breaking rendering on kubernetes side17:37
sbezverkI pinged you about that few days back17:38
Jeffrey4lsorry. not always online.17:38
Jeffrey4lso what' the detail?17:38
Jeffrey4li saw you fix patch.17:39
sbezverkthis line causes issues on kube side17:39
sbezverkopenstack_service_workers: "{{ [ansible_processor_vcpus, 5]|min if orchestration_engine == 'ANSIBLE' else '1'}}"17:39
Jeffrey4lyes. But your fix should fix it, right? sbezverk17:40
sbezverkit is in all.yml, we have to comment it out to be able to use kolla-kubernetes cli17:40
sbezverknope, I do not have fix for this17:40
Jeffrey4ldoes this break the k8s now?17:41
*** egonzalez90 has quit IRC17:41
sbezverkJeffrey4l: yes17:41
sbezverkthe workaround is easy but annoying17:41
Jeffrey4lwhat's the workaround?17:42
mntdevopssdake, thanks for looking over my blog.  Hope to continue working on it as time permits17:42
Jeffrey4li thought this PS fix it https://review.openstack.org/#/c/350356/17:42
patchbotJeffrey4l: patch 350356 - kolla - Change workers to be static when using kubernates (MERGED)17:42
sbezverkJeffrey4l: Commenting this line17:43
Jeffrey4lOK.17:43
*** inc0 has joined #openstack-kolla17:43
sbezverkhmm, really strange I hit that issue yesterday17:43
sbezverklet me run more tests17:44
*** senk_ has joined #openstack-kolla17:44
*** senk_ has quit IRC17:45
Jeffrey4lsbezverk, OK.17:45
*** dwalsh_ has joined #openstack-kolla17:46
*** egonzalez90 has joined #openstack-kolla17:47
inc0ok, I know what's wrong with cinder17:50
Mech422Anyone know what I should be googling if I want to control Openstack 'regions'(?) in multiple DCs from a single control plain? Is that cells ? availability zones ?17:51
inc0volume_type cannot be None17:51
inc0didn't we fix this one?17:51
Mech422inc0: it used to be defined to a non-existant volume type17:51
kfox1111regions are a totally seperate thing from the rest of those.17:51
inc0Mech422, so regions is single keystone, multiple novas and such17:51
kfox1111regions assume a seperate control plane.17:51
inc0cells are single everything besides nova, but they don't work very well as of now17:51
Mech422inc0: sounds like they the definition of it got removed, but not the usage ?17:51
jrich523where do the horizon logs end up? from what i can tell all the logs should be /var/log/kolla, but that folder isnt even there17:51
kfox1111you probably want either host aggregates, az's or cells.17:52
kfox1111though, cells really are a seperate control plane...17:52
inc0jrich523, try /var/lib/docker/volumes/kolla_logs/_data17:52
jrich523thanks17:52
inc0kfox1111, do you know anyone running cells with neutron tho?17:52
kfox1111neutron doesn't support them. you have to share a neutron with cells as far as I know.17:53
Mech422kfox1111: how about storage? can I put a local (small) ceph cluster in each DC and have the DC use that one to prevent long net traversals for i/o ?17:53
kfox1111cells are just a way to scale out nova a bit further.17:53
kfox1111as far as I know, no.17:53
Mech422I think I want something like the eucalyptus 'cluster' idiom...17:53
kfox1111regions really were designed around datacenters.17:54
kfox1111each datacenter getting a region.17:54
kfox1111totally seperate.17:54
kfox1111cause often you loose a datacenter at a time.17:54
Mech422kfox1111: yeah - that what I thought, but boss wants to know if there's any 'single mgmt' options..17:54
inc0"often you lose datacenter at a time"...uhh17:54
Mech422kfox1111, inc0 : thanks - I'll keep googling17:55
inc0how often is "often"?;)17:55
kfox1111sure. just depends on how much pain you want to take apon yourself. :)17:55
inc0Mech422, so regions are kinda single mgmt17:55
inc0tho17:55
kfox1111inc0: had an eagle drop a rabbit on a power transformer the other week. ;)17:55
inc0as you have single ks17:55
inc0and ks is managing endpoints well17:55
kfox1111you'd think the power folks would learn by now to put up some plastic shields... :)17:55
inc0so for end user it's pretty convenient17:55
inc0lol17:56
kfox1111sharding keystone in such a way that its multidata center resilient is fairly easy.17:56
wirehead_As far as I can tell, based on swapping stories with my mom who programmed mainframes, when they made the first mainframe, they also invented the truck-magnet.  Because data centers have been taken out by trucks since there were datacenters.17:56
kfox1111its harder for the other services.17:56
kfox1111just use fernet tokens and make sure you have one mariadb replica at each  of 3+ dc's.17:56
wirehead_Um, ask the rackers about the great data center truck incident, inc0. :D17:56
inc0will do wirehead_17:57
* kfox1111 chuckles17:57
kfox1111what can go wrong, will go wrong. :)17:57
*** berendt has joined #openstack-kolla17:57
inc0I have one story about the fastest DC migration in Polish history17:57
inc0less than 2 weeks17:57
Mech422sounds like regions is the place to start then...I'll dig into them more17:57
kfox1111so, yeah. you can span a region across multiple data centers.... even across the world. I just don't recommend it.17:57
inc0flood in old dc makes you move quicker17:57
wirehead_There’s ALWAYS a t-junction adjacent to a datacenter and trucks ALWAYS manage to miss that it’s a T-junction.  And then you put up bollards and the trucks knock the bollards over.17:57
*** daneyon_ has joined #openstack-kolla17:57
Mech422couple of years ago, when the gpu compute craze hit - we actually lost an air handler and setoff the sprinklers in a DC with all the gpu's we had packed in there17:58
inc0and yes, servers were prioritized based on rack position - lower ones went first;)17:58
wirehead_And, frankly, infra makes that kinda easy.  If the control plane for data center A lives in data center A and it all goes down, no harm no fowl.17:59
kfox1111we had an incident where some rear door chillers had a pipe bust.17:59
kfox1111hosed down the datacenter. :/17:59
kfox1111was on the top of the rack, so it flew good, and was a water/glycol mix, so not only wet, but sticky. :/18:00
*** ad_rien_ has joined #openstack-kolla18:01
*** daneyon has quit IRC18:01
Mech422kfox1111: yeah - I'm actually surprised we had water based fire suppression...but the story goes we had rusty standpipe water flying out the sprinklers and down all the racks18:02
*** senk_ has joined #openstack-kolla18:03
kfox1111us too. they value employees over machines, which is good. :)18:04
Mech422kfox1111: water + elect doesn't sound like they value much of anything to me....18:05
Mech422except maybe BBQ ?18:05
*** berendt has quit IRC18:05
sean-k-mooneyhalon is not that friendly to machines either but they react much better then humans18:05
kfox1111yeah, but you can breath long enough to escape more often. :)18:05
*** rmart04 has joined #openstack-kolla18:05
Mech422kfox1111: "cake or death" ? "suffocate or electrocute" ?18:06
*** huikang has joined #openstack-kolla18:07
kfox1111either way your having a bad day. :)18:07
*** fragatina has quit IRC18:07
Mech422sean-k-mooney: is there much halon left?  I thought they only had a 5 year grace to replace them when they were banned ?18:07
sean-k-mooneyi would guess most halon has been replaced with co2 or other gas based suppresion systmes. the ozone layer like halon even less then humans18:08
*** ad_rien_ has quit IRC18:09
Mech422sean-k-mooney: yeah - I know it got banned back in the 90s ...  I worked on wall st. in a halon 'protected' facility back then18:09
*** jemcevoy has joined #openstack-kolla18:10
Mech422same place had motion sensors inside the DC to unlock crash bars...18:10
sean-k-mooneyi mean if it wasnt for the increased uv form the impact to the ozone layer we would probably still be using it. it worked very well18:10
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Fix bad merge of ceph mariadb  https://review.openstack.org/35366918:10
Mech422so you could get in the DC by putting a piece of paper under the door, and flicking it with a finger so it tripped the motion sensor18:11
*** daneyon has joined #openstack-kolla18:11
sean-k-mooneyMech422: feature or bug :)18:11
Mech422sean-k-mooney: heh - depends on who you ask...the boss wasn't very happy about his 'secure' DC being breached with a piece of photo copy paper18:12
*** berendt has joined #openstack-kolla18:12
*** berendt has quit IRC18:13
jemcevoysdake, I am still stuck when ceph creates osd's on stable/mitaka.  Is there anyone who can help me past this?  I errors I am seeing are here: http://paste.openstack.org/show/553622/18:13
*** daneyon_ has quit IRC18:14
*** rmart04 has quit IRC18:14
*** senk_ has quit IRC18:14
*** ad_rien_ has joined #openstack-kolla18:14
Mech422jemcevoy: your using the same device for both a full disk OSD data partition AND its journal ?18:15
*** ad_rien_ has quit IRC18:15
Mech422jemcevoy: if your doing full disk ceph setup, IIRC kolla does the journal setup for you18:15
Mech422jemcevoy: (also, I'm not sure stable/mitaka supports partition based osd data for ceph - it _think_ its full disk only?)18:17
sean-k-mooneymitaka should support external journals18:19
sean-k-mooneysams blog that doumented it was deploying mitaka18:19
Mech422sean-k-mooney: oh? cool ...18:20
Mech422sean-k-mooney: ahh - I should did that up for jemcevoy - might help him18:20
sean-k-mooneyjemcevoy: this might help https://yaple.net/2016/03/20/deploying-openstack-mitaka-with-kolla-docker-and-ansible/18:21
dcwangmit01rhallisey: Hey yo ryan18:22
rhalliseydcwangmit01, hey!18:22
dcwangmit01Are the ubuntu-xenial gate jobs new? gate-kolla-kubernetes-docs-ubuntu-xenial18:22
dcwangmit01http://status.openstack.org/zuul/18:22
sbezverkJeffrey4l: jinja2.exceptions.UndefinedError: 'ansible_processor_vcpus' is undefined  and it is on latest kolla18:23
dcwangmit01Do we have control over those, or are they created by the openstack infra team?18:23
sean-k-mooneydcwangmit01: the upstream gate in other projects started swaping over in june18:23
rhalliseydcwangmit01, they've been there for a little.  I have to fix them18:23
sean-k-mooneydcwangmit01: Jeffrey4l  has a review open to enable them for kolla18:23
jemcevoyI do not have any SDDs.  Just some old lab systems with 4 - 1TB disks... I want to use three disks for ceph and the other for docker...18:24
rhalliseydcwangmit01, I got them from infra, but they script they run is wip18:24
Mech422pbourke: that reminds me - we need to decide what to do about the ceph-partiton/sgdisk patch - right now its hard-coded to use /usr/sbin/sgdisk, but ubuntu puts it in /sbin/sgdisk - sdace doesn't want to use shell path to locate the bin, and asked me to talk to you about it when you got back from PTO18:24
sean-k-mooneydcwangmit01: https://review.openstack.org/#/c/352089/18:24
patchbotsean-k-mooney: patch 352089 - openstack-infra/project-config - Enable ubuntu-xenial for Kolla18:24
dcwangmit01sean-k-mooney: rhallisey: So I queried the openstack-infra team yesterday about why the ubuntu-xenial jobs are queuing for 6 hours, as opposed to the jobs for the other platforms.  Apparently, there are limited nodes ofr ubuntu-xenial.  I don't remember them ever being there before.  Is there a way to run those jobs on another platform like they used to be?18:25
*** daneyon_ has joined #openstack-kolla18:25
sean-k-mooneydcwangmit01: oh wait sorry i missed  the kubernetes in the change i linked was for kolla-ansible18:25
dcwangmit01sean-k-mooney: I'm not sure you want to enable ubuntu-xenial.  It'll add 6 hours to your test pipeline18:26
sean-k-mooneythey enabled them by default for some projects18:26
rhalliseyhm18:26
sean-k-mooneynetworking-ovs-dpdk swaped over without me changing it too18:26
dcwangmit01All I know is that once ubuntu-xenial appeared, our kolla-kube test pipeline is suddently taking hours and hours.18:27
dcwangmit01I suppose this is fine, since it all eventually goes through18:28
dcwangmit01But it is a huge pain to wait hours, only to find out there is a merge conflict18:28
rhalliseydcwangmit01, weird. I didn't ask for those specifcally18:28
*** egonzalez90 has quit IRC18:28
rhalliseyjsut a subset of what kolla has18:28
*** egonzalez90 has joined #openstack-kolla18:29
*** egonzalez90 has quit IRC18:29
dcwangmit01rhallisey: Just check out any kolla or kolla-kube job in http://status.openstack.org/zuul/18:29
*** daneyon_ has quit IRC18:30
*** ayoung has quit IRC18:31
rhalliseythe pep8 ubuntu-xenial18:31
sean-k-mooneyrhallisey: i think the swapped over all the python-jobs to use xenial on master18:32
rhalliseyI think so too18:32
jemcevoysean-k-mooney,  Did I have to configure one of the disks for journals and cache?  I do not understand how to label the disks so they will do an OSD per disk?   I tried not creating a single partition and that did not work either...18:32
rhalliseydcwangmit01, I'm not sure what we can do on our end18:32
dcwangmit01rhallisey: That's okay.  I know they are working on adding more nodes in openstac-infra18:33
rhalliseyya we'll just have to be patient18:33
jemcevoyMech422 what does parted <dev> print look like for an OSD and Journal to be co resident on a disk?18:33
*** lrensing_ has quit IRC18:33
sean-k-mooneyjemcevoy: if you dont want to use external journals just  run parted $DISK -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -118:34
sean-k-mooneywhere disk is /dev/sdX18:34
*** shardy is now known as shardy_afk18:35
jemcevoysean-k-mooney, I was using the label KOLLA_CEPH_OSD_BOOTSTRAP_1 ,2,3... Do I just label all 9 disks the same  KOLLA_CEPH_OSD_BOOTSTRAP18:36
sean-k-mooneyjemcevoy: yes if you are not using external journals18:36
*** ad_rien_ has joined #openstack-kolla18:36
sean-k-mooneykolla will enumerate the disks when its bootstrapping.18:37
Mech422jemcevoy: I run master with ceph co-located on my boot disks... I'm a very bad man18:37
sean-k-mooneyMech422: :) i see no issue with this18:38
jrich523sorry... silly question.. how do i login to horizon? is there a default username/password?18:39
jemcevoyI am looking into getting some nvme sdd's for the production servers for the expansion to pod.penguincomputing.com18:39
sean-k-mooneyMech422: honestly though i thinkits valid to at lease share os and journals on the same ssd18:39
Mech422sean-k-mooney: :-)  We're just Rebels :-)18:39
sean-k-mooneyjrich523: yes you can find it in the passwords.tml18:39
jemcevoybtw. the expansion is falsom to mitaka or newton...  Adding like 4000 cores18:40
sean-k-mooneypasswords.yml18:40
jrich523ok thats what i thought, but those are encrypted, so how do i use them?18:40
Mech422jrich523: actually - they're not - the passwords ARE uuids18:40
Mech422jrich523: just copy and past them18:40
sean-k-mooneythey are not encrypted they are actuly that long string of charaters18:40
jrich523... oh... lol18:40
jrich523super18:40
*** ad_rien_ has quit IRC18:42
jrich523what are the default username and domain?18:42
sean-k-mooneydomain i think is default and username is admin18:42
jrich523hmm nope, that didnt fly18:43
*** fragatina has joined #openstack-kolla18:44
sean-k-mooneydid you run the post-deploy playbook18:44
sean-k-mooneythat will give you an openrc file with the correct values18:45
jrich523i believe i did that18:45
sean-k-mooneyif you look in /etc/kolla that should have created an admin-openrc.sh18:46
jrich523however based on the docs, it says the openrc file should end up in /etc/kolla, which i do not see, so, maybe i did that on a different attempt18:46
jrich523yeah18:46
jrich523dont see that, let me run the post deploy18:46
sean-k-mooneythis is what mine looks like18:48
sean-k-mooneyexport OS_PROJECT_DOMAIN_ID=default18:48
sean-k-mooneyexport OS_USER_DOMAIN_ID=default18:48
sean-k-mooneyexport OS_PROJECT_NAME=admin18:48
sean-k-mooneyexport OS_TENANT_NAME=admin18:48
sean-k-mooneyexport OS_USERNAME=admin18:48
sean-k-mooneyexport OS_PASSWORD=<removed>18:48
sean-k-mooneyexport OS_AUTH_URL=http://10.10.10.254:35357/v318:48
sean-k-mooneyexport OS_IDENTITY_API_VERSION=318:48
sean-k-mooneyso by default your domain should be "default" and user "admin"18:49
jrich523:D18:50
jrich523i actually got it up!18:50
jrich523and by "i" i mean that zhugaoxiao, sdake, and sean-k-mooney got it up! :)18:50
jrich523really appreciate the hand holding guys18:51
*** inc0 has quit IRC18:51
sean-k-mooneyjrich523: http://i.imgur.com/yOCM63a.png18:52
sean-k-mooneyjrich523: glad we could help18:52
*** ad_rien_ has joined #openstack-kolla18:52
*** ad_rien_ has quit IRC18:55
*** sean-k-mooney is now known as sean-k-mooneyAFK18:59
*** ad_rien_ has joined #openstack-kolla19:05
sean-k-mooneyAFKsdake: im going to try and rebase the bifrost work this evening and add the rel note/custimisation.19:05
sean-k-mooneyAFKsdake: at that point that series should be ready to merge but i might make some changes later if required based on more testing.19:07
sean-k-mooneyAFKsdake: but first dinner. good evening everyone.19:09
*** lrensing_ has joined #openstack-kolla19:12
*** ad_rien_ has quit IRC19:16
DavieyHow are you chaps using kolla with non-root logins?  I've tried setting ansible_become=true in the inventory for each host.. no joy.19:26
*** senk_ has joined #openstack-kolla19:29
*** mark-casey has quit IRC19:30
*** ppowell has quit IRC19:37
*** ayoung has joined #openstack-kolla19:44
*** JoseMello has joined #openstack-kolla19:49
jrich523sean-k-mooneyAFK lol i just got back from lunch, and i was watching archer while i ate lol19:50
sdakeyo20:01
sdakefolks using the osic cluster I dont think anyone ever answered my quetions how about to ssh into th vms20:02
*** senk_ has quit IRC20:02
*** mark-casey has joined #openstack-kolla20:06
*** haplo37__ has quit IRC20:07
*** srwilkers has quit IRC20:15
sdakeDaviey i use sudo20:22
sdakeDaviey but i know it can be done without20:22
sdakeDaviey but is undocumtend20:22
openstackgerritShaun Smekel proposed openstack/kolla: Add full support for fernet  https://review.openstack.org/34936620:24
*** inc0 has joined #openstack-kolla20:24
*** kbaegis has joined #openstack-kolla20:28
kbaegisHi all20:28
kbaegisHaving issues setting kolla up on gentoo20:28
kbaegisI know it's probably not a "supported" distro20:29
mark-caseykbaegis: https://i.imgflip.com/2/gjx1b.jpg20:29
mark-caseyno I'm kidding20:29
mark-caseyI'm not actually sure if anyone has tried that20:30
*** msimonin has joined #openstack-kolla20:30
kbaegis:)20:30
kbaegisWell I'll be the first then20:30
inc0kbaegis, sorry about that:P20:30
kbaegisKeep getting "ERROR! Unexpected Exception: name 'unicode' is not defined"20:30
inc0what's the issue>20:30
inc0eee?20:30
inc0python --version plz20:30
*** shardy_afk has quit IRC20:31
mark-caseykbaegis: I assume this is using it as the OS running kolla containers? not the OS in the container images?20:31
kbaegisPython 3.4.320:31
sdakeinc0 i can't ssh into vms20:31
kbaegisCorrect20:31
kbaegis:)20:31
sdakei want o ssh into vms to run bonnie within them20:31
sdakeinc0 any ideas?20:31
inc0sdake, you tried via namespace?20:31
*** coolsvap has quit IRC20:31
sdakeno i tiried via the deploy box20:32
kbaegisSo it's "NameError: name 'unicode' is not defined" from ansible/inventory/__init__.py", line 73620:32
inc0ok, jon me on tmux20:32
sdakei have meeting shortly20:32
inc0you wont be able to ssh from deployment20:32
sdakei see20:32
kbaegisAnd that'd be ansible/cli/playbook.py", line 15420:32
sdakei cant join you on tmux atm - have meeting shorty20:33
sdakewhen i gt wrapped up i'll ping ou20:33
inc0I'll leave ssh open in tmux20:33
inc0I'll have more meetings myself so just look at history what I did20:34
kbaegistraceback: http://pastebin.com/k8RKTPLV20:34
*** Jeffrey4l has quit IRC20:34
inc0sdake, on tmux 3 you'll be able to ssh20:35
*** Jeffrey4l has joined #openstack-kolla20:35
kbaegisI definitely have a config at /etc/kolla/globals.yml20:36
mark-caseykbaegis: what version of ansible you have installed?20:36
inc0kbaegis, so let me put it this way....unicode is a buildin function20:36
inc0to python20:36
kbaegisthe latest from pip2/pip320:36
inc0which makes me think that something's really wrong with your python20:36
kbaegisansible 2.1.1.020:36
mark-caseyinc0: kbaegis: this maybe? http://stackoverflow.com/a/36110612/659578720:37
inc0oh?20:37
mark-caseyI see ansible/cli stuff in the stacktrace. Ansible cli was a standalone/community thing recently added ansible official so perhaps that call to unicode needs to be renamed?20:38
inc0so I guess ansible is no-bueno on python 3.420:38
kbaegisI tried python2, but it wasn't working either20:38
sbezverkKolla cores, please check this https://review.openstack.org/#/c/336751/ The gate is super green :-)20:38
patchbotsbezverk: patch 336751 - kolla - Fix intf address for RabbitMQ and disable cluster ...20:38
inc0sbezverk, ooo even ubnutu-source20:38
inc0we managed to get thins libvirt thing fixed?20:38
inc0anyone can tell me what was the issue?20:39
*** michauds has joined #openstack-kolla20:40
kbaegisAlright, thanks guys.  I'll look through my packages and whatnot20:41
kbaegisSee if I can roll back or update20:41
mark-caseyinc0: kbaegis: https://github.com/ansible/ansible/issues/1409#issuecomment-9426610120:42
inc0kbaegis, I suggest to use virtualen20:42
inc0v20:42
inc0it helps control version of python and python libs20:43
kbaegisThanks20:43
mark-caseyoh I didn't realize it could do version. Yea. perhaps a little learning curve there but better than hacking up system python20:43
kbaegisSo I just need to roll back to python 3.3?20:47
*** shardy_afk has joined #openstack-kolla20:48
kbaegisI think I'll go that route20:48
mark-casey2.6/2.7 I think. My read of that was no Python 3 support at all yet, but generally yes... roll back to any version they list as supported20:48
kbaegisOh, I have python2.7 slotted20:48
kbaegisYeah, that was having other issues20:48
*** imcsk8 is now known as imcsk8|lunch20:50
inc0kbyrne, 2.7.x is the way to go20:50
sdakeany python nerds out there20:50
inc0I'm ex python nerd20:50
inc0I moved to deployment of openstack20:50
sdakewhat is it giong to take to get us to python 3.5 exclusively20:50
inc0why would we ever go to 3.5 exclusively?20:51
inc0especially if, as you see in log above, ansible is no bueno in 3+20:51
sdakehttps://review.openstack.org/#/c/349069/20:51
inc0so you'd need different python for kolla and ansible20:51
kbaegisWell, that's exactly the problem20:51
patchbotsdake: patch 349069 - governance - add ocata goal "support python 3.5"20:51
inc0which is kinda...duh20:51
inc0we can *support* it20:51
inc0but not exclusively20:52
kbaegispython 2.7 and 3.4 are treated as separate languages by devs20:52
inc0we can be compatible with both 2 and 320:52
inc0yes and no20:52
kbaegisYeah, but devs aren't20:52
inc0there are libs, like six20:52
sdakeinc0 read the proposal20:52
inc0that allows you to write cross-compatible code20:52
sdakerather then just the title20:52
*** ravig has joined #openstack-kolla20:53
inc0sdake, thing is...we'd say good bye to kolla-ansible20:53
sdakegoals are a new thing for projects coming in cocata20:53
sdakeyou mean the anible implementation?20:53
inc0I mean command20:54
sdakeso we could rewrite that, no?20:54
inc0no, we'd need to rewrite ansible20:54
inc0ansible will not run on 3.520:54
inc0and kolla-ansible calls ansible20:54
sdakeok thanks20:55
kbaegisI mean, the version split from 2-3 is a community wide python problem20:55
kbaegisNot just ansible20:55
kbaegisBut that's OT20:55
kbaegis:)20:55
inc0yeah, but for our particular case20:55
inc0it's fubar20:55
inc0and ansible folk is not a community that will be bound by these goals20:56
inc0sdake, unless you ask your wonderful wife to make it happen20:56
inc0I mean support for python3 in ansible20:56
mark-caseyinc0: it looks like they see the writing on the wall, even if they don't like python 320:56
mark-caseyI mean it may be a year and a half but it looks like it's coming20:57
inc0yeah, if it comes, it's cool20:57
inc0in fact, I think our code is python3 compliant even now20:57
inc0but if it's not, we don't have THAT much of it20:57
inc0it will be pretty fast move for us20:57
inc0but...ansible.20:57
inc0brb20:57
*** shardy_afk has quit IRC20:58
kbaegisWell, for the record I'd have greatly appreciated a note on http://docs.openstack.org/developer/kolla/quickstart.html stating that python 3.x is not necessarily compatible20:59
kbaegisEasy enough to switch over :)21:00
sbezverkkfox111121:00
sbezverkping21:00
mark-caseykbaegis: I think we've just learned that too :D  afaik most of the supported kolla platforms default to v221:01
*** dwalsh_ has quit IRC21:02
inc0kbaegis, you can make this note:) you'll get free admission to summit for it;)21:02
kbaegisI guess it's either that or booth duty21:03
inc0I don't think you'll get full pass on booth duty really21:03
inc0ATC gives you full access to summit, not just markeplace21:03
mark-caseyinc0: atc?21:03
sdakemark-casey - fwiw, project-wide goals are to be prioritized above all else, and to be finished prior to m1.21:03
sdake(of ocata)21:04
inc0active technical contributor21:04
inc0type of summit pass21:04
inc0sdake, not much we can do about that21:05
inc0we might want to talk with dhellman about it21:05
inc0I'll ping him21:06
sdakei am taking cre of it relax21:07
sdakefeel free to read my response to the review21:08
*** daneyon_ has joined #openstack-kolla21:08
*** fragatin_ has joined #openstack-kolla21:08
*** msimonin has quit IRC21:09
mark-caseysdake: for sure. I was only suggesting a reason that a python 3 note was not already in our docs21:10
inc0sdake, btw ssh is ready for you21:10
sdakestill waking up21:11
sdakegive me 15 mins21:11
sdakepls21:11
sdakethey had to call the crash cart in about 30 minutes ago21:11
*** fragatina has quit IRC21:12
*** daneyon_ has quit IRC21:12
*** rmart04 has joined #openstack-kolla21:12
*** msimonin has joined #openstack-kolla21:15
kbaegisWell, I've repaired my system. :) Looks like 2.7 is working for me21:16
mark-caseykbaegis: great. have you already built images?21:19
kbaegisMy first kolla deployment.  A beautiful thing21:19
kbaegisYes21:19
sdakekbaegis grats!21:19
*** rmart04 has quit IRC21:20
*** huikang has quit IRC21:21
kbaegisI was always unhappy with devstack21:21
kbaegisNice21:21
sdakewirehead_ any report from sv days?21:21
wirehead_I have concentrated rage from devstack.21:22
wirehead_Nothing too much.  Didn’t run into the guy from SAP yet.21:22
kbaegis:)21:22
kbaegisIt's the most hideously coded bash I've ever read21:22
sdakeconcetrated rage?21:23
sdakecould you expand?21:23
wirehead_I like Kubernetes because kube-up.sh is also hideously coded bash, but it works most of the time.21:23
mark-caseykbaegis: my brain isn't working right now... need more coffee... but sometimes there have been issues building for example centos images while running in ubuntu21:24
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Neutron bootstrap to create service/project/role  https://review.openstack.org/35374021:24
mark-caseyjust something to keep in mind if you hit weird stuff21:24
sdakemark-casey sucks when your brain doens't cooperate right ;)21:24
kbaegisI had issues with haproxy.  I'm rebuilding that image now21:24
mark-caseysdake: indeed21:24
kbaegis"INFO:kolla.image.build.base:Cannot open: http://repo.percona.com/release/7/RPMS/x86_64/percona-release-0.1-3.noarch.rpm. Skipping'21:24
sdakekbaegis that link wfm, maybe your firewall or proxy is getting in the wya21:25
wirehead_I spent too much time while at Rackspace futzing with Devstack not working so I never got anything committed to OpenStack.21:25
wirehead_Concentrated rage is overstating my feelings.21:25
wirehead_But I can be a drama queen.21:26
sdakecan't we all :)21:26
wirehead_Well, a considerable pool of software engineers claim to be totally rational, all the time.21:26
wirehead_Also, frequently “just making a joke, man"21:26
wirehead_:P21:26
wirehead_Like, for an OpenStack conference, Kubernetes has been discussed a LOT.21:28
wirehead_But most of the running OpenStack atop Kubernetes talks were yesterday.21:28
*** jtriley has quit IRC21:28
*** inc0 has quit IRC21:30
sean-k-mooneyAFKthe one thing ill say in defece of devstack for whtat it does( provide a dev enviorment for developing openstack) it is a good solution. i would like to find a way to use kolla the same way but even source build from git are not quite as simple to develop with21:31
*** ad_rien_ has joined #openstack-kolla21:31
kbaegissean-k Easy enough to do git clone from a container21:32
openstackgerritMerged openstack/kolla-kubernetes: Introducing DaemonSet for Neutron Openvswitch components  https://review.openstack.org/35243321:33
sean-k-mooneyAFKkbaegis: i have often built kolla containers from git then exec in and modify the code and restart the container to pick up the chagned.21:34
sean-k-mooneyAFKkbaegis: its still simpler to boot a vm on my kolla cluster and run devstack in it normally21:34
sean-k-mooneyAFKkbaegis: the main issue for me is how to debuge with a ide when the code is runing in a docker container21:35
kbaegisJust rebuilt the image from another docker host on other side21:36
*** lrensing_ has quit IRC21:37
kbaegisGlad I put up a image repo21:37
kbaegisAnyone know how much memory is taken by the controller playbook?21:38
kbaegisI have 2 nodes with 3-4 gigs I'll use if I can21:38
kbaegisThey have docker but they're atom.  Nice for failover, terrible for vms21:38
*** shardy has joined #openstack-kolla21:40
sean-k-mooneyAFKrabbitmq is a little ram hungrey but you should be able to get a controler deploy with 4GB but that is pushing it a little. 8GB would run fine. not sure about 4G21:40
*** inc0 has joined #openstack-kolla21:42
*** gfidente has quit IRC21:44
kbaegisI'm getting timeouts from my docker registry21:47
kbaegisAnyone know what the curl is to check the api for21:47
kbaegislike an ls21:47
kbaegisPing works21:47
sdakekbaegis are you deploying multinode?21:47
openstackgerritMerged openstack/kolla: Fix intf address for RabbitMQ and disable cluster for Kube  https://review.openstack.org/33675121:48
kbaegissdake I'm not actually.  Still using a registry21:48
sdakeor doing an aio eval?21:48
kbaegisNecessary here, as I can't get haproxy w/o percona on centos evidently21:48
kbaegisIt's my home lab.  I'm learning how kolla works :)21:48
kbaegisSort of21:49
sdakekbaegis roger - reecommend just using docker hub in that case21:49
kbaegisEww.  I prefer self-hosted :)21:49
sdakeok21:49
sdakedocker ps -ef | grep docker -> pate21:49
sdakepaste21:49
kbaegisregistry:2          "/entrypoint.sh /etc/"   2 hours ago         Up 2 hours          192.168.254.3:4000->5000/tcp   registry21:50
sdakesorry tired21:50
sdakeps -e  gerp docker -> paste21:50
kbaegisgrep right? :)21:51
sdakegerp/grep21:51
kbaegiss/er/re/g21:51
sdakebut you gotta hang tought when your all out there on your own21:51
kbaegiswait, is there a password generated for the21:52
kbaegisHmm21:52
kbaegishttps://192.168.254.3:4000/v1/_ping returns error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol21:52
kbaegisSo the registry is locked to the host that spawned it?21:52
kbaegisI'll figure this out21:53
kbaegisI think this is the "secure registry"21:53
kbaegisI don't have insecure set on the host uploading21:54
kbaegisYet another argument for my poor poor docker host21:55
*** inc0 has quit IRC21:56
sean-k-mooneyAFKkbaegis: i havent got around to impentat the bootstrap-build-host part of the kolla-host playbook but i should have that coded up by the end of the week. so hopefully setup the build host including the registry will be automate as a single command soon21:58
*** msimonin has quit IRC21:58
*** inc0 has joined #openstack-kolla21:59
*** vhosakot has quit IRC21:59
sdakekbaegis my speculation is if you run ps -ef | grep docker you will see it lacking the --insecure-registry flag22:00
*** ravig has quit IRC22:00
sdakekbaegis you need  that to run your own registry - unless yougo throug hthe trouble of setting up tls for your registry22:00
*** vhosakot has joined #openstack-kolla22:01
kbaegisI've already set one up :)22:02
kbaegisIt's timing out22:02
*** ravig has joined #openstack-kolla22:02
kbaegisAlready figured that one out earlier :)22:02
kbaegisYou're right, it was missing22:02
kbaegisI put it in and restarted.  It didn't help unfortunately22:03
kbaegisSo I'm just going to docker save, scp,22:03
*** kong has joined #openstack-kolla22:04
*** ravig has quit IRC22:07
sean-k-mooneyAFKkbaegis: what os are you using and where did you add the insecure-regestry setting22:07
sdakekbaegis are you sure insecure-registry is set- you can check with the ps command22:07
*** shardy has quit IRC22:07
kbaegisdocker ps -e doesn't give me anything22:09
kbaegisdocker ps --help22:09
sean-k-mooneyAFKkbaegis: ps -ef | grep docker22:10
sean-k-mooneyAFKkbaegis:  not docker ps22:10
kbaegis'/usr/bin/docker daemon -p /run/docker.pid --insecure-registry 192.168.254.3:4000'22:11
kbaegis'/usr/bin/docke daemon -p /run/docker.pid --bip=172.22.1.1/24 --fixed-cidr=172.22.1.0/24 --dns=172.21.5.254 --ip-forward=true --iptables=true --icc=true --insecure-registry 192.168.254.3:4000'22:11
kbaegisthe r got garbled by the terminal evidently22:12
sean-k-mooneyAFKkbaegis: that looks fine22:13
kbaegisYup22:13
*** michauds has quit IRC22:13
kbaegisstil somehow docker registry times out22:13
sean-k-mooneyAFKif you did not have the insecure-registrey set when you built the images they will not have been pushed22:13
mark-caseykbaegis: pushing during build or pulling during deploy?22:13
kbaegispushing after build22:13
sdakesean-k-mooneyAFK n++22:13
mark-caseywas the push flag set on build.py (and also what sean said could be it)22:14
kbaegisHost 2: python2 /usr/bin/kolla-build haproxy --registry 192.168.254.3:4000 --push22:14
kbaegisdifferent kolla git sync22:14
sean-k-mooneyAFKyou can push with bash after the fact. we probably should script that.22:15
sean-k-mooneyAFKlet me see if i still have the command in my bash history22:15
kbaegisdocker push --help wasn't very helpful22:15
kbaegisBad documentation by docker22:15
kbaegisAhh22:16
kbaegisThe manpage is very enlightening22:16
sean-k-mooneyAFKkbaegis: i belive this will push the images for you on the host that built them22:18
sean-k-mooneyAFKfor img in $(docker images | grep "192.168.254.3:4000" | awk '{ print $1 }'); do docker push $img; done22:18
*** JoseMello has quit IRC22:20
sean-k-mooneyAFKi should add that as a command to kolla-ansible as the counterpart to pull22:21
kbaegistcp 192.168.254.3:4000: i/o timeout22:21
kbaegisHmm22:21
ntpttr___inc0: ping22:22
kbaegisSince when did hard drives just start getting fatter rather than more densely packed?22:23
kbaegis*sigh*22:23
*** vhosakot has quit IRC22:23
kbaegisWhat's the "offline" way to do it?22:24
kbaegisPretend I had a sneakernet instead22:24
kbaegisHow do I export the image to a file, scp (which works fine), and upload it locally to the registry?22:25
sean-k-mooneyAFKthat or build the images on every host22:25
*** inc0 has quit IRC22:25
kbaegisOH!22:25
kbaegisConfident that this is a networking issue22:26
*** fragatin_ has quit IRC22:26
sean-k-mooneyAFKkbaegis: do you have a proxy?22:26
*** fragatina has joined #openstack-kolla22:27
kbaegisNo, I have multiple route tables per host22:27
kbaegisip route add 172.22.1.0/24 dev docker0 src 172.22.1.1 table os22:27
kbaegisI'll let ya know if that works :)22:27
kbaegisI keep my lab very isolated22:28
kbaegisNope22:28
kbaegisThat wasn't it22:28
kbaegis:/22:28
*** imcsk8|lunch is now known as imcsk822:28
*** rstarmer has joined #openstack-kolla22:29
kbaegiscurl http://192.168.254.3:4000/v2/_ping doesn't work from host222:30
kbaegisfrom the global route table22:30
kbaegisicmp to that address works fine22:30
sean-k-mooneyAFKin by working registry that curl command gives a 404 not found22:33
kbaegisYeah22:33
kbaegisI get no response whatsoever22:33
kbaegisSomething networking or iptables related I'm guessing22:33
*** stevemcquaid has joined #openstack-kolla22:33
sean-k-mooneyAFKwell i hate iptables so beyond iptables -F iptables -X and try again i cant help much on that front22:35
stevemcquaid@kfox1111 - @harlowja said you might be the person with the answers!22:35
harlowjaobviously!22:35
harlowjaha22:35
jrich523actually i had to modify iptables to allow access to horizon, which should probably be part of the deployment scripts :)22:36
sean-k-mooneyAFKjrich523: really i have never had to do that.22:36
jrich523it threw me for a loop22:36
jrich523its a basic install of CentOS 722:36
jrich523firewall-cmd --zone=public --add-port=80/tcp --permanent; firewall-cmd --reload22:37
stevemcquaidWRT: kubernetes, have you ever used ABAC access control before? We are running V1.2 and have not been able to do get a policy set to deny all, but allow svc accounts for ingress controller yet22:38
kfox1111stevemcquaid: hi.22:38
sean-k-mooneyAFKoh i think sdake said someting about disableing firewalld on centos was assumed to be done when deploying kolla22:38
*** sdake has quit IRC22:38
jrich523interesting assumption :)22:38
stevemcquaidhi @kfox1111 :)22:38
jrich523notice he leaves when you mention him!22:39
kfox1111whats up?22:40
jrich523im going to blow away what i have (or make another) so see if i can rebuild it again... and maybe... create a pull request for that 'quick start' doc :)22:40
harlowjareposting from a little older stevemcquaid message22:41
harlowjakfox1111  'kubernetes, have you ever used ABAC access control before? We are running V1.2 and have not been able to do get a policy set to deny all, but allow svc accounts for ingress controller yet'22:41
*** inc0 has joined #openstack-kolla22:41
* harlowja thought u might have an idea (or if u don't thats ok to)22:41
kbaegisWell, this is definitely my fault :)22:46
*** rstarmer has quit IRC22:47
*** zhubingbing has joined #openstack-kolla22:48
*** zhubingbing_ has joined #openstack-kolla22:48
*** inc0 has quit IRC22:49
kbaegisSo my dns wasn't enabled on my new lab server.  I think that's why it wasn't finding percona.22:50
kbaegisForgetting that it doesn't use the "hypervisor" dns22:50
openstackgerritzhubingbing proposed openstack/kolla: repair gnocchi  dockerfile  https://review.openstack.org/34838822:51
kbaegisWell hopefully knowing idiot admins like me are out there making mistakes like this is helpful to you too! :)22:52
*** sould has joined #openstack-kolla22:52
*** ad_rien_ has quit IRC22:54
*** ad_rien_ has joined #openstack-kolla22:54
souldgood evening everyone :) i had a question about kolla-kubernetes.  im following the quickstart guide, and im having an issue with the section regarding whether skydns is responding22:54
souldwhen i run 'nslookup kubernetes.default.svc.openstack.local 127.0.0.1', i get ';; connection timed out; no servers could be reached'.  is there some additional configuration required outside of the quickstart guide i need to poke around and be aware of?22:55
kbaegisWould be wayyyy to convenient if it was the problem I just solved, huh?22:56
kbaegisDOCKER_OPTS="--dns=<dns server>"?22:56
rhalliseysbezverk, ^^^22:56
*** daneyon_ has joined #openstack-kolla22:56
*** ad_rien_1 has joined #openstack-kolla22:56
kbaegisConfirmed, that was my problem guys22:56
*** inc0 has joined #openstack-kolla22:57
*** ad_rien_ has quit IRC22:57
*** inc0 has quit IRC22:57
*** daneyon_ has quit IRC23:00
*** inc0 has joined #openstack-kolla23:01
mark-caseykbaegis: ahhh, makes sense. nice catch23:01
rhalliseysould, could be the skydns container failing23:01
rhalliseydocker logs <skydns_contianer>23:01
*** ad_rien_1 has quit IRC23:02
*** ad_rien_ has joined #openstack-kolla23:02
kbaegisdeploy still failed.  I'm rebuilding everything23:03
kbaegisshould I be concerned about: 587 packages excluded due to repository priority protections?23:04
kbaegisKind of like, "please go away, you're too troublesome?23:05
mark-caseykbaegis: how failed? have you run prechecks?23:09
kbaegishaproxy was unable to pass a config23:09
kbaegisIt fails the cinder check, but I have cinder disable in globals23:10
kbaegisSpecifically, this is what failed: "TASK [haproxy : Copying over haproxy.cfg] **************************************23:14
kbaegisfailed: [localhost] (item=haproxy) => {"failed": true, "item": "haproxy", "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'ipv4'"}"23:14
kbaegisls23:14
*** inc0 has quit IRC23:17
mark-caseykbaegis: k. worth rebuilding once, but image build is more or less deterministic so you'll often get the same result.23:25
mark-caseyAt some point it becomes more about fixing interactions like between ansible on the deploy host and sshd or docker daemon on the target hosts, and you don't necessarily have to rebuild images unless you've made changes to, for example, the default dockerfiles23:25
mark-caseyin this case the haproxy.cfg can't be templated by ansible because a variable doesn't exist, which (shot in the dark) could be something like a missing python lib on target hosts23:25
mark-caseyso you'd install that and redeploy w/out rebuilding images23:26
*** sdake has joined #openstack-kolla23:26
kbaegisHmm23:27
mark-caseytwo things to keep in mind next little bit...: one is you have to run clean-containers and clean-images on the target hosts after making changes but before redeploying, because the nodes don't know you've started over; last time's containers still running, and two if you haven't already you should run kolla-ansible prechecks instead of going straight to deploy23:28
kbaegisAlright, I'll follow the guide :)23:30
*** ayoung has quit IRC23:30
kbaegisThis is really slick, and doesn't require 20 nodes like fuel23:31
kbaegisLiking it a lot23:31
openstackgerritDave Walker proposed openstack/kolla: Add --user-domain when adding heat role admin  https://review.openstack.org/35347123:31
openstackgerritMerged openstack/kolla-kubernetes: Fix bad merge of ceph mariadb  https://review.openstack.org/35366923:32
sdakesean-k-mooneyAFK ya we need to disable firewalld23:34
sean-k-mooneyAFKsdake: ok cool i dont have that in kolla-host currently but ill add it in another commit23:35
*** zhurong has joined #openstack-kolla23:38
mark-caseykbaegis: cool. I *should* be off and on tonight so if you're still at it in a few hours maybe we can see if there's a new roadbloack23:44
kbaegisthanks!23:46
*** rkrum has joined #openstack-kolla23:50
openstackgerritChristine Hicks proposed openstack/kolla: Following rst guidelines fixing the documentation for kolla  https://review.openstack.org/35378623:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!