| *** zhurong has quit IRC | 00:07 | |
| *** neilus has joined #openstack-kolla | 00:15 | |
| *** zhubingbing has joined #openstack-kolla | 00:18 | |
| *** zhubingbing has quit IRC | 00:19 | |
| *** neilus has quit IRC | 00:19 | |
| *** sdake has quit IRC | 00:28 | |
| *** duonghq has joined #openstack-kolla | 00:38 | |
| *** senk__ has quit IRC | 00:43 | |
| *** senk has joined #openstack-kolla | 00:44 | |
| *** hieulq has joined #openstack-kolla | 00:45 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: More services to deployments https://review.openstack.org/372189 | 00:49 |
|---|---|---|
| imcsk8_ | have you hit the "too many links" problem while building containers under a ext4 partition? | 00:51 |
| duonghq | noop | 00:58 |
| *** dwalsh has quit IRC | 00:58 | |
| *** williamcaban has joined #openstack-kolla | 01:03 | |
| *** zhubingbing has joined #openstack-kolla | 01:05 | |
| *** williamcaban has quit IRC | 01:07 | |
| *** zhurong has joined #openstack-kolla | 01:09 | |
| openstackgerrit | dupengfei proposed openstack/kolla: Fix keystone,database,memcache for k8s. https://review.openstack.org/372089 | 01:13 |
| *** david-lyle has joined #openstack-kolla | 01:27 | |
| *** senk has quit IRC | 01:31 | |
| *** senk__ has joined #openstack-kolla | 01:31 | |
| *** david-lyle has quit IRC | 01:31 | |
| *** senk has joined #openstack-kolla | 01:36 | |
| *** senk__ has quit IRC | 01:36 | |
| *** salv-orl_ has joined #openstack-kolla | 01:41 | |
| *** zhurong has quit IRC | 01:43 | |
| *** zhurong has joined #openstack-kolla | 01:44 | |
| *** salv-orlando has quit IRC | 01:45 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: More services to deployments https://review.openstack.org/372189 | 01:46 |
| *** jamesdenton has joined #openstack-kolla | 01:50 | |
| *** senk has quit IRC | 01:54 | |
| *** senk__ has joined #openstack-kolla | 01:54 | |
| *** coolsvap has joined #openstack-kolla | 02:01 | |
| duonghq | zhubingbing: ping | 02:04 |
| zhubingbing | hi | 02:04 |
| zhubingbing | i am here | 02:04 |
| duonghq | good morning | 02:04 |
| duonghq | about this bug: https://bugs.launchpad.net/kolla/+bug/1619209 | 02:04 |
| openstack | Launchpad bug 1619209 in kolla "Fail to build keystone ubuntu binary docker image" [Medium,Triaged] | 02:04 |
| duonghq | how did you reproduced this? | 02:04 |
| zhubingbing | ok | 02:05 |
| duonghq | I cleaned image | 02:05 |
| duonghq | and successful build | 02:05 |
| zhubingbing | sure | 02:08 |
| zhubingbing | i understand | 02:08 |
| zhubingbing | Can you answer him | 02:08 |
| zhubingbing | duonghq you can answer him , thanks | 02:09 |
| duonghq | zhubingbing: ok, thanks | 02:09 |
| zhubingbing | thank you | 02:10 |
| duonghq | zhubingbing: I did not try to reproduce this bug? | 02:11 |
| duonghq | *you | 02:11 |
| zhubingbing | yes | 02:11 |
| openstackgerrit | dupengfei proposed openstack/kolla-kubernetes: The configmaps of iscsid and tgtd are error. https://review.openstack.org/372200 | 02:11 |
| duonghq | ok, so I think we should not triaged in this case | 02:12 |
| duonghq | seem that we need put more attention in new bug, just notice this bug is reported from 2016-09-01, if the bug is really existed, it is critical bug indeed | 02:14 |
| zhubingbing | this not a bug | 02:14 |
| zhubingbing | good advice | 02:15 |
| duonghq | huh? what do you mean "this is no a bug" | 02:15 |
| duonghq | *not | 02:15 |
| zhubingbing | yes | 02:18 |
| *** fguillot is now known as fguillot_afk | 02:24 | |
| *** fguillot_afk has quit IRC | 02:26 | |
| *** senk__ has quit IRC | 02:28 | |
| *** senk has joined #openstack-kolla | 02:28 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the failure of ceilometer-api container https://review.openstack.org/372161 | 02:31 |
| *** zhurong has quit IRC | 02:39 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Delete the unnecessary parameter https://review.openstack.org/372137 | 02:39 |
| Jeffrey4l_ | morning guys | 02:39 |
| *** caowei has joined #openstack-kolla | 02:41 | |
| *** zhurong has joined #openstack-kolla | 02:43 | |
| *** AndChat-382025 has quit IRC | 02:44 | |
| *** senk has quit IRC | 02:47 | |
| *** senk has joined #openstack-kolla | 02:47 | |
| *** AndChat|382025 has joined #openstack-kolla | 02:50 | |
| zhubingbing | morning | 02:51 |
| *** daneyon has joined #openstack-kolla | 02:57 | |
| *** neilus has joined #openstack-kolla | 02:57 | |
| *** neilus has quit IRC | 03:02 | |
| *** daneyon has quit IRC | 03:02 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: More services to deployments https://review.openstack.org/372189 | 03:08 |
| *** zhubingbing has quit IRC | 03:11 | |
| *** zhubingbing has joined #openstack-kolla | 03:12 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Add "---" into tempest.yml https://review.openstack.org/372214 | 03:29 |
| *** senk has quit IRC | 03:36 | |
| *** huikang has joined #openstack-kolla | 03:48 | |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Remove duplicate lbaas provider config https://review.openstack.org/372216 | 03:54 |
| *** michauds has joined #openstack-kolla | 04:04 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Delete the unnecessary parameter https://review.openstack.org/372137 | 04:08 |
| *** tonanhngo has joined #openstack-kolla | 04:11 | |
| *** salv-orl_ has quit IRC | 04:12 | |
| *** salv-orlando has joined #openstack-kolla | 04:13 | |
| *** tonanhngo has quit IRC | 04:16 | |
| *** bdaca has joined #openstack-kolla | 04:20 | |
| zhubingbing | Who knows now magnum cluster-create command seems to have no | 04:27 |
| zhubingbing | magnum cluster-template-create | 04:27 |
| *** lulei has joined #openstack-kolla | 04:30 | |
| *** williamcaban has joined #openstack-kolla | 04:39 | |
| *** michauds has quit IRC | 04:40 | |
| *** tonanhngo has joined #openstack-kolla | 04:43 | |
| *** williamcaban has quit IRC | 04:43 | |
| *** neilus has joined #openstack-kolla | 04:45 | |
| *** tonanhngo has quit IRC | 04:47 | |
| *** neilus has quit IRC | 04:50 | |
| *** tonanhngo has joined #openstack-kolla | 04:52 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Source the openrc file automatically when running tools/init-runonce https://review.openstack.org/371725 | 04:57 |
| *** rstarmer has joined #openstack-kolla | 05:00 | |
| rstarmer | is there a way to "Become root" in the kolla containers? I want to look at the ovs status, and it seems that I don't have permissions to do this, or am I approaching this the wrong way? | 05:00 |
| *** ssurana has joined #openstack-kolla | 05:02 | |
| *** v1k0d3n has quit IRC | 05:18 | |
| *** senk has joined #openstack-kolla | 05:20 | |
| bdaca | rstarmer try using '-u 0' parameter when executing your 'decoker exec' command | 05:20 |
| rstarmer | bdaca: thanks, will give it a shot | 05:21 |
| bdaca | s /decoker/docker/ ;] | 05:21 |
| rstarmer | brilliant | 05:24 |
| rstarmer | docker exec -itu 0 {container} bash | 05:24 |
| bdaca | good :) | 05:25 |
| rstarmer | another question that's been pestering me: is it possible to mix/match "binary" vs. "source" containers in a deployment? | 05:26 |
| rstarmer | I.e., I like binary containers because my "upstream" OS vendor will support that packaging. But I'd like to use services that aren't yet packaged as RPMs/Debs | 05:28 |
| duonghq | rstarmer: currently: no way | 05:33 |
| rstarmer | duonghq: thanks, that's what I though, so perhaps I'll end up "patching" things to build "binary" non-binary containers :). We'll see, that won't be for a few weeks at least. | 05:34 |
| rstarmer | I guess this also falls into the "I have a container that I'd like to build that's not part of the current set of defined containers anyway" | 05:34 |
| zhubingbing | ping Liyingjun | 05:41 |
| bdaca | any ceph master here? | 05:42 |
| bdaca | or even advance ceph user? ;) | 05:43 |
| *** msimonin has joined #openstack-kolla | 05:47 | |
| *** AndChat|382025 has quit IRC | 05:48 | |
| *** msimonin has quit IRC | 05:49 | |
| *** huikang has quit IRC | 05:53 | |
| *** AndChat|382025 has joined #openstack-kolla | 05:55 | |
| *** senk has quit IRC | 06:02 | |
| *** tonanhngo has quit IRC | 06:07 | |
| zhubingbing | ping hanwei | 06:08 |
| *** AndChat-382025 has joined #openstack-kolla | 06:11 | |
| *** AndChat|382025 has quit IRC | 06:14 | |
| *** rstarmer has quit IRC | 06:20 | |
| *** rstarmer has joined #openstack-kolla | 06:20 | |
| *** egonzalez90 has joined #openstack-kolla | 06:21 | |
| zhubingbing | hi egonzalez90 | 06:22 |
| *** tonanhngo has joined #openstack-kolla | 06:22 | |
| zhubingbing | morning | 06:22 |
| zhubingbing | ping kevpn | 06:24 |
| egonzalez90 | morning | 06:26 |
| *** williamcaban has joined #openstack-kolla | 06:28 | |
| *** senk has joined #openstack-kolla | 06:29 | |
| *** williamcaban has quit IRC | 06:32 | |
| *** daneyon has joined #openstack-kolla | 06:33 | |
| *** daneyon has quit IRC | 06:38 | |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Fix vpnaas for neutron https://review.openstack.org/372250 | 06:45 |
| *** tonanhngo has quit IRC | 06:48 | |
| *** senk has quit IRC | 07:02 | |
| *** senk has joined #openstack-kolla | 07:02 | |
| *** ishu has joined #openstack-kolla | 07:03 | |
| *** senk has quit IRC | 07:04 | |
| *** berendt has joined #openstack-kolla | 07:11 | |
| berendt | morning guys. | 07:12 |
| *** coolsvap is now known as _coolsvap_ | 07:13 | |
| ishu | morning | 07:13 |
| berendt | bjolo we merged vpnaas a few days ago, becuase of that your old inventory file worked all the time | 07:13 |
| bjolo | berendt, ok but i dont get what you mean? yes i had issue with old inventory, but there was an issue with vpnaas as well? | 07:16 |
| berendt | bjolo you mentioned that your deployment worked after adding the missing vpnaas entires to the inventory file | 07:16 |
| berendt | I will open a bug report for https://review.openstack.org/#/c/372250/1 now to be able to backport the fixes | 07:17 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Fix vpnaas for neutron https://review.openstack.org/372250 | 07:19 |
| *** zouyapeng has joined #openstack-kolla | 07:20 | |
| *** egonzalez90 has quit IRC | 07:20 | |
| openstackgerrit | Merged openstack/kolla: Fix up aodh config templates https://review.openstack.org/371663 | 07:20 |
| *** msimonin has joined #openstack-kolla | 07:21 | |
| *** lulei has quit IRC | 07:22 | |
| openstackgerrit | Merged openstack/kolla: Add "---" into tempest.yml https://review.openstack.org/372214 | 07:23 |
| *** lulei has joined #openstack-kolla | 07:23 | |
| *** duonghq has quit IRC | 07:37 | |
| *** matrohon has joined #openstack-kolla | 07:37 | |
| *** ankush has joined #openstack-kolla | 07:40 | |
| *** neilus has joined #openstack-kolla | 07:40 | |
| *** neilus has quit IRC | 07:41 | |
| *** salv-orl_ has joined #openstack-kolla | 07:41 | |
| *** zhubingbing has quit IRC | 07:41 | |
| *** zhubingbing has joined #openstack-kolla | 07:42 | |
| *** neilus has joined #openstack-kolla | 07:43 | |
| *** salv-orlando has quit IRC | 07:44 | |
| *** senk has joined #openstack-kolla | 07:46 | |
| *** duonghq has joined #openstack-kolla | 07:46 | |
| *** neilus has quit IRC | 07:48 | |
| *** senk has quit IRC | 07:55 | |
| *** egonzalez90 has joined #openstack-kolla | 07:59 | |
| *** kong has quit IRC | 08:01 | |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Fix murano auth issue https://review.openstack.org/372279 | 08:02 |
| *** shardy has joined #openstack-kolla | 08:04 | |
| *** gfidente has joined #openstack-kolla | 08:07 | |
| *** egonzalez90 has quit IRC | 08:12 | |
| *** zhubingbing_ has joined #openstack-kolla | 08:12 | |
| *** mgoddard has joined #openstack-kolla | 08:13 | |
| *** zhubingbing has quit IRC | 08:13 | |
| *** msimonin1 has joined #openstack-kolla | 08:15 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Create the heat_stack_user/owner role in the heat role https://review.openstack.org/351220 | 08:17 |
| *** msimonin has quit IRC | 08:18 | |
| zhubingbing_ | ping prameswar | 08:19 |
| *** dwalsh has joined #openstack-kolla | 08:20 | |
| berendt | zhubingbing_ please review https://review.openstack.org/#/c/372250/ | 08:22 |
| zhubingbing_ | ok | 08:22 |
| zhubingbing_ | i review now | 08:23 |
| openstackgerrit | Merged openstack/kolla: Fix the kolla_docker issue with docker 1.12 https://review.openstack.org/358002 | 08:23 |
| openstackgerrit | Merged openstack/kolla: Fix reconfiguration issue for Keystone role https://review.openstack.org/371617 | 08:23 |
| zhubingbing_ | ok | 08:25 |
| zhubingbing_ | LATM | 08:25 |
| zhubingbing_ | LGTM | 08:25 |
| *** msimonin1 has quit IRC | 08:25 | |
| *** msimonin has joined #openstack-kolla | 08:25 | |
| *** athomas has joined #openstack-kolla | 08:26 | |
| zhubingbing_ | hi berendt | 08:28 |
| zhubingbing_ | https://review.openstack.org/#/c/369190/ | 08:28 |
| zhubingbing_ | https://bugs.launchpad.net/kolla/+bug/1622878 | 08:28 |
| openstack | Launchpad bug 1622878 in kolla "Deprecate nova-network container" [Critical,Confirmed] | 08:29 |
| berendt | zhubingbing_ what about this? | 08:30 |
| openstackgerrit | zhongshengping proposed openstack/kolla: Deprecate old auth parameters of ironic in nova.conf https://review.openstack.org/372306 | 08:32 |
| *** egonzalez90 has joined #openstack-kolla | 08:40 | |
| openstackgerrit | Merged openstack/kolla: Fix vpnaas for neutron https://review.openstack.org/372250 | 08:45 |
| *** egonzalez90 has quit IRC | 08:46 | |
| *** Serlex has joined #openstack-kolla | 08:48 | |
| openstackgerrit | Wei Cao proposed openstack/kolla: Fix ironic-inspector Dockerfile https://review.openstack.org/372314 | 08:48 |
| *** papacz has joined #openstack-kolla | 09:00 | |
| *** tonanhngo has joined #openstack-kolla | 09:00 | |
| *** Serlex has quit IRC | 09:01 | |
| *** Serlex has joined #openstack-kolla | 09:04 | |
| *** tonanhngo has quit IRC | 09:05 | |
| openstackgerrit | dupengfei proposed openstack/kolla: Fix keystone,database,memcache for k8s. https://review.openstack.org/372089 | 09:09 |
| *** dwalsh has quit IRC | 09:09 | |
| *** phuongnh has joined #openstack-kolla | 09:10 | |
| *** jmccarthy has quit IRC | 09:14 | |
| Jeffrey4l_ | zhubingbing_, this related to doc change not implement ansible role please do not mark it as invalid https://bugs.launchpad.net/kolla/+bug/1605566 | 09:14 |
| openstack | Launchpad bug 1605566 in kolla " implement Ansible Tempest role" [Medium,Invalid] - Assigned to Jeffrey Zhang (jeffrey4l) | 09:14 |
| *** jmccarthy has joined #openstack-kolla | 09:15 | |
| *** daneyon has joined #openstack-kolla | 09:16 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Source the openrc file automatically when running tools/init-runonce https://review.openstack.org/371725 | 09:16 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Deprecate nova-network container https://review.openstack.org/369190 | 09:19 |
| *** daneyon has quit IRC | 09:21 | |
| *** egonzalez90 has joined #openstack-kolla | 09:41 | |
| *** ankush has quit IRC | 09:42 | |
| *** ankush has joined #openstack-kolla | 09:44 | |
| *** awiddersheim has quit IRC | 09:45 | |
| zhubingbing_ | ping Liyingjun | 09:49 |
| *** kong has joined #openstack-kolla | 09:49 | |
| *** Marx314 has quit IRC | 09:54 | |
| *** lamt has quit IRC | 09:55 | |
| hogepodge | hi, so for a number of different tasks, it looks like I need to be root to deploy, yes? | 09:59 |
| hogepodge | work is in progress to enable become on a bunch of items, but in the meantime I need to be root? | 10:00 |
| *** zhurong has quit IRC | 10:00 | |
| hogepodge | based on looking at this topic https://review.openstack.org/#/q/topic:bp/ansible-specific-task-become | 10:01 |
| *** williamcaban has joined #openstack-kolla | 10:04 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Add "---" into ceilometer/tasks https://review.openstack.org/372366 | 10:07 |
| *** williamcaban has quit IRC | 10:09 | |
| *** egonzalez90 has quit IRC | 10:13 | |
| *** phuongnh has quit IRC | 10:15 | |
| bjolo | berendt, yes, after adding vpnaas to inventory, it worked. | 10:20 |
| bjolo | berendt, whats odd tho is that my deployments has worked for quite a while even w/o vpnaas in inventory (i do at least 1 fresh deployment per day) | 10:21 |
| *** ssurana has quit IRC | 10:24 | |
| *** egonzalez90 has joined #openstack-kolla | 10:25 | |
| *** duonghq has quit IRC | 10:26 | |
| openstackgerrit | Wei Cao proposed openstack/kolla: Delete extra spaces in gnocchi config.yml https://review.openstack.org/372378 | 10:27 |
| *** openstackstatus has quit IRC | 10:28 | |
| openstackgerrit | Md Nadeem proposed openstack/kolla: Added enable_option for some services https://review.openstack.org/372390 | 10:37 |
| openstackgerrit | Md Nadeem proposed openstack/kolla: Added Barbican in the list of services deployed by kolla https://review.openstack.org/372395 | 10:50 |
| openstackgerrit | Wei Cao proposed openstack/kolla: Fix ironic-inspector Dockerfile https://review.openstack.org/372314 | 10:51 |
| openstackgerrit | Wei Cao proposed openstack/kolla: Fix ironic Dockerfile https://review.openstack.org/372314 | 10:55 |
| *** ntpttr_ has quit IRC | 11:00 | |
| *** ntpttr_ has joined #openstack-kolla | 11:02 | |
| *** ntpttr_ has quit IRC | 11:02 | |
| *** ntpttr_ has joined #openstack-kolla | 11:04 | |
| *** manjeets_ has quit IRC | 11:04 | |
| *** pumaranikar has quit IRC | 11:04 | |
| *** jascott1 has joined #openstack-kolla | 11:05 | |
| bdaca | zhubingbing are you here? | 11:06 |
| bdaca | zhubingbing_ are you here? | 11:07 |
| zhubingbing_ | here | 11:07 |
| zhubingbing_ | i am here | 11:07 |
| zhubingbing_ | what 's matter | 11:07 |
| bdaca | hi | 11:07 |
| zhubingbing_ | hi | 11:07 |
| bdaca | can we talk about https://bugs.launchpad.net/kolla/+bug/1620374 for a second? | 11:07 |
| openstack | Launchpad bug 1620374 in kolla "Murano needs dedicated rabbitmq" [Critical,Won't fix] | 11:07 |
| *** dwalsh has joined #openstack-kolla | 11:08 | |
| bdaca | I added a comment for this bug, can you tell me what you think about it? | 11:08 |
| openstackgerrit | Wei Cao proposed openstack/kolla: Fix ironic Dockerfile https://review.openstack.org/372314 | 11:11 |
| *** dwalsh has quit IRC | 11:13 | |
| *** caowei has quit IRC | 11:18 | |
| *** AndChat-382025 has quit IRC | 11:21 | |
| openstackgerrit | dupengfei proposed openstack/kolla: Support shared path for glance images and nova instances disk https://review.openstack.org/372420 | 11:23 |
| *** senk has joined #openstack-kolla | 11:23 | |
| *** dwalsh has joined #openstack-kolla | 11:25 | |
| *** ccesario has joined #openstack-kolla | 11:26 | |
| *** egonzalez90 has quit IRC | 11:32 | |
| *** zhubingbing_ has quit IRC | 11:35 | |
| *** mliima has joined #openstack-kolla | 11:37 | |
| *** rstarmer has quit IRC | 11:41 | |
| *** ccesario has quit IRC | 11:42 | |
| *** eaguilar has joined #openstack-kolla | 11:43 | |
| openstackgerrit | Merged openstack/kolla: Added Barbican in the list of services deployed by kolla https://review.openstack.org/372395 | 11:44 |
| *** ccesario has joined #openstack-kolla | 11:45 | |
| *** sdake has joined #openstack-kolla | 11:49 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fixing cinder/iscsi/tgtd issues https://review.openstack.org/372429 | 11:49 |
| *** AndChat|382025 has joined #openstack-kolla | 11:52 | |
| *** egonzalez90 has joined #openstack-kolla | 11:52 | |
| pbourke | anyone seeing problems with merge_configs.py rendering l3_agent.ini ? | 11:56 |
| *** schwicht_at_work has quit IRC | 11:58 | |
| *** daneyon has joined #openstack-kolla | 11:58 | |
| *** rstarmer has joined #openstack-kolla | 11:58 | |
| *** shardy is now known as shardy_lunch | 12:01 | |
| *** bdaca has quit IRC | 12:02 | |
| *** daneyon has quit IRC | 12:03 | |
| *** egonzalez90 has quit IRC | 12:03 | |
| openstackgerrit | Merged openstack/kolla: corrected error in multinode.rst https://review.openstack.org/371961 | 12:09 |
| *** zhubingbing has joined #openstack-kolla | 12:13 | |
| zhubingbing | ping bdaca | 12:13 |
| *** sdake has quit IRC | 12:13 | |
| *** sdake has joined #openstack-kolla | 12:14 | |
| zhubingbing | hi sdake | 12:14 |
| sdake | sup zhubingbing | 12:15 |
| zhubingbing | I found a few problems in the finishing bug, you need to confirm the next | 12:16 |
| zhubingbing | 1. https://bugs.launchpad.net/kolla/+bug/1620374 | 12:16 |
| openstack | Launchpad bug 1620374 in kolla "Murano needs dedicated rabbitmq" [Critical,Won't fix] | 12:16 |
| *** srwilkers has joined #openstack-kolla | 12:17 | |
| *** ccesario has quit IRC | 12:17 | |
| *** rhallisey has joined #openstack-kolla | 12:18 | |
| zhubingbing | What status do you think the state of this bug should be set up | 12:18 |
| zhubingbing | ping <bdaca> | 12:18 |
| zhubingbing | ping bdaca | 12:18 |
| mliima | morning sdake | 12:20 |
| mliima | ops | 12:20 |
| mliima | morning all | 12:20 |
| mliima | :) | 12:20 |
| sdake | zhubingbing feature request | 12:21 |
| sdake | zhubingbing seems low priority | 12:21 |
| zhubingbing | yes | 12:21 |
| sdake | ut i know pbourke uses murano | 12:21 |
| zhubingbing | pbourke | 12:21 |
| sdake | zhubingbing so i leave it in his capable hand to make the final judgement call on thatone | 12:21 |
| sdake | sup mliima | 12:21 |
| zhubingbing | yes i think so | 12:22 |
| sdake | mliima - whatis your analysis o the state of kolla for things you work on? | 12:22 |
| *** rhallisey has quit IRC | 12:22 | |
| *** Serlex has quit IRC | 12:22 | |
| sdake | morning rh | 12:22 |
| *** rstarmer has quit IRC | 12:22 | |
| sdake | sbezverk i was thinkingof tagging 0.3.0 - to indicate OVS + nova work | 12:23 |
| sdake | sbezverk premature - or is that tag good to g? | 12:23 |
| zhubingbing | murano agents and murano engine is realised by the rabbitmq. | 12:23 |
| sdake | (of kolla-kubernetes) | 12:23 |
| sdake | zhubingbing i read the bug report yesterday | 12:24 |
| sdake | i didn't knwo what ot do with it either | 12:24 |
| sdake | so left it alone | 12:24 |
| zhubingbing | yes | 12:24 |
| sdake | if pbourke says "IDC"then it gets bounced to ocata -if pbouke sys IDOcare" then i'm surehis team can find a solutino thatworks | 12:25 |
| *** rhallisey has joined #openstack-kolla | 12:26 | |
| zhubingbing | so this bug status setting to opinion | 12:27 |
| *** ayoung has joined #openstack-kolla | 12:28 | |
| *** dave-mccowan has joined #openstack-kolla | 12:28 | |
| *** dave-mccowan has quit IRC | 12:28 | |
| *** dave-mccowan has joined #openstack-kolla | 12:29 | |
| mliima | sdake, I had some problems with the rabbitmq, but I'm trying to verify this now. | 12:29 |
| mliima | the container wasn't up | 12:29 |
| sdake | mliima expand - we recently changed rabbimq versions | 12:29 |
| mliima | yes | 12:30 |
| mliima | inc0 told me | 12:30 |
| mliima | i'm rebuilding now | 12:30 |
| mliima | kolla has working fine, the most of the problems I had was because of some mistake i did | 12:32 |
| mliima | :) | 12:32 |
| sdake | roger | 12:32 |
| sdake | can you tell me if mitaka or lierty are affected | 12:33 |
| *** fguillot has joined #openstack-kolla | 12:34 | |
| *** ccesario has joined #openstack-kolla | 12:37 | |
| mliima | sdake, about my issue with mariadb_bootstrap | 12:37 |
| mliima | https://bugs.launchpad.net/kolla/+bug/1609502 | 12:37 |
| openstack | Launchpad bug 1609502 in kolla "Container bootstrap_mariadb exits due to Galera failed" [Critical,Confirmed] | 12:37 |
| *** david-lyle has joined #openstack-kolla | 12:38 | |
| zhubingbing | mliima, this bug have solved ？ | 12:39 |
| mliima | i don't know | 12:39 |
| mliima | it was? | 12:39 |
| sdake | peple working gon that i think | 12:39 |
| sdake | huikang in particular | 12:40 |
| mliima | :) | 12:40 |
| *** caowei has joined #openstack-kolla | 12:41 | |
| *** srwilkers has quit IRC | 12:41 | |
| sdake | zhubingbing any other bugs of question? | 12:42 |
| zhubingbing | 2. magnum | 12:42 |
| zhubingbing | wait | 12:42 |
| sdake | essential priority | 12:42 |
| sdake | +critical priority | 12:42 |
| sdake | in other words - it needs top focus | 12:43 |
| zhubingbing | This bug looks like I'm going to change it. | 12:43 |
| *** david-lyle has quit IRC | 12:43 | |
| sdake | may make sense to take over vhosakot's patches | 12:43 |
| zhubingbing | vhosakot have long time not on irc | 12:45 |
| zhubingbing | https://bugs.launchpad.net/kolla/+bug/1551992 | 12:45 |
| openstack | Launchpad bug 1551992 in kolla "Magnum failed to create trustee" [Critical,In progress] - Assigned to zhubingbing (zhubingbing) | 12:45 |
| zhubingbing | For magnum I have a question, cluster-create magnum seems to have no this command | 12:46 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Document that new services have to be listed in the README file https://review.openstack.org/372460 | 12:46 |
| openstackgerrit | Merged openstack/kolla: Add "---" into ceilometer/tasks https://review.openstack.org/372366 | 12:46 |
| *** fguillot is now known as fguillot_afk | 12:47 | |
| *** fguillot_afk has quit IRC | 12:48 | |
| *** shardy_lunch is now known as shardy | 12:49 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Remove duplicate lbaas provider config https://review.openstack.org/372216 | 12:50 |
| openstackgerrit | Merged openstack/kolla: Delete extra spaces in gnocchi config.yml https://review.openstack.org/372378 | 12:52 |
| zhubingbing | http://egonzalez.org/magnum-in-rdo-openstack-liberty-manual-installation-from-source-code/ | 12:52 |
| *** fguillot has joined #openstack-kolla | 12:52 | |
| zhubingbing | 90 | 12:52 |
| *** rstarmer has joined #openstack-kolla | 12:53 | |
| zhubingbing | the blog look is egonzalez90 | 12:53 |
| *** jheroux has joined #openstack-kolla | 12:54 | |
| pbourke | I agree low priority is ok for that bug | 12:55 |
| zhubingbing | ok | 12:55 |
| zhubingbing | thanks pbourke | 12:55 |
| *** jtriley has joined #openstack-kolla | 12:55 | |
| zhubingbing | https://bugs.launchpad.net/kolla/+bug/1620374 | 12:56 |
| openstack | Launchpad bug 1620374 in kolla "Murano needs dedicated rabbitmq" [Low,Confirmed] | 12:56 |
| zhubingbing | pbourke | 12:57 |
| *** v1k0d3n has joined #openstack-kolla | 13:02 | |
| *** dwalsh has quit IRC | 13:03 | |
| *** papacz has quit IRC | 13:05 | |
| *** papacz has joined #openstack-kolla | 13:05 | |
| *** LamT_ has joined #openstack-kolla | 13:08 | |
| *** gfhellma has joined #openstack-kolla | 13:10 | |
| *** eaguilar has quit IRC | 13:13 | |
| *** zhubingbing has quit IRC | 13:16 | |
| *** fguillot is now known as fguillot_afk | 13:17 | |
| *** AndChat|382025 has quit IRC | 13:17 | |
| openstackgerrit | bjorn lofdahl proposed openstack/kolla: simple fix. Checking valid fernet expiry only need to run once https://review.openstack.org/372483 | 13:17 |
| *** dwalsh has joined #openstack-kolla | 13:19 | |
| *** schwicht has joined #openstack-kolla | 13:20 | |
| *** lrensing has joined #openstack-kolla | 13:21 | |
| *** zhubingbing has joined #openstack-kolla | 13:21 | |
| *** zhurong has joined #openstack-kolla | 13:23 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Use a named volume or a shared directory to store Cloudkitty states https://review.openstack.org/372490 | 13:24 |
| *** srwilkers has joined #openstack-kolla | 13:27 | |
| *** ishu has quit IRC | 13:31 | |
| *** eaguilar has joined #openstack-kolla | 13:31 | |
| *** dwalsh has quit IRC | 13:37 | |
| mliima | rabbitmq container still restartning | 13:39 |
| *** huikang has joined #openstack-kolla | 13:40 | |
| *** Marx314 has joined #openstack-kolla | 13:40 | |
| *** kong has quit IRC | 13:41 | |
| *** williamcaban has joined #openstack-kolla | 13:41 | |
| *** salv-orlando has joined #openstack-kolla | 13:42 | |
| *** salv-orl_ has quit IRC | 13:45 | |
| *** srwilkers has quit IRC | 13:45 | |
| *** williamcaban has quit IRC | 13:46 | |
| zhubingbing | hi | 13:46 |
| zhubingbing | ping huikang | 13:46 |
| huikang | pong | 13:46 |
| zhubingbing | ping mliima | 13:48 |
| openstackgerrit | Merged openstack/kolla: Remove duplicate lbaas provider config https://review.openstack.org/372216 | 13:48 |
| mliima | \o/ zhubingbing | 13:48 |
| zhubingbing | https://bugs.launchpad.net/kolla/+bug/1609502 | 13:48 |
| openstack | Launchpad bug 1609502 in kolla "Container bootstrap_mariadb exits due to Galera failed" [Critical,Confirmed] | 13:48 |
| mliima | ? | 13:49 |
| zhubingbing | huikang i see you confirm this bug, you can help mliim to solve this bug | 13:49 |
| zhubingbing | can you | 13:49 |
| zhubingbing | － － | 13:49 |
| zhubingbing | －） | 13:49 |
| *** inc0 has joined #openstack-kolla | 13:50 | |
| huikang | let me take a look | 13:50 |
| zhubingbing | ok | 13:51 |
| zhubingbing | thanks | 13:51 |
| *** Serlex has joined #openstack-kolla | 13:53 | |
| inc0 | good morning | 13:54 |
| mliima | morning inc0 | 13:54 |
| *** tonanhngo has joined #openstack-kolla | 13:54 | |
| huikang | morning everyone | 13:55 |
| *** huikang has quit IRC | 13:57 | |
| *** huikang has joined #openstack-kolla | 13:57 | |
| *** dave-mccowan has quit IRC | 14:00 | |
| *** AndChat|382025 has joined #openstack-kolla | 14:00 | |
| *** huikang has quit IRC | 14:01 | |
| *** huikang has joined #openstack-kolla | 14:04 | |
| *** gfhellma has quit IRC | 14:04 | |
| *** eaguilar_ has joined #openstack-kolla | 14:07 | |
| *** eaguilar has quit IRC | 14:07 | |
| *** egonzalez90 has joined #openstack-kolla | 14:08 | |
| *** dave-mccowan has joined #openstack-kolla | 14:11 | |
| *** huikang has quit IRC | 14:14 | |
| zhubingbing | good morning inc0 | 14:15 |
| *** huikang has joined #openstack-kolla | 14:15 | |
| *** fguillot has joined #openstack-kolla | 14:15 | |
| *** dave-mcc_ has joined #openstack-kolla | 14:15 | |
| *** huikang has quit IRC | 14:16 | |
| *** dave-mccowan has quit IRC | 14:16 | |
| *** huikang has joined #openstack-kolla | 14:16 | |
| *** wznoinsk has quit IRC | 14:20 | |
| *** eaguilar_ has quit IRC | 14:20 | |
| hogepodge | Hmm, it's really worrying how often bootstrap containers fail. Sometimes it's fatal, sometimes not. | 14:21 |
| *** schwicht has quit IRC | 14:22 | |
| hogepodge | Is there a way to do introspection on the failure? See what exactly went wrong? They all exit with non-zero status then the container is dead | 14:23 |
| openstackgerrit | Merged openstack/kolla: Enable CADF events in Keystone and introduce enable_cadf_notifications https://review.openstack.org/349006 | 14:23 |
| *** mickt has joined #openstack-kolla | 14:24 | |
| openstackgerrit | Merged openstack/kolla: Fix ironic Dockerfile https://review.openstack.org/372314 | 14:24 |
| inc0 | hogepodge, docker logs should have some info | 14:24 |
| inc0 | that and logs of services | 14:24 |
| *** wznoinsk has joined #openstack-kolla | 14:27 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Install extra packages for gnocchi using setuptools extra feature https://review.openstack.org/372529 | 14:30 |
| hogepodge | inc0: thanks, it looks like the problem is an unreliable database connection :-/ | 14:33 |
| *** strigazi has quit IRC | 14:33 | |
| inc0 | hogepodge, what was the error? | 14:33 |
| hogepodge | inc0: "'Lost connection to MySQL server during query" while doing db_sync | 14:34 |
| hogepodge | inc0: sometimes it makes it through to the end, sometimes it doesn't | 14:34 |
| inc0 | yeah, seems like network issue | 14:34 |
| inc0 | especially that I haven't seen behavior like that before, and nobody reported it | 14:34 |
| openstackgerrit | Andrew Widdersheim proposed openstack/kolla: Use ansible_version when available for precheck https://review.openstack.org/372535 | 14:35 |
| inc0 | so might be something local to your env | 14:35 |
| *** strigazi has joined #openstack-kolla | 14:35 | |
| inc0 | but let's be sure, if it's kolla issue, it's potentially critica | 14:35 |
| inc0 | l | 14:35 |
| hogepodge | inc0: yeah, it's an allinone on a nuc, and the network may be funny | 14:35 |
| inc0 | well if you're all in one it shouldn't matter right? | 14:35 |
| openstackgerrit | Merged openstack/kolla: fix config file permission to ironic instead of root https://review.openstack.org/372084 | 14:36 |
| inc0 | hogepodge, check mariadb logs too | 14:37 |
| *** openstackgerrit has quit IRC | 14:37 | |
| *** openstackgerrit has joined #openstack-kolla | 14:37 | |
| *** awiddersheim has joined #openstack-kolla | 14:40 | |
| *** david-lyle has joined #openstack-kolla | 14:40 | |
| sdake | 145 people in channel | 14:42 |
| sdake | going to be a busy week | 14:43 |
| inc0 | sdake, what's current record? | 14:44 |
| sdake | 148 from my informal poking | 14:44 |
| *** david-lyle has quit IRC | 14:45 | |
| *** awiddersheim has quit IRC | 14:46 | |
| *** zhubingbing has quit IRC | 14:47 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the failure of ceilometer-api container https://review.openstack.org/372161 | 14:47 |
| *** zhubingbing has joined #openstack-kolla | 14:48 | |
| zhubingbing | hello guys | 14:48 |
| zhubingbing | Let's get up. | 14:49 |
| *** awiddersheim has joined #openstack-kolla | 14:53 | |
| awiddersheim | found that doing builds with rhel as the base | 14:54 |
| awiddersheim | is not working | 14:54 |
| awiddersheim | not sure if that is a bug or desired | 14:54 |
| zhubingbing | reel ？ | 14:54 |
| zhubingbing | rhel ？ | 14:55 |
| awiddersheim | yes | 14:55 |
| zhubingbing | # Valid options are [ centos, fedora, oraclelinux, ubuntu ] | 14:55 |
| awiddersheim | https://github.com/openstack/kolla/blob/master/kolla/common/config.py#L21 | 14:56 |
| awiddersheim | right | 14:56 |
| awiddersheim | but rhel does have a version in the release dict | 14:56 |
| awiddersheim | and it's in the docs as being something possible | 14:56 |
| *** senk has quit IRC | 14:56 | |
| awiddersheim | https://github.com/openstack/kolla/blob/master/kolla/common/config.py#L24 | 14:57 |
| zhubingbing | emm | 14:57 |
| zhubingbing | right | 14:57 |
| awiddersheim | http://docs.openstack.org/developer/kolla/image-building.html#build-openstack-from-source | 14:57 |
| awiddersheim | has rhel as an option | 14:58 |
| awiddersheim | I can make pull request | 14:58 |
| zhubingbing | you can test redhet build images | 14:59 |
| zhubingbing | root@ip-172-31-22-78:~# kolla-build aodh --base redhat --type source | 15:01 |
| zhubingbing | argument --base: Invalid String(choices=['centos', 'ubuntu', 'oraclelinux', 'debian']) value: redhat | 15:01 |
| zhubingbing | i think it is a bug | 15:01 |
| awiddersheim | right | 15:01 |
| awiddersheim | ok | 15:01 |
| awiddersheim | I'll make pull request | 15:01 |
| zhubingbing | you can commit this a bug ? | 15:01 |
| zhubingbing | can you ? | 15:01 |
| *** jogam has quit IRC | 15:02 | |
| *** egonzalez90 has left #openstack-kolla | 15:04 | |
| awiddersheim | umm I"m nto sure | 15:04 |
| awiddersheim | like permissions wise? | 15:05 |
| zhubingbing | yes | 15:05 |
| zhubingbing | You can make a contribution to kolla. | 15:06 |
| awiddersheim | I've never tried to make a bug before | 15:06 |
| awiddersheim | so I dont' know | 15:06 |
| zhubingbing | i teach you | 15:07 |
| awiddersheim | great thanks | 15:07 |
| zhubingbing | https://bugs.launchpad.net/kolla | 15:07 |
| zhubingbing | you can open this link | 15:08 |
| awiddersheim | ok | 15:08 |
| zhubingbing | you have launchpad id ？ | 15:08 |
| awiddersheim | yes | 15:08 |
| zhubingbing | good | 15:08 |
| zhubingbing | you can choose report a bug | 15:08 |
| awiddersheim | ok | 15:09 |
| zhubingbing | Right at the top of the page | 15:09 |
| awiddersheim | yup got it | 15:09 |
| awiddersheim | on 'enter a summary' | 15:09 |
| *** bdaca has joined #openstack-kolla | 15:09 | |
| zhubingbing | Follow the prompts to describe your bug | 15:09 |
| zhubingbing | yes | 15:09 |
| awiddersheim | ok | 15:09 |
| bdaca | hello all | 15:10 |
| zhubingbing | Take a meaningful title | 15:10 |
| zhubingbing | hi bdaca | 15:10 |
| *** mgoddard_ has joined #openstack-kolla | 15:11 | |
| *** mgoddard has quit IRC | 15:11 | |
| *** caowei has quit IRC | 15:11 | |
| bdaca | zhubingbing thanks for your answer | 15:12 |
| awiddersheim | https://bugs.launchpad.net/kolla/+bug/1625229 | 15:12 |
| openstack | Launchpad bug 1625229 in kolla "Using rhel as a base image is not supported" [Undecided,New] | 15:12 |
| awiddersheim | done | 15:12 |
| zhubingbing | ok | 15:12 |
| zhubingbing | good | 15:12 |
| zhubingbing | you can commit a patch to solve this bug | 15:13 |
| awiddersheim | ok | 15:13 |
| awiddersheim | and I reference this bug | 15:13 |
| awiddersheim | in the patch | 15:13 |
| awiddersheim | commit message | 15:13 |
| awiddersheim | correct? | 15:13 |
| bdaca | ping pbourke | 15:14 |
| zhubingbing | http://docs.openstack.org/contributor-guide/index.html | 15:14 |
| zhubingbing | you can read this guide | 15:15 |
| *** dwalsh has joined #openstack-kolla | 15:15 | |
| zhubingbing | how to contribute to openstack | 15:15 |
| *** schwicht has joined #openstack-kolla | 15:15 | |
| zhubingbing | bdaca, you can contact sdake or pbourke | 15:16 |
| zhubingbing | hmm | 15:16 |
| pbourke | bdaca: hi | 15:16 |
| *** manjeets has joined #openstack-kolla | 15:16 | |
| pbourke | bdaca: can I help? | 15:16 |
| *** pbourke has quit IRC | 15:18 | |
| *** pbourke has joined #openstack-kolla | 15:18 | |
| openstackgerrit | Andrew Widdersheim proposed openstack/kolla: Fix using rhel as a base build image https://review.openstack.org/372570 | 15:18 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Use a named volume or a shared directory to store Cloudkitty states https://review.openstack.org/372490 | 15:18 |
| zhubingbing | wow http://docs.openstack.org/contributor-guide/index.html | 15:19 |
| zhubingbing | Andrew Widdersheim | 15:19 |
| zhubingbing | good advice | 15:19 |
| zhubingbing | commit | 15:19 |
| awiddersheim | thanks! | 15:19 |
| awiddersheim | thanks for your help | 15:20 |
| zhubingbing | np | 15:21 |
| sbezverk | akwasnie: ping | 15:21 |
| zhubingbing | This is what I should do, and I'm glad to see you submit. | 15:21 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Checking valid fernet expiry only need to run once https://review.openstack.org/372483 | 15:22 |
| *** msimonin has quit IRC | 15:23 | |
| *** dwalsh has quit IRC | 15:26 | |
| *** Marcellin__ has joined #openstack-kolla | 15:28 | |
| *** williamcaban has joined #openstack-kolla | 15:29 | |
| mliima | guys | 15:30 |
| mliima | after deploy, the container rabbitmq isn't up | 15:30 |
| mliima | http://paste.openstack.org/show/581144/ | 15:30 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the failure of ceilometer-api container https://review.openstack.org/372161 | 15:31 |
| mliima | i run the build using --no-cache | 15:31 |
| mliima | without --no-cache | 15:31 |
| mliima | but nothing works | 15:31 |
| *** david-lyle has joined #openstack-kolla | 15:32 | |
| *** fguillot has quit IRC | 15:32 | |
| *** vhosakot has joined #openstack-kolla | 15:34 | |
| Jeffrey4l_ | mliima, could u paste your docker images | 15:34 |
| mliima | ok | 15:34 |
| *** williamcaban has quit IRC | 15:34 | |
| Jeffrey4l_ | and docker logs rabbitmq | 15:34 |
| mliima | http://paste.openstack.org/show/581145/ | 15:34 |
| *** daneyon has joined #openstack-kolla | 15:35 | |
| mliima | docker logs rabbitmq -> http://paste.openstack.org/show/581150/ | 15:35 |
| Jeffrey4l_ | is there any log in /var/lib/docker/volumes/kolla_logs/_data/rabbitmq mliima ? | 15:36 |
| mliima | root@ubuntu:~/kolla# ls /var/lib/docker/volumes/kolla_logs/_data/rabbitmq/ | 15:37 |
| mliima | startup_err startup_log | 15:37 |
| mliima | ERROR: epmd error for host ubuntu: address (cannot connect to host/port) | 15:37 |
| zhubingbing | cannot connect to host/port) | 15:37 |
| mliima | the prechecks worked fine | 15:38 |
| sdake | i was paged zhubingbing ? | 15:38 |
| sdake | mliima when is lsat time you rebuit your containes | 15:38 |
| zhubingbing | hostname | 15:38 |
| sdake | oh i figured this one out | 15:38 |
| sdake | moment let me reach into the depths of my mind palace | 15:38 |
| zhubingbing | you can check your hostname | 15:38 |
| mliima | 1 hour | 15:38 |
| sdake | mliima pip show kolla | 15:38 |
| sdake | mliima run pip show kollaplease | 15:38 |
| mliima | http://paste.openstack.org/show/581151/ | 15:39 |
| sdake | mliima the docs have changed | 15:39 |
| sdake | mliima run pip uninstall kolla | 15:39 |
| *** daneyon has quit IRC | 15:39 | |
| sdake | mliima reference the delta in the docs between develoeprworkflow andoperator workflow | 15:40 |
| sdake | mliima rebuild your master images after pip uninstall kolla | 15:40 |
| sdake | and you will be good to go | 15:40 |
| Jeffrey4l_ | sdake, old kolla? | 15:41 |
| sdake | i wonder ifsomething is fishy with upgrades and rabbitmq 2.6.z | 15:41 |
| sdake | Jeffrey4l_ mliima is working on master | 15:41 |
| mliima | yes | 15:41 |
| Jeffrey4l_ | mliima, try `docker run 10.102.20.225:4000/kolla/ubuntu-source-rabbitmq dpkg -l | grep rabbitmq ` | 15:42 |
| *** ssurana has joined #openstack-kolla | 15:42 | |
| mliima | Unable to find image '10.102.20.225:4000/kolla/ubuntu-source-rabbitmq:latest' locally | 15:43 |
| mliima | Pulling repository 10.102.20.225:4000/kolla/ubuntu-source-rabbitmq | 15:43 |
| mliima | docker: Error: image kolla/ubuntu-source-rabbitmq:latest not found | 15:43 |
| Jeffrey4l_ | mliima, try `docker run 10.102.20.225:4000/kolla/ubuntu-source-rabbitmq:3.0.0 dpkg -l | grep rabbitmq ` | 15:43 |
| Jeffrey4l_ | sorry, left out the tag number. | 15:43 |
| mliima | root@ubuntu:~# docker run 10.102.20.225:4000/kolla/ubuntu-source-rabbitmq:3.0.0 dpkg -l | grep rabbitmq | 15:43 |
| mliima | ii rabbitmq-server 3.6.2-1 all Multi-protocol messaging broker | 15:43 |
| sdake | mliima the act that you have rabbitmq-server 3.6.2 is irrelevant to fixing your issue :) | 15:44 |
| sdake | act/fact | 15:44 |
| zhubingbing | wow | 15:45 |
| mliima | sdake | 15:46 |
| mliima | "rebuild your master images after pip uninstall kolla" | 15:46 |
| zhubingbing | docker run 10.102.20.225:4000/kolla/ubuntu-source-rabbitmq:3.0.0 dpkg -l | grep rabbitmq Docker can also be used in this way | 15:46 |
| pbourke | zhubingbing: Jeffrey4l_: any strong feelings on having a separate rabbitmq for murano-agent? | 15:47 |
| pbourke | zhubingbing: Jeffrey4l_: it seems a more secure way to go | 15:47 |
| mliima | so i shouldn't run pip install kolla? | 15:47 |
| Jeffrey4l_ | pbourke, for secure reason, we should. | 15:47 |
| *** harlowja_at_home has joined #openstack-kolla | 15:48 | |
| Jeffrey4l_ | pbourke, but how about use a different vhost for the same rabbitmq instance? | 15:48 |
| pbourke | Jeffrey4l_: I share the concern of some people on this thread (http://www.gossamer-threads.com/lists/openstack/operators/57816) that it could pose a risk to the stability of the cluster overall | 15:48 |
| *** huikang has quit IRC | 15:49 | |
| *** huikang has joined #openstack-kolla | 15:49 | |
| pbourke | Jeffrey4l_: going by the thread we may potentially want to allow both | 15:50 |
| Jeffrey4l_ | is there any recommendation from mirantis for this? ( i treat magnum as mirantis project :)) | 15:52 |
| sdake | mliima no pip kolla should not be installed for master development | 15:53 |
| *** huikang has quit IRC | 15:53 | |
| sdake | pbourke which thread - you mean in the bug? | 15:54 |
| mliima | ok | 15:54 |
| sdake | Jeffrey4l_ magnum is not a miranits project | 15:54 |
| sdake | murano is single vendor mostly suppoorted y mirantis | 15:54 |
| Jeffrey4l_ | sdake, sorry, i meaned murano.. | 15:54 |
| Jeffrey4l_ | type :) | 15:54 |
| sdake | roger | 15:54 |
| Jeffrey4l_ | a typo :) | 15:54 |
| *** zhurong has quit IRC | 15:55 | |
| Jeffrey4l_ | i hate the implementation that agent installed in guest os need talked with my openstack cluster :( | 15:55 |
| pbourke | Jeffrey4l_: as far as i know they recommend a separate one | 15:55 |
| pbourke | bdaca: ^ | 15:55 |
| Jeffrey4l_ | ya. that's will be more secure | 15:56 |
| bdaca | ok, will do this way | 15:56 |
| bdaca | thanks | 15:56 |
| *** huikang has joined #openstack-kolla | 15:56 | |
| *** athomas has quit IRC | 15:59 | |
| *** schwicht has quit IRC | 16:00 | |
| *** tonanhngo has quit IRC | 16:01 | |
| *** tonanhngo has joined #openstack-kolla | 16:02 | |
| *** athomas has joined #openstack-kolla | 16:03 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fixing cinder/iscsi/tgtd issues https://review.openstack.org/372429 | 16:06 |
| *** matrohon has quit IRC | 16:08 | |
| *** huikang has quit IRC | 16:09 | |
| sdake | Jeffrey4l_, pbourke as far as I see it there are three options | 16:09 |
| sdake | intentionally disable murano until we can sort it outproperly in ocata | 16:09 |
| *** huikang has joined #openstack-kolla | 16:09 | |
| *** ssurana has quit IRC | 16:09 | |
| sdake | implement a separate rabbitmq for murano (since its a security thing) | 16:10 |
| sdake | document that when using murano as is, the security of the deployment is compromised because "X" | 16:10 |
| *** tonanhngo has quit IRC | 16:10 | |
| sdake | which choice to make is up to the folks doing the implementation work | 16:10 |
| Jeffrey4l_ | option 2 :) | 16:10 |
| sdake | i have no strnog feelings about any of them although i prefer 1 to 3 | 16:11 |
| *** tonanhngo has joined #openstack-kolla | 16:11 | |
| Jeffrey4l_ | the base line is: we need enable murano in newton. I do not think we need block it until O | 16:12 |
| sdake | Jeffrey4l_ wfm | 16:12 |
| sdake | Jeffrey4l_ sounds like 2 it is | 16:12 |
| Jeffrey4l_ | yep. | 16:12 |
| sdake | pbourke in agreement? | 16:12 |
| openstackgerrit | Merged openstack/kolla-kubernetes: The configmaps of iscsid and tgtd are error. https://review.openstack.org/372200 | 16:13 |
| Jeffrey4l_ | sdake, about the rc1 dead line ( talk in the ML) i think we should move rc1 dead line later ( we are tail project ) | 16:13 |
| *** huikang has quit IRC | 16:14 | |
| sdake | Jeffrey4l_ rc1 has shipped | 16:14 |
| sdake | Jeffrey4l_ join #openstack-release and lock it in as a permanent channel | 16:15 |
| Jeffrey4l_ | one case i hit is: ceilometer-api moved to apache at the end of rc1 and we have no time to change that. :( | 16:15 |
| Jeffrey4l_ | ok. | 16:15 |
| sdake | see this review: | 16:15 |
| Jeffrey4l_ | i mean the ideal progress for next cycle | 16:15 |
| sdake | https://review.openstack.org/372504 | 16:15 |
| sdake | the guidance from the release team is minimize rcs as much as possible | 16:17 |
| sdake | our rc2 should be complete and as perfect as we can make it | 16:17 |
| Jeffrey4l_ | so we have free time branch time. then release scheduler mean nothing for us. | 16:17 |
| sdake | including whatever is need ed to support rdo and uca | 16:17 |
| Jeffrey4l_ | after we tag rc1/rc2, we may still need make big change. | 16:18 |
| *** ssurana has joined #openstack-kolla | 16:18 | |
| sdake | Jeffrey4l_ not precisely - the release schedule milestones are to be followed | 16:18 |
| sdake | we have latitude on the rcs | 16:18 |
| sdake | the latitude comes rom the need to minimize rcs | 16:18 |
| *** tonanhngo has quit IRC | 16:18 | |
| Jeffrey4l_ | rcs means? | 16:18 |
| sdake | release candidates | 16:18 |
| sdake | obvously rc1 is a pretty rough releasse candidate | 16:19 |
| sdake | rc2 should be complete, stand alone, and be taggable as the released version | 16:19 |
| *** huikang has joined #openstack-kolla | 16:19 | |
| sdake | and if we can't make it such, we will have to sort itout in the z streams | 16:19 |
| *** schwicht has joined #openstack-kolla | 16:19 | |
| sdake | but no more features, at all | 16:20 |
| sdake | if a current feature is broken, thatis a bug - and that is perfectly acceptable | 16:20 |
| Jeffrey4l_ | ya. I know the rc means. but for tailing project, rc2 is 26/9 and final release date dead line is 21/10. isn't is too long between rc2 and final release? | 16:20 |
| sdake | it sounds like his murano case is it is broken | 16:20 |
| sdake | doug indicated we should tag rc2 when its "final" | 16:21 |
| Jeffrey4l_ | cool. that's what i want :) | 16:21 |
| sdake | and i we can't do that, we make it a source only release and handle binary in the zstreams | 16:21 |
| *** jascott1_ has joined #openstack-kolla | 16:22 | |
| Jeffrey4l_ | zstreams means? | 16:22 |
| *** jascott1_ has quit IRC | 16:22 | |
| *** tonanhngo has joined #openstack-kolla | 16:23 | |
| sdake | x.y.z | 16:23 |
| sdake | z= zstream | 16:23 |
| sdake | 2.0.2 is a zstream release | 16:23 |
| Jeffrey4l_ | got. | 16:23 |
| *** jascott1_ has joined #openstack-kolla | 16:23 | |
| pbourke | sdake: we'll go with 2), bdaca will implement | 16:23 |
| Jeffrey4l_ | pbourke, cool thanks bdaca | 16:23 |
| *** huikang has quit IRC | 16:24 | |
| bdaca | pleasure ;) | 16:24 |
| sdake | pbourke canyou mrk the bug critical and confirmed please | 16:25 |
| sdake | and i we/and if we | 16:25 |
| sdake | sorry for all the typos - my keyboard is worn out | 16:25 |
| bdaca | sdake can you take a look on https://review.openstack.org/#/c/367566/2 again? | 16:25 |
| sdake | bdaca still a -2 :( | 16:27 |
| *** jascott1_ has quit IRC | 16:27 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Fixing cinder/iscsi/tgtd issues https://review.openstack.org/372429 | 16:27 |
| *** jascott1_ has joined #openstack-kolla | 16:27 | |
| ntpttr | hey kolla folks, I have a question about upgrades | 16:29 |
| bdaca | sdake thanks ;) but forgive me, I don't understand :( this is cherry-pick from the master (it is fixed there by mliima), I just cherry-picked it to stable/mitaka and resolved the conflicts | 16:30 |
| ntpttr | I was doing some testing of rolling upgrades, and was curious about what happens to running processes, like an in progress backup happening in cinder or something. I asked around and learned that oslo.service will let running processes finish when it's shut down gracefully | 16:30 |
| ntpttr | does kolla make use of this in its upgrades, or is it wiping away old containers w/o sending a shutdown signal to the service they're running? | 16:31 |
| *** ssurana has quit IRC | 16:31 | |
| zhubingbing | bdaca ,Looks like there's been a plan for murano | 16:31 |
| *** gfidente is now known as gfidente|afk | 16:32 | |
| *** harlowja_at_home has quit IRC | 16:33 | |
| *** AndChat|382025 has quit IRC | 16:34 | |
| openstackgerrit | Andrew Widdersheim proposed openstack/kolla: Use ansible_version when available for precheck https://review.openstack.org/372535 | 16:34 |
| ntpttr | inc0 you worked on the kolla upgrades right, do you know the answer to this :)? ^ | 16:35 |
| Jeffrey4l_ | pbourke, sdake this PS https://review.openstack.org/372003 | 16:35 |
| Jeffrey4l_ | it block the mitaka gate. | 16:36 |
| sdake | unblocked | 16:36 |
| sdake | ntpttr docker sends a sigterm - waits for 10 seconds - does a segkill | 16:37 |
| inc0 | back | 16:37 |
| sdake | ntpttr if cinder finishes in this timeframe, the answer is yes | 16:37 |
| sdake | if not no ;) | 16:37 |
| Jeffrey4l_ | so fast, thaks. | 16:37 |
| sdake | bdaca oh - my bad | 16:38 |
| sdake | i didn't kno wit wsa a cherrypick | 16:38 |
| zhubingbing | go to sleep , good bye guys | 16:38 |
| ntpttr | hmm a backup likely won't be finish in 10 seconds | 16:38 |
| inc0 | sdake, Jeffrey4l_ so we need to make sure it's more than 10s | 16:38 |
| bdaca | bye zhubingbing | 16:38 |
| inc0 | we need cinder to be sigtermed indefinetly | 16:38 |
| inc0 | you think we can make dump init do this? | 16:38 |
| *** jascott1_ has quit IRC | 16:38 | |
| Jeffrey4l_ | docker stop -t xxx <container> | 16:38 |
| sdake | bdaca +2 | 16:38 |
| sdake | sleep well zhubingbing | 16:39 |
| Jeffrey4l_ | inc0, we need change the kolla_docker if we want., | 16:39 |
| *** jascott1_ has joined #openstack-kolla | 16:39 | |
| Jeffrey4l_ | -t, --time int Seconds to wait for stop before killing it (default 10) | 16:39 |
| *** zhubingbing has quit IRC | 16:39 | |
| *** AndChat|382025 has joined #openstack-kolla | 16:39 | |
| sdake | Jeffrey4l_ validate that is availalble in docker 1.10.0 pls | 16:39 |
| bdaca | sdake thanks ;) thought that I still don't understand the workflow ;) | 16:39 |
| Jeffrey4l_ | ok. | 16:40 |
| *** AndChat|382025 has quit IRC | 16:40 | |
| sdake | i'd like to keep us on docker 1.10.0 if possibble | 16:40 |
| sdake | as a minimum dep | 16:40 |
| sdake | docker moves fasat and 1.10 was the first version of docker that had everthing we need for kolla in operational shape | 16:41 |
| *** jascott1_ has quit IRC | 16:41 | |
| *** jascott1_ has joined #openstack-kolla | 16:41 | |
| inc0 | yeah for backup it take much longer... | 16:41 |
| inc0 | ntpttr does backup have any "retry" mechanism? | 16:42 |
| openstackgerrit | Merged openstack/kolla: fix Marathon bust the gate https://review.openstack.org/372003 | 16:42 |
| ntpttr | inc0: hmm, I don't think so, I think retrying would just be starting a new backup and deleting the failed oone | 16:43 |
| sdake | sweet - elrog 845 tubes on the way - arriving tuesday :) | 16:44 |
| *** jascott1_ has quit IRC | 16:45 | |
| *** AndChat|382025 has joined #openstack-kolla | 16:46 | |
| *** AndChat|382025 has quit IRC | 16:48 | |
| openstackgerrit | Merged openstack/kolla: Added enable_option for some services https://review.openstack.org/372390 | 16:50 |
| openstackgerrit | Bartłomiej Daca proposed openstack/kolla: Repair swift reconfiguration https://review.openstack.org/372631 | 16:50 |
| *** msimonin has joined #openstack-kolla | 16:51 | |
| *** dave-mcc_ has quit IRC | 16:51 | |
| Jeffrey4l_ | sdake, inc0 docker has such parameter in 1.10.0 | 16:53 |
| sdake | Jeffrey4l_ cool so tht is an option | 16:53 |
| *** jascott1_ has joined #openstack-kolla | 16:53 | |
| Jeffrey4l_ | ya. | 16:53 |
| sdake | i think limiting to just cinder would be wise | 16:53 |
| sdake | and putting some upper timebound limit on it wuld also be wise | 16:53 |
| openstackgerrit | Merged openstack/kolla: Checking valid fernet expiry only need to run once https://review.openstack.org/372483 | 16:54 |
| sdake | and notifying the uer in the task "This task may take awhile" is wise | 16:54 |
| *** Serlex has quit IRC | 16:54 | |
| *** jascott1_ has quit IRC | 16:54 | |
| Jeffrey4l_ | sdake, which task will use longer time to stop ? | 16:54 |
| *** msimonin has quit IRC | 16:55 | |
| inc0 | I asked on cinder how to solve it upstream | 16:55 |
| Jeffrey4l_ | do we want drop the support for debian? sdake inc0 | 16:55 |
| *** jascott1_ has joined #openstack-kolla | 16:56 | |
| Jeffrey4l_ | iirc, we talked this in one meeting. | 16:56 |
| *** jascott1_ has quit IRC | 16:56 | |
| Jeffrey4l_ | i prefer to drop it because: no gate and no one complain debian. | 16:56 |
| Jeffrey4l_ | the same case for rhel :( | 16:56 |
| sdake | Jeffrey4l_ i thiink that dserves a vote | 16:56 |
| inc0 | ideally I'd limit our supporting distros to centos, ubuntu, oracle and later some lightweight distro made for dockers | 16:57 |
| sdake | (deprecation) | 16:57 |
| *** ntpttr_ has quit IRC | 16:57 | |
| inc0 | sdake, I'd rather build policies on what we support/accept | 16:57 |
| inc0 | and based on these policies deprecate anything that doesn't meet them | 16:57 |
| sdake | the argument for deprecation is it doesn' work, and nobody seems to be interested | 16:57 |
| sdake | inc0 feels a bit heavy handed | 16:58 |
| sdake | but if you want to target that if your elected ptl go forit :) | 16:58 |
| inc0 | well, there are lots of distros out there, we need to know what will we accept/decline | 16:58 |
| inc0 | and we need to have guidelines for that call | 16:58 |
| inc0 | it's more about future proofing | 16:59 |
| sdake | for now lets make a decision about fedora and debian | 16:59 |
| Jeffrey4l_ | fedora is dropped already | 16:59 |
| sdake | droppeed or deprecated? | 16:59 |
| *** ntpttr_ has joined #openstack-kolla | 17:00 | |
| sdake | we can't drop implementation at end of cycle - that runs afoul o the deprection policies | 17:00 |
| inc0 | we can for example base our policy on "gates has to be operational during same release as acceptance of new distro" | 17:00 |
| Jeffrey4l_ | hmm. i saw a PS dropped fedora. seems it is not merged. | 17:00 |
| sdake | inc0 that seems unfriendly to many of our continuency | 17:00 |
| inc0 | so debian, since no gates are written, will enter deprecation unless somebody will write gates and sign up for supporting it | 17:00 |
| Jeffrey4l_ | it is not right now. | 17:00 |
| *** ntpttr_ has quit IRC | 17:00 | |
| sdake | Jeffrey4l_ recommend -2 - recommend a vote for debian vote for fedora as seaprate votes | 17:01 |
| sdake | Jeffrey4l_ or inc0 can one of you kick off that vote | 17:01 |
| Jeffrey4l_ | np. | 17:01 |
| sdake | use word deprecation rather then drop support | 17:01 |
| *** dave-mccowan has joined #openstack-kolla | 17:01 | |
| inc0 | yeah, regardless what we decide, we need 1 release of deprecation | 17:02 |
| inc0 | and we need to put up warning messages in Newto | 17:02 |
| inc0 | n | 17:02 |
| Jeffrey4l_ | so for the result, if we support debian/fedora, some need set up gate and fix all the issue ASAP. | 17:02 |
| inc0 | so we need to make final decision before 3.0.0 | 17:02 |
| *** ntpttr is now known as ntpttr_ | 17:03 | |
| Jeffrey4l_ | if we will deprecated debian/fedora, should we mark it deprecated now and remove in O? | 17:03 |
| Jeffrey4l_ | sdake, inc0 ^^ | 17:03 |
| Jeffrey4l_ | opinion? | 17:03 |
| *** ntpttr_ is now known as ntpttr | 17:03 | |
| inc0 | yeah | 17:03 |
| inc0 | and by now I mean 3.0.0 | 17:04 |
| Jeffrey4l_ | yep. before 3.0.0 tag | 17:04 |
| *** ntpttr is now known as ntpttr_away | 17:04 | |
| inc0 | sdake, why unfriendly? | 17:04 |
| inc0 | call it a QA requirement | 17:04 |
| *** ntpttr_away is now known as ntpttr | 17:04 | |
| Jeffrey4l_ | let me prepare the mail :), ok? | 17:05 |
| sdake | inc0 we decided in a votte a long time ago that we would accept things wihtout gating | 17:05 |
| inc0 | we can have partial support for stuff that's not gated at | 17:05 |
| *** ntpttr is now known as ntpttr_wfh | 17:05 | |
| sdake | making that directive woudl imply nothing goes in without ci | 17:05 |
| sdake | huge undertaking | 17:05 |
| *** ntpttr_wfh is now known as ntpttr | 17:05 | |
| sdake | can't be completed in one cycle | 17:05 |
| inc0 | sdake, no distro goes without CI | 17:05 |
| sdake | slippery slope ;) | 17:05 |
| inc0 | yeah, but if something is not gated at, we can't really assure it's quality | 17:06 |
| Jeffrey4l_ | if we support one distro, we need set up gate at least in next cycle. | 17:06 |
| inc0 | we can have "core" and "partial-support" | 17:06 |
| inc0 | anyway, that's one thing we should talk in summit | 17:06 |
| inc0 | I have issues with our support policy anyway | 17:06 |
| inc0 | we're growing too big, not scallable | 17:07 |
| sdake | inc0 we are scalable with ci | 17:07 |
| sdake | which we lack | 17:07 |
| Jeffrey4l_ | without gate, we can not guarantee the quality. | 17:07 |
| sdake | but making such a directive will lock out other projectss | 17:07 |
| inc0 | either move to plugin model - we'll have some "core" services/distros which will be heavily supported | 17:07 |
| sdake | and it will piss them off - because it would nger me that people got into kolla and then kolla locked the doors on us | 17:07 |
| sdake | (putting myself in the shoes of a project that isn't in the project) | 17:08 |
| inc0 | on the other hand we need to protect ourselves from "hey, I deployed watches using debian and it's not working" | 17:08 |
| sdake | in the project/in kolla | 17:08 |
| inc0 | (probably because nobody ever deployed this mix | 17:09 |
| sdake | you mean watcher? | 17:09 |
| inc0 | yeah | 17:09 |
| sdake | right - so my suggestion there is lets fix the ci problem, then make the directive | 17:09 |
| inc0 | put any openstack-project name which is less used than compute cit | 17:09 |
| inc0 | kit | 17:09 |
| Jeffrey4l_ | if no one or very few guys use/care about fedora/debian distro, why we need support that? | 17:09 |
| inc0 | we can slowly introduce these directives | 17:10 |
| sdake | then everyone is on a level playing field | 17:10 |
| inc0 | Jeffrey4l_, all it takes is one prod running debian | 17:10 |
| inc0 | we don't want to screw people over | 17:10 |
| *** gfhellma has joined #openstack-kolla | 17:10 | |
| inc0 | we can start deprecation, but it's only fair to give them way to keep the support | 17:10 |
| sdake | lets seprate out debian and fedora from directive making :) | 17:10 |
| inc0 | if they want to help | 17:10 |
| *** vhosakot has quit IRC | 17:11 | |
| inc0 | if someone is heavy debian user, and have resources to help Kolla to keep debian up with quality | 17:11 |
| inc0 | hell, why not? | 17:11 |
| sdake | well we do hae to balance our teams needs with the needs of the community | 17:11 |
| inc0 | so imho we need to start by making quality guidelines for people to meet to be in "kolla core" | 17:11 |
| sdake | we dont want to get into any one-off implementations | 17:11 |
| sdake | inc0 supportive of the idea; lets keep that idea for ocata | 17:12 |
| inc0 | this might be need of our community, all I'm saying, we should give Debian a chance to be supported | 17:12 |
| sdake | and focus on debain/fedora deprecation in the vote | 17:12 |
| inc0 | ofc, Newton is done | 17:12 |
| *** imcsk8_ is now known as imcsk8 | 17:12 | |
| inc0 | we don't change anything yet | 17:12 |
| inc0 | I thing this would make great summit session | 17:12 |
| sdake | possibly contentious :) | 17:13 |
| sdake | nonetheless a discussion worth having | 17:13 |
| *** unicell has quit IRC | 17:15 | |
| inc0 | contentious is good for summit | 17:15 |
| inc0 | if it's not contentious usually openstack-dev discussion is enough;) | 17:15 |
| *** vhosakot has joined #openstack-kolla | 17:18 | |
| *** williamcaban has joined #openstack-kolla | 17:18 | |
| Jeffrey4l_ | Ce | 17:21 |
| mliima | sdake, rabbitmq still restarting | 17:22 |
| mliima | : | 17:22 |
| mliima | :/ | 17:22 |
| sdake | mliima did you rebuid? | 17:22 |
| mliima | i uninstall kolla and rebuild | 17:22 |
| mliima | yes | 17:22 |
| *** williamcaban has quit IRC | 17:22 | |
| *** williamcaban has joined #openstack-kolla | 17:24 | |
| *** AndChat|382025 has joined #openstack-kolla | 17:25 | |
| *** AndChat|382025 has quit IRC | 17:28 | |
| *** harlowja has joined #openstack-kolla | 17:30 | |
| *** huikang has joined #openstack-kolla | 17:33 | |
| *** kangh_ has joined #openstack-kolla | 17:34 | |
| *** lamt has joined #openstack-kolla | 17:36 | |
| *** senk has joined #openstack-kolla | 17:37 | |
| *** huikang has quit IRC | 17:38 | |
| *** neilus has joined #openstack-kolla | 17:39 | |
| *** huikang has joined #openstack-kolla | 17:40 | |
| sdake | mlima run pip show kolla | 17:41 |
| sdake | mliima ^^ | 17:41 |
| *** unicell has joined #openstack-kolla | 17:42 | |
| mliima | i'm doing all from scrath | 17:43 |
| sdake | mliima how do you build your containers (exact command typed)? | 17:44 |
| mliima | ./tools/build.py --base ubuntu --type source --registry 10.102.20.225:4000 --push --no-cache | 17:45 |
| sdake | mliima that looks good | 17:48 |
| sdake | i've seen people append 4000/kolla on the end of registry | 17:48 |
| sdake | which results in kollakolla/abc | 17:48 |
| *** msimonin has joined #openstack-kolla | 17:50 | |
| *** tonanhngo has quit IRC | 17:51 | |
| mliima | hmm | 17:51 |
| *** shardy has quit IRC | 17:52 | |
| sdake | look ata dockerimages | 17:56 |
| sdake | and see their age | 17:56 |
| sdake | docker images -> pasate | 17:56 |
| sdake | have meeting in 3 minutes | 17:56 |
| *** inc0 has quit IRC | 17:59 | |
| *** inc0 has joined #openstack-kolla | 17:59 | |
| imcsk8 | hello, i keep getting a "too many links" error while running kolla-build: https://paste.fedoraproject.org/431087/47430799/ | 18:00 |
| inc0 | imcsk8, if you do a lot of rebuilds that will happen at some point | 18:01 |
| imcsk8 | yup | 18:01 |
| inc0 | I suggest running kolla-ansible destroy with image cleanup | 18:01 |
| inc0 | over night. It will take some time | 18:01 |
| inc0 | took about 5 hrs for me | 18:01 |
| inc0 | I had 60k something images;) | 18:01 |
| imcsk8 | i have just 160 images | 18:02 |
| imcsk8 | the problem is that overlay creates a lot of hardlinks for some files and hits ext4 limit (65000) | 18:02 |
| sdake | imcsk8 try ulimit | 18:04 |
| sdake | or setrlimit | 18:04 |
| sdake | or some combo of those likely already intgrated into docker proper | 18:04 |
| *** tonanhngo has joined #openstack-kolla | 18:05 | |
| imcsk8 | ulimit does not seem to manage hard links | 18:06 |
| *** bjolo_ has joined #openstack-kolla | 18:13 | |
| inc0 | imcsk8, but every time you rebuild it will create stale layers | 18:13 |
| inc0 | docker images -a |wc -l | 18:13 |
| *** vhosakot has quit IRC | 18:13 | |
| imcsk8 | 761 | 18:14 |
| inc0 | I had more... | 18:14 |
| inc0 | anyway, remove images | 18:14 |
| inc0 | and it solves it | 18:14 |
| imcsk8 | ok, thanks | 18:14 |
| inc0 | for image in $(docker images -a -q); do docker rmi $image; done | 18:15 |
| *** ssurana has joined #openstack-kolla | 18:15 | |
| *** vhosakot has joined #openstack-kolla | 18:16 | |
| imcsk8 | i'll remove the images | 18:17 |
| *** athomas has quit IRC | 18:18 | |
| *** lamt has quit IRC | 18:20 | |
| openstackgerrit | bjorn lofdahl proposed openstack/kolla: Only check once for empty passwords in passwords.yml https://review.openstack.org/372678 | 18:21 |
| *** sdake_ has joined #openstack-kolla | 18:23 | |
| *** sdake has quit IRC | 18:25 | |
| *** LamT_ has quit IRC | 18:31 | |
| *** senk has quit IRC | 18:33 | |
| *** williamcaban has quit IRC | 18:35 | |
| *** inc0 has quit IRC | 18:46 | |
| *** inc0 has joined #openstack-kolla | 18:46 | |
| *** neilus has quit IRC | 18:47 | |
| inc0 | sdake_, did you talk to dhellman about stable/newton? | 18:48 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: More test stuff, create nova-metadata svc, simplify instructions https://review.openstack.org/371148 | 18:50 |
| openstackgerrit | bjorn lofdahl proposed openstack/kolla: Check only once the number of network agents https://review.openstack.org/372703 | 18:53 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Add container debug feature https://review.openstack.org/371931 | 18:54 |
| *** neilus has joined #openstack-kolla | 18:57 | |
| sbezverk | inc0: ping | 18:58 |
| inc0 | I'm here | 18:58 |
| sbezverk | sdake mentioned that you have some slides which can be used for kube presentation | 18:59 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Set ownership of barbican configuration files to barbican https://review.openstack.org/372705 | 19:01 |
| *** lamt has joined #openstack-kolla | 19:01 | |
| *** neilus has quit IRC | 19:01 | |
| sdake_ | inc0 yes | 19:05 |
| sdake_ | seee scrollbak discussoin with Jeffrey4l_ | 19:05 |
| openstackgerrit | bjorn lofdahl proposed openstack/kolla: run_once Checking mongodb backend for ceilometer https://review.openstack.org/372709 | 19:08 |
| *** msimonin has quit IRC | 19:08 | |
| *** neilus has joined #openstack-kolla | 19:09 | |
| *** msimonin has joined #openstack-kolla | 19:09 | |
| inc0 | sdake_, I have read your discussion with dhellman | 19:10 |
| inc0 | missed it in morning | 19:10 |
| inc0 | so, what's the plan? | 19:10 |
| sdake_ | no plan as of yet | 19:10 |
| inc0 | we branch out when we tag rc2? | 19:10 |
| sdake_ | yes - release tam wants rcs compressed | 19:11 |
| inc0 | imho if we wait for rdo, centos and uca to have mitaka | 19:11 |
| inc0 | newton* | 19:11 |
| inc0 | we'll wait for months | 19:11 |
| sdake_ | rdo will be ready | 19:11 |
| sdake_ | not sure for uca | 19:11 |
| inc0 | ehh | 19:12 |
| inc0 | voes of having binary build | 19:12 |
| *** neilus has quit IRC | 19:13 | |
| inc0 | anyway, I'd not put out money on it, we can change repos to newton and until gates are green we will put a warning on | 19:14 |
| sdake_ | that is what i told doug | 19:20 |
| *** williamcaban has joined #openstack-kolla | 19:20 | |
| *** dwalsh has joined #openstack-kolla | 19:23 | |
| bjolo_ | berendt, saw your comment now to fix all run_once in one patch, sorry | 19:23 |
| bjolo_ | berendt, should not be any more patches now | 19:24 |
| berendt | bjolo_ think this makes more sense to not end in tons of patchets | 19:24 |
| bjolo_ | im a newbie on this :) | 19:24 |
| berendt | the barbican role is not working, has anybody tested it before the merge? | 19:24 |
| *** williamcaban has quit IRC | 19:24 | |
| bjolo_ | keystone_fernet also seems broken | 19:25 |
| sdake_ | zhubingbing said he did | 19:25 |
| sdake_ | fernet - not sure | 19:25 |
| sdake_ | pleae file bugs, mark them as critical, and mark them as confirmed | 19:25 |
| berendt | sdake_ definitive no.. barbican used the paste file for the keystone credentials and this file is untouched | 19:25 |
| sdake_ | whatever was tested or wasn't tested we need to make a concerted effort as a team to make sure all the new playbook roles are teted | 19:26 |
| berendt | sdake_ cleaning up at the moment, first patch is already pending ;) | 19:27 |
| *** huikang has quit IRC | 19:27 | |
| *** huikang has joined #openstack-kolla | 19:28 | |
| *** huikang_ has joined #openstack-kolla | 19:30 | |
| *** huikang has quit IRC | 19:31 | |
| berendt | Other point with barbican that makes no sense to me: by default barbican uses a file based secret store on the local filesystem. | 19:34 |
| berendt | This means that we required a shared directory for this store to make the driver workable when using multiple control nodes | 19:35 |
| berendt | bug for disabled authentication: https://bugs.launchpad.net/kolla/+bug/1625337 | 19:35 |
| openstack | Launchpad bug 1625337 in kolla "Authentication for Barbican not enabled" [Critical,Confirmed] | 19:35 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add pulling ceilometer-collector image https://review.openstack.org/372727 | 19:35 |
| sdake_ | berendt barbicn includes ax509 certificate feature which is ha | 19:35 |
| sdake_ | and does not rely on hrdwae | 19:35 |
| sdake_ | or perhaps tht is magnum | 19:36 |
| berendt | #. Barbican has a plugin architecture which allows the deployer to store secrets in | 19:36 |
| berendt | a number of different back-end secret stores. By default, Barbican is configured to | 19:36 |
| berendt | store secrets in a basic file-based keystore. This key store is NOT safe for | 19:36 |
| berendt | production use. | 19:36 |
| sdake_ | the reason barbican is in kolla is because among other things it offers of value, its a hard dep for magnum | 19:36 |
| sdake_ | dave-mccowan any thoughts on the above ^^ | 19:37 |
| berendt | at the moment this file based keystore is used in the ansible role | 19:37 |
| sdake_ | berendt dave-mccowan can provide clarity here | 19:37 |
| berendt | even if it is fine to use the file based keystore the implementation is broken because we do not required a shared directory for the store | 19:38 |
| sdake_ | file bugs | 19:38 |
| sdake_ | pls | 19:38 |
| sdake_ | so the core team can prioirtize them | 19:39 |
| dave-mccowan | i agree. default config of barbican is developer friendly, not production ready. there are a few config tweaks that need to be made to "turn on security". | 19:41 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: fix horizon fernet container https://review.openstack.org/372730 | 19:42 |
| *** _coolsvap_ has quit IRC | 19:42 | |
| *** salv-orl_ has joined #openstack-kolla | 19:42 | |
| berendt | https://bugs.launchpad.net/kolla/+bug/1625340 | 19:43 |
| openstack | Launchpad bug 1625340 in kolla "Barbican default basic file-based keystone not safe for production" [Critical,Confirmed] | 19:43 |
| *** eaguilar has joined #openstack-kolla | 19:43 | |
| berendt | dave-mccowan does it makes sense to use the file based store when running multiple control nodes? | 19:43 |
| *** huikang_ has quit IRC | 19:43 | |
| berendt | dave-mccowan without a shared directory | 19:43 |
| *** salv-orlando has quit IRC | 19:45 | |
| dave-mccowan | database backend would be a better choice. | 19:45 |
| kfox1111 | I think kolla-trunk just broke kolla-kubernetes. | 19:46 |
| inc0 | kfox1111, elaborate | 19:46 |
| kfox1111 | fresh checkout, and genconfig | 19:47 |
| kfox1111 | TASK [keystone : Copying over keystone.conf] *********************************** | 19:47 |
| kfox1111 | fatal: [localhost]: FAILED! => {"failed": true, "msg": "'dict object' has no attribute u'ansible_eth0'"} | 19:47 |
| kfox1111 | looks like it grew a transport_uri | 19:47 |
| kfox1111 | looks like it grew a transport_url | 19:47 |
| kfox1111 | needs something like: | 19:48 |
| berendt | dave-mccowan can you take care of the barbican role to fix the issues? | 19:49 |
| kfox1111 | http://paste.openstack.org/show/YkJKeRl7j5dZmsZG1xmw/ | 19:49 |
| berendt | dave-mccowan those tweaks, db backend, wrong paste pipeline | 19:49 |
| inc0 | kfox1111, looks like memcache uri fail? | 19:49 |
| *** sdake has joined #openstack-kolla | 19:49 | |
| dave-mccowan | berendt yes. i'll work with some barbicaneers to come up with a good set of defaults for kolla. is it just the one bug open now? | 19:51 |
| kfox1111 | inc0: transport for cadf rabbit. | 19:51 |
| berendt | there are two related to the barican configuration | 19:51 |
| berendt | https://bugs.launchpad.net/kolla/+bug/1625340 | 19:51 |
| openstack | Launchpad bug 1625340 in kolla "Barbican default basic file-based keystone not safe for production" [Critical,Confirmed] | 19:51 |
| berendt | https://bugs.launchpad.net/kolla/+bug/1625337 | 19:51 |
| openstack | Launchpad bug 1625337 in kolla "Authentication for Barbican not enabled" [Critical,Confirmed] | 19:51 |
| *** sdake_ has quit IRC | 19:52 | |
| inc0 | a yeah I see it | 19:52 |
| berendt | cadf notifications should be disabled by default? | 19:52 |
| *** huikang has joined #openstack-kolla | 19:54 | |
| *** huikang has quit IRC | 19:54 | |
| *** huikang has joined #openstack-kolla | 19:54 | |
| kfox1111 | berendt: I didn't enable them. | 19:54 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the failure of ceilometer-api container https://review.openstack.org/372161 | 19:58 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: integrate gnocchi with ceilometer https://review.openstack.org/372737 | 19:58 |
| *** ssurana has quit IRC | 19:59 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla: Fix cadf for kolla-kubernetes https://review.openstack.org/372738 | 20:00 |
| kfox1111 | inc0: please have a look at that review. ^--- should fix the issue. | 20:01 |
| kfox1111 | the fix is copy/pasted from nova.conf.j2 which it looks like the cadf one was copy/pasted in from an earlier version. | 20:01 |
| *** tonanhngo has quit IRC | 20:04 | |
| berendt | kfox1111 doh.. i am sorry :( | 20:05 |
| berendt | kfox1111 yes this was a pretty old review.. probably because of that the transport url was not up to date | 20:06 |
| kfox1111 | bradjones: no worries. if we can get that fix in quick, we're good. :) | 20:07 |
| kfox1111 | I've working on getting a kolla-kubernetes setup kolla genconfig gate test going. so in the future we catch these things earlier. | 20:08 |
| berendt | kfox1111 i will hijack your review, there is an other issue with the transport_url parameter | 20:09 |
| berendt | there are 2 of them | 20:09 |
| *** bdaca has quit IRC | 20:10 | |
| sdake | inc0 did ttx post a public room allocation graphical representation? | 20:12 |
| inc0 | sdake, haven't seen it | 20:13 |
| inc0 | I just seen spreadsheet | 20:13 |
| inc0 | hold on | 20:13 |
| *** gfhellma has quit IRC | 20:13 | |
| *** vhosakot has quit IRC | 20:13 | |
| *** vhosakot has joined #openstack-kolla | 20:14 | |
| inc0 | sdake, https://docs.google.com/spreadsheets/d/1TQ-RSlbiBBEclkonIbfUP7R1ExZSJylF1uiEKV2G_Cw/pubhtml?gid=1107826458&single=true | 20:14 |
| sdake | inc0 thanks | 20:15 |
| inc0 | sdake, so part of what I've said today was fridays meetup | 20:16 |
| inc0 | we have full day of it | 20:16 |
| inc0 | so less sessions | 20:16 |
| kfox1111 | bradjones: sounds good. thanks for looking into it. | 20:24 |
| kfox1111 | unrelated... | 20:24 |
| kfox1111 | are there any known issues with cinder with genconfig with ansible 2.1.1.0? | 20:25 |
| kfox1111 | sdake: Can you poke us before you cut a final release so we can test against it? | 20:26 |
| kfox1111 | that way we can ensure 3.0.0 works for kolla-kubernetes and a last minute breakage doesn't slip in. | 20:26 |
| kfox1111 | for the cinder breakage, when I have this in globals: | 20:27 |
| kfox1111 | enable_cinder: "yes" | 20:27 |
| kfox1111 | enable_cinder_backend_lvm: "yes" | 20:27 |
| kfox1111 | I get: | 20:28 |
| kfox1111 | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TemplateRuntimeError: {{ cinder_backends|selectattr('enabled', 'equalto', true)|list }}: no test named 'equalto' | 20:28 |
| *** gfhellma has joined #openstack-kolla | 20:28 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Fix Keystone CADF notifications https://review.openstack.org/372738 | 20:28 |
| sdake | we have a half day meetup | 20:29 |
| mliima | guys, the rabbitmq was changed in 3.0.0rc1? | 20:29 |
| sdake | inc0 ^ | 20:29 |
| sdake | mliima wfm | 20:29 |
| sdake | mliima works multinode | 20:29 |
| mliima | multinode? | 20:29 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Fix Keystone CADF notifications https://review.openstack.org/372738 | 20:30 |
| mliima | i'm trying to deploy aio | 20:30 |
| sdake | rabbitmq was teted on ubuntu and centos aio and multinode without issue | 20:30 |
| mliima | and using master the rabbitmq container is restarting | 20:30 |
| mliima | ok | 20:30 |
| mliima | i'll use 3.0.0rc1 | 20:30 |
| kfox1111 | berendt: isn't the [oslo_messaging_notifications] the new way to do it? | 20:30 |
| sdake | i built containers yetersday and tested for sbezverk a different issue | 20:30 |
| sdake | mliima aren't you using master? | 20:31 |
| berendt | kfox1111 i just tested it and when the transport_url inside osl_messaging_notifications is set i god rabbitmq connections refues | 20:31 |
| mliima | no, i'm using master | 20:31 |
| berendt | when i moved it into the default section it works | 20:31 |
| kfox1111 | berendt: ah. weird. ok. | 20:31 |
| mliima | but now i will not | 20:31 |
| kfox1111 | berendt: your seeing the mesages out of rabbit thogh? or is it just ignoring it now? | 20:31 |
| berendt | kfox1111 yes confusing.. not sure | 20:31 |
| berendt | i get the messages in ceilometer | 20:32 |
| kfox1111 | ok. cool. | 20:32 |
| berendt | let's run a final test | 20:32 |
| kfox1111 | sdake: do you think that's something you could do? | 20:33 |
| sdake | which kfox1111 / | 20:33 |
| *** msimonin has quit IRC | 20:33 | |
| kfox1111 | sdake: can you let us know before you cut the final 3.0.0 release so we can test it against kolla-kubernetes so we don't have a broken 3.0.0 release? | 20:34 |
| mliima | on wednesday it was working fine, but now not | 20:34 |
| sdake | kfox1111 test against rc2 - that should be nearly identical to 3.0.0 | 20:34 |
| *** Jeffrey4l_ has quit IRC | 20:34 | |
| kfox1111 | sdake: trunk's broken. we have a fix going. | 20:34 |
| sdake | but yes, when I start the process of tagging 3.0.0 i'lll let ou know | 20:34 |
| kfox1111 | this is the second breakage. | 20:34 |
| kfox1111 | sdake: thanks. | 20:35 |
| berendt | works for me | 20:35 |
| berendt | kfox1111 you can show those events with ceilometer event-list --query event_type=identity.user.created | 20:35 |
| kfox1111 | I've got basically the logic for at least genconfig working I think, so I think we can make a kolla-kubernetes-genconfig job against kolla to make sure it doesn't break in the future. | 20:35 |
| kfox1111 | but it will take a few weeks to finish probably. :/ | 20:35 |
| *** Jeffrey4l_ has joined #openstack-kolla | 20:35 | |
| *** tonanhngo has joined #openstack-kolla | 20:35 | |
| kfox1111 | berendt: nice. | 20:36 |
| kfox1111 | we're going to be enabling cadf on one of our other systems soon. | 20:36 |
| berendt | cadf is only enabled for keystone, other events are working out of the box | 20:36 |
| kfox1111 | nice. :) | 20:37 |
| mliima | good night guys | 20:37 |
| kfox1111 | anyone ever see: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TemplateRuntimeError: {{ cinder_backends|selectattr('enabled', 'equalto', true)|list }}: no test named 'equalto | 20:37 |
| mliima | \o | 20:37 |
| mliima | bye | 20:37 |
| *** mliima has quit IRC | 20:38 | |
| kfox1111 | https://github.com/ansible/ansible/issues/10123 maybe... | 20:38 |
| *** eaguilar has quit IRC | 20:40 | |
| kfox1111 | it looks like that issue, but it says 2.8 fixes. it and I'm running 2.8. | 20:41 |
| kfox1111 | any other ideas? | 20:41 |
| rhallisey | yo | 20:44 |
| kfox1111 | rhallisey: hey. | 20:44 |
| kfox1111 | rhallisey: running into a weird issue with kolla genconfig with cinder enabled. | 20:45 |
| rhallisey | kfox1111, I think the genconfig piece would be tested within an overall deploy jonb | 20:45 |
| rhallisey | I'm not surprised | 20:45 |
| kfox1111 | rhallisey: yes, but that's a hard thing to do. | 20:45 |
| *** ccesario has quit IRC | 20:45 | |
| rhallisey | we'll it would be require to do as part of the config | 20:45 |
| kfox1111 | rhallisey: most of the breakages we've seen though coudl be convered simply by doing a genconfig and watch it burn. :/ | 20:45 |
| rhallisey | s/config/deploy/ | 20:46 |
| kfox1111 | it would be a relatively easy gate check against kolla to write I think. | 20:46 |
| rhallisey | not saying it's a bad idea. I think the more gates the better | 20:46 |
| kfox1111 | we can do a full kolla-kubernetes check once we have an automatable workflow done. | 20:46 |
| kfox1111 | true true. :) | 20:47 |
| kfox1111 | have you seen that cinder genconfig issue before? | 20:47 |
| rhallisey | yes | 20:47 |
| rhallisey | coming soon | 20:47 |
| rhallisey | not sure the time table yet | 20:47 |
| rhallisey | :) | 20:47 |
| kfox1111 | cool. | 20:47 |
| *** dwalsh has quit IRC | 20:48 | |
| sdake | wirehead_ ping | 20:48 |
| kfox1111 | https://github.com/ansible/ansible/issues/8836 looks like the issue too | 20:49 |
| kfox1111 | and it mentions: https://github.com/ansible/ansible/pull/9645 | 20:49 |
| kfox1111 | does the test plugin have to be loaded independently now? | 20:49 |
| *** berendt has quit IRC | 20:51 | |
| kfox1111 | ah. maybe ansible's too new? | 20:54 |
| kfox1111 | is trunk still requiring 2.0? | 20:54 |
| *** dwalsh has joined #openstack-kolla | 20:55 | |
| *** schwicht has quit IRC | 20:57 | |
| *** inc0 has quit IRC | 21:00 | |
| kfox1111 | confirmed: yum remove -y ansible; pip install "ansible<2.1" worked. | 21:00 |
| *** lrensing has quit IRC | 21:03 | |
| *** awiddersheim has quit IRC | 21:04 | |
| *** huikang has quit IRC | 21:07 | |
| *** msimonin has joined #openstack-kolla | 21:08 | |
| *** msimonin has quit IRC | 21:08 | |
| *** kangh_ has quit IRC | 21:10 | |
| *** inc0 has joined #openstack-kolla | 21:13 | |
| sdake | sbezverk slides should be incoming | 21:13 |
| sdake | rhallisey if you could share with wirehead_ as well tht would be fantastic | 21:13 |
| rhallisey | sure | 21:14 |
| *** dwalsh has quit IRC | 21:14 | |
| sdake | kfox i run with 2.2.0.sothing and it works | 21:14 |
| kfox1111 | sdake: odd. well, whatever's in epel doesn't work. | 21:15 |
| *** schwicht has joined #openstack-kolla | 21:16 | |
| *** kong has joined #openstack-kolla | 21:17 | |
| *** rhallisey has quit IRC | 21:21 | |
| *** msimonin has joined #openstack-kolla | 21:23 | |
| *** salv-orl_ has quit IRC | 21:25 | |
| *** salv-orlando has joined #openstack-kolla | 21:25 | |
| *** schwicht has quit IRC | 21:30 | |
| *** schwicht has joined #openstack-kolla | 21:32 | |
| *** williamcaban has joined #openstack-kolla | 21:35 | |
| *** jtriley_ has joined #openstack-kolla | 21:38 | |
| *** williamcaban has quit IRC | 21:40 | |
| *** jtriley has quit IRC | 21:42 | |
| *** jheroux has quit IRC | 21:43 | |
| bjolo_ | gonna test out keystone_fernet again. Do i need to do a new deploy or would a reconfigure be sufficient | 21:43 |
| *** jtriley_ has quit IRC | 21:45 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Document more stuff https://review.openstack.org/371148 | 21:48 |
| kfox1111 | the rabbit patch broke more then just genconfig. :/ | 21:50 |
| kfox1111 | the bootstrap jobs are failing now too.... | 21:50 |
| *** msimonin has quit IRC | 21:51 | |
| *** harlowja has quit IRC | 21:51 | |
| *** wznoinsk has quit IRC | 21:51 | |
| *** gfidente|afk has quit IRC | 21:51 | |
| *** sbezverk has quit IRC | 21:51 | |
| *** harbie has quit IRC | 21:51 | |
| *** wznoinsk has joined #openstack-kolla | 21:51 | |
| *** sbezverk has joined #openstack-kolla | 21:51 | |
| *** msimonin has joined #openstack-kolla | 21:51 | |
| *** gfidente|afk has joined #openstack-kolla | 21:51 | |
| *** neilus has joined #openstack-kolla | 21:51 | |
| kfox1111 | ah.. I think I know why. I thin kthe current posted patch fixes it. | 21:55 |
| *** harlowja has joined #openstack-kolla | 21:55 | |
| *** schwicht has quit IRC | 21:55 | |
| kfox1111 | yup. that fixed it. | 21:57 |
| *** williamcaban has joined #openstack-kolla | 22:00 | |
| *** vhosakot has quit IRC | 22:01 | |
| *** harbie has joined #openstack-kolla | 22:02 | |
| *** msimonin has quit IRC | 22:04 | |
| *** neilus has quit IRC | 22:07 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: More services to deployments https://review.openstack.org/372189 | 22:08 |
| *** williamcaban has quit IRC | 22:14 | |
| *** ssurana has joined #openstack-kolla | 22:14 | |
| *** lamt has quit IRC | 22:15 | |
| kfox1111 | hmm... | 22:16 |
| kfox1111 | where's https://hub.docker.com/r/kolla/centos-binary-iscsid/? | 22:16 |
| kfox1111 | oh... did iscsi stuff move in 3.x? | 22:17 |
| kfox1111 | maybe it has a different 2.x name? | 22:17 |
| *** rstarmer has quit IRC | 22:23 | |
| kfox1111 | did kolla mitaka not have lvm suport at all? | 22:26 |
| imcsk8 | does somebody knows what does this error mean? ERROR:kolla.image.build:trove-conductor Failed with status: matched | 22:27 |
| *** neilus has joined #openstack-kolla | 22:28 | |
| inc0 | imcsk8, real error is somewhere above | 22:31 |
| inc0 | probably in trove-base | 22:31 |
| *** gfidente|afk has quit IRC | 22:31 | |
| kfox1111 | I don't see a reference to iscsi at all. I guess its a 3.x feature only. | 22:31 |
| inc0 | kfox1111, nope | 22:31 |
| inc0 | it's Newton | 22:31 |
| kfox1111 | oh well. no one I know uses lvm anyway. | 22:31 |
| *** tonanhngo has quit IRC | 22:32 | |
| kfox1111 | just gota expidite ceph support for kolla. | 22:32 |
| inc0 | I know some | 22:32 |
| kfox1111 | kolla-kubernetes. | 22:32 |
| kfox1111 | inc0: really? risky. | 22:32 |
| inc0 | well, if somebody don't like their data | 22:32 |
| kfox1111 | heh. yeah. right? :) | 22:32 |
| openstackgerrit | Andrew Widdersheim proposed openstack/kolla: Update EPEL version in base https://review.openstack.org/372797 | 22:32 |
| kfox1111 | I've physically moved a ceph cluster twice now. live. :) | 22:32 |
| inc0 | I tried to discuss "what if your volume controler dies" | 22:32 |
| inc0 | that's nice:) I'm just learning my ceph-fu now | 22:33 |
| kfox1111 | like, move one node at a time to a different rack. :) | 22:33 |
| bjolo_ | reconfigure takes forever and when it comes to nova it goes in circles. anyone else seen this? | 22:33 |
| bjolo_ | http://paste.openstack.org/show/581351/ | 22:33 |
| kfox1111 | 0 downtime. :) | 22:33 |
| sdake | sbezverk do you haee the slides | 22:33 |
| inc0 | http://grafana.openstack.org/dashboard/db/nodepool-osic with this - provider cloud8 | 22:33 |
| sdake | wirehead_ do you have the slides | 22:33 |
| *** neilus has quit IRC | 22:35 | |
| sdake | inc0 newton is 3.x | 22:35 |
| bjolo_ | ok reconfigure got passed nova, but it took about 1h. does that sound normal? | 22:35 |
| sdake | bjolo_ definately not | 22:35 |
| inc0 | bjolo_, no | 22:35 |
| sdake | normal would be 5-10 secondss | 22:36 |
| inc0 | bjolo_, how many nodes? | 22:36 |
| sdake | bjolo which storage driver backend areou using? | 22:36 |
| sdake | inc0 64 iirc | 22:36 |
| inc0 | ok so it might take some longer time | 22:36 |
| inc0 | if you use it heavily | 22:36 |
| bjolo_ | 48 compute, 3 control, 3 network | 22:36 |
| sdake | but not 1 hour | 22:36 |
| bjolo_ | storage driver backend? | 22:37 |
| inc0 | yeah, way too long | 22:37 |
| sdake | bjolo docker graph storage driver is what? | 22:37 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Document more stuff https://review.openstack.org/371148 | 22:37 |
| sdake | bjolo docker info iirc produces that | 22:37 |
| inc0 | bjolo which task in particular took this time | 22:37 |
| inc0 | ? | 22:37 |
| bjolo_ | http://paste.openstack.org/show/581351/ | 22:38 |
| bjolo_ | about evenly spread | 22:38 |
| bjolo_ | 5-10 min on each | 22:38 |
| sdake | bjolo storage driver - it makes a difference | 22:39 |
| *** tonanhngo has joined #openstack-kolla | 22:39 | |
| sdake | bjolo run docker info | 22:39 |
| bjolo_ | Server Version: 1.12.1 | 22:39 |
| bjolo_ | Storage Driver: btrfs | 22:39 |
| bjolo_ | Build Version: Btrfs v3.19.1 | 22:39 |
| sdake | is /var/lib/kolla formattedd as btrfs? | 22:40 |
| inc0 | does it need to sdake ? | 22:40 |
| bjolo_ | fresh install, everything works fine. Enable TLS in globals -> kolla-ansible reconfigure | 22:40 |
| sdake | docker does not do that automatically | 22:40 |
| inc0 | sdake, you mean /var/lib/docker;) | 22:40 |
| sdake | inc0 so yes it needs to be formatted | 22:41 |
| *** awiddersheim has joined #openstack-kolla | 22:41 | |
| bjolo_ | i have root fs as btrfs | 22:41 |
| inc0 | bjolo_, did deploy took long time? | 22:41 |
| sdake | is /var on a separat partition by chance? | 22:41 |
| sdake | bjolo i dont know if anyone has ever tested reconfigure going from non-tls to tls | 22:41 |
| sdake | i have not | 22:41 |
| sdake | reconfigure is about reconfiguring /etc/kolla/config | 22:42 |
| sdake | not so much abut globals.yml | 22:42 |
| bjolo_ | deploy took 28min | 22:42 |
| sdake | your total time on reconfgiure was 1 hour | 22:42 |
| sdake | or the nova time? | 22:42 |
| bjolo_ | reconfigure is still going | 22:42 |
| bjolo_ | on neutron now | 22:42 |
| inc0 | going faster or about the same? | 22:43 |
| sdake | bjolo when you get done - can you kolla-ansible destry and compare the delta of reconfigure without modifying tls optiosn (or any options) | 22:43 |
| inc0 | bjolo_, also, all the nodes runs btrfs? | 22:43 |
| inc0 | hold on sdake, his problems seems to be docker-ish | 22:44 |
| bjolo_ | now it is pretty fast, but i dont know. | 22:44 |
| inc0 | as it's start container that took a lot of time | 22:44 |
| bjolo_ | all nodes are equal in every aspect | 22:44 |
| sdake | bjolo indicates they each took about 5 minutes | 22:44 |
| bjolo_ | besides control and network have ssd drives | 22:44 |
| inc0 | yeah, each start container | 22:45 |
| sdake | i thgout that wass targeted at the roles | 22:45 |
| bjolo_ | compute have normal 10KRPM SAS drives | 22:45 |
| inc0 | each nova service, we have lot of those | 22:45 |
| inc0 | bjolo_, and btrfs? | 22:45 |
| sdake | Jeffrey4l_ had said reconfigure needs some attention in newton | 22:45 |
| inc0 | hmm | 22:45 |
| inc0 | well, storage can be a bitch | 22:45 |
| sdake | siad it is pokey on centos | 22:45 |
| bjolo_ | kolla master node is running pretty heavy on ansible threads | 22:45 |
| inc0 | bjolo_, paste your ansible.cfg plz | 22:46 |
| bjolo_ | http://paste.openstack.org/show/581352/ | 22:47 |
| inc0 | bjolo_, change pipelining to True | 22:47 |
| inc0 | should help ansible a bit | 22:47 |
| bjolo_ | ok | 22:48 |
| bjolo_ | will do when the reconfigure is done | 22:48 |
| *** daneyon has joined #openstack-kolla | 22:48 | |
| sdake | bjolo i'd like to see a fresh deploy without tls in the reconfigure mix if you can do so | 22:49 |
| inc0 | bjolo_, so at your scale, with slowish spindles and 30 forks without pipelining, it might add up | 22:49 |
| sdake | ie. kolla-ansible deploy | 22:50 |
| sdake | kolla-ansible reconfigure | 22:50 |
| sdake | with no intermediate steps | 22:50 |
| inc0 | I have feeling that it's not TLS | 22:50 |
| sdake | lets eliminate it as a cause rather then rely on feelings | 22:51 |
| sdake | the only unique thing bjolo has done is setup tls and run recnofigure | 22:51 |
| bjolo_ | mmmm | 22:51 |
| inc0 | no, not only | 22:51 |
| sdake | that is a huge untested model | 22:51 |
| bjolo_ | http://paste.openstack.org/show/581353/ | 22:51 |
| bjolo_ | strace -p on one ansible thread | 22:52 |
| inc0 | looks like some strange loop | 22:53 |
| *** schwicht has joined #openstack-kolla | 22:53 | |
| *** daneyon has quit IRC | 22:53 | |
| bjolo_ | why is reconfigure spending time on things i dont run? | 22:53 |
| inc0 | what do you mean? | 22:53 |
| bjolo_ | http://paste.openstack.org/show/581354/ | 22:54 |
| bjolo_ | neutron-sfc agent | 22:54 |
| bjolo_ | 3-4 min on that | 22:54 |
| bjolo_ | for example | 22:54 |
| sdake | bjolo sfc-agent reconfigure may be broken | 22:55 |
| sdake | file a bug | 22:55 |
| sdake | pls | 22:55 |
| inc0 | bjolo_, it doesn't skip this thing? | 22:55 |
| sdake | inc0 sfc is treated ass an agent (like ovs and linuxbridge) | 22:56 |
| sdake | as/as | 22:56 |
| sdake | as/as | 22:56 |
| sdake | ass/as | 22:56 |
| sdake | groan | 22:56 |
| sdake | inc0 logic probablywrong there | 22:56 |
| sdake | bjolo - my recommended course of action is to test reconfgiure independent of tls | 22:56 |
| bjolo_ | can someone else do a reconfigure on a multinode setup and se what happens | 22:56 |
| bjolo_ | real 96m47.403s | 22:57 |
| bjolo_ | user 531m39.461s | 22:57 |
| bjolo_ | sys 29m34.047s | 22:57 |
| bjolo_ | i can just kick it off again right, no config changes what so ever | 22:58 |
| sdake | bjolo if you reconfigure without tls - i'd be curious o see the results | 22:58 |
| sdake | make sure your tls cloud is working | 22:58 |
| inc0 | bjolo_ pipelining plz | 22:58 |
| sdake | one change at a time | 22:58 |
| sdake | bjolo pick your change | 22:58 |
| inc0 | start with pipelining as it won't require redeploy | 22:58 |
| bjolo_ | ok | 22:59 |
| inc0 | you can bump number of forks up too | 22:59 |
| inc0 | when you're at it | 22:59 |
| *** ssurana has quit IRC | 22:59 | |
| sdake | one change at time | 22:59 |
| inc0 | it's ok to be big if you're using pipelining | 22:59 |
| sdake | making a bunch of changes and then declaring victory doesn't find root cause | 22:59 |
| inc0 | well, if configuring ansible helps, that's what we should be pointing at | 23:00 |
| bjolo_ | ok piplining enables, everything else the same | 23:01 |
| sdake | my money is on tls | 23:01 |
| bjolo_ | all target nodes are calm, no full filesystems, etc, etc | 23:01 |
| sdake | but pipelining will be a good exercise | 23:01 |
| inc0 | btw did you see any siginficant time difference between controller node services and compute node srrvices | 23:01 |
| inc0 | ? | 23:01 |
| bjolo_ | inc0, wasnt really paying attention for the first 30min | 23:02 |
| bjolo_ | should i rm -rf /root/.ansible/ as well? | 23:02 |
| inc0 | well, if you have ansible.cfg there that's one is going to be used | 23:02 |
| bjolo_ | ansible.cfg is in /etc/ansible | 23:03 |
| bjolo_ | to clear the cache | 23:03 |
| inc0 | hmm nova does seem to be looping around | 23:03 |
| inc0 | took 8min for my little 5vm setup | 23:06 |
| bjolo_ | ok goes pretty smooth, then it kinda stalls on TASK [common : Copying over heka config files | 23:06 |
| *** neilus has joined #openstack-kolla | 23:07 | |
| inc0 | but it was kinda strange as it seemed it does same nova task multiple times | 23:08 |
| inc0 | ahh I know why...serial | 23:10 |
| inc0 | it will do reconfigure on 30% of nodes at the time | 23:10 |
| inc0 | well let me know if pipelining helps | 23:11 |
| bjolo_ | hmm that could explain why strace shows a lot of ansible thread spinn locking on futex | 23:11 |
| bjolo_ | waiting for go ahead | 23:12 |
| inc0 | yeah it probably spawns tons of ansible threads | 23:12 |
| inc0 | per node | 23:12 |
| inc0 | and only 30% of them are doing stuff | 23:12 |
| inc0 | rest is just hanging out | 23:12 |
| *** fguillot has joined #openstack-kolla | 23:12 | |
| inc0 | and then you have your compute nodes on splindle | 23:13 |
| inc0 | so it will take good couple of seconds to restart one service on them | 23:13 |
| inc0 | it might just add up | 23:13 |
| inc0 | hows your networking bjolo_? 10Gig? | 23:14 |
| *** tonanhngo has quit IRC | 23:15 | |
| bjolo_ | all 10Gb | 23:15 |
| bjolo_ | compute are HP BL460 G8, 2x E5-2690v2 and 192GB memory | 23:16 |
| sdake | i'm about 90% sure its tls | 23:16 |
| sdake | and the relationsship with reconfigure | 23:16 |
| *** tonanhngo has joined #openstack-kolla | 23:16 | |
| bjolo_ | sdake, ok | 23:17 |
| sdake | beccause that has not been tested | 23:17 |
| sdake | we dont test int eh gate that scenario | 23:17 |
| sdake | i have never tessted that particular usage | 23:17 |
| sdake | and that isn't what reconfigure was designed to solve originally | 23:17 |
| sdake | although we do want to get there some day | 23:17 |
| sdake | also Jeffrey4l_ did say reconfigure was broken in some way | 23:18 |
| sdake | but he was vague on details other then he is working on it | 23:18 |
| bjolo_ | so how should i enable TLS? | 23:18 |
| sdake | bjolo_ lets do a baseline test | 23:18 |
| sdake | disable it | 23:18 |
| sdake | detroy | 23:18 |
| sdake | deploy | 23:18 |
| sdake | reconfigure | 23:18 |
| bjolo_ | reason for doing this was that i got an error in horizon when trying to login "Unable to establish connection to keystone endpoint." | 23:19 |
| bjolo_ | so i redeploy w/o TLS. everything worked great | 23:20 |
| sdake | bjolo_ not sure what the cause of that is | 23:20 |
| sdake | bjolo_ already done with destro and deploy? | 23:20 |
| bjolo_ | still doing the reconfigure | 23:20 |
| bjolo_ | should i kill that? | 23:20 |
| sdake | did you destroy first | 23:20 |
| bjolo_ | nope, we did the pipline and reconfigure again | 23:20 |
| bjolo_ | no other changes | 23:21 |
| sdake | detroy, deploy, recofnigure | 23:21 |
| bjolo_ | ok | 23:21 |
| sdake | after disabling tls in your config | 23:21 |
| sdake | time the deploy time the reconfigure | 23:21 |
| sdake | bjolo_ reconfigure for tls is a very complex problem | 23:22 |
| sdake | not saying we can't or wont solve it for newton | 23:22 |
| bjolo_ | ok | 23:22 |
| sdake | only sayiing the generic reconfigure logic we use all over kolla likely not to work for that | 23:22 |
| bjolo_ | TLS worked on kolla 2 | 23:22 |
| sdake | tls works on master a well | 23:22 |
| inc0 | I honestly don't believe it has anything to do with TLS | 23:23 |
| sdake | ok well lets eliminate it | 23:24 |
| sdake | that is what debugging is all about eliminating theories | 23:24 |
| bjolo_ | ok im getting tired here... so you want me to do; destroy; disable TLS; time deploy; time reconfigure | 23:24 |
| sdake | bjolo_ you got it | 23:24 |
| *** gfhellma has quit IRC | 23:24 | |
| sdake | bjolo_ i'd do it myself but i dont have a 64 node cluster as you do :) | 23:25 |
| bjolo_ | :) | 23:25 |
| bjolo_ | ok here it goes | 23:25 |
| *** gfhellma has joined #openstack-kolla | 23:26 | |
| *** lamt has joined #openstack-kolla | 23:27 | |
| bjolo_ | i need to got to bed (1:30 in the morning for me) will report in tomorrow how this goes | 23:28 |
| bjolo_ | 0600 i have 2 kids shaking me to back to consiousness | 23:29 |
| sdake | bjolo_ thanks | 23:30 |
| sdake | see ya tonight ;) | 23:30 |
| bjolo_ | thank you :) | 23:30 |
| bjolo_ | g night folks | 23:30 |
| bjolo_ | ~ | 23:30 |
| *** lamt has quit IRC | 23:31 | |
| *** tonanhngo has quit IRC | 23:37 | |
| *** tonanhngo has joined #openstack-kolla | 23:37 | |
| *** inc0 has quit IRC | 23:41 | |
| *** openstackgerrit has quit IRC | 23:42 | |
| *** tonanhngo has quit IRC | 23:42 | |
| *** neilus has quit IRC | 23:43 | |
| *** zhurong has joined #openstack-kolla | 23:43 | |
| *** openstackgerrit has joined #openstack-kolla | 23:46 | |
| *** williamcaban has joined #openstack-kolla | 23:50 | |
| *** openstackgerrit has quit IRC | 23:51 | |
| *** hogepodge has quit IRC | 23:53 | |
| *** openstackgerrit has joined #openstack-kolla | 23:54 | |
| *** neilus has joined #openstack-kolla | 23:55 | |
| *** williamcaban has quit IRC | 23:55 | |
| *** hogepodge has joined #openstack-kolla | 23:56 | |
| *** awiddersheim has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!