Tuesday, 2017-05-23

*** yingjun has joined #openstack-kolla00:01
*** salv-orlando has quit IRC00:01
openstackgerritMerged openstack/kolla master: Add qdrouterd container  https://review.openstack.org/46514500:01
kbaegisHey, has anyone had the horizon container fail to deploy, incorrect timezone?00:02
kbaegisstable/ocata?00:02
kbaegis"ValueError: Incorrect timezone setting: UTC"00:03
*** chas has joined #openstack-kolla00:04
*** huzhengchuan has joined #openstack-kolla00:08
*** huzhengchuan_ has joined #openstack-kolla00:08
*** masber has joined #openstack-kolla00:09
*** chas has quit IRC00:09
*** ducttape_ has joined #openstack-kolla00:11
*** ducttape_ has quit IRC00:16
*** rwellum has quit IRC00:21
*** chas has joined #openstack-kolla00:25
*** ducttape_ has joined #openstack-kolla00:26
*** numans has quit IRC00:27
*** ducttape_ has quit IRC00:30
*** chas has quit IRC00:30
*** numans has joined #openstack-kolla00:32
*** tonanhngo has quit IRC00:36
*** tovin07_ has joined #openstack-kolla00:37
*** itlinux has joined #openstack-kolla00:40
*** itlinux has quit IRC00:41
*** feisky has joined #openstack-kolla00:42
*** itlinux has joined #openstack-kolla00:42
*** tonanhngo has joined #openstack-kolla00:43
*** chas has joined #openstack-kolla00:46
*** tonanhngo_ has joined #openstack-kolla00:47
*** tonanhngo has quit IRC00:48
*** tonanhngo_ has quit IRC00:51
*** chas has quit IRC00:51
*** daidv has joined #openstack-kolla00:53
*** vhosakot_ has quit IRC00:55
daidvinc0, morning, can you review for my commits related to mixing image type deployment blueprint?00:56
daidvhttps://review.openstack.org/#/q/mixing-binary-and-source-image00:56
*** zhuzeyu has quit IRC00:59
*** zhuzeyu has joined #openstack-kolla01:01
*** bmaltais has quit IRC01:04
*** jtriley has joined #openstack-kolla01:05
*** chas has joined #openstack-kolla01:07
*** chas has quit IRC01:12
*** vhosakot has joined #openstack-kolla01:13
vhosakotkbaegis: I think the "Incorrect timezone setting" is from django, not from horizon itself.  Can you check the horizon logs in the horizon container at /var/log/httpd or /var/log/apache2.01:19
*** gfidente has quit IRC01:19
*** ducttape_ has joined #openstack-kolla01:27
*** emccormick has quit IRC01:27
*** itlinux has quit IRC01:28
*** chas has joined #openstack-kolla01:28
*** zhuzeyu has quit IRC01:30
*** zhuzeyu has joined #openstack-kolla01:30
*** hrw has quit IRC01:32
*** chas has quit IRC01:33
*** hrw has joined #openstack-kolla01:34
kbaegisvhosakot: I just had to rerun deploy01:34
kbaegisSeemed to work01:34
vhosakotcool01:34
*** dave-mccowan has quit IRC01:36
*** vhosakot has quit IRC01:37
*** japestinho has joined #openstack-kolla01:38
duonghqinc0, hello, will we follow openstack release schedule for Pike?01:39
*** zhuzeyu has quit IRC01:39
sbezverksdake: ping01:41
*** caowei has joined #openstack-kolla01:42
*** chas has joined #openstack-kolla01:49
*** chas has quit IRC01:53
*** ducttape_ has quit IRC01:55
*** ducttape_ has joined #openstack-kolla01:56
inc0duonghq: we missed P-1, but I'd hope we'll release Pike-201:57
inc0Jeffrey4l is our release liason01:57
duonghqinc0, roger01:57
celo_rHi, building kolla-ansible 4.0.1 centos from source, kafka fails to build: INFO:kolla.image.build.kafka:source not yet available for centos01:58
sbezverkinc0: https://review.openstack.org/#/c/454841 was addressed, could you please review it?01:59
*** unicell has quit IRC02:00
inc0we didn't figure out why nova can't use nova rbd user?02:01
celo_rLooks like binary is not available either: INFO:kolla.image.build.kafka:binary not yet available for centos. What is kafka needed for?02:06
*** chas has joined #openstack-kolla02:10
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible stable/ocata: Mount cinder volume for cinder-volume and cinder-backup all the time  https://review.openstack.org/46668302:13
*** chas has quit IRC02:14
*** kbaegis1 has joined #openstack-kolla02:23
*** kbaegis has quit IRC02:26
sdakecelo_r nothing crucial02:28
sdakecelo_r if build binary it wont build - warning may be ignored02:28
celo_rThank you @sdake, will try to deploy the way it is.02:29
sdakeucs host upgrade utility slower then mud02:29
*** chas has joined #openstack-kolla02:30
*** ducttape_ has quit IRC02:31
openstackgerritMerged openstack/kolla-kubernetes master: v4 gate.  https://review.openstack.org/45484102:33
*** tovin07__ has joined #openstack-kolla02:34
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements  https://review.openstack.org/46439302:35
*** chas has quit IRC02:36
*** eanylin has quit IRC02:37
*** tovin07_ has quit IRC02:37
*** xinliang has quit IRC02:40
*** Pavo has joined #openstack-kolla02:40
*** ArminderSingh- has joined #openstack-kolla02:41
*** Pavo has quit IRC02:42
*** Pavo has joined #openstack-kolla02:43
*** ArminderSingh has quit IRC02:44
*** Pavo has quit IRC02:50
*** ducttape_ has joined #openstack-kolla02:51
*** chas has joined #openstack-kolla02:51
*** xinliang has joined #openstack-kolla02:52
*** xinliang has quit IRC02:52
*** xinliang has joined #openstack-kolla02:52
*** ducttape_ has quit IRC02:55
*** chas has quit IRC02:56
*** eanylin has joined #openstack-kolla02:58
*** huzhengchuan_ has quit IRC03:00
*** huzhengchuan has quit IRC03:01
*** chas has joined #openstack-kolla03:12
*** chas has quit IRC03:17
*** huzhengchuan has joined #openstack-kolla03:25
*** huzhengchuan_ has joined #openstack-kolla03:27
*** itlinux has joined #openstack-kolla03:30
*** chas has joined #openstack-kolla03:33
*** stefan_bo has quit IRC03:35
Jeffrey4lsdake, around?03:37
Jeffrey4lis helm is competitive with etcd-operator? sdake sbezverk ?03:38
*** chas has quit IRC03:38
*** huzhengchuan_ has quit IRC03:40
*** huzhengchuan has quit IRC03:41
*** huzhengchuan_ has joined #openstack-kolla03:45
*** huzhengchuan has joined #openstack-kolla03:46
*** japestinho has quit IRC03:48
*** kbaegis1 has quit IRC03:51
*** chas has joined #openstack-kolla03:54
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla-ansible master: [WIP] Add dragonflow ansible role  https://review.openstack.org/46586203:56
*** tovin07_ has joined #openstack-kolla03:58
*** chas has quit IRC03:59
duonghqJeffrey4l, we have many *-operator thing beside of etcd-operator03:59
*** tovin07__ has quit IRC04:00
*** caowei has quit IRC04:05
Jeffrey4lduonghq, i am talking this project  of etcd-op04:11
Jeffrey4lhttps://github.com/coreos/etcd-operator04:11
*** david-lyle has quit IRC04:14
*** chas has joined #openstack-kolla04:15
Jeffrey4lisn't is a Orchestration tools created by coreos?04:15
*** chas has quit IRC04:19
*** itlinux has quit IRC04:25
*** chas has joined #openstack-kolla04:36
*** chas has quit IRC04:40
duonghqJeffrey4l, kind of it, but helm is package manager04:41
duonghqI think "operator" is like concept more04:41
*** caowei has joined #openstack-kolla04:49
*** chas has joined #openstack-kolla04:57
*** chas has quit IRC04:57
*** chas has joined #openstack-kolla04:57
*** caowei has quit IRC05:04
*** ducttape_ has joined #openstack-kolla05:07
*** chas has quit IRC05:08
Jeffrey4lis kolla-k8s using helm and etcd-operator? duonghq05:10
duonghqJeffrey4l, currently, kolla-k8s is using helm05:11
*** ducttape_ has quit IRC05:11
Jeffrey4lis there any future plan? duonghq05:12
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: Add images for novajoin services  https://review.openstack.org/46626805:12
duonghqJeffrey4l, I'm not pretty sure05:13
Jeffrey4laha, thx ;)05:13
Jeffrey4li am trying to under the kubernetes works ;)05:14
Jeffrey4lworld*05:14
*** jaosorior_away is now known as jaosorior05:24
*** chas has joined #openstack-kolla05:28
*** chandankumar has joined #openstack-kolla05:30
*** chas has quit IRC05:33
*** caowei has joined #openstack-kolla05:34
openstackgerritZeyu Zhu proposed openstack/kolla-kubernetes master: Remove the invariable value in string.format()  https://review.openstack.org/46701005:35
openstackgerritZeyu Zhu proposed openstack/kolla-kubernetes master: Remove the invariable value in string.format()  https://review.openstack.org/46701005:38
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600705:48
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600705:48
*** chas has joined #openstack-kolla05:50
*** caowei has quit IRC05:52
*** caowei has joined #openstack-kolla05:52
*** chas has quit IRC05:54
*** stefan_bo has joined #openstack-kolla06:01
*** ducttape_ has joined #openstack-kolla06:08
*** chas has joined #openstack-kolla06:11
*** magicboiz has joined #openstack-kolla06:12
*** ducttape_ has quit IRC06:13
sdakesbezverk for your c220,s od you have "blanks" for storage devices and the n"caddys" for the storage devices that are actually there?06:13
*** chas has quit IRC06:15
*** magicboiz has quit IRC06:16
*** tvignaud_afk is now known as tvignaud06:16
*** magicboiz has joined #openstack-kolla06:24
*** huzhengchuan has quit IRC06:25
*** huzhengchuan_ has quit IRC06:25
*** jmccarthy has joined #openstack-kolla06:30
*** chas has joined #openstack-kolla06:32
*** sbezverk has quit IRC06:33
*** sbezverk has joined #openstack-kolla06:33
*** chas has quit IRC06:36
*** huzhengchuan has joined #openstack-kolla06:36
*** huzhengchuan_ has joined #openstack-kolla06:36
*** pcaruana has joined #openstack-kolla06:40
*** skramaja has joined #openstack-kolla06:43
*** chas has joined #openstack-kolla06:52
*** jascott1 has joined #openstack-kolla06:55
*** chas has quit IRC06:56
*** matrohon has joined #openstack-kolla06:57
*** chas has joined #openstack-kolla07:03
*** Serlex has joined #openstack-kolla07:05
*** Serlex has quit IRC07:05
*** Serlex has joined #openstack-kolla07:05
*** korzen has joined #openstack-kolla07:05
*** salv-orlando has joined #openstack-kolla07:10
*** unicell has joined #openstack-kolla07:24
*** salv-orlando has quit IRC07:25
*** manheim_ has joined #openstack-kolla07:29
berendtkolla-slack: do we have a kolla slack channel?07:37
*** huzhengchuan_ has quit IRC07:44
*** huzhengchuan has quit IRC07:44
*** blallau has joined #openstack-kolla07:49
manheim_not that i know of07:53
*** huzhengchuan_ has joined #openstack-kolla07:58
*** huzhengchuan has joined #openstack-kolla07:58
*** unicell has quit IRC07:58
*** skramaja is now known as skramaja_afk07:59
*** shardy has joined #openstack-kolla08:07
*** ducttape_ has joined #openstack-kolla08:10
*** ducttape_ has quit IRC08:15
*** egonzalez has joined #openstack-kolla08:15
zhubingbingsup egonzalez08:16
zhubingbing;)08:16
egonzalezhey zhubingbing , how are you?08:16
zhubingbingi am fine08:16
zhubingbing;)08:17
*** lpetrut has joined #openstack-kolla08:17
openstackgerritJuan Badia Payno proposed openstack/kolla master: Sensu images  https://review.openstack.org/46359308:18
openstackgerritJuan Badia Payno proposed openstack/kolla master: WIP - Modified fluentd Image  https://review.openstack.org/46707508:18
*** openstackgerrit has quit IRC08:18
manheim_egonzalez when i do kolla-ansible pull i can see it goes through ansible/site.yml, right?08:20
*** magicboiz has quit IRC08:21
*** magicboiz has joined #openstack-kolla08:21
*** athomas has joined #openstack-kolla08:21
egonzalezmanheim, yep, when you use deploy/pull/upgrade in goes to site.yml, then on each role main.yml have {{ action }}.yml08:21
egonzalezeach action is a file called upgrade,pull or deploy08:21
manheim_ok, so i do the following thing08:22
*** jascott1 has quit IRC08:22
manheim_1. put the hyper nodes in the inventory file08:22
manheim_2. pull08:22
manheim_fatal: [192.168.10.15]: FAILED! => {"failed": true, "reason": "the file_name '/usr/local/share/kolla-ansible/ansible/pull.yml' does not exist, or is not readable"}08:22
manheim_this happens08:22
manheim_that is the IP of the hyperv node08:23
*** jascott1 has joined #openstack-kolla08:23
manheim_TASK [nova-hyperv : include]08:23
egonzalezhrm, add an empty pull.yml with --- at the begining08:23
*** mandre is now known as mandre|away08:23
jaosorioregonzalez: hey! could you check this out if you have some time https://review.openstack.org/#/c/466268/ ?08:23
manheim_at the begining of what08:23
egonzalezof the file, to say is a yaml file but with empty content08:24
egonzalezsimilar to https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/aodh/tasks/check.yml08:25
egonzalezalso add the check.yml file ;)08:25
manheim_ok, i see now08:25
manheim_what is the purpose of the file more exactly, when is it used?08:25
egonzalezfor sanity checks, example create a volume and remove it08:27
*** jascott1 has quit IRC08:27
egonzalezis wip yep08:27
manheim_ok, thanks for the help08:28
*** pbourke has joined #openstack-kolla08:29
*** ipsecguy_ has joined #openstack-kolla08:30
*** openstackgerrit has joined #openstack-kolla08:32
openstackgerritZeyu Zhu proposed openstack/kolla-kubernetes master: Remove the invariable value in string.format()  https://review.openstack.org/46701008:32
*** caowei has quit IRC08:32
*** ipsecguy has quit IRC08:33
*** kbaegis has joined #openstack-kolla08:34
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Generalize api_interface_address variable usage  https://review.openstack.org/45747908:35
*** skramaja_afk is now known as skramaja08:36
*** kbaegis1 has joined #openstack-kolla08:37
*** caowei has joined #openstack-kolla08:38
openstackgerritMerged openstack/kolla-ansible stable/ocata: Mount cinder volume for cinder-volume and cinder-backup all the time  https://review.openstack.org/46668308:38
*** kbaegis has quit IRC08:40
*** vcn[m] has quit IRC08:46
*** retr0h has quit IRC08:46
*** bachp has quit IRC08:46
*** nhlfr has quit IRC08:46
*** imcsk8 has quit IRC08:46
*** imcsk8_ has joined #openstack-kolla08:46
jaosorioregonzalez: thanks for checking it out; got a question about one file though, commented there08:47
*** sambetts|afk has quit IRC08:48
*** Kimmo_ has quit IRC08:48
*** vcn[m] has joined #openstack-kolla08:49
egonzalezjaosorior, replied08:49
jaosorioroooh, right, now I get it08:50
jaosoriorthanks08:50
egonzalezbasically is that source images are mostly the same for .deb and .rpm distros, unless there is a missing library on one of them08:50
matrohonegonzalez, hi08:51
matrohonegonzalez, I have another question concerning upgrades08:52
*** sambetts_ has joined #openstack-kolla08:52
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: Add images for novajoin services  https://review.openstack.org/46626808:52
jaosorioregonzalez: well, ipa-admintools is missing from debian based distros08:52
jaosorioregonzalez: so novajoin can't be used there.08:53
jaosoriorat least it's missing on ubuntu08:53
egonzalezjaosorior, maybe is this package? https://packages.ubuntu.com/search?keywords=freeipa-admintools08:53
*** nhlfr has joined #openstack-kolla08:53
*** retr0h has joined #openstack-kolla08:53
*** bachp has joined #openstack-kolla08:53
egonzalezmatrohon, sup08:54
*** kbaegis1 has quit IRC08:54
jaosorioregonzalez: might be. I was told by the team that maintains it that they had missed the deadline for packaging it for ubuntu.08:54
matrohonegonzalez, let's say I want to upgrade only one component that I tag with 4.0.2.0 and push on my local repo, how can I upgrade only this component?08:55
*** jistr is now known as jistr|call08:56
jaosorioregonzalez: I'll give it a try08:57
egonzalezmatrohon, example of upgrade senlin-engine only08:58
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: Add images for novajoin services  https://review.openstack.org/46626808:58
egonzalezin every defaults of each role, you have a section with tag/name of the images08:58
egonzalezhttps://github.com/openstack/kolla-ansible/blob/master/ansible/roles/senlin/defaults/main.yml#L3608:58
egonzalezjust override senlin_engine_tag: <your_tag> and deploy/upgrade08:58
matrohonegonzalez, in globals.yml?08:59
egonzalezmatrohon, I'd put it there just to remember I changed the value ;) but can be in any place who overrides default variables09:00
matrohonegonzalez, great, I'll give it a try09:00
egonzalezjaosorior, added a few other comments on a previous PS than current09:04
jaosorioregonzalez: thought I had addressed them09:04
jaosorioradded the source and pip install packages, handled the debian based distros (hopefully), removed the nova user comment09:05
*** shardy has quit IRC09:05
jaosorioregonzalez: the thing is, we rely on the openstack-nova-common package to create the /etc/ bits09:06
jaosoriorsince the configuration is in /etc/nova09:06
jaosoriorwe don't have an /etc/novajoin directory09:06
jaosoriorit worked just fine for the binary image... but I guess this is an issue for the source one09:06
egonzalezone if the issues is the cp /novajoin/etc/nova which path not exists in novajoin repo09:07
jaosorioroh, right09:08
egonzalezand the other is nova-common binary into source images that are not allowed to mix09:08
jaosoriorcrap09:08
jaosoriorwell, any recommendations?09:08
jaosoriorI'm honestly never gonna use the source image :/09:08
egonzaleznova-common is only needed for the files? or for some dependency?09:08
jaosoriornor will we ever support it in tripleo09:08
jaosorioregonzalez: files, users and permissions.09:09
*** huzhengchuan has quit IRC09:10
egonzaleznova user is being created at base-image, novajoin will need to be in the nova group guess09:10
*** huzhengchuan_ has quit IRC09:10
jaosorioregonzalez: it even uses the same user09:10
egonzalezjaosorior, I'd use plugins for adding nova source code into novajoin, check an example of how to do it https://review.openstack.org/#/c/454743/09:11
egonzalezthen, remove novajoin user creation from config.py, will avoid you a bunch of merge conflicts09:12
jaosorioregonzalez: I'm not sure what the plugins bit would be used for09:12
jaosorioror how09:13
jaosorioregonzalez: or, you mean that novajoin should use the base nova image?09:14
*** shardy has joined #openstack-kolla09:15
egonzaleznot sure how close novajoin is to nova, maybe is worth be a child from nova-base and be included into nova/ folder09:17
*** jistr|call is now known as jistr09:19
matrohonegonzalez, you took senlin as an example, but can I do the same with openstack services, I want to do it for nova09:21
jaosorioregonzalez: it's a vendordata plugin. So it's close-ish... but not that much09:22
egonzalezmatrohon, nope in a single variable (is WIP atm) but can override each nova variable09:22
jaosorioregonzalez: it can be ran fairly independently and merely requires the /etc/nova folder along with the user and permissions. But that's all.09:22
jaosoriorI did think about using the nova base image... but there'a a LOT of stuff we don't really need, and the image would end up quite bloated :/ needlessly09:23
*** bjolo has joined #openstack-kolla09:24
*** huzhengchuan has joined #openstack-kolla09:24
*** masber has quit IRC09:24
*** sambetts_ is now known as sambetts09:24
bjoloJeffrey4l, sup09:25
Jeffrey4lhey bjolo09:25
matrohonegonzalez, for instance, what is the variable for specifying the tag to use for nova_scheduler? I can't find the code where this can be overridden09:25
egonzalezmatrohon, https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/defaults/main.yml#L19209:26
matrohonegonzalez, great thanks09:26
*** gfidente has joined #openstack-kolla09:27
*** gfidente has quit IRC09:27
*** gfidente has joined #openstack-kolla09:27
bjoloJeffrey4l, we have rolled back and tried older versions of kernel, docker, etc, but same issue09:27
Jeffrey4lhrm, intresting.09:27
bjolodid you try with newer images?09:27
Jeffrey4lnot yet :( . but i have a new deployment multi node now. let me test this now.09:28
Jeffrey4lmom09:28
Jeffrey4lmonent.09:28
bjolotnx09:28
*** powerd has joined #openstack-kolla09:33
jaosoriorHow are services that use genconfig handled?09:36
*** huzhengchuan_ has joined #openstack-kolla09:36
bjoloJeffrey4l, https://bugs.launchpad.net/kolla-ansible/+bug/169285209:39
openstackLaunchpad bug 1692852 in kolla-ansible "nova_libvirt restart fails when there are runing VMs" [Undecided,New]09:39
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a Kolla 'devstack' mode  https://review.openstack.org/45469009:43
Jeffrey4lbjolo, it works.09:45
Jeffrey4lcould u try docker-engine from docker official? rather than docker from centos packages.09:48
zhubingbingsup sdake09:52
*** tovin07_ has quit IRC09:55
*** ducttape_ has joined #openstack-kolla09:57
*** ducttape_ has quit IRC10:02
bjoloJeffrey4l, have tried 17.05 as well10:04
openstackgerritMerged openstack/kolla-ansible master: Generalize api_interface_address variable usage  https://review.openstack.org/45747910:04
bjolodifferent error with 17.05 tho10:04
bjolonova_libvirt restarts but you can not launch any new VMs on that node or attach volumes untill all running vms have been restarted. i.e. moved back to nova_libvirt as parent pid10:05
bjoloJeffrey4l, in the ticket, can you please write down your setup? os, kernel, docker, storage, etc10:06
manheim_hmm, i have an interesting error while using 'tunnel_interface'10:07
manheim_"msg": "{{ hostvars[inventory_hostname]['ansible_' + tunnel_interface]['ipv4']['address'] if orchestration_engine == 'ANSIBLE' else '0.0.0.0' }}: 'dict object' has no attribute 'ipv4'"10:07
manheim_but I did not set any IP on it10:07
manheim_i put the interface on manual10:07
bjolomanheim_, how did u do the deploy? with --limit?10:08
manheim_no10:08
manheim_simple deploy10:08
bjolohmm ok10:08
*** rmart04 has joined #openstack-kolla10:08
bjoloive seen similar errors when using --limit10:08
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Fix zun deployment  https://review.openstack.org/45625610:10
*** yingjun has quit IRC10:11
egonzalezmanheim, is that failing on control nodes where tunnel_interface is not present?10:12
pesterinstalled kolla 4.0.0 with magnum, trying to run bay, error - "from kubernetes import client as k8s_config" "ImportError: cannot import name client"10:12
Jeffrey4lbjolo, could u show me "rpm -qa | grep docker"10:13
pesterand I did "pip install python-kubernetes" in conductor container10:13
hrwmorning10:13
*** yee379 has quit IRC10:13
*** yee379 has joined #openstack-kolla10:14
manheim_i have only one control node, that interface is present10:14
egonzalezpester, magnum code had a requirement bug missing kubernetes library, if using images from docker hub 4.0.0 have that issue, if you build your own should be fixed10:14
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: Add images for novajoin services  https://review.openstack.org/46626810:14
pesterI got this issue, how to fix it ? I tried  "pip install python-kubernetes"  it solved first error, but now next...10:15
manheim_the tunnel interface is set up the same as for br-ex, and br-ex has no issues10:21
*** lpetrut has quit IRC10:22
*** lpetrut has joined #openstack-kolla10:22
pesterok, to fix magnum run `pip install kubernetes && pip install python-kubernetes`10:23
pesterin conductor container10:23
egonzalezpester, was fixed upstream early ocata, but not present in published images. btw, ensure kubernetes is kubernetes>=1.0.0b1 # Apache-2.010:25
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' for only necessary tasks (default roles)  https://review.openstack.org/39868410:25
*** caowei has quit IRC10:28
*** duonghq has quit IRC10:29
*** stefan_bo has quit IRC10:33
*** daidv has quit IRC10:42
*** Kimmo_ has joined #openstack-kolla10:48
*** salv-orlando has joined #openstack-kolla10:49
openstackgerritMartin Mágr proposed openstack/kolla master: Sensu images  https://review.openstack.org/46359310:51
openstackgerritMerged openstack/kolla-ansible master: Updated from global requirements  https://review.openstack.org/46525810:52
*** salv-orlando has quit IRC10:53
openstackgerritMerged openstack/kolla-ansible master: Add VMware DataStore support to glance  https://review.openstack.org/45217610:57
bjoloJeffrey4l, http://paste.openstack.org/show/610439/10:57
manheim_so am I require to have an IP on the tunnel interface?11:01
manheim_and why11:01
*** huzhengchuan has quit IRC11:02
*** huzhengchuan_ has quit IRC11:02
openstackgerritMerged openstack/kolla-ansible master: Advertise public heat endpoint for wait conditions  https://review.openstack.org/46265511:09
*** Pavo has joined #openstack-kolla11:11
*** Pavo has quit IRC11:11
*** huzhengchuan has joined #openstack-kolla11:14
*** huzhengchuan_ has joined #openstack-kolla11:14
openstackgerritMerged openstack/kolla-ansible master: Fix wrong variable used in trove role  https://review.openstack.org/46325811:15
openstackgerritMerged openstack/kolla-ansible master: Fix secure_proxy_ssl_header option in nova  https://review.openstack.org/45206211:16
*** rwellum has joined #openstack-kolla11:21
openstackgerritMerged openstack/kolla-ansible master: Add copy ceph keyring files conditions in cinder  https://review.openstack.org/45836611:23
openstackgerritMerged openstack/kolla-ansible master: Optimize reconfigure action for rabbitmq  https://review.openstack.org/42270511:24
openstackgerritMerged openstack/kolla master: Replace invalid link for Docker images in README.rst  https://review.openstack.org/46347911:24
rwellumMorning11:26
openstackgerritPavel Gluschak (scsnow) proposed openstack/kolla-ansible master: Make cinder-backup service optional  https://review.openstack.org/46462011:29
*** bjolo has quit IRC11:29
*** bjolo has joined #openstack-kolla11:29
*** awebber has joined #openstack-kolla11:41
*** salv-orlando has joined #openstack-kolla11:42
awebbermy goal is to launch more instances per host. is this achieved with 'cpu_allocation_ratio' and what is the best way to change it in kolla?11:46
*** mnasiadka has joined #openstack-kolla11:51
*** salv-orlando has quit IRC11:52
*** ducttape_ has joined #openstack-kolla11:58
*** srwilkers has joined #openstack-kolla12:03
egonzalezawebber, create a file at /etc/kolla/config/nova.conf with the section, option, value of the value you want to override and do kolla-ansible reconfigure12:04
*** ducttape_ has quit IRC12:04
*** rhallisey has joined #openstack-kolla12:04
awebberegonzalez: thanks, ill give it a try...12:05
awebberegonzalez: i already have a file called nova-compute.conf. will it get merged if i create a nova.conf in the same directory?12:07
egonzalezawebber, you can override values for each node, per service or global for nova, all depends where you put the values12:08
*** bachp has quit IRC12:09
*** vcn[m] has quit IRC12:09
*** nhlfr has quit IRC12:09
*** retr0h has quit IRC12:09
egonzalezso, you can change schedulers only for nova-scheduler service, virt_type for computes and have a nova.conf to configure something else on both services12:09
egonzalezs/both/all12:09
*** mrunge has quit IRC12:11
*** dgonzalez has quit IRC12:12
*** aolwas has quit IRC12:12
*** iniazi_ has joined #openstack-kolla12:13
*** vcn[m] has joined #openstack-kolla12:13
*** mrunge has joined #openstack-kolla12:14
*** dgonzalez has joined #openstack-kolla12:14
*** dave-mccowan has joined #openstack-kolla12:14
sdakemorning12:16
*** iniazi has quit IRC12:17
*** bachp has joined #openstack-kolla12:17
*** retr0h has joined #openstack-kolla12:17
*** nhlfr has joined #openstack-kolla12:17
egonzalezmorning sdake12:19
rwellumMorning sdake12:21
Jeffrey4lbjolo, you are using docker from centos. could u have a try official docker?12:22
openstackgerritBertrand Lallau proposed openstack/kolla-ansible stable/ocata: Fix Telegraf container restarting issue  https://review.openstack.org/46403912:25
manheim_I have an issue with tempest12:27
manheim_running minimum basic scenario but erorrs out with IOError: [Errno 2] No such file or directory: '/opt/stack/new/devstack/files/images/cirros-0.3.1-x86_64-uec/cirros-0.3.1-x86_64-vmlinuz'12:27
manheim_dunno why it looks there as I modified tempest.conf accordingly12:27
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Panko.conf cannot be copied in ceilometer role  https://review.openstack.org/45853412:28
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Allow serial per service  https://review.openstack.org/46009012:30
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing some issues with v4 gates  https://review.openstack.org/46653612:32
*** bjolo has quit IRC12:32
*** bmaltais has joined #openstack-kolla12:33
*** bjolo has joined #openstack-kolla12:34
*** tomatopeel has quit IRC12:40
openstackgerritJuan Badia Payno proposed openstack/kolla master: WIP - Modified fluentd Image  https://review.openstack.org/46721912:43
*** Pavo has joined #openstack-kolla12:48
*** salv-orlando has joined #openstack-kolla12:48
*** salv-orlando has quit IRC12:53
*** bjolo has quit IRC12:53
*** jtriley has quit IRC12:55
manheim_found my tempest issue, scenario was missing12:56
*** jaosorior has quit IRC12:57
*** jaosorior has joined #openstack-kolla12:57
*** m1dev has joined #openstack-kolla13:06
*** celo_r has quit IRC13:06
awebberegonzalez: the values got copied over but it didnt reflect in the horizon dashboard. I am on an 3.0.0 though, could be an older bug...13:06
egonzalezhrm, iirc nova didnt change those values once deployed13:07
*** huzhengchuan has quit IRC13:09
*** huzhengchuan_ has quit IRC13:10
bmaltaisGoo morning. I have a strange issue I have not been able to troubleshoot yet. It has to do with memcached13:11
bmaltaisFor some reason memcached stay in Restarting (71) for ever during and after a successfull deployment with kolla-ansible13:11
bmaltaisI tried everything to fix that. Is there a log file I can look into to see whay it keep failing like that?13:12
bmaltaisEverything was working yesterday... but I tried to rebuild me environment to confirm my documentation and now the environment won't deploy properly.13:13
bmaltaisI have a feeling it has to do with DNS/hosts file13:13
bmaltaisThe end result is that horizon display "Something went wrong!" upon successful login.13:14
*** salv-orlando has joined #openstack-kolla13:15
bmaltaisIn the docker log I see "Error setting up exec command in container memcached: "13:16
egonzalezbmaltais, strange, the only thing memcached changed recently is remove verbose from logs13:16
*** lucasxu has joined #openstack-kolla13:17
*** lucasxu has quit IRC13:17
*** spsurya has quit IRC13:17
bmaltaisegonzalez: I am using "openstack_release: "4.0.0""13:17
egonzalezbmaltais, whats the content of /etc/kolla/memcached/config.json13:18
egonzalezhrm, then is not the same13:18
egonzalezmemcached is a simple service, just start on its IP address and port, nothing else13:19
bmaltaisegonzalez: http://paste.openstack.org/show/610451/13:19
egonzalezthat is good unless the IP is not in the host or the port used by any other service13:20
*** salv-orlando has quit IRC13:20
bmaltaisegonzalez: netstat | grep 11211 return nothing13:21
bmaltaisegonzalez: Same for 500013:21
bmaltaisegonzalez: Been pulling my hair since yesterday's afternoon on that13:22
*** eanylin has quit IRC13:22
egonzalezbmaltais, whats the docker info memcahed y docker logs memcached output?13:23
*** shardy has quit IRC13:25
bmaltaisegonzalez: http://paste.openstack.org/show/610452/13:25
bmaltaisegonzalez: docker info http://paste.openstack.org/show/610453/13:27
*** shardy has joined #openstack-kolla13:29
*** lucasxu has joined #openstack-kolla13:30
*** chas has quit IRC13:30
egonzalezbmaltais, may be something related to max open files in ulimit13:30
*** chas has joined #openstack-kolla13:30
egonzalezby default we set 500013:31
egonzalezsorry, mean inspect memcached instead of docker info13:31
bmaltaisegonzalez: Where can I change this value? Is it in the node or the container?13:31
egonzalezbmaltais, make a test just changing /etc/kolla/memcached/config.json , change 5000 to a higher value and restart the container and check if fixes the issue13:32
*** skramaja is now known as skramaja_afk13:32
bmaltaisegonzalez: ok, let see13:33
*** fguillot has joined #openstack-kolla13:33
bmaltaisegonzalez: So the new reported command is: "Running command: '/usr/bin/memcached -vv -l 10.10.10.231 -p 11211 -c 8000'"13:34
bmaltaisegonzalez: Increased -c to 800013:34
*** eanylin has joined #openstack-kolla13:35
bmaltaisegonzalez: Same result13:35
egonzalezcan you check ulimit -a?13:35
*** chas has quit IRC13:35
bmaltaisegonzalez: http://paste.openstack.org/show/610455/13:36
egonzalezthe host have open files                      (-n) 102413:36
bmaltaisegonzalez: So this is too low? I never changed this value before... perhaps something new with a recent centos7 update?13:37
bmaltaisegonzalez: Changing -c to 1000 allow the container to start13:40
bmaltaisegonzalez: Not sure why this is happening out of nowhere13:40
egonzalezwell, i would increase host max open files instead of lowering memcached13:41
egonzalezmemcached open a lot of files13:41
bmaltaisegonzalez: I tried ulimit -n 8000 and with -c 5000 it was still a no go13:41
bmaltaisWhat is your ulimit for files at your end?13:42
egonzalezulimit -n13:42
egonzalez102413:42
egonzalezlet me test memcached container13:42
bmaltaisegonzalez: Wow... this is bizare13:42
egonzalezbmaltais, inside memcached container ulimit -n13:44
egonzalez6553613:44
bmaltaisegonzalez: Changing -c to 4000 also work with ulimit -n 102413:45
bmaltaisegonzalez: So is it a possible container issue?13:45
bmaltaisegonzalez: I tought they were not changing once published13:46
egonzalezbmaltais, not sure, cannot confirm that13:46
egonzalezthe container didnt changed, and the config only removed --verbose13:46
egonzaleznothing else afik13:46
bmaltaisegonzalez: What is in your config.json for memcached? 5000 or 4000?13:46
egonzalezdefault 500013:47
*** dave-mcc_ has joined #openstack-kolla13:47
bmaltaisegonzalez: Wow... this is weird. If I change the value to 4000 all work. Not sure why going to 5000 trip it to fail13:47
egonzalez5000 goes beyond the limits, where those limits came, no idea :/13:48
bmaltaisegonzalez: I will try to find it as it might be good to add to the guide13:48
bmaltaisegonzalez: I am sure others will also run into this. Can't be alone. Maybe a trend caused by a new centos7 update13:49
*** dave-mccowan has quit IRC13:49
*** ducttape_ has joined #openstack-kolla13:49
*** tomatopeel has joined #openstack-kolla13:49
*** dave-mccowan has joined #openstack-kolla13:50
*** ducttape_ has quit IRC13:51
*** ducttape_ has joined #openstack-kolla13:53
*** dave-mcc_ has quit IRC13:53
egonzalezbmaltais, actually -c is for max connections not open files13:55
*** jtriley has joined #openstack-kolla13:57
bmaltaisegonzalez: So possibly related to the network connections limit on the node13:58
*** rhallisey has quit IRC13:59
*** shardy_ has joined #openstack-kolla14:00
*** rhallisey has joined #openstack-kolla14:04
*** JoseMello has joined #openstack-kolla14:05
pesterfluentd have UI ? how to access it ?14:06
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Support OSprofile usage  https://review.openstack.org/45562814:07
*** salv-orlando has joined #openstack-kolla14:10
*** jtriley has quit IRC14:10
*** jtriley has joined #openstack-kolla14:11
*** chas has joined #openstack-kolla14:12
*** salv-orlando has quit IRC14:14
*** eaguilar has joined #openstack-kolla14:16
*** chas has quit IRC14:16
*** emccormick has joined #openstack-kolla14:20
Serlex@pester, for logs?14:20
Serlexyou probably want kibana to view them14:21
bmaltaisegonzalez: Changing the value to 4096 also work14:21
bmaltaisSo possibly some limit imposed by nproc.d14:21
egonzalezbmaltais, dont know where those limits come14:22
bmaltaisAre you running centos7?14:22
egonzalezif you found it, let me know so i learn from this too14:22
egonzalezyep14:22
bmaltaisegonzalez: Did you try to do yum update on one of your working node to see if it would break after the update?14:22
bmaltaisegonzalez: I am really wondering why I am the only one with this issue. I am booting my VM from the same template that always used to work14:23
bmaltaisegonzalez: So it must be something external to the state it always was14:23
egonzalezbmaltais, only have docker upgrade, upgraded a few days ago14:24
bmaltaisegonzalez: It would have been introduced sometime yesterday... as it was working the day before14:24
*** ducttape_ has quit IRC14:24
egonzalezbmaltais, going to upgrade a node and see if is the only breaking14:26
sbezverkk8s-kolla cores: this one is ready to review and merge https://review.openstack.org/466536 most of experimental jobs are green now..14:28
*** eaguilar has quit IRC14:28
*** klindgren has quit IRC14:29
bmaltaisegonzalez: OK. If it stay fine than the issue is somewhere else.14:31
egonzalezsomething weird, but had docker-engine 1.12 with docker-selinux 15 and was working :S14:32
*** chas has joined #openstack-kolla14:33
egonzalezbmaltais, still working after upgrade14:33
*** klindgren has joined #openstack-kolla14:35
*** mattmceuen has joined #openstack-kolla14:35
bmaltaisegonzalez: OK. If it stay fine than the issue is somewhere else.14:36
egonzalezanyone used bootstrap-servers role to install remote nodes?14:37
bmaltaisegonzalez: This is really strange. I guess I will have to manually change -c to 4096 until the source of the issue is found14:37
*** chas has quit IRC14:37
egonzalezcan check docker-selinux installed version?14:37
bmaltaisegonzalez: Humm... not sure how to do that14:38
egonzalezbmaltais, that was a general question to everybody deploying servers using bootsrap-servers command14:38
bmaltaisegonzalez: OK ;-)14:39
egonzalezsimply rpm -qa | grep docker or the corresponding on .deb distros14:39
bmaltaisegonzalez: docker-engine-selinux-17.05.0.ce-1.el7.centos.noarch14:40
egonzaleznot sure if was my mistake while upgrading other things or a bug in baremetal role14:40
bmaltaisegonzalez: That got upgraded with yum update14:40
bmaltaisegonzalez: But I had the same issue before the update anyway14:40
egonzalezyep, but I had docker-engine 1.12 and selinux in 17.0514:40
bmaltaisegonzalez: I also used the bootsrap-servers command to configure my nodes14:41
egonzalezbootstrap pinpoint to docker-engine 1.1214:42
bmaltaisegonzalez: I did notice some strange things with the result. For example, the hosts file looked wrong14:42
*** eaguilar has joined #openstack-kolla14:42
bmaltaisegonzalez: But it gor upgraded when using yum update14:42
bmaltaisegonzalez: The host file contained localhost for my node IP instead of the proper node names14:43
egonzalezbmaltais, may have localhost as hostname?14:45
egonzalezopened a bug for etc/hosts with wrong formatting https://bugs.launchpad.net/kolla-ansible/+bug/169291314:46
openstackLaunchpad bug 1692913 in kolla-ansible "wrong etc/hosts format with baremetal role" [Low,New]14:46
egonzalezquite easy if someone wants to take a bug ;)14:46
openstackgerritSam Betts proposed openstack/kolla-kubernetes master: Switch ironic-dnsmasq to Deployment to ensure only one replica  https://review.openstack.org/46727214:49
*** shardy has quit IRC14:49
*** shardy_ has quit IRC14:50
openstackgerritBertrand Lallau proposed openstack/kolla master: Remove all Heka related remaining stuff  https://review.openstack.org/44603614:50
*** shardy has joined #openstack-kolla14:51
bmaltaisegonzalez: Correct. I tought the script would take care of setting the hostname to the provided node name14:52
*** shardy_ has joined #openstack-kolla14:52
bmaltaisegonzalez: I guess it does not and it need to be set 1st before running the script14:52
*** chas has joined #openstack-kolla14:54
openstackgerritSam Betts proposed openstack/kolla-kubernetes master: Make sure pathfinder can find passwords.yml in etc/kolla  https://review.openstack.org/46727614:54
openstackgerritSam Betts proposed openstack/kolla-kubernetes master: Switch ironic-dnsmasq to Deployment to ensure only one replica  https://review.openstack.org/46727214:55
*** huzhengchuan has joined #openstack-kolla14:56
*** huzhengchuan_ has joined #openstack-kolla14:57
*** lpetrut has quit IRC14:57
*** itlinux has joined #openstack-kolla14:57
*** chas has quit IRC14:58
bmaltaisegonzalez: Looking at the mariadb galera.cfg file something hit me14:59
bmaltaisegonzalez: The guide is not clear about using dhcp vs static for nodes interfaces (or I did not saw it).15:00
bmaltaisegonzalez: The way the configs are written it use static IP vs fqdn15:00
bmaltaisegonzalez: So if the DHCP renew change the IP everything will collapse15:00
*** ducttape_ has joined #openstack-kolla15:01
pestermagnum on kolla 4.0.0 - got error "InvalidParameterValue: ERROR: Missing required credential: roles ['heat_stack_owner']"15:02
pesterbut role is present - openstack role create heat_stack_owner15:03
pesterDuplicate entry found with name heat_stack_owner15:03
*** ducttape_ has quit IRC15:03
*** ducttape_ has joined #openstack-kolla15:04
egonzalezpester, need to add your user into heat_stack_owner role15:04
egonzalezbmaltais, yep, that is something we can document on networking guide, ie, rabbit relies on hostname instead of ip15:05
pesternow error is "Keystone API connection failed: no password, trust_id or token found"15:06
bmaltaisegonzalez: Make sense. I might propose a change to the guide to address this.15:06
*** ducttape_ has quit IRC15:07
bmaltaisegonzalez: Changing -c to 4096 fixed my issue with memcached15:07
*** ducttape_ has joined #openstack-kolla15:07
egonzalezprobably we assume that users will use static IPs, probably too many assumptions on our side :)15:08
bmaltaisLet see if others run into it. A possible fix might be to reduce the ansible config from 5000 to 4096 to prevent it15:08
egonzalezi never had an issue with 5000 :/15:08
bmaltaisegonzalez: Static is probably a best practive for any DC servers ;-)15:08
bmaltaisegonzalez: Lab environment... not so much15:08
egonzalezbmaltais, actually I have dhcp for my test vms15:09
egonzalezmay be lucky but IPs never changed ;)15:09
bmaltaisegonzalez: As long as users know that if DHCP is used it should be "fixed" or else problems will occur.15:10
*** blallau has quit IRC15:11
bmaltaisegonzalez: On a different topic, when I try to deploy using the git clone it always fail. I have to use the pip kolla-ansible version for it to complete.15:12
bmaltaisegonzalez: Is there a known deployment issue with the git clone project?15:12
egonzalezbmaltais, nope, i always use the git version15:13
*** shardy_ has quit IRC15:13
bmaltaisegonzalez: OK. So it might be worth trying again and reporting the issue I am seeing.15:13
*** shardy_ has joined #openstack-kolla15:13
egonzalezbmaltais, just need this commands http://paste.openstack.org/show/610467/15:14
*** chas has joined #openstack-kolla15:14
*** lpetrut has joined #openstack-kolla15:14
bmaltaisegonzalez: Will give it a try15:15
*** hieulq_ has joined #openstack-kolla15:15
*** Serlex has quit IRC15:16
bmaltaisegonzalez: I was not installing it like that before... perhaps this was the issue... I was doing python setup.py build ; python setup.py install15:18
bmaltaisegonzalez: Was this wrong?15:18
egonzalezyep, local installs only need requirements15:19
*** chas has quit IRC15:19
egonzalezand execute commands locally in the git repo15:19
bmaltaisegonzalez: local install... like in AIO? or local install of the git project15:20
egonzalezgit repo, sorry15:20
pesterhm, it should work ? "/etc/kolla/config/magnum/magnum.conf &&  kolla-ansible reconfigure --tags magnum" ??15:21
*** eaguilar has quit IRC15:21
bmaltaisegonzalez: I see. OK. Failed again with the same error...15:22
bmaltaisegonzalez: http://paste.openstack.org/show/610472/15:23
egonzalezpester, nope, /etc/kolla/config/magnum.conf should15:23
*** tovin07 has joined #openstack-kolla15:23
egonzalezbmaltais, jinja version?15:24
*** tovin07 has left #openstack-kolla15:24
bmaltaisegonzalez: On the deploy node or the controller nodes?15:25
egonzalezdeploy15:25
bmaltaisegonzalez: Doing rpm -qa | grep jinja return nothing...15:26
egonzalezpip freeze | grep -i jinja15:27
bmaltaisJinja2==2.9.615:28
openstackgerritJuan Badia Payno proposed openstack/kolla master: WIP - Modified fluentd Image  https://review.openstack.org/46721915:28
egonzalezbmaltais, try pip install Jinja2==2.8.2 and deploy again15:28
bmaltaisegonzalez: Is 2.8.1 OK? I get an error with 2.8.2: "Could not find a version that satisfies the requirement Jinja2==2.8.2 (from versions: 2.0rc1, 2.0, 2.1, 2.1.1, 2.2, 2.2.1, 2.3, 2.3.1, 2.4, 2.4.1, 2.5, 2.5.1, 2.5.2, 2.5.3, 2.5.4, 2.5.5, 2.6, 2.7, 2.7.1, 2.7.2, 2.7.3, 2.8, 2.8.1, 2.9, 2.9.1, 2.9.2, 2.9.3, 2.9.4, 2.9.5, 2.9.6) No matching distribution found for Jinja2==2.8.2"15:29
egonzalez2.8.115:29
bmaltaisegonzalez: Deploying...15:30
*** mgoddard has joined #openstack-kolla15:30
*** mnasiadka has quit IRC15:33
hrwdoes someone remember price for attending PTG?15:33
kfox1111morning.15:33
egonzalezhrw, was about 150$ iirc15:33
bmaltaisegonzalez: Same crash15:33
hrwtrying to gather costs for my 3 weeks US trip15:33
sbezverkkfox1111: good morning, how is your yesterday's fire? extinguished I hope?15:34
kfox1111heh. ongoing. :/15:34
kfox1111narrowing in on it though I think.15:34
hrwegonzalez: that's with contributor discount?15:34
kfox1111"there are 2 hard things in computer science. naming things, cache invalidation, and off by one errors"15:34
kfox1111this one's a cache thing I think. :/15:35
*** mgoddard has quit IRC15:35
egonzalezhrw, full access was 100, there was no contributor discount15:35
*** chas has joined #openstack-kolla15:35
hrwegonzalez: ah right. discount was for summit15:36
sbezverkkfox1111: yeah I watch one api design course, and one famous dude said that nameing is 90% of programming ;)15:37
kfox1111hehe. yeah, I know right?15:37
kfox1111takes so long to come up with a good name.15:38
bmaltaisegonzalez: Trying to replace globals.yml with the one from new git repo and apply my config on it15:38
bmaltaisegonzalez: Let see if this help15:38
egonzalezbmaltais, also check the inventory file15:38
sbezverkkfox1111: :) if you have a couple of minutes, please check https://review.openstack.org/#/c/466536/15:39
bmaltaisegonzalez: The inventory file... where is it located?15:39
*** itlinux has quit IRC15:39
bmaltaisYou mean the deployment file?15:39
bmaltaisLike multinode15:39
egonzalezif using allinone in ansible/inventory but if have custom nodes, need a file with all node and groups on it15:40
egonzalezbmaltais, yep15:40
*** chas has quit IRC15:40
bmaltaisegonzalez: OK, will copy the example file from git repo and apply my node names on it and see it that help15:41
kfox1111sbezverk: looks really good. one question on the review.15:44
*** lucasxu has quit IRC15:44
openstackgerritsean mooney proposed openstack/kolla master: enable the kolla build system to use out of tree docker files  https://review.openstack.org/46613215:44
*** sean-k-mooney has joined #openstack-kolla15:45
sbezverkkfox1111: nameing nameing ;) yeah I will change it..15:46
sean-k-mooneyJeffrey4l: https://review.openstack.org/#/c/466132/ should now work the way you suggested15:46
kfox1111sbezverk: thanks. :)15:46
*** pcaruana has quit IRC15:47
Jeffrey4lsean-k-mooney, nice. LGTM at fist glance. will test it later. thx. ;D15:47
*** mnasiadka has joined #openstack-kolla15:49
bmaltaisegonzalez: Yeah... the issue was the inventory file I guess15:49
*** lucasxu has joined #openstack-kolla15:49
bmaltaisThere must be a significant diff that was causing the crash15:49
*** mnasiadka has left #openstack-kolla15:49
bmaltaisegonzalez: Now that this work I will zero everything and re-install everything from scratch to see if the memcached error is gone using the git repo version15:50
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing some issues with v4 gates  https://review.openstack.org/46653615:52
sbezverkkfox1111: done15:52
*** jrich523 has joined #openstack-kolla15:52
kfox1111sbezverk: thanks. looks good to me. :)15:53
*** chas has joined #openstack-kolla15:56
*** manheim_ has quit IRC15:57
kfox1111sdake: inc0: https://review.openstack.org/#/c/466398/16:00
kfox1111plz16:00
*** chas has quit IRC16:01
*** scheuk has joined #openstack-kolla16:02
bmaltaisegonzalez: When running kolla-ansible git repo... should I also install the kolla repo to get the gen-password tools?16:02
egonzalezbmaltais, all is in the repo at tools16:02
egonzalezpython generate-passwords.py16:02
scheukI just got into an interesting situation, I performed a kolla-ansible reconfigure with 3.0.3 and now the nova user is missing and I can't docker exec -it into some of the containers16:03
bmaltaisI appear to be missing some dependancies then... "ImportError: No module named Crypto.PublicKey"16:03
*** rhallisey has quit IRC16:03
sdakekfox1111 looking16:04
*** jamesbenson has joined #openstack-kolla16:04
sdakekfox1111 looks like it merged16:04
*** rhallisey has joined #openstack-kolla16:04
kfox1111sdake: thanks.16:04
bmaltaisegonzalez: pip install pycrypto16:04
kfox1111srwilkers: thanks. :)16:05
*** cristicalin has joined #openstack-kolla16:05
egonzalezbmaltais, should be installed with requirements.txt16:05
bmaltaisegonzalez: Should I propose a change to requirements.txt file to add it?16:05
bmaltaisegonzalez: Was not ;-(16:05
openstackgerritMerged openstack/kolla-kubernetes master: Remove workaround for broken 1.6.3  https://review.openstack.org/46639816:06
jamesbensonGood morning everyone!16:06
kfox1111jamesbenson: good morning. :)16:07
jamesbensonI have a question and I was wondering if someone could help... I've been struggling with it for a few days now and can't seem to get it working (or if it is possible at this version)16:07
*** ducttap__ has joined #openstack-kolla16:07
jamesbensonI'm using MaaS to deploy servers and pre-created the KOLLA_CEPH_DATA_0 and KOLLA_CEPH_DATA_0_J partitions (with the xfs and ceph format)...but ansible doesn't like it....16:08
jamesbensonI'm familiar with Ansible, but can't seem to get it to jive.16:08
*** ducttape_ has quit IRC16:09
egonzalezbmaltais, found a bug ;), pycrypto is being installed with kolla repo not with kolla-ansible16:09
kfox1111sdake: see thread "Poroposal for shipping binaries and containers"16:10
*** lucasxu has quit IRC16:14
bmaltaisegonzalez: Glad I can help debug this ;-)16:15
*** matrohon has quit IRC16:15
bmaltaisegonzalez: One last question Ihope... when I try to run "./kolla-ansible bootstrap-servers -i ~/multinode"16:15
egonzalezjamesbenson, ceph need partitions labelled as KOLLA_CEPH_OSD_BOOTSTRAP_J https://github.com/openstack/kolla-ansible/blob/master/doc/ceph-guide.rst#using-an-external-journal-drive16:16
egonzalezjamesbenson, if not just will ignore those partitions and wont create any osd16:16
bmaltaisegonzalez: One last question Ihope... when I try to run "./kolla-ansible bootstrap-servers -i ~/multinode" I get "Command failed ansible-playbook -i /root/multinode -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla  -e action=bootstrap-servers /root/kolla-ansible/ansible/kolla-host.yml"16:16
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a Kolla 'devstack' mode  https://review.openstack.org/45469016:17
*** chas has joined #openstack-kolla16:17
bmaltaisegonzalez: Another missing dependency: pip install -U ansible16:18
sean-k-mooneybmaltais: do you have any more info on how the bootstrap-server command fails?16:18
jamesbenson@egonzalez, I know, but that's assuming you have more than one drive dedicated for ceph.  Part of our policy is to run in raid 5/6 so we have one large drive (8 Tb).  So I'm separating 7 Tb for ceph16:18
*** vhosakot has joined #openstack-kolla16:19
*** jmccarthy has quit IRC16:19
*** shardy has quit IRC16:19
egonzalezbmaltais, follow quickstart-guide to install deps ;)16:19
bmaltaissean-k-mooney: Because my nodes hostnames were not set to start with16:19
bmaltaissean-k-mooney: The bootstrap script does not set it as part of the boot-strap16:19
sean-k-mooneybmaltais: it will but in the host you run boot strap server you first either need to have them set in /etc/hosts or in the inventory16:20
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements  https://review.openstack.org/46439316:21
jamesbenson@bmaltais, be sure to use the internal nic IP's too if you have an option....16:21
bmaltaisegonzalez: Right ;-) I was expecting too much I guess ;-) Now on to the next issue ;-)16:21
*** skramaja_afk has quit IRC16:21
sean-k-mooneybmaltais: it assummes that all host listed in the inventory either have ther ips listed in the inventory or that the hostnames are resolvable on the local system16:21
*** lpetrut has quit IRC16:22
bmaltaissean-k-mooney: I was using the pip kolla-ansible version... and it might have been the cause.16:22
*** chas has quit IRC16:22
bmaltaissean-k-mooney: I am now trying using the latest hit repo version. Perhaps I should unset the hostname to validate that it does set them16:22
sdakekfox1111 have you given any thought to password rotation16:23
kfox1111sdake: what kind of password rotation?16:23
kfox1111like fernet rotation or something else?16:23
sdakekfox1111 genpwd -> replace all passwords in the system with the new passwords.yml secret contents16:24
sdake(and associated configmaps)16:24
bmaltaissean-k-mooney: The bootstrap script now fails with: * fatal: [kolla1.ducourrier.com]: FAILED! => {"failed": true, "msg": "the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: 'dict object' has no attribute u'ansible_ens160'\n\n16:24
kfox1111sdake: thats not a use case openstack really supports. :/16:24
sdakenot high priority - just something i'd like to see16:24
sdakekfox1111 it can be done :)16:24
kfox1111it should be done, but really needs the ability for an account to have 2 passwords.16:24
kfox1111and old an a new, so it can be rotated properly.16:24
kfox1111as you can't update all places instantaniously.16:25
sdakei was thinking a really smooth way of doing so would be to have oslo.cfg read kubernetes secretes directly16:25
kfox1111keystone needs support for that though.16:25
sdakethere would obviously be some minimal outage16:25
bmaltaissean-k-mooney: ens160 is my primary NIC name16:25
* sdake whines - can someone upgrade my bios for me :)16:26
kfox1111+1 to getting oslo.cfg to be able to seperate secrets from the rest of the config.16:26
jamesbensonbmaltais:  pull your ansible hosts inventory and see what it is calling your nics...16:26
jamesbensonbmaltais: ansible all -i multinode -m setup --tree /tmp/facts16:27
sdakecould you expand on an account having 2 passwords?  Is this for a seamless password rotation?  a single password should work with a little bit of downtime (under a few seconds)16:27
jamesbensonbmaltais: I had something like that as well...I was trying virtual eths....16:27
sdakekfox1111 slack when you get a chance16:27
kfox1111k.16:28
sean-k-mooneybmaltais: do you have ens160 on all servers you are bootstraping?16:28
jamesbensonegonzalez: I definitely understand the importance of it using the one named partition, so it doesn't destroy the rest of your drive unintentially.  But no way to recognize partitions that are all ready created and manipulate those if they match?16:29
bmaltaissean-k-mooney: Yes, all nodes have ens160 and ens19216:29
bmaltais jamesbenson: I see: "ansible_default_ipv4": {             "address": "10.10.10.231",             "alias": "ens160",16:29
sean-k-mooneyis ens160 set as network_interface or neutron_external_interface16:31
*** stefan_bo has joined #openstack-kolla16:31
sean-k-mooneyi assume network_interface16:31
bmaltaissean-k-mooney: network_interface16:31
*** salv-orlando has joined #openstack-kolla16:31
bmaltaissean-k-mooney: In globals.yml: network_interface: "ens160"16:32
sean-k-mooneyso the error means that on whatever host it failed on it could not find ens160 and there for did not create teh corresponding host_vars16:32
sean-k-mooneycan you ssh into the failed host and run ifconfig ens160 to confirm16:33
bmaltaissean-k-mooney: I checked and it exist on both. This is how ansible contact each nodes16:34
sean-k-mooneybmaltais: what is the title of the failing task?16:34
*** manheim_ has joined #openstack-kolla16:35
bmaltaisTASK [baremetal : Generate /etc/hosts for all of the nodes]16:35
bmaltaisThe error appears to have been in '/root/kolla-ansible/ansible/roles/baremetal/tasks/pre-install.yml': line 1916:35
*** stefan_bo has quit IRC16:35
*** Mannheim has joined #openstack-kolla16:36
bmaltaissean-k-mooney: Full output of error: http://paste.openstack.org/show/610477/16:37
*** ducttap__ has quit IRC16:37
*** awebber has quit IRC16:38
*** chas has joined #openstack-kolla16:38
*** ducttape_ has joined #openstack-kolla16:38
*** manheim_ has quit IRC16:39
bmaltaissean-k-mooney: Should I have a host entry for each nodes on the deployment hosts?16:39
bmaltaissean-k-mooney: Right now I rely on DNS lookup to resolve16:39
bmaltaissean-k-mooney: Perhaps this is the issue?16:39
sean-k-mooneyyes it shoudl be either resolveable via dns or your have to have an entry in /etc/host of the deployment host or you need to define the connection info in the inventory file16:40
jamesbensonthe bootstrap will create the hosts on each machine as long as the deploy node can reach them16:40
bmaltaissean-k-mooney: No, same error even with hosts file entries16:40
sean-k-mooneyif you ping kolla1 or kolla1.ducourrier.com dose it work from the deployment host?16:40
bmaltaisYes. I can also ssh with no password in each as root16:40
egonzalezbmaltais, your deployer node also have the interface?16:41
sean-k-mooneyand "ssh kolla1 ifconfig ens160" show the nic info16:41
bmaltaisens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 150016:42
bmaltaisYep, plus a bunch of other lines I did not paste16:42
*** chas has quit IRC16:42
bmaltaisegonzalez: Yes, same with the exception of ens192 because it does not need it16:43
bmaltaisThis is the command I run: "./kolla-ansible bootstrap-servers -i ~/multinode"16:43
bmaltaisAnd the multinode top part of the file I modified: http://paste.openstack.org/show/610478/16:44
sean-k-mooneybmaltais: can you run something like ansible -i ~/multinode kolla1 -m debug -a "var=hostvars[inventory_hostname]"16:44
bmaltaisI get: [WARNING]: No hosts matched, nothing to do16:45
bmaltaisRunning "ansible -i ~/multinode kolla1 -m debug -a "var=hostvars[inventory_hostname]""16:45
openstackgerritSam Betts proposed openstack/kolla-kubernetes master: Make sure pathfinder can find passwords.yml in etc/kolla  https://review.openstack.org/46727616:45
bmaltaisBut running "ansible -i ~/multinode kolla1.ducourrier.com -m debug -a "var=hostvars[inventory_hostname]"" I get lot's of info16:46
bmaltaisDiff is hostname vs FQDN16:46
sean-k-mooneybmaltais: how do you have the host listed in the invenotry file? hostname or fqdn16:46
bmaltaisFQDN16:46
*** egonzalez has quit IRC16:47
sean-k-mooneycan you try swapping to hostname16:47
bmaltaissean-k-mooney: Sure16:47
sean-k-mooneyalso if you grep the output for the fqdn did it containte the ansible_ens... var16:47
bmaltaissean-k-mooney: Same error ;-(16:49
bmaltaissean-k-mooney: Grep returned no match for ansible_ens16:49
*** lpetrut has joined #openstack-kolla16:49
bmaltaissean-k-mooney: So it is not part of the returned info16:50
bmaltaissean-k-mooney: It used to bootstrap fine with the old kolla-ansible as server by pip install kolla-ansible16:53
sean-k-mooneyjust checked on one of my sytems and that info is not included either let me see if i can figure out how to dump all the info16:54
sean-k-mooneyoh you would need to also run gather facts before the debug command16:56
*** harlowja has joined #openstack-kolla16:56
bmaltaissean-k-mooney: Sorry for asking... but how do I gather facts?16:58
sean-k-mooneyansible -i ~/multinode kolla1 -m setup16:59
bmaltaisansible all -i ~/multinode -m setup --tree /tmp/facts ?16:59
*** chas has joined #openstack-kolla16:59
sean-k-mooneyansible -i ~/multinode kolla1 -m setup | grep ens160?16:59
rwellumHey sdake - I've noticed lately when I apply canal.yaml, my master node is 'automatically' untainted.16:59
bmaltaissean-k-mooney: ansible all -i ~/multinode -m setup --tree /tmp/facts | grep ansible_ens return a bunch of ansible_ens160 lines17:00
sean-k-mooneybmaltais: so ansible should be able to discover the values17:00
bmaltaissean-k-mooney: I guess... not sure why it does not17:01
bmaltaisI could do a webex if you want to poke around17:02
sean-k-mooneyi dont think im going to have time to debug much further if you have all the host in dns you could jsut turn off the hsot file generation17:03
sdakerwellum ok - not sure what to make of that :)17:03
*** salv-orlando has quit IRC17:03
sean-k-mooneyit is only need when not use dns17:03
*** chas has quit IRC17:03
*** cristicalin has quit IRC17:04
bmaltaissean-k-mooney: OK, I will try that and see. Should still figure out at some point.17:04
bmaltaissean-k-mooney: So just edit the "/root/kolla-ansible/ansible/roles/baremetal/tasks/pre-install.yml" then?17:05
sean-k-mooneybmaltais: yes when we wrote the bootstrap playbook it was only tested in enviornments wehre dns was not available17:05
sean-k-mooneybmaltais: just add customize_etc_hosts: False in the global.yml17:05
bmaltaisOK, Added customize_etc_hosts and now moving along17:07
bmaltaisThank's for the help17:07
sean-k-mooneyno worries. sorry its not going more smothly17:07
bmaltaisThat is part of the fun ;-)17:07
vhosakotbmaltais: I've seen ansible errors with hostname/fqdn inconsistencies too.  I just dump all the hostvars and check if ansible_hostname is what the task expects.17:09
*** cristicalin has joined #openstack-kolla17:10
sean-k-mooneyvhosakot: yes that might be the issue the task assumes ansible_hostname will match the value of running hostname on the server which will not return the fqdn17:11
vhosakotsean-k-mooney: yes, I have experienced that pain few times :)17:11
bmaltaissean-k-mooney: Let me know if I can provide anything to help with that17:11
bmaltaissean-k-mooney: I noticed another small issue now with kolla-ansible prechecks...17:12
sean-k-mooneybmaltais: well if the node called kolla1 does not output kolla1 when you run ssh kolla1 hostname i think it will fail17:12
bmaltaissean-k-mooney: It checks for a version of docker on the local deployment node (localhost) that does not have it install... resulting in a fail report17:13
bmaltaissean-k-mooney: It returns: kolla1.ducourrier.com17:13
sean-k-mooneyya that technically is not a host name17:14
sean-k-mooneyi think ansible will be expecting just kolla117:14
sean-k-mooneynot sure though17:14
*** feisky has quit IRC17:14
sean-k-mooneywhen you ran kolla-ansible prechecks did you add -1 ~/multinode e.g. kolla-ansible prechecks -i ~/multinode17:15
sean-k-mooneyif not it was running the prechecks on the localhost17:15
bmaltaissean-k-mooney: Yes, -i ~/multinode17:15
sean-k-mooneyas an FYI precheck are not expected to pass untill a you manually complete all the steps in the quickstart or you have successfully ran the bootstrap play17:16
vhosakothi17:16
bmaltaisIt just happen to contain an entry for [deployment] localhost       ansible_connection=local17:16
vhosakothaha, sorry, wrong window :)17:16
*** rmart04 has quit IRC17:16
bmaltaissean-k-mooney: As part of the git repo ansible/inventory example17:16
*** cristicalin has quit IRC17:17
*** jaosorior is now known as jaosorior_away17:17
bmaltaissean-k-mooney: Perhaps it is not required in a multinode setup?17:17
*** korzen has quit IRC17:17
bmaltaissean-k-mooney: Especially when the deployment host does not run any containers17:17
sean-k-mooneybmaltais: ya that is a new bug17:18
vhosakotbmaltais: FYI, I don't see any issue with 1024 ulimit size for open files too.17:18
sean-k-mooneybefore there was not a group for deployment17:18
sean-k-mooneyso when it listed all hosts it did not run on local host17:18
*** ducttape_ has quit IRC17:18
sean-k-mooneymany people use the deployment host as one of the controlers17:19
*** salv-orlando has joined #openstack-kolla17:19
bmaltaisvhosakot: Look good now when using the latest git repo code17:19
sean-k-mooneythe prechecks should probably skipp the deployment group17:19
vhosakotbmaltais: cool, cool :)17:19
bmaltaisvhosakot: Might have been all related to using the old version deployed by using pip install kolla-ansible17:19
*** chas has joined #openstack-kolla17:20
bmaltaissean-k-mooney: I agree... unless it verifies dependencies17:20
bmaltaisGot to go grab lunch, be back17:21
sean-k-mooneybmaltais: well the only dependecy the deployment host has is that ansible is installed17:21
sean-k-mooneybmaltais: if you can run the precheck you have met that dependecy implicitly17:21
sean-k-mooneybmaltais: enjoy17:22
vhosakotbmaltais: I don't see any issue with "pip install kolla-ansible". I've done many times with no issues.  Before re-insalling kolla, just need to do pip uninstall properly so that the next pip installation of kolla does not conflict with few packages like jina/pyctypto/etc..17:23
*** salv-orlando has quit IRC17:23
*** powerd has quit IRC17:24
vhosakotpycrypto*17:24
*** itlinux has joined #openstack-kolla17:24
*** chas has quit IRC17:24
rwellumsdake: yeah it doesn't hurt because you just get a warning there's no taint to untaint. But it is odd.17:33
bmaltaisrwellum: Correct. I was able to deploy successfully.17:34
rwellumAh did you see the same thing bmaltais ?17:34
bmaltaisrwellum: You mean the prechecks fail for the localhost deployment node?17:35
rwellumbmaltais: well this is kolla-kubernetes - sorry if I've cross-threaded here :)17:36
bmaltaisrwellum: My bad ;-) I play in both ansible and kubernetes camp ;-)17:37
*** tomatopeel has quit IRC17:38
*** tomatopeel has joined #openstack-kolla17:40
*** chas has joined #openstack-kolla17:41
rwellumnp bmaltais17:41
rwellumWell ahead of me there17:41
*** shardy_ has quit IRC17:42
*** lucasxu has joined #openstack-kolla17:43
*** chas has quit IRC17:45
*** jrich523 has quit IRC17:48
*** salv-orlando has joined #openstack-kolla17:51
*** manheim_ has joined #openstack-kolla17:53
*** hongbin has joined #openstack-kolla17:55
sean-k-mooneyare there any issues with the gate currently?17:58
sean-k-mooneyit looks like several jobs are failing on anything other then ubuntu17:59
*** jamesbenson has quit IRC17:59
vhosakotsean-k-mooney: which gate?  we have 3 :)   kolla, kolla-ansible, kolla-k8s?18:00
sean-k-mooneygood point kolla18:00
sean-k-mooneytrying to figure out why https://review.openstack.org/#/c/466132/ failed18:00
sean-k-mooneyin the gate config this code should not execute so it should not be able to cause a build failure18:01
*** chas has joined #openstack-kolla18:01
sean-k-mooneyvhosakot: well copy_dir will execute but on the first call dest is not a directory so it just calls shutil.copytree(src, dest) which is what the old code did18:02
*** itlinux has quit IRC18:02
vhosakotsean-k-mooney: I see gate is failing here -->  http://logs.openstack.org/32/466132/2/check/gate-kolla-dsvm-build-centos-binary-centos-7-nv/821cec6/console.html#_2017-05-23_16_31_43_794263.   I'd add debugs in tests/test_build.py in runTest before line #71.18:04
*** lpetrut has quit IRC18:05
*** chas has quit IRC18:06
*** srwilkers has quit IRC18:06
sean-k-mooneyvhosakot: hum ok18:06
sbezverksrwilkers: https://review.openstack.org/#/c/466536 please check when you have a chance18:06
vhosakotsean-k-mooney: looks like mariadb is not building in gate ? -->  http://logs.openstack.org/32/466132/2/check/gate-kolla-dsvm-build-centos-binary-centos-7-nv/821cec6/console.html#_2017-05-23_16_31_42_92675618:07
sean-k-mooneyvhosakot: ERROR:kolla.image.build.mariadb:Error'd with the following message18:08
sean-k-mooneyERROR:kolla.image.build.mariadb:The command '/bin/sh -c yum --setopt skip_missing_names_on_install=False -y install expect galera hostname percona-xtrabackup pv rsync socat tar MariaDB-client MariaDB-Galera-server && yum clean all' returned a non-zero code: 118:08
sean-k-mooneyvhosakot: so ya on centos the yum install is failing18:09
vhosakotsean-k-mooney: where do you see that error? :)18:09
sean-k-mooneyvhosakot: No package MariaDB-Galera-server available.18:09
sean-k-mooneyvhosakot: http://logs.openstack.org/32/466132/2/check/gate-kolla-dsvm-build-centos-binary-centos-7-nv/821cec6/logs/build/mariadb.txt.gz18:09
*** eaguilar has joined #openstack-kolla18:10
vhosakotsean-k-mooney: yes, looks like a mariadb issue in gate, unrelated to your patch.18:11
sean-k-mooneyvhosakot: so ya good news i didnt break the gate bad news gate is still broken18:11
chandankumarHello18:12
sean-k-mooneyvhosakot: i wonder if the package name changed or the repo is unavailable18:12
chandankumarDo we run tempest with kolla?18:12
vhosakotsean-k-mooney: yes, gate is busted, I'll see if I can debug more.  Jeffrey4l is our gate expert :)18:12
*** itlinux has joined #openstack-kolla18:12
*** ducttape_ has joined #openstack-kolla18:12
vhosakotsean-k-mooney: yes, you are right about the classification of good and bad news ;)18:13
vhosakotinc0: gate is down :)18:13
vhosakotsean-k-mooney: looks like just a matter of using the right (new) package name instead of MariaDB-Galera-server, right?18:13
sean-k-mooneyvhosakot: looking at http://yum.mariadb.org/10.0/centos7-amd64/rpms/ i dont see MariaDB-Galera-server18:14
sean-k-mooneyvhosakot: i think it just galara now?18:14
vhosakotsean-k-mooney: does gate pull from there?18:14
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Add instruction on shutting down and restarting complete kolla-ansible environment.  https://review.openstack.org/46688118:14
sean-k-mooneythat is the default in https://github.com/openstack/kolla/blob/master/docker/base/MariaDB.repo18:14
vhosakotah right.18:14
vhosakotsean-k-mooney: mind submitting a patch?  busy for next few hours, I'll +2 if gate fixes :)18:15
sean-k-mooneyill see if changing the name works but im not 100% shure it correct. i guess the gate will tell us18:16
vhosakotsean-k-mooney: yep18:16
*** rmart04 has joined #openstack-kolla18:17
*** hieulq_ has quit IRC18:19
*** powerd has joined #openstack-kolla18:20
*** rmart04 has quit IRC18:22
*** chas has joined #openstack-kolla18:23
*** rmart04 has joined #openstack-kolla18:23
*** powerd has quit IRC18:25
*** chas has quit IRC18:28
*** eaguilar has quit IRC18:33
*** itlinux has quit IRC18:34
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600718:34
*** jrich523 has joined #openstack-kolla18:34
*** krtaylor has quit IRC18:35
*** goni has joined #openstack-kolla18:38
goniHi. I am trying to access the openstack dashboard. My deployment was successful using kolla-ansible18:40
goniIs the URL to access dashboard http://<kolla-internal-vip-address>:80?18:41
*** tonanhngo has joined #openstack-kolla18:42
*** eaguilar has joined #openstack-kolla18:42
vhosakotgoni: yes, if you want to access from within/inside the cluster.18:42
*** itlinux has joined #openstack-kolla18:42
*** chas has joined #openstack-kolla18:43
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Grafana post_config checks only need to run locally  https://review.openstack.org/46381018:43
*** Pavo has quit IRC18:46
*** salv-orlando has quit IRC18:46
gonivhosakot: I get this error. http://paste.openstack.org/raw/610483/18:46
*** egonzalez has joined #openstack-kolla18:47
vhosakotgoni: can you curl <kolla-internal-vip-address>:80 on the controller?18:47
*** chas has quit IRC18:48
gonivhosakot I tried doing that but no response :/18:48
*** manheim_ has quit IRC18:48
vhosakotgoni: hmmm :)  can you see who owns port 80?      lsof -i :8018:49
*** tonanhngo has quit IRC18:49
sean-k-mooneymy guess from that error is that you will find a more detailed error in the apache error log18:49
gonivhosakot apache218:50
vhosakotgoni: right, I'd dig /var/log/httpd or var/log/apache218:50
gonilet me check18:51
*** JoseMello has quit IRC18:55
openstackgerritsean mooney proposed openstack/kolla master: fixed mariadb package name for yum distors  https://review.openstack.org/46736018:56
*** salv-orlando has joined #openstack-kolla18:56
sean-k-mooneyvhosakot: https://review.openstack.org/#/c/467360/ allows me to build locally but we should see what the gate says.18:57
vhosakotsean-k-mooney: perfect!18:57
sean-k-mooneyrunning  yum install MariaDB-Galera-server in the final container give me the same No package MariaDB-Galera-server available error we are seeing in the gate so it looks like we should just use MariaDB-server now18:58
sean-k-mooneyvhosakot: im going to go have dinner but hopefullly that will fix it. chat tommorow18:59
vhosakotsean-k-mooney: o/  night :)18:59
*** itlinux has quit IRC19:01
*** zhubingbing has quit IRC19:01
*** zhubingbing has joined #openstack-kolla19:02
*** salv-orlando has quit IRC19:02
*** lucasxu has quit IRC19:02
*** chas has joined #openstack-kolla19:04
gonivhosakot: var/log/apache2 did not exist. I re-installed apache2 and tried again. I can get to the error log now but doesn't seem to give much information http://paste.openstack.org/show/610484/19:08
*** srwilkers has joined #openstack-kolla19:09
*** chas has quit IRC19:09
*** salv-orlando has joined #openstack-kolla19:10
*** salv-orlando has quit IRC19:11
bmaltaisHi, I feel bad to ask this... seem like I always ask for help... I am trying to get ceph working following the guide https://github.com/openstack/kolla-ansible/blob/master/doc/ceph-guide.rst19:11
bmaltaisEverything looked good with the deploy until: TASK [nova : Pulling cinder cephx keyring for libvirt] ************************************************************************************************************************************************************************ fatal: [kolla1 -> kolla1]: FAILED! => {"changed": false, "cmd": ["docker", "exec", "ceph_mon", "ceph", "auth", "get-key", "client.cinder"], "delta": "0:00:00.19:11
bmaltais["Error ENOENT: don't have client.cinder"], "stdout": "", "stdout_lines": []}19:12
*** rmart04 has quit IRC19:12
bmaltaisAny pointer as to what might be causing this?19:12
*** cristicalin has joined #openstack-kolla19:15
*** lucasxu has joined #openstack-kolla19:15
rwellumsdake: sbezverk: filed a bug with canal: https://github.com/projectcalico/canal/issues/7719:20
*** sgrasley has quit IRC19:21
*** sgrasley has joined #openstack-kolla19:22
*** chas has joined #openstack-kolla19:25
*** scheuk has quit IRC19:27
*** krtaylor has joined #openstack-kolla19:28
*** chas has quit IRC19:29
jemcevoyI have a couple questions on host specific configs for Newton...19:34
jemcevoyIf I have a host that is a little different and has a nvidia K2 Grid card instead of an M60 Grid card like the rest of the systems19:34
jemcevoyWhat exactly do I name the nova.conf so it gets configures PCI passthrough for 10de:11bf rather then the M60's 10de:13f2 value19:34
jemcevoyAnd do I need to put all configs that are general with the nova.conf which need to be different?19:34
jemcevoyIf the hostname is etbc6-vmhost1 does the config go in /etc/kolla/configs/etbc-vmhost1/nova.conf19:35
vhosakotjemcevoy: here is a blog I wrote about deploying PCI pass-through in OpenStack.  I've specified the configs needed under the section "Below are the neutron and nova configurations needed to enable SR-IOV in OpenStack".  you might need to add the vendor-scpecific parts in those configs tho.19:36
vhosakotjemcevoy: https://communities.cisco.com/community/developer/openstack/blog/2017/01/13/how-to-deploy-sr-iov-pci-pass-through-in-openstack-using-ciscos-ucsm-ml2-neutron-plugin19:36
*** powerd has joined #openstack-kolla19:38
egonzalezvhosakot good info :)19:38
vhosakotegonzalez: thanks :)19:39
egonzalezjemcevoy the path  is /etc/kolla/config/nova/<inventory_hostname>/nova.conf19:39
jemcevoyvhosakot : I have the pci passthrough working well for the new M60 cards My question is on the kolla specifics config directory19:39
jemcevoyThanks egonzalez19:40
vhosakotjemcevoy: you might need appropriate configs in ml2_conf as well.  At least, I needed them in ml2_conf for SR-IOV.19:40
vhosakotjemcevoy: ah, then, add those card-specific/vendor-specific config bits in /etc/kolla/config/nova/<inventory_hostname>/nova.conf, what egonzalez said.19:41
bmaltaisTried a few things but still get an error when seting a basic ceph deployment.19:42
bmaltaisWhen running "docker exec ceph_mon ceph auth get-key client.cinder" I get Error ENOENT: don't have client.cinder19:42
jemcevoyvhosakot: I am passing through an nVidia GPU for high performance VDI used for CADD design of very large models... The VMs are 12 core 128GB big guys19:42
bmaltaisThe strange thing is that I can get a key for nove...19:42
bmaltaiss/nove/nova19:42
vhosakotjemcevoy: cool, high perf is well suited usecase for PCI pass-thru19:43
bmaltaisAnyone has seen this before?19:43
vhosakotbmaltais: haven't done much ceph ;)19:43
bmaltaisvhosakot: Thanks. This is my 1st attempt. Tought I crossed all T but I must be missing something19:45
vhosakotneutron is what I've done the most as everyone keeps saying "it's the network's fault" ;)19:45
*** chas has joined #openstack-kolla19:46
egonzalezbmaltais is cinder deployed?19:48
*** itlinux has joined #openstack-kolla19:48
*** sayantan_ has joined #openstack-kolla19:49
bmaltaisJust saw that globals.yml had cinder set to "no" as the default...19:49
bmaltaisI am changing that to yes and will try again ;-)19:50
*** chas has quit IRC19:50
vhosakotcinder is the front-end for ceph, so it will be needed19:52
*** bmaltais has quit IRC19:54
*** goni has quit IRC19:54
jemcevoyvhosakot... Cisco question for you... This cluster has a re-purposed 1Gig switch for the BMCs and admin access.  Ports on this switch keep dropping the link for an unknown reason... There are no vlans or anything special on this switch.  Is there a command to reset the switch to behave like an unmanaged switch?   The 10Gig side is all Cumulus and Cumulus configs makes sense to me.19:56
jemcevoyWe just need a dumb switch to connect to a  jump host19:57
vhosakotjemcevoy: "unmanaged switch".. does that ever exist?  I'd remove all configs, and reboot to go back to day-1 factory-defaults.19:58
*** AzazeL_ has quit IRC20:00
*** AzazeL_ has joined #openstack-kolla20:00
openstackgerritMerged openstack/kolla-kubernetes master: Fixing some issues with v4 gates  https://review.openstack.org/46653620:02
*** Mannheim has quit IRC20:05
*** chas has joined #openstack-kolla20:07
jemcevoyvhisakot :  where are the configs?  Can I just do an rm *.conf or something?   I have not touched a Cisco switch since before Y2K...20:07
*** sambetts is now known as sambetts|afk20:08
vhosakotjemcevoy: depends on the OS on the box :)  erase configs, clear configs on startup, reboot, go back to day-1.  do you know which box?  IOS, NXOS, XR, XE, CSR, ISR, virtual appliance?20:09
*** numans has quit IRC20:10
*** dave-mccowan has quit IRC20:10
*** chas has quit IRC20:11
*** numans has joined #openstack-kolla20:12
jemcevoyI do not know which box or NOS.. vhosakot... And the switch is in Sunnyvale and I'm on Portland... I'll have to get their networking team to look into it... But that can take a week or so...20:13
*** manheim_ has joined #openstack-kolla20:13
*** sayantan_ has quit IRC20:13
emccormickjemcevoy: show version will tell you what OS it is20:15
emccormickassuming you have enable access to the switch20:15
*** lucasxu has quit IRC20:17
emccormickand pre y2k commands generally still work. If it's IOS you can probably do "write erase" and then "reload", but it will truly go back to no config and you won't be able to connect to it again until someone sets an IP and password for it.20:17
vhosakot^^^20:17
emccormickIt could be that the switch hates your NICs and is set to auto detect speed / duplex. You may want to do a show interfaces to see what they are negotiated at20:18
emccormickAnd now for the reason I woke up. Does anyone know where I should go fish to solve this error during prechecks (kolla 3.0.3)?20:19
emccormickfatal: [compute2.cc.osdev.cirrusseven.net]: FAILED! => {"failed": true, "msg": "The conditional check ''169.254.' not in kolla_internal_vip_address and kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3]) is none' failed. The error was: error while evaluating conditional ('169.254.' not in kolla_internal_vip_address and kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3]) is none):20:19
emccormicklist object has no element 3"}20:19
*** sayantan_ has joined #openstack-kolla20:20
emccormickI set custom interfaces for compute2 like so: compute2.cc.osdev.cirrusseven.net network_interface=enp3s0f1 api_interface=enp3s0f1 tunnel_interface=enp3s0f020:20
emccormickand for everything else it seems to pick things up OK, but that one check fails every time20:21
*** lucasxu has joined #openstack-kolla20:21
*** AzazeL_ has quit IRC20:23
*** sayantan_ has quit IRC20:24
*** numans has quit IRC20:26
*** sayantan_ has joined #openstack-kolla20:26
*** chas has joined #openstack-kolla20:27
*** sayantan_ has quit IRC20:31
*** manheim_ has quit IRC20:32
*** numans has joined #openstack-kolla20:32
*** chas has quit IRC20:32
*** harlowja_ has joined #openstack-kolla20:32
*** harlowja has quit IRC20:33
*** mattmceuen has quit IRC20:34
*** manheim_ has joined #openstack-kolla20:39
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate  https://review.openstack.org/46600720:41
*** ducttape_ has quit IRC20:42
*** ducttap__ has joined #openstack-kolla20:42
*** manheim_ has quit IRC20:47
*** chas has joined #openstack-kolla20:48
*** ducttape_ has joined #openstack-kolla20:49
*** manheim_ has joined #openstack-kolla20:50
*** ducttap__ has quit IRC20:52
*** chas has quit IRC20:53
*** fguillot has quit IRC20:53
*** david-lyle has joined #openstack-kolla20:54
*** jtriley has quit IRC20:57
*** vhosakot_ has joined #openstack-kolla20:59
*** itlinux has quit IRC21:00
*** vhosakot has quit IRC21:00
*** lucasxu has quit IRC21:03
*** bmace__ has quit IRC21:04
*** harlowja_ has quit IRC21:05
*** bmace__ has joined #openstack-kolla21:05
*** jamesbenson has joined #openstack-kolla21:05
*** rhallisey has quit IRC21:06
*** harlowja has joined #openstack-kolla21:08
*** chas has joined #openstack-kolla21:09
vhosakot_sean-k-mooney: I see mariadb built successfully at gate, but failed to deploy at gate during bootstrap -->  http://logs.openstack.org/60/467360/1/check/gate-kolla-dsvm-deploy-centos-binary-centos-7-nv/1546f0b/console.html#_2017-05-23_19_29_16_88654421:10
vhosakot_I'll dig the logs and see what's going on.21:10
*** david-lyle has quit IRC21:10
*** chas has quit IRC21:14
*** eaguilar has quit IRC21:16
-openstackstatus- NOTICE: The logserver has filled up, so jobs are currently aborting with POST_FAILURE results; remediation is underway.21:21
*** ChanServ changes topic to "The logserver has filled up, so jobs are currently aborting with POST_FAILURE results; remediation is underway."21:21
*** david-lyle has joined #openstack-kolla21:22
*** chas has joined #openstack-kolla21:30
*** chas has quit IRC21:35
*** ducttap__ has joined #openstack-kolla21:39
*** krtaylor has quit IRC21:41
*** ducttape_ has quit IRC21:43
*** jroll has quit IRC21:47
*** jroll has joined #openstack-kolla21:47
*** jroll has quit IRC21:49
*** salv-orlando has joined #openstack-kolla21:50
*** chas has joined #openstack-kolla21:51
*** ducttap__ has quit IRC21:51
*** ducttape_ has joined #openstack-kolla21:52
*** jroll has joined #openstack-kolla21:53
*** chas has quit IRC21:56
*** vhosakot_ has quit IRC21:58
*** david-lyle has quit IRC22:01
*** jamesbenson has quit IRC22:05
*** srwilkers has quit IRC22:10
*** chas has joined #openstack-kolla22:12
*** chas has quit IRC22:16
*** jascott1 has joined #openstack-kolla22:18
*** rwellum has quit IRC22:19
*** jascott1 has quit IRC22:23
*** chas has joined #openstack-kolla22:33
*** peterjenkins__ has quit IRC22:33
*** stefan_bo has joined #openstack-kolla22:33
*** peterjenkins__ has joined #openstack-kolla22:34
*** karlamrhein has quit IRC22:34
*** bmaltais has joined #openstack-kolla22:34
*** NobodyCam has quit IRC22:35
*** NobodyCam has joined #openstack-kolla22:35
bmaltaisHi. I have a question regarding a good minimum ceph enable setup for testing. Well at least a minimum global.yml config for proper ceph support.22:36
bmaltaisI have deployed ceph with what I think is required for ceph but when I run init-runonce it is stuck at the glance upload phase22:36
bmaltaisAnd horizon show that the image is being saved...22:36
*** karlamrhein has joined #openstack-kolla22:36
bmaltaisWhat options in global.yml are you enabling for a basic ceph support?22:37
*** chas has quit IRC22:38
*** stefan_bo has quit IRC22:38
*** ducttap__ has joined #openstack-kolla22:38
bmaltaisHere is a paste bin of what I changed for my setup: http://paste.openstack.org/show/610497/22:39
bmaltaisDo you see anything missing or that I should add?22:39
*** ducttape_ has quit IRC22:41
openstackgerritAnthony Lin proposed openstack/kolla-kubernetes master: Script to Deploy Kolla Kubernetes AIO  https://review.openstack.org/46431222:42
*** sgrasley1 has joined #openstack-kolla22:43
*** egonzalez has quit IRC22:43
*** iniazi_ has quit IRC22:44
*** dave-mccowan has joined #openstack-kolla22:44
*** sgrasley has quit IRC22:44
*** iniazi_ has joined #openstack-kolla22:45
*** salv-orlando has quit IRC22:48
*** chas has joined #openstack-kolla22:54
*** manheim_ has quit IRC22:58
*** chas has quit IRC22:59
*** cristicalin has quit IRC23:02
*** athomas has quit IRC23:04
*** chas has joined #openstack-kolla23:15
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role  https://review.openstack.org/41636723:15
*** powerd has quit IRC23:16
*** chas has quit IRC23:20
*** rkrum has joined #openstack-kolla23:23
*** huzhengchuan has quit IRC23:23
*** huzhengchuan_ has quit IRC23:25
*** rkrum has quit IRC23:35
*** chas has joined #openstack-kolla23:36
*** chas has quit IRC23:40
bmaltaisAnyone on the channel running kolla-ansible with ceph storage?23:43
*** hongbin has quit IRC23:45
bmaltaisLooking for some tips on globals.yml to get a basic ceph storage working.23:45
bmaltaisCurrently I can deploy but loading glance image is stuck at saving... so something is broken with the storage.23:46
bmaltaisLooking for info on what you enabled in globals.yml to get going with a sound options selection.23:46
*** chas has joined #openstack-kolla23:56
*** salv-orlando has joined #openstack-kolla23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!