Wednesday, 2016-04-06

*** daneyon has quit IRC00:00
*** Marga_ has joined #openstack-kolla00:01
*** daneyon has joined #openstack-kolla00:01
*** Marga_ has quit IRC00:02
*** Marga_ has joined #openstack-kolla00:03
*** Marga_ has quit IRC00:03
*** Marga_ has joined #openstack-kolla00:03
sdake_http://buildlogs.centos.org/centos/7/virt/x86_64/kvm-common/centos-release-qemu-ev-1.0-1.el7.noarch.rpm00:04
*** weiyu has quit IRC00:10
dave-mccowansomething's gone wrong with my all-in-one kolla server.  running deploy gives me: msg: The requested image does not exist: kollaglue/centos-binary-heka:2.0.0, but the images show under "docker images".  i'm using a local registry and it is set insecure in my docker startup.  any ideas what i've missed?00:12
*** diogogmt has quit IRC00:14
*** mark-casey has quit IRC00:15
kklimondahave you set docker_registry in kolla's global.yml?00:17
kklimondaglobals.yml* even00:17
*** daneyon has quit IRC00:17
sdake_did you run genpwd?00:19
sdake_and did you set docker_registry in global.yml?00:19
sdake_and when you built did you use --registery locaol-server-ip:4000 ?00:19
*** andrewvwebber has joined #openstack-kolla00:20
sdake_dave-mccowan ^^00:20
dave-mccowanyes to reg set in global and --reg used on build. in passwords.yaml i have docker_registry_password: null.  (is that right for no password)00:21
sdake_sbezverk centos-release-ceph-hammer.noarch00:23
*** mwheckmann has joined #openstack-kolla00:25
*** andrewvwebber has quit IRC00:26
sbezverksdake here is the bug to track qemu version: https://bugs.launchpad.net/kolla/+bug/156658800:26
openstackLaunchpad bug 1566588 in kolla "integrate new qemu 2.3.0.X version into Kolla " [Undecided,New]00:26
*** sdake has joined #openstack-kolla00:32
*** sdake_ has quit IRC00:34
manjeetssdake_ not sure what i did not getting a single error like image not found for past 10 20 runs of creating vms and destroying them00:42
sdakehuh00:43
sdakedo you mean the problem you had disappared?00:43
manjeetsi just change max_scheduler00:43
manjeets= 1000:43
manjeetsmax_scheduler_attempts = 1000:43
sdakeya00:44
sdakethat variable definately matters00:44
manjeetsthats what i mean00:44
manjeetsi don't know why split brain is not happening00:45
manjeetsall my machine are getting one port and one ip00:45
manjeetstried it upto 200000:45
manjeetsbut I am not using horizon at all00:45
manjeetseverything through cli00:45
manjeetsand deleting 500 vms is like 11 mins00:46
*** Jeffrey4l has joined #openstack-kolla00:47
manjeetsnova delete vm1 vm2 ..................vm_n is fastest way so far i've seen00:47
*** dave-mcc_ has joined #openstack-kolla00:50
*** sdake_ has joined #openstack-kolla00:51
*** sdake has quit IRC00:51
*** dave-mccowan has quit IRC00:51
*** ccesario___ has quit IRC00:52
*** weiyu has joined #openstack-kolla00:54
*** iceyao has joined #openstack-kolla00:56
sdake_its not spit brain00:57
sdake_manjeets its a different error, if you read the bug log my analysis is in there00:58
*** rhallisey has quit IRC01:00
*** yuanying has quit IRC01:01
*** ccesario___ has joined #openstack-kolla01:01
*** daneyon has joined #openstack-kolla01:01
sdake_sbezverk nova compute builds for me01:02
*** yuanying has joined #openstack-kolla01:02
*** yuanying has quit IRC01:04
*** daneyon has quit IRC01:07
*** diogogmt has joined #openstack-kolla01:07
*** yuanying has joined #openstack-kolla01:13
sdake_https://bugs.launchpad.net/kolla/+bug/156659401:15
openstackLaunchpad bug 1566594 in kolla mitaka "enable hammer and qemu-ev properly" [Medium,Confirmed] - Assigned to Steven Dake (sdake)01:15
*** ccesario___ has quit IRC01:17
*** ccesario___ has joined #openstack-kolla01:17
*** ccesario___ has quit IRC01:24
*** unicell1 has quit IRC01:25
*** yuanying has quit IRC01:25
*** yuanying has joined #openstack-kolla01:28
manjeetssdake i just got fastest delete of nova vms01:30
manjeets500 vms got killed in 4 m01:30
*** yuanying has quit IRC01:30
manjeetsits something some times not freeing up resources01:30
*** salv-orlando has joined #openstack-kolla01:32
*** Jeffrey4l has quit IRC01:34
*** yuanying has joined #openstack-kolla01:39
*** Jeffrey4l has joined #openstack-kolla01:42
*** yuanying has quit IRC01:42
*** yuanying has joined #openstack-kolla01:43
*** weiyu has quit IRC01:47
*** salv-orlando has quit IRC01:47
sdake_manjeets what made deletes faster?01:50
sdake_any change you made?01:50
*** iceyao has quit IRC01:50
*** iceyao has joined #openstack-kolla01:51
*** vhosakot has joined #openstack-kolla01:53
*** tfukushima has joined #openstack-kolla01:54
*** weiyu_ has joined #openstack-kolla01:54
*** sdake has joined #openstack-kolla01:55
*** mwheckmann has quit IRC01:55
*** sdake_ has quit IRC01:56
*** yuanying has quit IRC02:02
kjellyHi review, what is the palce for putting unit test code for build.py? https://review.openstack.org/#/c/290423/02:03
patchbotkjelly: patch 290423 - kolla - add unit tests for build script02:03
kjellyHi reviewers, what is the palce for putting unit test code for build.py?02:04
*** yuanying has joined #openstack-kolla02:05
*** kjelly_ has joined #openstack-kolla02:23
*** banix has joined #openstack-kolla02:33
*** achanda has joined #openstack-kolla02:34
*** weiyu_ has quit IRC02:39
*** kjelly has quit IRC02:40
*** tobe has joined #openstack-kolla02:42
*** tobe has quit IRC02:42
*** ccesario___ has joined #openstack-kolla02:43
*** unicell has joined #openstack-kolla02:49
*** yuanying has quit IRC02:50
*** weiyu_ has joined #openstack-kolla02:53
Jeffrey4lkjelly_, I am OK put it in or out the kolla folder. But my point is there is only one place to put the UTs.02:54
kjelly_Jeffrey4l: Got it. send next patch as soon as possible.02:56
*** weiyu_ has quit IRC02:56
*** iceyao_ has joined #openstack-kolla02:57
*** weiyu_ has joined #openstack-kolla02:59
*** iceyao has quit IRC03:01
*** sdake_ has joined #openstack-kolla03:01
*** salv-orlando has joined #openstack-kolla03:03
Jeffrey4lthanks.03:03
*** sdake has quit IRC03:04
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Add  iscsid/tgtd specific parameters to Kolla infrastructure  https://review.openstack.org/30000403:04
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Adds ansible code for iscsid and tgtd containers  https://review.openstack.org/29999503:04
*** salv-orlando has quit IRC03:13
*** weiyu_ has quit IRC03:21
kklimondahmm, it seems like accessing variables via hostvars doesn't take globals.yml into account? i.e. {{ api_interface }} and {{ hostvars[ansible_hostname]['api_interface'] }} don't match. Am I missing something?  Ansible 1.9.503:22
*** weiyu has joined #openstack-kolla03:22
*** achanda has quit IRC03:27
*** banix has quit IRC03:33
*** coolsvap has joined #openstack-kolla03:35
coolsvapJeffrey4l: hi regarding https://review.openstack.org/#/c/30095503:38
coolsvapis your selinux enforcing?03:38
Jeffrey4lcoolsvap, yes03:39
coolsvapits not working for me on centos03:39
coolsvapare you checking on centos?03:39
Jeffrey4lcoolsvap, yep.03:40
Jeffrey4lcoolsvap, see this http://paste.openstack.org/show/493098/03:40
Jeffrey4li really deployed the kolla successfully.03:40
coolsvapbecause i tried the /etc/hosts change but it was not working for me unless i have selinux!=enforcing i will check again03:41
sdake_coolsvap you may need to relabel03:46
*** yuanying has joined #openstack-kolla03:46
coolsvapsdake_: yes, i am trying the same now03:51
Jeffrey4lcoolsvap, kolla do not work with selinux for sure, https://github.com/openstack/kolla/blob/master/doc/selinux.rst. But it not cause your issue03:53
coolsvapJeffrey4l: I will double check, I currently do not have access to the system i tested earlier, i am setting up new currently03:54
coolsvapi will check on that system again tonight03:54
Jeffrey4lroger03:54
kklimondahmm, what is a correct ansible version to deploy kolla master? docs say it's <2.0 but commit 903c7d4c seems to be 2.0 only, as it doesn't work with ansible 1.9.503:55
kklimonda(well, ansible runs without error, but this way of accessing api_interface produces wrong result in 1.9.5)03:56
* sdake_ GROANS03:57
sdake_hardpin to 1.9.4 or 1.9.5> < 2.0.0 i guess03:58
sdake_kklimonda atm kola is tested extensivly only on ansible 1.9.403:59
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Install proper security libs  https://review.openstack.org/29560204:03
kklimondasdake_: installing 1.9.4 didn't change anything, I'm still seeing same result: https://dpaste.de/OBCy04:10
sdake_kklimonda can you show deploy output pls04:12
sdake_i have no idea what your showign in that paste04:12
kklimondasure04:12
sdake_also which version are you running04:12
sdake_master, mitaka, lierty?04:12
kklimondamaster04:12
*** weiyu has quit IRC04:16
kklimondaok, so deploy output doesn't really show much - there are no errors, or warnings but haproxy.cfg has wrong IPs for mariadb backend servers (and most likely others too).04:17
*** weiyu has joined #openstack-kolla04:18
kklimondahttp://paste.debian.net/424617/ - this is output from two extra debug tasks I've added to roles/haproxy/tasks/configure.yml04:20
sdake_dave-mcc_ ^6^04:20
sdake_dave is working on that problem04:20
*** weiyu has quit IRC04:21
dave-mcc_sdake_ that looks different than what i'm looking at.04:21
dave-mcc_mine is specific to cinder and glance.  not sure what would cause mariadb to have the wrong address.04:21
sdake_kklimonda show me your /etc/host file on the nodes04:22
sdake_control nodes04:22
sdake_paying 1 usd to someone to document how /etc/host should be configured in qsg04:25
dave-mcc_i remember someone saying that all the XX_interface options may not work.   kklimonda is there a reason you're customizing api_interface?  i don't think that is common.04:25
kklimondasdake_: it's pretty empty, it has 127.0.1.1 node001.[fqdn] node001 on each node, other nodes can resolve node001.[fqdn] with dns server's help (both fqdn and hostname, search is set in resolv.conf)04:28
sdake_127.0.1.1 should not resolve to node000104:30
kklimondadave-mcc_: I want mariadb (and other services) to be listening on a "internal IP", that isn't routable to our network04:30
sdake_it should reolve to the real ip address of the machine04:30
sdake_127.0.1.1 is a made up address to make gnome work properly04:30
sdake_for ubuntu machines04:30
sdake_bad hack imo04:30
sdake_but it is what it is ;)04:30
*** jasonsb has joined #openstack-kolla04:30
*** salv-orlando has joined #openstack-kolla04:31
sdake_my ip ddress of the machine, i mean the ip managemenet ip address04:32
kklimondayeah, I agree - I don't have that in production, but this is pretty barebone installation, and fixing it doesn't seem to make a difference anyway04:32
sdake_it should have all hsots in it04:32
sdake_my/by04:33
*** diogogmt has quit IRC04:33
sdake_it will break how mariadb works04:33
sdake_and all the addresses it gets04:33
sdake_it may not be your exact problem04:33
kklimondahow will it break given that my dns resolves everything (both host->ip and ip->host) anyway, and galera uses IP addresses?04:35
kklimondabut that doesn't really seem to be an issue right now anyway04:35
dave-mcc_kklomonda i agree that is important.  the way i do that is to set kolla_internal_vip_address to a non-routable.  this will keep internal URLs to opentack, mariadb, and rabbit all internal-only.  then set kolla_external_vip_address to your routable and set kolla_external_vip_interface to be the outside i/f, leaving network_interface to be internal only.04:35
kklimondammm, sounds like that would work04:37
sdake_dave++04:37
*** daneyon has joined #openstack-kolla04:38
kklimondado you also set network_interface to your internal interface then?04:38
kklimondaok04:38
kklimondaI should get some sleep04:39
kklimondaI could swear you didn't write that04:39
*** salv-orlando has quit IRC04:39
kklimondayeah, with that change I get haproxy.cfg I want, thanks04:41
sdake_thanks dave-mcc_04:41
dave-mcc_yes, with two vips network_inteface is the internal one.04:41
sdake_kklimonda mind running it throug hthe 1.9.5 grinder when you wrap up04:41
sdake_also you will need a matching /etc/host file on th control nodess04:42
sdake_without 127.0.1.1 ;)04:42
openstackgerritVikram Hosakote proposed openstack/kolla: Add prechecks to validate kolla_internal_vip_address and kolla_external_vip_address  https://review.openstack.org/30202604:42
*** daneyon has quit IRC04:42
vhosakotsdake_: ^^^ I've further hardened prechecks04:43
sdake_nice04:43
sdake_fix bugsp lz :)04:44
sdake_we got a slew of em04:44
vhosakotI've added the following three additional prechecks04:44
vhosakotChecking if kolla_internal_vip_address is assigned to any interface on deploy node04:44
vhosakotChecking if kolla_internal_vip_address is in the same network as network_interface04:44
vhosakotChecking if kolla_external_vip_address is assigned to any interface on deploy node04:44
sdake_how do yu odetermine same network for #204:45
sdake_ /8, /16, /24?04:45
vhosakotansible has a builtit module to check if an IP address belong to a range04:45
sdake_which range are you specifying04:45
sdake_i think we dont know that range ahead of time04:46
vhosakotrange of network_interface04:46
vhosakotwe can find in code04:46
sdake_ok if it works wfm ;)04:46
sdake_i'e added 2.3.0 qemu for those that care04:46
sdake_which is sbezverk atm ;)04:46
vhosakotyep, I tested 9 combinations with the PS...04:46
vhosakotsdake_: nice!04:46
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Enable the kibana access from the external network  https://review.openstack.org/30202704:46
vhosakotI will bug the next bug now04:47
sdake_vhosakot which version of qemu is used in ubuntu containers?04:48
vhosakotlet me check...04:48
sdake_my intenet is working again04:48
sdake_at night04:48
sdake_when its cool04:48
sdake_i hoep its not a heat problem04:48
sdake_Jeffrey4l do yo hae a ceph setup available?04:52
vhosakotsdake_: how can I check kvm version... is it tied to kernel version (uname -r).. I am using Ubuntu kernel 4.2.0-30-generic LTS wily04:52
sdake_dpkg -l | grep qemu04:52
sdake_qemu-kvm is the centos package name04:52
vhosakotah package... 1 sec04:52
sdake_not sure what its called in ubuntian04:52
vhosakotsdake_: "dpkg -l | grep qemu"  worked... It shows   qemu  1:2.5+dfsg-5ubuntu4~cloud004:54
vhosakot1:2.504:54
sdake_grep for kvm04:54
sdake_ca nyou check wha the latest upstream versoin is?04:55
vhosakotnothing.. I dont see anything when I grep kvm04:55
vhosakotsdake_: http://wiki.qemu.org/Download  says 2.6.0-rc104:56
sdake_nice thanks04:56
sdake_looks like centos is behind04:56
sdake_but 2.3 better then 1.5.3 + 1000 patches or more04:56
vhosakotsdake_: I see kvm in kernerl module (lsmod)05:00
vhosakot$ lsmod | grep kvm05:00
vhosakotkvm_intel             163840  305:00
vhosakotkvm                   507904  1 kvm_intel05:00
*** achanda has joined #openstack-kolla05:00
sdake_vhosakot i'm good with the info thanks05:00
vhosakotcool05:00
*** vhosakot has quit IRC05:07
*** tfukushima has quit IRC05:10
*** weiyu has joined #openstack-kolla05:15
coolsvapcinder is not building in gate05:22
*** weiyu has quit IRC05:30
*** sekrit has quit IRC05:30
*** Marga_ has quit IRC05:34
*** iceyao has joined #openstack-kolla05:34
*** Marga_ has joined #openstack-kolla05:37
*** iceyao_ has quit IRC05:38
*** Marga_ has quit IRC05:38
*** Marga_ has joined #openstack-kolla05:38
*** sdake has joined #openstack-kolla05:40
*** sdake_ has quit IRC05:42
*** Marga_ has quit IRC05:43
*** salv-orlando has joined #openstack-kolla05:46
*** sdake_ has joined #openstack-kolla05:49
*** Marga_ has joined #openstack-kolla05:52
*** sdake has quit IRC05:52
*** salv-orlando has quit IRC05:55
*** iceyao has quit IRC05:55
*** iceyao has joined #openstack-kolla05:56
*** Marga_ has quit IRC05:56
*** dave-mcc_ has quit IRC05:56
openstackgerritSteven Dake proposed openstack/kolla: Increase max scheduling attempts to 10  https://review.openstack.org/30091405:58
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203605:58
*** Marga_ has joined #openstack-kolla06:01
*** reddy has quit IRC06:08
*** reddy has joined #openstack-kolla06:09
*** mikelk has joined #openstack-kolla06:13
*** mdnadeem has quit IRC06:14
*** weiyu_ has joined #openstack-kolla06:21
*** sekrit has joined #openstack-kolla06:23
*** tfukushima has joined #openstack-kolla06:24
*** daneyon has joined #openstack-kolla06:26
*** andrewvwebber has joined #openstack-kolla06:27
*** kjelly_ has quit IRC06:28
*** coolsvap has quit IRC06:28
*** daneyon has quit IRC06:31
*** andrewvwebber has quit IRC06:31
*** dave-mcc_ has joined #openstack-kolla06:33
*** coolsvap has joined #openstack-kolla06:41
*** Kennan has joined #openstack-kolla06:45
*** achanda has quit IRC06:47
*** dave-mccowan has joined #openstack-kolla06:49
Jeffrey4lsdake_, you means deploy ceph using kolla?06:51
*** achanda has joined #openstack-kolla06:53
Jeffrey4lsdake_, the answer is yes.06:53
sdake_huh06:53
sdake_oh ya i want to try out this 23. patch when i finish getting it working with ceph06:54
sdake_but i dont have a ceph setup06:54
*** dave-mcc_ has quit IRC06:54
Jeffrey4lsdake_, so what can i do to help?06:55
sdake_nothign yet06:56
sdake_but fix bugs :)06:56
sdake_but when i get it  worign locally i'll let you know06:56
sdake_then try a deploy with ceph06:56
Jeffrey4lsdake_, cool. btw, is there any bugs exist in ceph?06:56
sdake_no clue06:57
Jeffrey4lroger06:57
sdake_you might ramp up by trying out ceph06:57
sdake_i should have patch ready by tomorrow06:57
sdake_dealing with permissions issues on /var/lib/noa06:57
Jeffrey4lcool06:58
sdake_just added qemu.conf to our templating06:58
*** achanda has quit IRC07:00
sdake_do you know wher enamed volume sget their default permisions from?07:00
Jeffrey4l"get their defautl permission" what this? isnt the defautl permissions root?07:01
*** athomas has quit IRC07:02
sdake_i dont know07:02
*** athomas has joined #openstack-kolla07:02
sdake_i'd think docker would be smart enough to see the directory is owned by nova:nova and mount it as such07:02
*** openstackgerrit has quit IRC07:02
*** openstackgerrit has joined #openstack-kolla07:03
sdake_did you look at that patch?07:03
Jeffrey4lsdake_, i do not know so. how can he know it should be owned by nova?07:03
Jeffrey4lnova may be easy, change the owner to the container running user.07:03
sdake_let em push mylaet version in a moment07:03
sdake_oh but your on ubuntu07:04
Jeffrey4lBut how to handle the horizon? the apache running at root, however, horizon is runing under horizon user.07:04
*** ccesario___ has quit IRC07:04
*** ccesario___ has joined #openstack-kolla07:04
sdake_horizon has nothing to do with libvirt ;)07:05
*** Marga_ has quit IRC07:05
Jeffrey4lsdake_, which patch?07:05
sdake_moment i'll push a new version07:05
Jeffrey4lcool.07:05
*** Jeffrey4l has left #openstack-kolla07:06
*** Jeffrey4l has joined #openstack-kolla07:06
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203607:08
sdake_Jeffrey4l ^^07:09
Jeffrey4lroger07:10
sdake_not read yto go yet07:10
Jeffrey4lgot.07:11
sdake_but it changes ceph versions slightlhy07:11
sdake_to oficially built images07:11
*** salv-orlando has joined #openstack-kolla07:12
sdake_if you want to work on  acritical bug right now07:12
sdake_i hear linuxbridge is busted07:13
sdake_that is 50% of our deployment s07:13
Jeffrey4lsdake_, is there any bug filed on the launchpad?07:14
sdake_yes i think but not certian07:14
sdake_if not file a new one07:14
sdake_and i'l sort ou the dups07:14
Jeffrey4lsdake_, ok. I will try to deploy a AIO env to seem what happen07:15
Jeffrey4lsdake_, i am working on the rax-iad node issue. It is really annoying. I think I find the root cause.07:17
sdake_nice07:21
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203607:23
*** gfidente has joined #openstack-kolla07:23
*** mgoddard has joined #openstack-kolla07:25
*** andrewvwebber has joined #openstack-kolla07:28
*** jmccarthy1 has joined #openstack-kolla07:32
*** jmccarthy1 has left #openstack-kolla07:32
*** andrewvwebber has quit IRC07:32
openstackgerritJeffrey Zhang proposed openstack/kolla: Handle multi Ethernets case in the gate  https://review.openstack.org/29023807:40
*** jmccarthy has joined #openstack-kolla07:41
*** unicell has quit IRC07:47
*** andrewvwebber has joined #openstack-kolla07:50
*** shardy has joined #openstack-kolla07:51
*** shardy has quit IRC07:52
*** mbound has joined #openstack-kolla07:52
*** ccesario___ has quit IRC07:52
*** shardy has joined #openstack-kolla07:53
*** ccesario___ has joined #openstack-kolla07:53
*** mgoddard_ has joined #openstack-kolla07:59
*** achanda has joined #openstack-kolla08:00
*** mgoddard has quit IRC08:02
*** achanda has quit IRC08:05
*** brad[] has quit IRC08:26
*** ccesario___ has quit IRC08:26
*** brad[] has joined #openstack-kolla08:27
*** pmisiak has joined #openstack-kolla08:39
openstackgerritMerged openstack/kolla: Log the SHA of the Git ref for source builds  https://review.openstack.org/30179308:40
Jeffrey4lsdake_, ping08:41
sdake_sup08:42
sdake_i just got my patch working08:42
*** dave-mccowan has quit IRC08:43
*** salv-orlando has quit IRC08:45
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203608:45
sdake_Jeffrey4l sup dude08:45
Jeffrey4lsdake_, how could we handle the liberty branch now? I saw many backport from master to liberty.08:46
Jeffrey4lsdake_, shouldn't we wait for the `deploy liberty using mitaka`?08:47
sdake_wait for what08:47
openstackgerritMerged openstack/kolla: Add nova_ssh in cleanup containers script  https://review.openstack.org/30164808:47
Jeffrey4lsdake_, we will drop the liberty branch and use mitaka branch to deploy liberty code. is this right?08:47
sdake_we will copy mitaka into liberty branch08:48
sdake_make liberty deploy liberty instead of mitaka08:48
sdake_not a great way to handle a stable branch08:48
sdake_i'd liek o nver do that again08:49
Jeffrey4lsdake_, so we should stop backport code to the liberty branch, right? we need wait for the merge of `copy mitaka into liberty branch`08:49
sdake_right08:49
Jeffrey4lsdake_, I know the root cause( yea, this is a not good idea.)08:49
sdake_yes we should ignore liberty for the moemnt08:50
Jeffrey4lsdake_, But I saw may backport from master to liberty. may be we need make some annoucement for this now.08:50
Jeffrey4ls/may/many/08:51
sdake_master to mitaka08:51
sdake_not to liberty08:51
sdake_if they are in teh queue comment on them08:51
*** pbourke has quit IRC08:51
Jeffrey4lsdake_, https://review.openstack.org/30179208:51
Jeffrey4llike this.08:51
Jeffrey4lgot it.08:51
*** pbourke has joined #openstack-kolla08:51
Jeffrey4lI have make comment for serveral PS.08:51
sdake_i see08:52
openstackgerritMichal Rostecki proposed openstack/kolla: Output image statuses to log instead of return value of main function  https://review.openstack.org/27885308:53
sdake_Jeffrey4l can you test my patch please08:53
sdake_https://review.openstack.org/30203608:53
sdake_you have ubuntu correct?08:53
Jeffrey4lsdake_, no. I only test the centos+source in my env. (A aio env)08:54
sdake_i mean your 10 node cluster?08:54
sdake_is centos?08:54
Jeffrey4lsdake_, right.08:54
sdake_cool can you give it a spin08:54
sdake_i believe yu ony hae to reuldi the *nova* containers08:55
Jeffrey4lI can not clean that env to do this test. that's a test env, some custom is evaluating it. :(08:56
Jeffrey4lsdake_, ^08:56
sdake_i see08:56
Jeffrey4lbut, I can make a AIO test.08:56
sdake_fiar enough08:56
sdake_i need an uuntu test is what i really need - centoss aio works for me08:56
sdake_but if you repicate that would make me feel better about it08:57
sdake_its a significant last minute change08:57
sdake_not sure how I feel aobut it for mitaka08:57
Jeffrey4lsdake_, you mean the OS of the host machina is Ubuntu rather than using ubuntu+source install?08:57
sdake_both08:58
sdake_but if your doing centos + centos/source thats ok too08:58
sdake_i'll ask vikram o test ubuntu in the morning08:58
pmisiakHi guys, can I use Ansible 2.0 with Kolla?08:58
sdake_pmisiak no08:58
Jeffrey4lsdake_, I am cuoristy why we need that change? any issue it raise? especially in the qemu group part.08:58
sdake_1.9.4 is what we run with mostly08:58
Jeffrey4lpmisiak, no.08:58
pmisiakok, thanks :)08:59
sdake_Jeffrey4l we use qemu 1.5 currently08:59
sdake_this goes to 2.3.008:59
Jeffrey4lsdake_, I know the qemu version part. I have no idea about that why we changed the qemu group.09:00
Jeffrey4lsdake_, I make a comment in you last PS.09:00
sdake_which part of the patch09:00
*** weiyu_ has quit IRC09:01
Jeffrey4lsdake_, https://review.openstack.org/#/c/302036/3/ansible/roles/nova/templates/qemu.conf.j209:01
patchbotJeffrey4l: patch 302036 - kolla - Use proper CentOS repositories for ceph and qemu09:01
sdake_we change to group kolla09:02
sdake_so /var/lib/nova/* can be written to09:02
sdake_by qemu which runs as user qemu:kolla09:02
Jeffrey4lsdake_, if not, it will raise any issue?09:02
*** weiyu_ has joined #openstack-kolla09:02
sdake_ya permission error starting vm09:03
sdake_otherwise libvirt tries to write to /var/lib/nova as the non-nova user09:03
sdake_previously qemu ran as root09:03
Jeffrey4lsdake_, but I never experience that09:03
sdake_rather libvirt09:04
sdake_and tarted instances as root09:04
sdake_so it alway s had permissions to write everywhere09:04
sdake_we could change the defualts to root;root in qemu.conf09:04
sdake_then the permission changes arenot needed09:04
sdake_but upstream docs recommend agains tthis09:05
sdake_the defaults changed fro mqemu 1.5 to 2.309:05
*** Marga_ has joined #openstack-kolla09:05
sdake_in 1.5 root:root09:05
sdake_in 2.3 qemu;qemu09:05
Jeffrey4lGot. thanks for explaination.09:05
Jeffrey4lso ubuntu test is really neccessary.09:06
sdake_right09:06
sdake_and ceph since that ws requierd to be changed  to work with new kvm09:06
Jeffrey4lah, a little big change.  :D09:07
sdake_same version different build09:07
sdake_but ya different nonetheless09:07
sdake_if i would have known about this 6 months ago we woudl have changed it  then09:08
sdake_sbezverk todl me about it at 4;3009:08
sdake_i've been working on it for 8 hours09:08
*** Marga_ has quit IRC09:10
Jeffrey4lin my 10 nodes env(centos + source), i checked the qemu process user, it is qemu not root.09:10
Jeffrey4lsdake_, ^09:11
sdake_ok09:11
Jeffrey4levent though in the qemu.conf, the user and group is root with comment out.09:11
sdake_well it doesn't work with that09:11
sdake_i tried that first obviously :)09:11
*** ccesario_ has joined #openstack-kolla09:12
Jeffrey4lOK. anyway, i will try to deploy you PS first. thanks.09:12
*** ccesario_ has quit IRC09:12
*** coolsvap has quit IRC09:13
*** salv-orlando has joined #openstack-kolla09:15
sdake_looks like it is not working09:15
sdake_i cleaned it up efore i submitted it09:15
sdake_and must have removed something crucial09:15
Jeffrey4lok. cool.09:15
sdake_why dont you try it09:16
*** salv-orlando has quit IRC09:19
Jeffrey4lsdake_, try your PS? I will try that.09:20
*** wuhg has joined #openstack-kolla09:20
*** tfukushima has quit IRC09:28
*** mgoddard_ has quit IRC09:31
*** achanda has joined #openstack-kolla09:33
*** andrewvwebber has quit IRC09:35
*** mgoddard has joined #openstack-kolla09:37
*** achanda has quit IRC09:38
sdake_might be too big for mitaka09:39
sdake_i'll have to think about it and consult with the team09:39
sdake_i am outof steam09:39
sdake_bedtime09:39
sdake_night Jeffrey4l09:40
sdake_report in patchset if yo uwouldn;t; mind your results09:40
sdake_btw mistral isnt building09:40
* sdake_ groans09:40
Jeffrey4lsdake_, good night09:43
sdake_i knwo why the patch wasnt working in my environment09:50
sdake_i yum installed libvirt to check config otpoins09:50
sdake_which started libvirt... lol09:50
openstackgerritMichal Rostecki proposed openstack/kolla: Output image statuses to log instead of return value of main function  https://review.openstack.org/27885309:52
*** b_bezak has joined #openstack-kolla10:02
*** salv-orlando has joined #openstack-kolla10:02
sdake_nihgt all10:02
*** sdake_ has quit IRC10:02
*** salv-orlando has quit IRC10:03
*** salv-orlando has joined #openstack-kolla10:03
*** salv-orlando has quit IRC10:18
pbourkeJeffrey4l: didn't know the same problem was happening on centos10:21
pbourkewrt rabbitmq10:21
*** weiyu_ has quit IRC10:21
pbourkeit fails locally also for me, but with a different startup error10:22
*** salv-orlando has joined #openstack-kolla10:26
*** salv-orlando has quit IRC10:30
*** weiyu_ has joined #openstack-kolla10:32
*** achanda has joined #openstack-kolla10:35
*** achanda has quit IRC10:41
*** salv-orlando has joined #openstack-kolla10:47
*** salv-orlando has quit IRC10:48
*** ccesario_ has joined #openstack-kolla10:52
*** salv-orlando has joined #openstack-kolla10:57
*** salv-orlando has joined #openstack-kolla10:57
*** daneyon has joined #openstack-kolla10:57
*** salv-orlando has quit IRC10:57
*** salv-orlando has joined #openstack-kolla10:58
*** salv-orlando has quit IRC10:58
*** rhallisey has joined #openstack-kolla11:01
*** ccesario___ has joined #openstack-kolla11:01
*** daneyon has quit IRC11:01
*** ccesario_ has quit IRC11:05
*** weiyu_ has quit IRC11:13
*** Marga_ has joined #openstack-kolla11:15
*** Marga_ has quit IRC11:16
*** Marga_ has joined #openstack-kolla11:16
*** ccesario___ has quit IRC11:20
*** Serlex has joined #openstack-kolla11:24
*** reddy has quit IRC11:25
*** ccesario has quit IRC11:30
*** ccesario___ has joined #openstack-kolla11:30
*** reddy has joined #openstack-kolla11:37
*** Marga_ has quit IRC12:07
*** Serlex has quit IRC12:08
*** Jeffrey4l has quit IRC12:09
*** Marga_ has joined #openstack-kolla12:09
*** salv-orlando has joined #openstack-kolla12:10
*** reddy has quit IRC12:15
*** dwalsh has joined #openstack-kolla12:15
sbezverkmandre ping12:17
sbezverkgood morning12:18
*** iceyao has quit IRC12:18
*** ccesario___ has quit IRC12:18
*** SiRiuS has joined #openstack-kolla12:18
*** wuhg has quit IRC12:19
*** ccesario___ has joined #openstack-kolla12:25
*** salv-orlando has quit IRC12:26
*** Liuqing has joined #openstack-kolla12:27
*** unicell has joined #openstack-kolla12:27
*** ccesario___ has quit IRC12:35
*** Marga_ has quit IRC12:36
*** Liuqing has quit IRC12:37
*** Liuqing has joined #openstack-kolla12:38
*** achanda has joined #openstack-kolla12:38
*** mwheckmann has joined #openstack-kolla12:39
*** Liuqing has quit IRC12:42
*** Liuqing has joined #openstack-kolla12:43
*** achanda has quit IRC12:43
*** Liuqing has quit IRC12:45
*** Liuqing has joined #openstack-kolla12:49
*** reddy has joined #openstack-kolla12:55
*** Serlex has joined #openstack-kolla12:55
*** ccesario has joined #openstack-kolla12:58
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: [WIP] Add Heka support  https://review.openstack.org/28630113:00
*** banix has joined #openstack-kolla13:04
*** salv-orlando has joined #openstack-kolla13:09
*** sdake has joined #openstack-kolla13:15
sdakemorning13:15
sbezverkgood morning13:15
sbezverksdake I really like your qemu patch.. It fixed one issue I have been having.13:16
sbezverkCannot access storage file '/var/lib/nova/instances/7aa33696-389a-4a90-a3b0-67591e94cb33/disk' (as uid:107, gid:107): Permission denied13:16
sbezverksince you changed group to kolla for user qemu, it works much better.13:17
*** salv-orlando has quit IRC13:17
*** sdake_ has joined #openstack-kolla13:19
*** sdake has quit IRC13:19
*** ccesario has quit IRC13:22
*** sdake_ has quit IRC13:24
*** Liuqing has quit IRC13:26
*** sdake has joined #openstack-kolla13:27
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Add LVM2/iSCSI documentation  https://review.openstack.org/30082413:29
*** Liuqing has joined #openstack-kolla13:33
mandrehey sbezverk13:33
sbezverkmandre thank you for pinging me back13:34
sbezverkI saw your comment and wanted to explain the situation when this option is required.13:34
mandreyeah, I'm slowly catching up on what was merged recently and stumbled upon your commit13:35
sbezverkmandre when my repository contains multiple versions, it is the only way for me to tell kolla deploy which version to deploy.. I could not find any other way.13:35
mandreright, so nothing actually prevented you to add it in your local globals.yaml before13:37
mandreyour change, if I remember correctly is to add back a commented out parameter, correct?13:38
mandreit's documentation only, and considered to be only for advanced users13:38
sbezverkmandre correct13:38
mandreif you read the comments on the patchset I linked in your issue, you'll see the reasons why it was removed13:39
*** daneyon has joined #openstack-kolla13:39
*** Jeffrey4l has joined #openstack-kolla13:42
*** diogogmt has joined #openstack-kolla13:42
*** jtriley has joined #openstack-kolla13:43
*** iceyao has joined #openstack-kolla13:43
*** ccesario___ has joined #openstack-kolla13:44
*** daneyon has quit IRC13:44
sbezverkmandre yep, I saw that but when I chatted with sdake actually it was him who suggested to restore it..13:45
sbezverkand I agree :-)13:45
*** ccesario has joined #openstack-kolla13:45
mandreok, I wanted to make sure you knew about the prior discussion13:46
sbezverkmandre sure, thank for brining it up.13:46
*** mlima_ has joined #openstack-kolla13:47
*** mlima has quit IRC13:50
sdakehuh13:51
sdakemandre fwiw i think it should stay - so that operators may upstream their alpha streams as they see fit13:52
sdakebut ya its docs only13:52
*** inc0 has joined #openstack-kolla13:52
sdakeand it was me who requested it13:52
sdakemy upstream/my update13:52
sdakeholy shit I just used upstream as a  veerb13:52
sdakesomeone shoot me13:52
inc0my upstream, my update...sounds like title of a political manifesto13:53
inc0hello13:53
sdakeneed anestaphine from the medlock!13:53
sdakehey inc013:53
*** ccesario___ has quit IRC13:54
sdakei modfieid kolla to use qemu 2.3.0 last night13:54
inc0mein patch13:54
sdakenot sure if we want it for mitaka or not13:54
inc0probably not13:54
sdakeits pianful but my thinking innitially is we ight as well get the pain over with13:54
mandresdake: apparently Sam had concerns about potential issues during upgrades13:54
inc0well, upgrade of qemu means restart of vms13:54
sdakemandre was talking about something else inc013:55
mandrehey inc0, long time no see :)13:55
*** diogogmt has quit IRC13:55
inc0yeah, mandre looking at clock...is it France already?:)13:55
mandreinc0: this patch was reverted https://review.openstack.org/#/c/274408/13:55
patchbotmandre: patch 274408 - kolla - Remove the openstack_release option in the globals... (MERGED)13:55
sdakeinc0 ubuntu is on 2.5, centos is on 1.513:55
sdakethe change moves centos to 2.313:55
sdakeusing official repos13:56
mandreinc0: indeed, I'm fully french now13:56
mandrebonjour tout le monde13:56
sdaketo be fully french you have to hate the us people ;)13:56
inc0sdake, I think british have more reasons to13:56
*** diogogmt has joined #openstack-kolla13:56
sdakethey have gotten over  the evolution13:57
mandresdake, like I said :)13:57
mandrejk13:57
inc0yeah I kinda think that openstack_release should be there13:58
sdakeinc0 mandre bikeshed plz13:58
inc0we are using reno for version and such13:58
sdakehttps://review.openstack.org/#/c/302036/13:58
patchbotsdake: patch 302036 - kolla - Use proper CentOS repositories for ceph and qemu13:58
sdakesbezverk your welcome - 2am working on that one13:58
mandresame here, I don't really have objection on the patch itself, I just wanted to make sure sbezverk read the previous discussion about it, which he did13:59
*** ccesario has quit IRC14:00
sdakei didnt'14:00
sbezverksdake there is one issue though14:00
sdakei guess i should14:00
sdakelink?14:00
sdake7am lazy tired can barely function14:00
sdakei could come up with more excuses if you like :)14:00
sbezverksdake ImportError: No module named kolla.common14:00
sbezverksdake but when I do pip install kolla/14:00
sbezverkit starts working, build.py used to work without pip install kolla/14:01
sdakegit log it14:01
sdakethat should work proeprly14:01
mandresdake, where did the Oracle Linux gate go?14:01
sdakegit log biuld.py14:01
sdakemandre there is no oracle gate atm that is running all the time14:02
mandreI remember we had an experimental one not so long ago14:02
sdaketo trigger it i think ou leave check experimental14:02
mandreok, good to know14:02
sdakenote i  doubt my patch workso n oracle linux14:03
sdakei hit the wall last night14:03
sdakeso didn't test it14:03
sdakework to be done14:03
mandreI just triggered the experimental gate14:04
mandrewe'll see14:04
sdakeit definately wont work14:04
sdakethe constraints in versionlock are different14:04
sdakethis patch needs testing on ubuntu14:04
sdakeanyone up for that14:04
sdaketesting being does a vm boot14:05
sbezverksdake build.py was changed august of last year.14:05
sdakesbezverk ok well you have some other problem then ;)14:05
sdakecan only handle about 15 emergencies at once14:05
sdakeat capacity ;)14:05
sbezverksdake np I will dig into it..14:06
sdakefwiw my main motive for making this change is rhos ships with qemu 2.3, and i want to match with what is going in rhel osp14:10
sdakei dont know i qemu 2.3 is really any better or not14:10
sdakesbezverk had some arguments for it but i forgot them alredy except they involved numa :)14:10
sdakeif we decide not to make this change, I think we should make atleast qemu.conf configurable14:11
sdakeas it is done now14:11
sdakethat is a big wide ass gap in our config system14:11
sdakeconfigurable/templateable14:11
sdakelet me sort out an ol solution real quick14:12
*** ccesario has joined #openstack-kolla14:13
mwheckmann@sdake: FWIW: The newer versions of qemu (2.x+) that come from oVirt/RHEV derived CentOS repos are a must14:15
sdakemwheckmann mind adding some argumentatin to that assertion14:16
sdakeso I have some ammo to support the change14:16
mwheckmannThe main thing for us is advanced HyperV enlightenments for Windows VMs14:16
mwheckmannin particular, the hv_time one14:16
mwheckmannOpenstack will take advantage of that automatically since Kilo14:16
mwheckmannwe're talking 3x+ performance improvement for some CPU bound workloads on Windows14:17
mwheckmannones that use a particular Windows perf counter14:17
mwheckmannThere were a bunch of patches added to Nova around Kilo to take advantage of this stuff.14:18
sdakepeple ue windows in openstack vms?14:18
sdake;-)14:18
mwheckmannyes :)14:18
sdakeok anything else14:18
mwheckmannSome IO stuff that will benefit Linux VMs too. I don't remember specifically though.14:19
mwheckmanniothreads for one, but that's not turned on by Openstack automatically (yet)14:20
mwheckmannmuch improved virtio-scsi support I believe too.14:20
sdakevirtio-scsi = win14:21
mwheckmann?14:21
sdakemwheckmann you deploying kolla or just watching its progress?14:21
sdakei mean improving virtio is a good thing14:21
*** diogogmt has quit IRC14:23
mwheckmann@sdake: mostly watching for now. Using it for dev/test.14:24
sdakemwheckmann nice14:24
mwheckmannprod is still on traditionally managed Ansible.14:24
sdakemwheckmann some folks in the commuity want to mke hte dev/tewt cycle faster with kolla by bind mounting source trees optionally14:25
sdakeany thoughts on that14:25
mwheckmannno thoughts unfortunatly.14:25
sdake./build --devtest otheroptions14:25
sdakewould build you a devtest enviornment with your source trees external14:26
mwheckmannsure. why not?14:26
*** achanda has joined #openstack-kolla14:26
sdakemwheckmann since you have played with kolla, can you tell me your top 3 pain points at present14:26
mwheckmannI really have no thoughts though. Right we're building packages from upstream repos using delorean14:27
sbezverkmwheckmann 100% agree about qemu 2.3 performance versus 1.5.X, for developers it is no big deal I guess ;-) but for people who uses opensatck for somthing produictive it matters a lot14:27
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203614:27
mwheckmannI barely started. So far everything went really smooth. The only thing was confusion about where to put config changes. I.e /etc/kolla/config/keystone.conf vs. /etc/kolla/keystone/keystone.conf (source vs. dest)14:28
mwheckmannpretty damn confusion14:28
*** diogogmt has joined #openstack-kolla14:28
mwheckmannconfusing.14:28
mwheckmannI had to re-read that Ansible task a couple of times to figure out why it wasn't picking up my changes14:29
mwheckmann@sbezverk: Exactly. I'm not sure but I think all the NUMA pinning stuff might require the newer qemu as well.14:31
sdakemwheckmann ya docs issue, we know we need operational docs14:31
sdakeanything else?14:31
sdakesbezverk do you have a ceph setup14:32
sbezverkmwheckmann vhost, vhost-user14:32
sdakethis patch needs to be tested against a fresh ceph sestup14:32
mwheckmannPS: this is a great high level overview of some of the qemu differences between the vanilla CentOS version and the more advanced one: https://videos.cdn.redhat.com/summit2015/presentations/12752_red-hat-enterprise-virtualization-hypervisor-kvm-now-in-the-future.pdf14:32
sdaketo make sure the centos ceph packages dont trigger breakage14:32
mwheckmann@sdake: no not yet, but I'll be sure to bring anything up here :)14:33
sbezverksdake nope, but if it is not urgent I could work on it.. I need to confirm lvm2/iscsi conditional volumes first14:33
mwheckmannanyway, for Qemu, point is that a lot of the more advanced Openstack features depend on the newer version.14:34
mwheckmanndaily stand-up time for me. back later.14:34
sdakemwheckmann thanks for the link thatsh elpful14:35
*** reddy has quit IRC14:36
*** ccesario has quit IRC14:37
sdakebrb loggin itn ovpn14:38
*** ccesario___ has joined #openstack-kolla14:41
*** sdake_ has joined #openstack-kolla14:42
*** sdake has quit IRC14:42
sdake_nothing like an 8am meeting which is cancelled at 7:4014:42
* sdake_ groans14:42
*** sdake_ is now known as sdake14:43
sdakewtb acks https://review.openstack.org/#/c/300914/14:44
patchbotsdake: patch 300914 - kolla - Increase max scheduling attempts to 1014:44
*** sdake_ has joined #openstack-kolla14:48
*** ccesario___ has quit IRC14:50
*** ccesario___ has joined #openstack-kolla14:50
*** sdake has quit IRC14:51
*** salv-orlando has joined #openstack-kolla14:51
*** achanda has quit IRC14:53
*** reddy has joined #openstack-kolla14:54
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203614:55
sdake_mlima_ ping14:56
*** pmisiak has quit IRC14:57
*** dougs1 has joined #openstack-kolla15:00
*** Jeffrey4l has quit IRC15:01
*** dougs has quit IRC15:01
*** mbound has quit IRC15:01
*** mbound has joined #openstack-kolla15:01
sdake_sbezverk do you have a ceph setup15:03
sdake_sbezverk can you test that new patch versoin please15:03
*** dougs has joined #openstack-kolla15:06
*** b_bezak has quit IRC15:06
*** Jeffrey4l has joined #openstack-kolla15:07
*** mark-casey has joined #openstack-kolla15:07
sdake_a cherrypick needing acks: https://review.openstack.org/#/c/301974/115:08
patchbotsdake_: patch 301974 - kolla (stable/mitaka) - Change waiting on mariadb.pid15:08
*** mbound has quit IRC15:09
*** dougs1 has quit IRC15:09
*** blahRus has joined #openstack-kolla15:10
*** inc0 has quit IRC15:10
*** absubram has joined #openstack-kolla15:11
*** Jeffrey4l has quit IRC15:12
*** vhosakot has joined #openstack-kolla15:14
*** Liuqing has quit IRC15:15
vhosakotmorning!15:15
*** dave-mccowan has joined #openstack-kolla15:15
mlima_i'm here sdake_15:16
sdake_mlima_ do you have a ceph setup available for testing?15:16
mlima_yes15:16
*** Marga_ has joined #openstack-kolla15:16
sdake_can you test a patch for me?  I dont have my lab back together yet15:16
mlima_yes, i can15:17
sdake_https://review.openstack.org/30203615:17
mlima_but i have only aio15:17
sdake_that wfm15:17
sdake_vhosakot can you test that patch on ubuntu15:17
*** daneyon has joined #openstack-kolla15:19
GonZo2000Hi on kibana which index patther should i configure ?15:19
mlima_sdake_, i haven't centos available here15:19
GonZo2000pattern*15:19
vhosakotsdake_: sure, can we setup loopback devices (losetup) for ceph disks similar to swift ?15:19
*** dwalsh has quit IRC15:22
*** salv-orlando has quit IRC15:22
sbezverksdake will do the test and let you know..15:23
*** Jeffrey4l has joined #openstack-kolla15:24
sdake_mlima_ oh ok15:25
sdake_GonZo2000 log-*15:25
sdake_GonZo2000 we are working on a dashboard :)15:25
sdake_vhosakot i just want yo uto test ubuntu works or not15:25
sdake_not if ceph works15:26
vhosakotsdake_: ah ok...15:26
sdake_the patch only afects centos ceph15:26
sdake_and by works i mean nova boot works15:26
sdake_and creates an active vm15:26
vhosakotsdake_: got it, will test https://review.openstack.org/302036 and post results in comments.. cool15:26
mlima_well, i can do a centos deployment to test it sdake_15:27
*** sdake has joined #openstack-kolla15:29
*** dave-mcc_ has joined #openstack-kolla15:30
*** stvnoyes has joined #openstack-kolla15:30
*** sdake_ has quit IRC15:30
vhosakotmeeting in 1 hr15:31
*** shardy has quit IRC15:31
*** dave-mccowan has quit IRC15:32
*** ccesario___ has quit IRC15:32
*** haplo37 has joined #openstack-kolla15:34
*** mikelk has quit IRC15:36
*** sdake_ has joined #openstack-kolla15:40
*** daneyon has quit IRC15:40
*** mlima has joined #openstack-kolla15:42
*** banix has quit IRC15:42
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203615:42
*** sdake has quit IRC15:43
*** mlima_ has left #openstack-kolla15:43
*** banix has joined #openstack-kolla15:44
*** iceyao has quit IRC15:51
*** salv-orlando has joined #openstack-kolla15:53
*** ccesario has joined #openstack-kolla15:53
*** salv-orlando has quit IRC15:56
*** salv-orlando has joined #openstack-kolla15:58
gmmahahi, had a quick Q with ceph.. i deployed iwth ceph enabled, and did a tear down and re-deploy and bootstrap_ceph_mom container fails to start and logs show this. http://paste.openstack.org/show/493191/16:02
gmmahais there a special cleanup i have to do ?16:02
*** stvnoyes has left #openstack-kolla16:03
mwheckmannhow are config subdirectories handled in Kolla?16:04
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Add  iscsid/tgtd specific parameters to Kolla infrastructure  https://review.openstack.org/30000416:04
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Adds iscsid and tgtd docker containers  https://review.openstack.org/29128516:04
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Adds ansible code for iscsid and tgtd containers  https://review.openstack.org/29999516:04
mwheckmannI'm thinking of things like Keystone domain specific drivers: http://docs.openstack.org/developer/keystone/configuration.html#domain-specific-drivers16:05
sdake_gmmaha ya you need to clean up oproperly16:05
sdake_gmmaha the cleanup scripts dont do the job16:05
gmmahasdake_: aaah16:05
sdake_you need to remove the fstab entry for ceph16:05
sdake_you need to delete /etc/kolla/*ceph*16:05
gmmahaaah i thought that part was added to the cleanup-host script. (the fstab cleanup)16:06
gmmahai will check /etc/kolla/*ceph*'16:06
mwheckmannso I would want to have /etc/keystone/domains in my Keystone container16:06
mwheckmannsimilar for fernet_tokens: "key_repository = /etc/keystone/fernet-keys/"16:06
gmmahamwheckmann: on the deploy host you can add contents to /etc/kolla/config/<service>.conf and it gets appended to the respective conf file16:07
gmmahais that what you are loooking for>16:07
mwheckmann@gmmaha: no. I understand that part.16:07
mwheckmannbut if I need to add a whole new subdir for config files16:07
sdake_we hae not solved that scenario16:08
mwheckmannfor example, configs for Keystone domain specific drivers need to be in a separate directory16:08
sdake_what is the use case besides fernet?16:08
sdake_so plugins is the use case?16:08
mwheckmanndomain specific drivers16:08
sdake_is that like a plugin?16:08
mwheckmannsort of.16:08
sdake_I can assure you in newton we are going to solve the plugin prolem earlier rather hten later16:09
mwheckmannusing different identity backends for different domains16:09
sdake_it is my primary technical concern at the moemnt16:09
mwheckmann@sdake: is there already an approach in mind?16:09
sdake_looking to the teeam for answers16:09
sdake_but we are wrapping up mitaka16:10
sdake_so not looking immediately :)16:10
mwheckmannok. fair enough :)16:10
*** dwalsh has joined #openstack-kolla16:11
mwheckmannin the case, all that would be required is the ability to specify source directory (instead of individual files) for service configs16:11
mwheckmannand do a recursive copy.16:11
mwheckmannsafely of course16:11
dave-mcc_i want to set debug to True in nova.conf and update the nova-api container.  but, kolla seems to override my config.  is there an option i'm missing?16:11
gmmahasdake_: thanks for the tip.. flushing /etc/kolla/*ceph* on the deployment targets did the job16:12
mwheckmann@dave-mcc_: you need to put your specific change in /etc/kolla/config/nova/nova.conf16:13
sdake_gmmaha ya file bug plz16:13
mwheckmannit will then merge it with what is in the nova.conf template.16:13
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MITAKA BRANCH  https://review.openstack.org/29918216:13
gmmahasdake_: done..16:14
*** daneyon has joined #openstack-kolla16:14
sdake_gmmaha you shoul dgive the qemu 2.3 work a spin16:16
sdake_if your interested in performance16:16
*** banix has quit IRC16:16
gmmahasdake_: for sure..16:16
sdake_need link?16:16
gmmahayes please16:16
sdake_https://review.openstack.org/30203616:16
gmmahasdake_: this only affects CentOS / RHEL >16:17
gmmaha?16:17
sdake_oh right16:17
sdake_your using debuntian16:17
sdake_that has qemu 2.5 already16:17
gmmahaYup.. :)16:17
sdake_nm then16:18
gmmahasweet.. thanks16:18
dave-mcc_mwheckmann thanks.  i missed part of that path.16:19
openstackgerritMerged openstack/kolla: Enable the kibana access from the external network  https://review.openstack.org/30202716:20
gmmahasdake_: alright.. odd problem.. probably cause i dont know about ceph..16:25
gmmahai flushed /etc/kolla/*ceph*. re-deployed and it installed16:25
gmmahabut now disk space comes up as 0bytes16:26
sdake_no idea16:27
sdake_did you relabel?16:27
gmmahadont think i did16:27
gmmahaaaah haaa16:27
gmmahadidnt realize i have to do that everytime16:28
sdake_ceph is bust in my patch16:29
* sdake_ groans16:29
sdake_but feel free to test ubuntu without ceph - i'd like to knwo that works16:29
vhosakotmeeting in a bit16:29
gmmahasdake_: works without ceph.. been doing that..16:31
*** thumpba has joined #openstack-kolla16:31
gmmahaneeded ceph to get more disk space for VMs16:31
sdake_gmmaha we have our meeting now in openstack-meeting-4 so i'll have to get back to you but your welcome to join in :)16:31
gmmahasdake_: for sure.. thanks..16:32
* gmmaha goes to join16:32
*** Marga_ has quit IRC16:34
*** Marga_ has joined #openstack-kolla16:37
*** SiRiuS has quit IRC16:38
*** SiRiuS has joined #openstack-kolla16:38
ccesarioa bit question guys...16:40
ccesarioto setup an environment var, just put ENV VARNAME varvalu inside  docker/base/Dockerfile.j2 and rebuild all images.... right!?16:40
ccesarioI meant ... "a environment var in all containers "16:42
*** jasonsb has quit IRC16:42
*** Marga_ has quit IRC16:44
*** ccesario___ has joined #openstack-kolla16:48
*** coolsvap has joined #openstack-kolla16:48
*** achanda has joined #openstack-kolla16:53
*** ccesario___ has quit IRC16:53
*** ccesario___ has joined #openstack-kolla16:53
*** Serlex has quit IRC16:55
*** banix has joined #openstack-kolla16:56
*** ayoung has quit IRC16:59
*** mgoddard has quit IRC17:03
*** mgoddard has joined #openstack-kolla17:03
*** banix has quit IRC17:06
mark-caseyI have a bare metal setup that will be "sort of production" (no customers on it, but in the future our devs will have a bad day if it's not working). I'm using MAAS inside Vagrant to PXE boot the target hosts, and will be attempting to run the private registry and the deployment tasks themselved via Vagrant too. I've been documenting this here: https://github.com/ropsoft/kolla_from_vagrant17:10
mark-caseyI have no particular plans for this outside of our use but if its useful I'd be happy to contribute it/keep working on it. So far it's ubuntu-only but there are supported CentOS images for MAAS so it seemed like a good fit17:11
*** salv-orlando has quit IRC17:12
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MITAKA BRANCH  https://review.openstack.org/29918217:13
*** daneyon has quit IRC17:14
*** mdnadeem___ has joined #openstack-kolla17:14
*** banix has joined #openstack-kolla17:15
Jeffrey4lpls stop to backport to liberty branch. Any PS to do this, pls -1 / -217:15
*** mdnadeem___ has quit IRC17:15
Jeffrey4lvhosakot, sbezverk let talk about the general_log .17:16
vhosakotsure... let me -2 first :)17:16
*** mdnadeem___ has joined #openstack-kolla17:17
Jeffrey4lso I want to revert that change in the master. what do you think?17:17
Jeffrey4llog every query to the ELK is not a good idea and meaningless in prod env.17:18
vhosakotJeffrey4l: yes... I was thinking about it17:18
*** mdnadeem___ has quit IRC17:18
vhosakotJeffrey4l: so, currently, mariadb logs just errros and nothing else.. is this fine ?17:18
Jeffrey4lIf someone want to debug the sql query, just enable it and run `reconfigure` is enough. vhosakot17:18
mlimaguys, https://review.openstack.org/29953417:18
mlimareview it17:19
Jeffrey4lvhosakot, I do not know that. I do think current log is a little less.17:19
Jeffrey4lBut I am sure that log the query sql is not acceptable.17:19
*** rajathagasthya has joined #openstack-kolla17:20
sbezverkvhosakot on a busy opensatck cloud you will chew up disk space very quickly17:20
vhosakotJeffrey4l: yes, current log is less.. agreed, general_log is bad.. I'm looking if mysql has logs with less verbosity than general_log17:20
vhosakotsbezverk: yes17:20
Jeffrey4lvhosakot, before you have now idea. I think we should revert the change in the master branch.17:21
Jeffrey4ls/now/new/17:21
sbezverkvhosakot you should match to what regular opensatck default to17:21
sdake_which change17:21
*** sdake_ is now known as sdake17:21
Jeffrey4lsdake, https://review.openstack.org/29023817:21
vhosakothttps://review.openstack.org/#/c/301975/117:21
patchbotvhosakot: patch 301975 - kolla (stable/mitaka) - Add general_log to mariadb container and make heka...17:21
vhosakotsbezverk: what is the default OpenStack setting ?17:21
Jeffrey4lsorry, I paste the wrong link sdake17:22
vhosakotsure, I can revert in master17:22
vhosakotbut, is more logging needed ?17:22
vhosakotsdake: this is causing disk space issues - https://review.openstack.org/#/c/299133/17:23
patchbotvhosakot: patch 299133 - kolla - Add general_log to mariadb container and make heka... (MERGED)17:23
Jeffrey4lvhosakot, cool appreciate that.17:23
sdakeyes revert17:23
vhosakotok cool17:23
sdakedont need a new bug id, just use git revert operation17:23
vhosakotah ok, will do17:24
vhosakotJeffrey4l: sdake what about this cehrry pick ? shall I abandon it ?17:24
Jeffrey4lvhosakot, more log can be acceptable, if you can find better solution. But query sql is not.17:24
Jeffrey4lvhosakot, abandon is enough.17:24
vhosakotJeffrey4l: cool..17:24
sdakeya abanodn17:24
sdakebrb17:24
vhosakotcoo.. thanks for the info...17:24
vhosakotJeffrey4l: sbezverk:, I did not measure disk space .. good catch... I will revert it so that master is good :)17:25
Jeffrey4lvhosakot, cool17:26
Jeffrey4lcoolsvap, ping17:26
*** achanda has quit IRC17:27
Jeffrey4lcoolsvap, if you are back, please check the comments I made here https://review.openstack.org/29724717:27
sbezverkvhosakot check this link, it talk about different level of logging in mariadb https://mariadb.com/kb/en/mariadb/log-files/17:27
coolsvapJeffrey4l: let me check17:28
*** Marga_ has joined #openstack-kolla17:28
vhosakotsbezverk: thanks, yep, that is where I found general_log :).... but, it is too verbose and noisy and eating up disk space...17:28
Jeffrey4lcoolsvap, thanks.17:28
vhosakotI agree with Jeffrey4l.. let mariadb have just error logs, if mariadb issues are found, the operator needs to enable general_log and run mariadb reconfigure17:28
Jeffrey4lvhosakot, sbezverk i think no other log can be enabled except for error-log in the prod env.17:28
vhosakotJeffrey4l: yes, error-log is good for prod (and gate as well)17:29
*** sdake_ has joined #openstack-kolla17:29
*** sdake has quit IRC17:29
coolsvapJeffrey4l: i read that in mail notification, yeah we can abandon this change , I will have a look at your change as well17:29
Jeffrey4lcoolsvap, thanks.17:29
*** mwheckmann has quit IRC17:30
mark-caseyhttps://mariadb.com/kb/en/mariadb/binary-log-formats/#statement-based17:31
mark-caseyI dunno if Maria does, but MySQL rotates binlogs for you...17:32
vhosakotmark-casey: thanks for the link... binray logs need to be decoded right... can a human read it.. say, an operator wants to read binary logs17:33
mark-caseyperhaps, but in statement-based mode what you decode is the SQL that was run. In row mode you get little info... perhaps just what table and what rows changed and then a binary diff17:34
mark-caseybut... as I put that I'm wondering if Galera would tolerate that17:34
vhosakotmark-casey: good point...  I see all mysql setting can be used under the [mysqld] section in my.cnf for galera17:35
vhosakotmark-casey: but, don't know if this too ends up too verbose and eats up the disk17:35
vhosakotmariadb/mysql is one of the busiest process in any cloud... lots of DB operations happen to create just one VM...17:36
vhosakotmark-casey: I will test binary logs and post the results soon17:36
Jeffrey4ldave-mcc_, around?17:37
mark-caseyvhosakot: expire_logs_days=1 would help but even that might not be quick enough on the cleanup17:37
vhosakotmark-casey: nice, let me check it17:38
Jeffrey4labout this PS https://review.openstack.org/300852 do you have any info/doc to explain the interface and track_interface?17:38
Jeffrey4ldave-mcc_, ^17:38
mark-caseyand cool. this as discussed should work. Its just that if you set it to statement-based logging and it breaks Galera, then yes... you could do all the same things with the harder to read logs. You get some good info but its nothing like the actual queries17:38
Jeffrey4lmark-casey, are you try to send the log to the ELK? if so, the expire_logs_days is not helpful.17:39
dave-mcc_Jeffrey4l pong17:39
Jeffrey4ldave-mcc_,  about this PS https://review.openstack.org/300852 do you have any info/doc to explain the interface and track_interface?17:39
Jeffrey4ldave-mcc_, pls check the comments i made in it.17:39
vhosakotmark-casey: I see17:40
dave-mcc_Jeffrey4l yes... sorry, i wrote a reply, but i don't see it.  i must have missed a step.17:40
mark-caseyJeffrey4l: oh I don't know. I just saw some logging discussion and I have to deal with that fairly frequently so I chimed in.17:40
sbezverksdake I just finished testing iscsi with conditional volumes, it works in both scenarios when iscsi configured  yes or no, in case NO no iscsi and tgtd containers are started and no volumes they create are mount to nova or cinder.17:40
Jeffrey4ldave-mcc_, LOL17:40
dave-mcc_Jeffrey4l https://github.com/acassen/keepalived/blob/010cc929aa000b74afd52e2cbcdbf34334701f28/keepalived/include/vrrp.h#L29917:41
dave-mcc_there's the code that keepalived uses.  it checks the primary first, then goes through the track list.17:41
mark-caseythat would be right though. going that expire route would preclude them from going to ELK and still keeping the disk space down17:41
sbezverksdake I starting to test it on ubuntu17:41
*** daneyon has joined #openstack-kolla17:42
*** sdake has joined #openstack-kolla17:42
vhosakotmark-casey: Jeffrey4l: yes, expired logs will reduce the logs collected by ELK drastically17:42
*** SiRiuS has quit IRC17:42
dave-mcc_Jeffrey4l i also added the comment to the bug.17:43
Jeffrey4ldave-mcc_, also forgot the C language. need a little time to consume it.17:43
dave-mcc_Jeffrey4l (at least i think i did) :-)17:43
*** daneyon has quit IRC17:43
dave-mcc_Jeffrey4l me too. i removed c from my CV. :-)17:43
Jeffrey4ldave-mcc_, but, how a about another case? like me write a cause, wait for a seconds.17:43
*** sdake_ has quit IRC17:44
Jeffrey4ldave-mcc_, check this case http://paste.openstack.org/show/493213/17:46
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203617:47
Jeffrey4ldave-mcc_, sorry, i am wrong. you are right, no need add the api_interface to the track_interface section.17:48
dave-mcc_Jeffrey4l thanks for the discussion.  hooray for peer reviews!  it will be my turn to be wrong next time. ;-)17:50
Jeffrey4lvhosakot, no. the expired logs only reduce the logs on the hard disk. the ELK will keep it, unless you tell the elasticsearch to remove the log explicit.17:50
sdakepbourke ping17:50
vhosakotJeffrey4l: won't expire_logs_days=1 make ELK see just one day's logs and not all the logs ?17:50
Jeffrey4ldave-mcc_, :D thanks17:51
sdakepbourke does oraclelinux mirror the virtualiation and storage sigs for centos for ole?17:51
Jeffrey4lexpire_logs_days=1 is the mariadb parameter, right? vhosakot17:51
vhosakotyes17:51
Jeffrey4lwhy the elasticsearch need follow this parameter?17:52
*** ccesario___ has quit IRC17:53
vhosakotJeffrey4l: ELK collects what mariadb writes when central_logging is enabled.. so, if mariadb re-writes logs every 24 hrs, ELK will not see logs older than24 hrs17:53
vhosakotJeffrey4l: for now, I will revert master.. we can think about improving mariadb logs (if needed) later, and I will implement what is less verbose for mariadb17:54
Jeffrey4lvhosakot, that not true. ELK will log every line log mariadb write. when you remove the log file, the data wrote in the ELK never be removed. unless you tell the elasticsearch to remove the log explicit by using its API.17:55
vhosakotah i see17:56
Jeffrey4lneed go to bed. Bye guys.17:59
*** achanda has joined #openstack-kolla17:59
vhosakotgood night Jeffrey4l17:59
ccesariogood afternoon guys :)18:00
Jeffrey4lthanks. bye18:00
*** Jeffrey4l has quit IRC18:05
*** daneyon has joined #openstack-kolla18:11
*** ccesario___ has joined #openstack-kolla18:12
vhosakotI now see the mariadb <--> heka <--> elasticsearch connection in code... cool18:16
*** daneyon has quit IRC18:16
*** inc0 has joined #openstack-kolla18:17
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203618:18
*** thumpba has quit IRC18:29
*** inc0 has quit IRC18:29
*** inc0 has joined #openstack-kolla18:31
*** thumpba has joined #openstack-kolla18:31
*** mlima_ has joined #openstack-kolla18:34
*** coolsvap has quit IRC18:37
*** mwheckmann has joined #openstack-kolla18:37
*** mlima has quit IRC18:37
*** inc0 has quit IRC18:38
openstackgerritVikram Hosakote proposed openstack/kolla: Revert "Add general_log to mariadb container and make heka collect it"  https://review.openstack.org/30240118:38
*** gfidente has quit IRC18:41
*** daneyon has joined #openstack-kolla18:42
ccesarioguys, could someone pls confirm it ?   to setup an environment var, just put ENV VARNAME varvalue inside  docker/base/Dockerfile.j2 and rebuild all images.... right!?18:43
vhosakotsbezverk: sdake: could you review the revert in master  https://review.openstack.org/30240118:46
vhosakotccesario: put it in base if you want to affect all the containers... if not, I'f just put the ENV in the Dockerfile of the container that needs it...18:51
vhosakotccesario: both the following shoud work18:51
vhosakotENV key value18:51
vhosakotENV key=value18:51
ccesariovhosakot, I just would confirm it, because even I doing it, and rebuild my images the ENV config does not show inside container18:53
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203618:53
ccesariovhosakot, thanks by info.... I will try discovery because this does not happen as expected18:54
*** rajathagasthya has quit IRC18:55
vhosakotccesario: I see the ENV variable inside the container18:56
vhosakot$ docker exec neutron_server printenv | grep KOLLA18:56
vhosakotKOLLA_CONFIG_STRATEGY=COPY_ALWAYS18:56
vhosakotKOLLA_BASE_DISTRO=ubuntu18:56
vhosakotKOLLA_INSTALL_TYPE=source18:56
vhosakotKOLLA_INSTALL_METATYPE=mixed18:56
vhosakotccesario: refer https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L9-L1118:56
manjeetssdake i want to make one point here out of my observation18:57
manjeetswhen i am not using horizon doing everything thing through cli i don't see multiple ports and ip getting created for vms18:58
manjeetsI see only single ip and port so far for upto 2000 vms when command is issued through cli18:58
sdakeya horizon creates overload18:59
sdakenova vm client does not18:59
sdakenova doesn't handle overload18:59
ccesariovhosakot, yes, I did exactly it  above this line https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L11 I put ENV MYVAR value12318:59
sdakei wrote write einteh bug what needs to be fixed18:59
sdakesome variable or naother needs to be tuned18:59
ccesariovhosakot, then I run docker rmi -f  on all images and rebuild the images again19:00
manjeetssomething might needs to be done how kolla handles horizon  with osa horizon does not do this19:00
sdakethey hve tuned around it19:01
sdakewhich is what we are gong to do19:01
sdakeas soon as i hae two seconds to come up for breath i'll do the work19:01
ccesariotools/build.py .... and re-deploy  ..... the images are deployed without this var..... untouched19:01
*** inc0 has joined #openstack-kolla19:02
sdakemanjeets how long will you be around today19:06
sdakeif your available now i can spend  2 hours on your problem until i take daughter to dr19:07
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203619:07
sdakemanjeets that is an sqlalchemy default19:15
*** ayoung has joined #openstack-kolla19:16
sdakemanjeets http://docs.sqlalchemy.org/en/latest/core/pooling.html19:18
*** SiRiuS has joined #openstack-kolla19:19
openstackgerritMerged openstack/kolla: Start the haproxy by using haproxy-system-wrapper  https://review.openstack.org/29470119:23
*** cbits has joined #openstack-kolla19:26
openstackgerritSteven Dake proposed openstack/kolla: Increase max pool size so conductor doesn't implode  https://review.openstack.org/30241519:32
*** daneyon has quit IRC19:33
sdakemanjeets there you go give that a spin19:33
dave-mcc_sdake I patched 302036 on my code, but i still can boot VMs.  is that the only patch needed?19:34
sdakedave-mcc_ it may be broken atm19:35
sdakei am working on it19:35
sdakecan only unfortunately do one thing at a time19:35
*** salv-orlando has joined #openstack-kolla19:37
sdakedave-mcc_ i'll lety ou kno whwne 302306 works in my enviornment with ceph19:40
*** cbits has left #openstack-kolla19:44
*** salv-orlando has quit IRC19:44
*** rajathagasthya has joined #openstack-kolla19:45
*** thumpba has quit IRC19:47
*** mlima__ has joined #openstack-kolla19:50
*** mlima_ has quit IRC19:53
*** salv-orlando has joined #openstack-kolla19:54
*** salv-orlando has joined #openstack-kolla19:55
*** daneyon has joined #openstack-kolla20:07
vhosakotsbezverk: ping20:11
*** banix has quit IRC20:11
*** daneyon has quit IRC20:12
*** mat128 has joined #openstack-kolla20:13
*** mat128 has left #openstack-kolla20:19
*** SiRiuS has quit IRC20:22
*** mbound has joined #openstack-kolla20:25
*** mbound has quit IRC20:25
*** mbound has joined #openstack-kolla20:25
rajathagasthyaHi! I can't see any logs in the Kibana dashboard. I'm see this error when I do docker logs heka: http://paste.openstack.org/show/493232/. Any idea how to resolve that?20:30
rajathagasthyaFYI, Elasticsearch is reachable from the heka container. curl 172.28.128.254:9200 work fine.20:31
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203620:31
sbezverkvhosakot pong20:35
vhosakotrajathagasthya: those error does not look related to kibana... can you curl the kibana port ?    curl -v 172.28.128.254:560120:36
rajathagasthyavhosakot: Kibana works fine, I can access the dashboard from browser. Problem is heka isn't sending logs to ES.20:37
*** sdake_ has joined #openstack-kolla20:37
vhosakotsbezverk: log-error is still there, it as just mariadb in file_match since there is  log-error=/var/log/kolla/mariadb/mariadb.log  in galera.cnf20:38
*** sdake has quit IRC20:39
vhosakotsbezverk: I did "git revert" and this is how it was before   https://review.openstack.org/#/c/299133/  was merged20:39
patchbotvhosakot: patch 299133 - kolla - Add general_log to mariadb container and make heka... (MERGED)20:39
vhosakotrajathagasthya: oh ok.. so, you see no logs in Kiabna dashboard ?20:40
rajathagasthyavhosakot: Nope, and that's because heka can't connect to ES. It's strange since I can reach ES from inside Heka container.20:40
vhosakotrajathagasthya: keystone_apache_logstreamer_input is failing in heka20:41
vhosakotrajathagasthya: can you send the full output of "docker logs heka"20:41
rajathagasthyavhosakot: Sure, one moment.20:42
sbezverkvhosakot ok, so you will keep mariadb erro log, right? if it is the case I will change my review right away20:42
*** sdake has joined #openstack-kolla20:42
*** ccesario___ has quit IRC20:42
rajathagasthyavhosakot: http://paste.openstack.org/show/493234/20:43
vhosakotsbezverk: yep, log-error is not removed... the name of the file is not mariadb-error.log anymore, as the revert made it mariadb.log and it was like mariadb.log before...20:43
*** sdake_ has quit IRC20:44
vhosakotsbezverk: file_match = 'mariadb/mariadb\.log\.?(?P<Seq>\d*)$'    was there before https://review.openstack.org/#/c/302401/1 was merged20:44
patchbotvhosakot: patch 302401 - kolla - Revert "Add general_log to mariadb container and m...20:44
sbezverkvhosakot got it20:44
vhosakotsbezverk:  are you saying we need name the file mariadb-error.log instead of mariadb.log as they are error logs ?20:44
vhosakotelemoine_ names error logs as mariadb.log as part of central logging20:45
openstackgerritRyan Hallisey proposed openstack/kolla: Set db connection retry to infinity  https://review.openstack.org/30243420:45
vhosakotsbezverk: thanks for reverting -1.. should we have mariadb-error.log as the name as it is more approporiate ?20:46
vhosakotsbezverk: ^^20:46
sbezverkvhosakot no it is fine, I double check my normal openstack and it uses mariadb.log20:47
openstackgerritHui Kang proposed openstack/kolla: Add etcd Docker container  https://review.openstack.org/29845120:49
elemoine_rajathagasthya: does the elasticsearch_output error keep repeating in the Heka logs? If no then these occurs just occurred because ES was not yet started when Heka started. If they repeat then it means Heka cannot connect to ES...20:50
vhosakotsbezverk: ah ok... thanks for checking...20:50
vhosakotsbezverk: cool, mariadb.log name is good... the operator will know from galera.cnf that mariadb.log is written from the log-error setting20:51
*** salv-orl_ has joined #openstack-kolla20:51
rajathagasthyaelemoine_: This doesn't repeat. But how do I control which container starts first?20:51
openstackgerritHui Kang proposed openstack/kolla: Add etcd Docker container  https://review.openstack.org/29845120:52
*** ccesario___ has joined #openstack-kolla20:53
vhosakotrajathagasthya: when you say you can reach ES from inside heka container, you mean "docker exec heka curl curl 172.28.128.254:9200" works ?20:53
rajathagasthyavhosakot: Yep, that's what I meant.20:54
elemoine_rajathagasthya: Heka container is started by the common role, so Ansible will start it before the ES container. Hence the few ES output errors we get when Heka starts.20:54
*** salv-orlando has quit IRC20:55
rajathagasthyaelemoine_: So in that case, I will eventually get the logs in Kibana?20:55
elemoine_rajathagasthya: so it sounds like that Heka does send logs to Elasticsearch20:55
openstackgerritHui Kang proposed openstack/kolla: Add etcd Docker container  https://review.openstack.org/29845120:55
elemoine_rajathagasthya: Yes, just select log-* as the log index pattern in Kibana20:56
rajathagasthyaelemoine_: Yeah, that's what it looks like to me as well. All the logs are present in heka is /var/log/kolla/20:56
elemoine_rajathagasthya: that does not mean Heka collects them and send them to Elasticsearch20:57
*** ccesario___ has quit IRC20:57
*** ccesario__ has joined #openstack-kolla20:57
elemoine_rajathagasthya: most services write their logs to this directory, and Heka reads them from there20:57
elemoine_rajathagasthya: so that dir would include logs even if Heka did not work20:58
rajathagasthyaelemoine_: Okay, now I get it. I missed the log-* index creation part. :)20:59
*** ccesario__ has quit IRC20:59
rajathagasthyaelemoine_: Thanks!21:00
elemoine_rajathagasthya: cool, we know Kolla needs default Kibana dashboards!21:00
rajathagasthyaelemoine_: That's what I'm looking at right now. To have default Kibana dashboard. And I wanted to discuss a couple of things regarding how to do that.21:01
elemoine_rajathagasthya: please suggest dashboards if you come with nice and useful ones :)21:01
rajathagasthyaelemoine_: I thought we were going to do this? https://github.com/openstack/fuel-plugin-elasticsearch-kibana/blob/master/deployment_scripts/puppet/modules/lma_logging_analytics/templates/kibana_dashboards/logs.json21:03
rajathagasthyaAs a starter, at least.21:03
elemoine_yes it's a source of inspiration, but those are Kibana 3 dashboards21:03
elemoine_and we probably don't need more complex dashboards at this point21:04
elemoine_go for it!21:04
elemoine_bed time for me, bye21:04
rajathagasthyaelemoine_: Sure, goodnight.21:05
vhosakotelemoine_: thanks bye rajathagasthya: what is this log-* index creation part ? is it done in code on the dashboard.. is it documented ?21:05
rajathagasthyavhosakot: No, it's not documented and right now I think we have to do it manually in Kibana. But when we introduce default dashboard, we don't need to do it. It redirects automatically.21:06
vhosakotah ok, until the default dashboard is implemented, I think we need to document the steps to create the log-* index in the Kibana dashboard..21:08
vhosakotrajathagasthya: would you mind adding documentation for that in https://github.com/openstack/kolla/blob/master/specs/logging-with-heka.rst...21:09
vhosakotrajathagasthya: if you're busy, could you send the steps, I can add them, and add you as co-author on the patch set21:10
rajathagasthyavhosakot: Sure, I can do that.21:11
vhosakotrajathagasthya: nice! thanks.. so, in  https://github.com/openstack/kolla/blob/master/specs/logging-with-heka.rst,  please add all the steps to create the log-* index in the Kibana dashboard...21:12
vhosakotrajathagasthya: is there a bug/blueprint that tracks the default Kiabna dashboard you're working on ?21:12
*** ccesario__ has joined #openstack-kolla21:12
rajathagasthyavhosakot: No, there's no bug that tracks it.21:13
vhosakotcool, I will check sdake and keep you psted21:13
sdakefile one21:13
sdakeblueprint21:13
sdakeplz21:13
vhosakotsdake: ok, will do.. thanks21:14
rajathagasthyavhosakot: I think it's better to document those Kibana steps in quickstart.rst (Debugging Kolla section). The file you mentioned is more of a spec file. What do you think?21:15
vhosakotrajathagasthya: yes21:15
sdakewe need to fix our docs with sudo21:15
sdakeit makes it look like we always run as root21:15
sdakewhich i sure dont21:15
vhosakotrajathagasthya: add in quiskstart as it is what operators refer than specs21:15
sdakenobody looks at specs after they hit the repository21:16
vhosakotsdake: running docs as root ? what is that ?21:16
rajathagasthyavhosakot: Cool.21:16
vhosakotrajathagasthya: so, after creating the log-* index in the Kibana dashboard, do you see all the logs ?21:16
vhosakotin the Kibana dashboard ?21:16
*** ccesario__ has quit IRC21:18
rajathagasthyavhosakot: Correct.21:18
*** ccesario__ has joined #openstack-kolla21:19
vhosakotrajathagasthya: including keystone logs that threw errors in "docker logs heka" ?21:19
*** inc0 has quit IRC21:20
rajathagasthyavhosakot: No, the error logs were thrown by heka process, not Keystone. I don't know if those are collected as well.21:21
vhosakotrajathagasthya: cool, I will check that.. thanks for all the help!21:22
rajathagasthyavhosakot: No problem!21:22
*** jtriley has quit IRC21:24
*** ccesario___ has joined #openstack-kolla21:29
*** ccesario__ has quit IRC21:29
mark-caseycould anyone tell me where to specify the ssh private key so 'kolla-ansible prechecks' will use it?21:33
sdakedave-mcc_ your using --flavor 2 to launch your instances correct?21:34
sdakemark-casey you put it in authorized_keys on every node (the public key ofy our private key)21:35
sdakein the .ssh directory21:35
sdakemake sure its mode 60021:35
sdakeand owned by the user21:35
sdaketypically root21:35
sdakesince kolla-ansible atm unforutnately must run as root21:35
dave-mcc_sdake i chose "medium" from horizon21:35
sdakedave-mcc_ hmm21:35
sdakei dont use horizon to boot21:36
mark-caseysdake: thx. so it has to be id_rsa or I can give it some other name?21:36
sdakemark_casey you copy your id_rsa.pub to authorized_key or authorized_keys on every node in the .ssh dir21:36
sdakei dont recall which unfortunately21:37
*** ccesario___ has quit IRC21:37
mark-caseyright I have that part21:37
sdakegoogle internet for ssh setup there is probaboy a bajilliona rticles21:37
dave-mcc_sdake i have a tiny cirros image i'm using.  should i try a different image or flavor?21:37
sdakeif your running under sudo, it needs to be in your root directory as well21:37
sdakedave-mcc_ moment21:37
mark-casey...bah21:38
mark-caseyI see what I did21:38
sdakeso the flavors are gone apparently21:38
mark-caseyI'm trying to run my deployer inside docker and it dropped me into / not /root21:38
sdakein maste ratleast21:38
mark-caseythx21:38
sdakethis was posted to the ml21:38
sdakei didn't think they would do so fast21:38
sdakegotta take daughter to dr bbl21:39
*** ayoung has quit IRC21:39
vhosakotsdake: rajathagasthya: https://blueprints.launchpad.net/kolla/+spec/default-kibana-dashboard21:41
vhosakotrajathagasthya:   https://discuss.elastic.co/t/how-to-set-a-dashboard-as-default-in-kibana-4-2/3366221:43
rajathagasthyavhosakot: This is what we'd want to do: https://github.com/openstack/fuel-plugin-elasticsearch-kibana/blob/master/deployment_scripts/puppet/modules/lma_logging_analytics/manifests/kibana_dashboard.pp#L35-L3621:44
vhosakotrajathagasthya: ah ok...21:46
*** sdake has quit IRC21:54
*** daneyon has joined #openstack-kolla21:56
*** dwalsh has quit IRC21:57
vhosakotmark-casey: http://docs.ansible.com/ansible/intro_getting_started.html#your-first-commands  has steps to setup passwordless SSH for ansible.. basically, add public SSH key of deploy node in authorized_keys on each target host..21:58
vhosakotmark-casey: ansible also has --private-key option to specify a pem file instead21:58
vhosakotmark-casey: http://docs.ansible.com/ansible/intro_configuration.html#private-key-file21:59
*** ccesario___ has joined #openstack-kolla22:00
*** daneyon has quit IRC22:01
mark-caseyvhosakot: thank you. I wasn't seeing an equivalent to ansible-playbook's --private-key option in kolla-ansible, but I22:02
mark-casey*i've just given the key the default id_rsa name and it found it :)22:02
vhosakotmark-casey: kolla-ansible has then -e option to pass Ansible variables https://github.com/openstack/kolla/blob/master/tools/kolla-ansible#L37... dont know if it can be used to pass the private key location thru to Ansible22:03
vhosakotmark-casey: I see it in the output of :ansible-playbook --help"22:05
vhosakot$ ansible-playbook --help | grep private22:05
vhosakot  --private-key=PRIVATE_KEY_FILE22:05
vhosakotmark-casey: so, you could try:      kolla-ansible deploy -e private-key=~/.ssh/id_rsa22:06
mark-caseyvhosakot: cool thx. I will22:07
vhosakotmark-casey: cool, let me know if it works.. may be worth adding documentation for it22:08
vhosakotmark-casey: thanks!22:08
*** stvnoyes has joined #openstack-kolla22:14
*** haplo37 has quit IRC22:23
mark-caseyvhosakot: ping22:31
vhosakotmark-casey: pong22:31
*** achanda has quit IRC22:32
mark-caseyIt didn't like that option for the key, but it is working well enough for now as is. I did have another question... did you all ever solve...   "APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/images/create?tag=2.0.0&fromImage=controller%3A4000%2Fkollaglue%2Fubuntu-source-heka',),)]"22:32
mark-caseyI got that error and saw it in http://eavesdrop.openstack.org/irclogs/%23kolla/%23kolla.2016-03-18.log.html22:32
mark-caseyre: TASK: [common | Starting heka container] **************************************22:33
vhosakotmark-casey: share the /run mount22:33
vhosakotmount --make-shared /run22:33
vhosakoton UBuntu22:33
vhosakotmark-casey: search "mount" in http://docs.openstack.org/developer/kolla/quickstart.html22:33
mark-caseyahhhh. ok. I must have scrolled past it. I'd made a mental note to come back to that and I guess it got erased :D22:34
mark-caseythanks22:34
vhosakotcool, np22:34
vhosakotmark-casey: I'm curious do you use curl/urlib to search in kolla logs, or manually search22:34
vhosakotI meant in kolla IRC logs22:35
openstackgerritRajath Agasthya proposed openstack/kolla: Added Kibana instructions to quickstart.rst  https://review.openstack.org/30245822:35
mark-caseyvhosakot: usually google :/22:37
mark-caseykubernetes site:http://eavesdrop.openstack.org/irclogs/#kolla/22:37
mark-caseyoops, that was supposed to be one whole thing22:37
vhosakotcool22:38
*** achanda has joined #openstack-kolla22:38
*** cbits has joined #openstack-kolla22:40
manjeetssdake vhosakot inc0 i am going to install openstack using linux bridge instead ovs22:40
manjeetsshould i cleanup registry ?22:40
manjeetsor cleanning up images and containers is enough22:41
*** daneyon has joined #openstack-kolla22:46
*** vhosakot has quit IRC22:46
sbezverkmanjeets I do not think you need to clean up the registrym you will be using the same images with the exception of neutron for linux bridge22:51
sbezverkjust make sure it was built and you can see in the list22:51
sbezverkdocker images | grep -i linux22:51
manjeetssbexverk thanks giving it a shot22:54
*** mwheckmann has quit IRC22:55
*** pabelanger has joined #openstack-kolla22:56
sbezverkmanjeets good luck!22:57
pabelangerGreetings, wanted to see when kolla is expecting to tag its mitaka release22:57
pabelangerover in openstack-infra we're looking at migrating your devstack-centos7 jobs to centos-7 dibs. Which requires some downtime22:58
*** wmiller has quit IRC23:05
*** wmiller has joined #openstack-kolla23:06
*** blahRus has quit IRC23:07
*** cbits has quit IRC23:08
*** ayoung has joined #openstack-kolla23:09
*** mgoddard has quit IRC23:09
*** mgoddard has joined #openstack-kolla23:10
*** achanda has quit IRC23:15
*** cbits has joined #openstack-kolla23:18
*** achanda has joined #openstack-kolla23:22
manjeetssbezverk  what other changes i need to do other than modifying global.yml and all,yml23:26
manjeetsi had linuxbridge in those but in deployment i still se neutron ovs agent got started23:26
manjeetssbezverk i just got success how do i make sure linux bridge is done properly23:28
*** rajathagasthya has quit IRC23:28
manjeetshttp://paste.openstack.org/show/493250/23:29
manjeetsi guess its configured properly23:29
*** ccesario___ has quit IRC23:34
*** ccesario___ has joined #openstack-kolla23:35
*** salv-orl_ has quit IRC23:36
*** sdake has joined #openstack-kolla23:44
*** rhallisey has quit IRC23:45
*** sdake_ has joined #openstack-kolla23:46
*** sdake has quit IRC23:49
*** sdake has joined #openstack-kolla23:50
sdakegood news is ceph works single node with my qemu 2.3 patch23:51
*** sdake_ has quit IRC23:52
* manjeets just got deploy with lb works23:54
sdakemanjeets did you try that patch23:56
sdakethe one i posted for the bug you reported wit hte horizon 2000 vms23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!