| *** daneyon has quit IRC | 00:00 | |
| *** Marga_ has joined #openstack-kolla | 00:01 | |
| *** daneyon has joined #openstack-kolla | 00:01 | |
| *** Marga_ has quit IRC | 00:02 | |
| *** Marga_ has joined #openstack-kolla | 00:03 | |
| *** Marga_ has quit IRC | 00:03 | |
| *** Marga_ has joined #openstack-kolla | 00:03 | |
| sdake_ | http://buildlogs.centos.org/centos/7/virt/x86_64/kvm-common/centos-release-qemu-ev-1.0-1.el7.noarch.rpm | 00:04 |
|---|---|---|
| *** weiyu has quit IRC | 00:10 | |
| dave-mccowan | something's gone wrong with my all-in-one kolla server. running deploy gives me: msg: The requested image does not exist: kollaglue/centos-binary-heka:2.0.0, but the images show under "docker images". i'm using a local registry and it is set insecure in my docker startup. any ideas what i've missed? | 00:12 |
| *** diogogmt has quit IRC | 00:14 | |
| *** mark-casey has quit IRC | 00:15 | |
| kklimonda | have you set docker_registry in kolla's global.yml? | 00:17 |
| kklimonda | globals.yml* even | 00:17 |
| *** daneyon has quit IRC | 00:17 | |
| sdake_ | did you run genpwd? | 00:19 |
| sdake_ | and did you set docker_registry in global.yml? | 00:19 |
| sdake_ | and when you built did you use --registery locaol-server-ip:4000 ? | 00:19 |
| *** andrewvwebber has joined #openstack-kolla | 00:20 | |
| sdake_ | dave-mccowan ^^ | 00:20 |
| dave-mccowan | yes to reg set in global and --reg used on build. in passwords.yaml i have docker_registry_password: null. (is that right for no password) | 00:21 |
| sdake_ | sbezverk centos-release-ceph-hammer.noarch | 00:23 |
| *** mwheckmann has joined #openstack-kolla | 00:25 | |
| *** andrewvwebber has quit IRC | 00:26 | |
| sbezverk | sdake here is the bug to track qemu version: https://bugs.launchpad.net/kolla/+bug/1566588 | 00:26 |
| openstack | Launchpad bug 1566588 in kolla "integrate new qemu 2.3.0.X version into Kolla " [Undecided,New] | 00:26 |
| *** sdake has joined #openstack-kolla | 00:32 | |
| *** sdake_ has quit IRC | 00:34 | |
| manjeets | sdake_ not sure what i did not getting a single error like image not found for past 10 20 runs of creating vms and destroying them | 00:42 |
| sdake | huh | 00:43 |
| sdake | do you mean the problem you had disappared? | 00:43 |
| manjeets | i just change max_scheduler | 00:43 |
| manjeets | = 10 | 00:43 |
| manjeets | max_scheduler_attempts = 10 | 00:43 |
| sdake | ya | 00:44 |
| sdake | that variable definately matters | 00:44 |
| manjeets | thats what i mean | 00:44 |
| manjeets | i don't know why split brain is not happening | 00:45 |
| manjeets | all my machine are getting one port and one ip | 00:45 |
| manjeets | tried it upto 2000 | 00:45 |
| manjeets | but I am not using horizon at all | 00:45 |
| manjeets | everything through cli | 00:45 |
| manjeets | and deleting 500 vms is like 11 mins | 00:46 |
| *** Jeffrey4l has joined #openstack-kolla | 00:47 | |
| manjeets | nova delete vm1 vm2 ..................vm_n is fastest way so far i've seen | 00:47 |
| *** dave-mcc_ has joined #openstack-kolla | 00:50 | |
| *** sdake_ has joined #openstack-kolla | 00:51 | |
| *** sdake has quit IRC | 00:51 | |
| *** dave-mccowan has quit IRC | 00:51 | |
| *** ccesario___ has quit IRC | 00:52 | |
| *** weiyu has joined #openstack-kolla | 00:54 | |
| *** iceyao has joined #openstack-kolla | 00:56 | |
| sdake_ | its not spit brain | 00:57 |
| sdake_ | manjeets its a different error, if you read the bug log my analysis is in there | 00:58 |
| *** rhallisey has quit IRC | 01:00 | |
| *** yuanying has quit IRC | 01:01 | |
| *** ccesario___ has joined #openstack-kolla | 01:01 | |
| *** daneyon has joined #openstack-kolla | 01:01 | |
| sdake_ | sbezverk nova compute builds for me | 01:02 |
| *** yuanying has joined #openstack-kolla | 01:02 | |
| *** yuanying has quit IRC | 01:04 | |
| *** daneyon has quit IRC | 01:07 | |
| *** diogogmt has joined #openstack-kolla | 01:07 | |
| *** yuanying has joined #openstack-kolla | 01:13 | |
| sdake_ | https://bugs.launchpad.net/kolla/+bug/1566594 | 01:15 |
| openstack | Launchpad bug 1566594 in kolla mitaka "enable hammer and qemu-ev properly" [Medium,Confirmed] - Assigned to Steven Dake (sdake) | 01:15 |
| *** ccesario___ has quit IRC | 01:17 | |
| *** ccesario___ has joined #openstack-kolla | 01:17 | |
| *** ccesario___ has quit IRC | 01:24 | |
| *** unicell1 has quit IRC | 01:25 | |
| *** yuanying has quit IRC | 01:25 | |
| *** yuanying has joined #openstack-kolla | 01:28 | |
| manjeets | sdake i just got fastest delete of nova vms | 01:30 |
| manjeets | 500 vms got killed in 4 m | 01:30 |
| *** yuanying has quit IRC | 01:30 | |
| manjeets | its something some times not freeing up resources | 01:30 |
| *** salv-orlando has joined #openstack-kolla | 01:32 | |
| *** Jeffrey4l has quit IRC | 01:34 | |
| *** yuanying has joined #openstack-kolla | 01:39 | |
| *** Jeffrey4l has joined #openstack-kolla | 01:42 | |
| *** yuanying has quit IRC | 01:42 | |
| *** yuanying has joined #openstack-kolla | 01:43 | |
| *** weiyu has quit IRC | 01:47 | |
| *** salv-orlando has quit IRC | 01:47 | |
| sdake_ | manjeets what made deletes faster? | 01:50 |
| sdake_ | any change you made? | 01:50 |
| *** iceyao has quit IRC | 01:50 | |
| *** iceyao has joined #openstack-kolla | 01:51 | |
| *** vhosakot has joined #openstack-kolla | 01:53 | |
| *** tfukushima has joined #openstack-kolla | 01:54 | |
| *** weiyu_ has joined #openstack-kolla | 01:54 | |
| *** sdake has joined #openstack-kolla | 01:55 | |
| *** mwheckmann has quit IRC | 01:55 | |
| *** sdake_ has quit IRC | 01:56 | |
| *** yuanying has quit IRC | 02:02 | |
| kjelly | Hi review, what is the palce for putting unit test code for build.py? https://review.openstack.org/#/c/290423/ | 02:03 |
| patchbot | kjelly: patch 290423 - kolla - add unit tests for build script | 02:03 |
| kjelly | Hi reviewers, what is the palce for putting unit test code for build.py? | 02:04 |
| *** yuanying has joined #openstack-kolla | 02:05 | |
| *** kjelly_ has joined #openstack-kolla | 02:23 | |
| *** banix has joined #openstack-kolla | 02:33 | |
| *** achanda has joined #openstack-kolla | 02:34 | |
| *** weiyu_ has quit IRC | 02:39 | |
| *** kjelly has quit IRC | 02:40 | |
| *** tobe has joined #openstack-kolla | 02:42 | |
| *** tobe has quit IRC | 02:42 | |
| *** ccesario___ has joined #openstack-kolla | 02:43 | |
| *** unicell has joined #openstack-kolla | 02:49 | |
| *** yuanying has quit IRC | 02:50 | |
| *** weiyu_ has joined #openstack-kolla | 02:53 | |
| Jeffrey4l | kjelly_, I am OK put it in or out the kolla folder. But my point is there is only one place to put the UTs. | 02:54 |
| kjelly_ | Jeffrey4l: Got it. send next patch as soon as possible. | 02:56 |
| *** weiyu_ has quit IRC | 02:56 | |
| *** iceyao_ has joined #openstack-kolla | 02:57 | |
| *** weiyu_ has joined #openstack-kolla | 02:59 | |
| *** iceyao has quit IRC | 03:01 | |
| *** sdake_ has joined #openstack-kolla | 03:01 | |
| *** salv-orlando has joined #openstack-kolla | 03:03 | |
| Jeffrey4l | thanks. | 03:03 |
| *** sdake has quit IRC | 03:04 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Add iscsid/tgtd specific parameters to Kolla infrastructure https://review.openstack.org/300004 | 03:04 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Adds ansible code for iscsid and tgtd containers https://review.openstack.org/299995 | 03:04 |
| *** salv-orlando has quit IRC | 03:13 | |
| *** weiyu_ has quit IRC | 03:21 | |
| kklimonda | hmm, it seems like accessing variables via hostvars doesn't take globals.yml into account? i.e. {{ api_interface }} and {{ hostvars[ansible_hostname]['api_interface'] }} don't match. Am I missing something? Ansible 1.9.5 | 03:22 |
| *** weiyu has joined #openstack-kolla | 03:22 | |
| *** achanda has quit IRC | 03:27 | |
| *** banix has quit IRC | 03:33 | |
| *** coolsvap has joined #openstack-kolla | 03:35 | |
| coolsvap | Jeffrey4l: hi regarding https://review.openstack.org/#/c/300955 | 03:38 |
| coolsvap | is your selinux enforcing? | 03:38 |
| Jeffrey4l | coolsvap, yes | 03:39 |
| coolsvap | its not working for me on centos | 03:39 |
| coolsvap | are you checking on centos? | 03:39 |
| Jeffrey4l | coolsvap, yep. | 03:40 |
| Jeffrey4l | coolsvap, see this http://paste.openstack.org/show/493098/ | 03:40 |
| Jeffrey4l | i really deployed the kolla successfully. | 03:40 |
| coolsvap | because i tried the /etc/hosts change but it was not working for me unless i have selinux!=enforcing i will check again | 03:41 |
| sdake_ | coolsvap you may need to relabel | 03:46 |
| *** yuanying has joined #openstack-kolla | 03:46 | |
| coolsvap | sdake_: yes, i am trying the same now | 03:51 |
| Jeffrey4l | coolsvap, kolla do not work with selinux for sure, https://github.com/openstack/kolla/blob/master/doc/selinux.rst. But it not cause your issue | 03:53 |
| coolsvap | Jeffrey4l: I will double check, I currently do not have access to the system i tested earlier, i am setting up new currently | 03:54 |
| coolsvap | i will check on that system again tonight | 03:54 |
| Jeffrey4l | roger | 03:54 |
| kklimonda | hmm, what is a correct ansible version to deploy kolla master? docs say it's <2.0 but commit 903c7d4c seems to be 2.0 only, as it doesn't work with ansible 1.9.5 | 03:55 |
| kklimonda | (well, ansible runs without error, but this way of accessing api_interface produces wrong result in 1.9.5) | 03:56 |
| * sdake_ GROANS | 03:57 | |
| sdake_ | hardpin to 1.9.4 or 1.9.5> < 2.0.0 i guess | 03:58 |
| sdake_ | kklimonda atm kola is tested extensivly only on ansible 1.9.4 | 03:59 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Install proper security libs https://review.openstack.org/295602 | 04:03 |
| kklimonda | sdake_: installing 1.9.4 didn't change anything, I'm still seeing same result: https://dpaste.de/OBCy | 04:10 |
| sdake_ | kklimonda can you show deploy output pls | 04:12 |
| sdake_ | i have no idea what your showign in that paste | 04:12 |
| kklimonda | sure | 04:12 |
| sdake_ | also which version are you running | 04:12 |
| sdake_ | master, mitaka, lierty? | 04:12 |
| kklimonda | master | 04:12 |
| *** weiyu has quit IRC | 04:16 | |
| kklimonda | ok, so deploy output doesn't really show much - there are no errors, or warnings but haproxy.cfg has wrong IPs for mariadb backend servers (and most likely others too). | 04:17 |
| *** weiyu has joined #openstack-kolla | 04:18 | |
| kklimonda | http://paste.debian.net/424617/ - this is output from two extra debug tasks I've added to roles/haproxy/tasks/configure.yml | 04:20 |
| sdake_ | dave-mcc_ ^6^ | 04:20 |
| sdake_ | dave is working on that problem | 04:20 |
| *** weiyu has quit IRC | 04:21 | |
| dave-mcc_ | sdake_ that looks different than what i'm looking at. | 04:21 |
| dave-mcc_ | mine is specific to cinder and glance. not sure what would cause mariadb to have the wrong address. | 04:21 |
| sdake_ | kklimonda show me your /etc/host file on the nodes | 04:22 |
| sdake_ | control nodes | 04:22 |
| sdake_ | paying 1 usd to someone to document how /etc/host should be configured in qsg | 04:25 |
| dave-mcc_ | i remember someone saying that all the XX_interface options may not work. kklimonda is there a reason you're customizing api_interface? i don't think that is common. | 04:25 |
| kklimonda | sdake_: it's pretty empty, it has 127.0.1.1 node001.[fqdn] node001 on each node, other nodes can resolve node001.[fqdn] with dns server's help (both fqdn and hostname, search is set in resolv.conf) | 04:28 |
| sdake_ | 127.0.1.1 should not resolve to node0001 | 04:30 |
| kklimonda | dave-mcc_: I want mariadb (and other services) to be listening on a "internal IP", that isn't routable to our network | 04:30 |
| sdake_ | it should reolve to the real ip address of the machine | 04:30 |
| sdake_ | 127.0.1.1 is a made up address to make gnome work properly | 04:30 |
| sdake_ | for ubuntu machines | 04:30 |
| sdake_ | bad hack imo | 04:30 |
| sdake_ | but it is what it is ;) | 04:30 |
| *** jasonsb has joined #openstack-kolla | 04:30 | |
| *** salv-orlando has joined #openstack-kolla | 04:31 | |
| sdake_ | my ip ddress of the machine, i mean the ip managemenet ip address | 04:32 |
| kklimonda | yeah, I agree - I don't have that in production, but this is pretty barebone installation, and fixing it doesn't seem to make a difference anyway | 04:32 |
| sdake_ | it should have all hsots in it | 04:32 |
| sdake_ | my/by | 04:33 |
| *** diogogmt has quit IRC | 04:33 | |
| sdake_ | it will break how mariadb works | 04:33 |
| sdake_ | and all the addresses it gets | 04:33 |
| sdake_ | it may not be your exact problem | 04:33 |
| kklimonda | how will it break given that my dns resolves everything (both host->ip and ip->host) anyway, and galera uses IP addresses? | 04:35 |
| kklimonda | but that doesn't really seem to be an issue right now anyway | 04:35 |
| dave-mcc_ | kklomonda i agree that is important. the way i do that is to set kolla_internal_vip_address to a non-routable. this will keep internal URLs to opentack, mariadb, and rabbit all internal-only. then set kolla_external_vip_address to your routable and set kolla_external_vip_interface to be the outside i/f, leaving network_interface to be internal only. | 04:35 |
| kklimonda | mmm, sounds like that would work | 04:37 |
| sdake_ | dave++ | 04:37 |
| *** daneyon has joined #openstack-kolla | 04:38 | |
| kklimonda | do you also set network_interface to your internal interface then? | 04:38 |
| kklimonda | ok | 04:38 |
| kklimonda | I should get some sleep | 04:39 |
| kklimonda | I could swear you didn't write that | 04:39 |
| *** salv-orlando has quit IRC | 04:39 | |
| kklimonda | yeah, with that change I get haproxy.cfg I want, thanks | 04:41 |
| sdake_ | thanks dave-mcc_ | 04:41 |
| dave-mcc_ | yes, with two vips network_inteface is the internal one. | 04:41 |
| sdake_ | kklimonda mind running it throug hthe 1.9.5 grinder when you wrap up | 04:41 |
| sdake_ | also you will need a matching /etc/host file on th control nodess | 04:42 |
| sdake_ | without 127.0.1.1 ;) | 04:42 |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Add prechecks to validate kolla_internal_vip_address and kolla_external_vip_address https://review.openstack.org/302026 | 04:42 |
| *** daneyon has quit IRC | 04:42 | |
| vhosakot | sdake_: ^^^ I've further hardened prechecks | 04:43 |
| sdake_ | nice | 04:43 |
| sdake_ | fix bugsp lz :) | 04:44 |
| sdake_ | we got a slew of em | 04:44 |
| vhosakot | I've added the following three additional prechecks | 04:44 |
| vhosakot | Checking if kolla_internal_vip_address is assigned to any interface on deploy node | 04:44 |
| vhosakot | Checking if kolla_internal_vip_address is in the same network as network_interface | 04:44 |
| vhosakot | Checking if kolla_external_vip_address is assigned to any interface on deploy node | 04:44 |
| sdake_ | how do yu odetermine same network for #2 | 04:45 |
| sdake_ | /8, /16, /24? | 04:45 |
| vhosakot | ansible has a builtit module to check if an IP address belong to a range | 04:45 |
| sdake_ | which range are you specifying | 04:45 |
| sdake_ | i think we dont know that range ahead of time | 04:46 |
| vhosakot | range of network_interface | 04:46 |
| vhosakot | we can find in code | 04:46 |
| sdake_ | ok if it works wfm ;) | 04:46 |
| sdake_ | i'e added 2.3.0 qemu for those that care | 04:46 |
| sdake_ | which is sbezverk atm ;) | 04:46 |
| vhosakot | yep, I tested 9 combinations with the PS... | 04:46 |
| vhosakot | sdake_: nice! | 04:46 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Enable the kibana access from the external network https://review.openstack.org/302027 | 04:46 |
| vhosakot | I will bug the next bug now | 04:47 |
| sdake_ | vhosakot which version of qemu is used in ubuntu containers? | 04:48 |
| vhosakot | let me check... | 04:48 |
| sdake_ | my intenet is working again | 04:48 |
| sdake_ | at night | 04:48 |
| sdake_ | when its cool | 04:48 |
| sdake_ | i hoep its not a heat problem | 04:48 |
| sdake_ | Jeffrey4l do yo hae a ceph setup available? | 04:52 |
| vhosakot | sdake_: how can I check kvm version... is it tied to kernel version (uname -r).. I am using Ubuntu kernel 4.2.0-30-generic LTS wily | 04:52 |
| sdake_ | dpkg -l | grep qemu | 04:52 |
| sdake_ | qemu-kvm is the centos package name | 04:52 |
| vhosakot | ah package... 1 sec | 04:52 |
| sdake_ | not sure what its called in ubuntian | 04:52 |
| vhosakot | sdake_: "dpkg -l | grep qemu" worked... It shows qemu 1:2.5+dfsg-5ubuntu4~cloud0 | 04:54 |
| vhosakot | 1:2.5 | 04:54 |
| sdake_ | grep for kvm | 04:54 |
| sdake_ | ca nyou check wha the latest upstream versoin is? | 04:55 |
| vhosakot | nothing.. I dont see anything when I grep kvm | 04:55 |
| vhosakot | sdake_: http://wiki.qemu.org/Download says 2.6.0-rc1 | 04:56 |
| sdake_ | nice thanks | 04:56 |
| sdake_ | looks like centos is behind | 04:56 |
| sdake_ | but 2.3 better then 1.5.3 + 1000 patches or more | 04:56 |
| vhosakot | sdake_: I see kvm in kernerl module (lsmod) | 05:00 |
| vhosakot | $ lsmod | grep kvm | 05:00 |
| vhosakot | kvm_intel 163840 3 | 05:00 |
| vhosakot | kvm 507904 1 kvm_intel | 05:00 |
| *** achanda has joined #openstack-kolla | 05:00 | |
| sdake_ | vhosakot i'm good with the info thanks | 05:00 |
| vhosakot | cool | 05:00 |
| *** vhosakot has quit IRC | 05:07 | |
| *** tfukushima has quit IRC | 05:10 | |
| *** weiyu has joined #openstack-kolla | 05:15 | |
| coolsvap | cinder is not building in gate | 05:22 |
| *** weiyu has quit IRC | 05:30 | |
| *** sekrit has quit IRC | 05:30 | |
| *** Marga_ has quit IRC | 05:34 | |
| *** iceyao has joined #openstack-kolla | 05:34 | |
| *** Marga_ has joined #openstack-kolla | 05:37 | |
| *** iceyao_ has quit IRC | 05:38 | |
| *** Marga_ has quit IRC | 05:38 | |
| *** Marga_ has joined #openstack-kolla | 05:38 | |
| *** sdake has joined #openstack-kolla | 05:40 | |
| *** sdake_ has quit IRC | 05:42 | |
| *** Marga_ has quit IRC | 05:43 | |
| *** salv-orlando has joined #openstack-kolla | 05:46 | |
| *** sdake_ has joined #openstack-kolla | 05:49 | |
| *** Marga_ has joined #openstack-kolla | 05:52 | |
| *** sdake has quit IRC | 05:52 | |
| *** salv-orlando has quit IRC | 05:55 | |
| *** iceyao has quit IRC | 05:55 | |
| *** iceyao has joined #openstack-kolla | 05:56 | |
| *** Marga_ has quit IRC | 05:56 | |
| *** dave-mcc_ has quit IRC | 05:56 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Increase max scheduling attempts to 10 https://review.openstack.org/300914 | 05:58 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 05:58 |
| *** Marga_ has joined #openstack-kolla | 06:01 | |
| *** reddy has quit IRC | 06:08 | |
| *** reddy has joined #openstack-kolla | 06:09 | |
| *** mikelk has joined #openstack-kolla | 06:13 | |
| *** mdnadeem has quit IRC | 06:14 | |
| *** weiyu_ has joined #openstack-kolla | 06:21 | |
| *** sekrit has joined #openstack-kolla | 06:23 | |
| *** tfukushima has joined #openstack-kolla | 06:24 | |
| *** daneyon has joined #openstack-kolla | 06:26 | |
| *** andrewvwebber has joined #openstack-kolla | 06:27 | |
| *** kjelly_ has quit IRC | 06:28 | |
| *** coolsvap has quit IRC | 06:28 | |
| *** daneyon has quit IRC | 06:31 | |
| *** andrewvwebber has quit IRC | 06:31 | |
| *** dave-mcc_ has joined #openstack-kolla | 06:33 | |
| *** coolsvap has joined #openstack-kolla | 06:41 | |
| *** Kennan has joined #openstack-kolla | 06:45 | |
| *** achanda has quit IRC | 06:47 | |
| *** dave-mccowan has joined #openstack-kolla | 06:49 | |
| Jeffrey4l | sdake_, you means deploy ceph using kolla? | 06:51 |
| *** achanda has joined #openstack-kolla | 06:53 | |
| Jeffrey4l | sdake_, the answer is yes. | 06:53 |
| sdake_ | huh | 06:53 |
| sdake_ | oh ya i want to try out this 23. patch when i finish getting it working with ceph | 06:54 |
| sdake_ | but i dont have a ceph setup | 06:54 |
| *** dave-mcc_ has quit IRC | 06:54 | |
| Jeffrey4l | sdake_, so what can i do to help? | 06:55 |
| sdake_ | nothign yet | 06:56 |
| sdake_ | but fix bugs :) | 06:56 |
| sdake_ | but when i get it worign locally i'll let you know | 06:56 |
| sdake_ | then try a deploy with ceph | 06:56 |
| Jeffrey4l | sdake_, cool. btw, is there any bugs exist in ceph? | 06:56 |
| sdake_ | no clue | 06:57 |
| Jeffrey4l | roger | 06:57 |
| sdake_ | you might ramp up by trying out ceph | 06:57 |
| sdake_ | i should have patch ready by tomorrow | 06:57 |
| sdake_ | dealing with permissions issues on /var/lib/noa | 06:57 |
| Jeffrey4l | cool | 06:58 |
| sdake_ | just added qemu.conf to our templating | 06:58 |
| *** achanda has quit IRC | 07:00 | |
| sdake_ | do you know wher enamed volume sget their default permisions from? | 07:00 |
| Jeffrey4l | "get their defautl permission" what this? isnt the defautl permissions root? | 07:01 |
| *** athomas has quit IRC | 07:02 | |
| sdake_ | i dont know | 07:02 |
| *** athomas has joined #openstack-kolla | 07:02 | |
| sdake_ | i'd think docker would be smart enough to see the directory is owned by nova:nova and mount it as such | 07:02 |
| *** openstackgerrit has quit IRC | 07:02 | |
| *** openstackgerrit has joined #openstack-kolla | 07:03 | |
| sdake_ | did you look at that patch? | 07:03 |
| Jeffrey4l | sdake_, i do not know so. how can he know it should be owned by nova? | 07:03 |
| Jeffrey4l | nova may be easy, change the owner to the container running user. | 07:03 |
| sdake_ | let em push mylaet version in a moment | 07:03 |
| sdake_ | oh but your on ubuntu | 07:04 |
| Jeffrey4l | But how to handle the horizon? the apache running at root, however, horizon is runing under horizon user. | 07:04 |
| *** ccesario___ has quit IRC | 07:04 | |
| *** ccesario___ has joined #openstack-kolla | 07:04 | |
| sdake_ | horizon has nothing to do with libvirt ;) | 07:05 |
| *** Marga_ has quit IRC | 07:05 | |
| Jeffrey4l | sdake_, which patch? | 07:05 |
| sdake_ | moment i'll push a new version | 07:05 |
| Jeffrey4l | cool. | 07:05 |
| *** Jeffrey4l has left #openstack-kolla | 07:06 | |
| *** Jeffrey4l has joined #openstack-kolla | 07:06 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 07:08 |
| sdake_ | Jeffrey4l ^^ | 07:09 |
| Jeffrey4l | roger | 07:10 |
| sdake_ | not read yto go yet | 07:10 |
| Jeffrey4l | got. | 07:11 |
| sdake_ | but it changes ceph versions slightlhy | 07:11 |
| sdake_ | to oficially built images | 07:11 |
| *** salv-orlando has joined #openstack-kolla | 07:12 | |
| sdake_ | if you want to work on acritical bug right now | 07:12 |
| sdake_ | i hear linuxbridge is busted | 07:13 |
| sdake_ | that is 50% of our deployment s | 07:13 |
| Jeffrey4l | sdake_, is there any bug filed on the launchpad? | 07:14 |
| sdake_ | yes i think but not certian | 07:14 |
| sdake_ | if not file a new one | 07:14 |
| sdake_ | and i'l sort ou the dups | 07:14 |
| Jeffrey4l | sdake_, ok. I will try to deploy a AIO env to seem what happen | 07:15 |
| Jeffrey4l | sdake_, i am working on the rax-iad node issue. It is really annoying. I think I find the root cause. | 07:17 |
| sdake_ | nice | 07:21 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 07:23 |
| *** gfidente has joined #openstack-kolla | 07:23 | |
| *** mgoddard has joined #openstack-kolla | 07:25 | |
| *** andrewvwebber has joined #openstack-kolla | 07:28 | |
| *** jmccarthy1 has joined #openstack-kolla | 07:32 | |
| *** jmccarthy1 has left #openstack-kolla | 07:32 | |
| *** andrewvwebber has quit IRC | 07:32 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Handle multi Ethernets case in the gate https://review.openstack.org/290238 | 07:40 |
| *** jmccarthy has joined #openstack-kolla | 07:41 | |
| *** unicell has quit IRC | 07:47 | |
| *** andrewvwebber has joined #openstack-kolla | 07:50 | |
| *** shardy has joined #openstack-kolla | 07:51 | |
| *** shardy has quit IRC | 07:52 | |
| *** mbound has joined #openstack-kolla | 07:52 | |
| *** ccesario___ has quit IRC | 07:52 | |
| *** shardy has joined #openstack-kolla | 07:53 | |
| *** ccesario___ has joined #openstack-kolla | 07:53 | |
| *** mgoddard_ has joined #openstack-kolla | 07:59 | |
| *** achanda has joined #openstack-kolla | 08:00 | |
| *** mgoddard has quit IRC | 08:02 | |
| *** achanda has quit IRC | 08:05 | |
| *** brad[] has quit IRC | 08:26 | |
| *** ccesario___ has quit IRC | 08:26 | |
| *** brad[] has joined #openstack-kolla | 08:27 | |
| *** pmisiak has joined #openstack-kolla | 08:39 | |
| openstackgerrit | Merged openstack/kolla: Log the SHA of the Git ref for source builds https://review.openstack.org/301793 | 08:40 |
| Jeffrey4l | sdake_, ping | 08:41 |
| sdake_ | sup | 08:42 |
| sdake_ | i just got my patch working | 08:42 |
| *** dave-mccowan has quit IRC | 08:43 | |
| *** salv-orlando has quit IRC | 08:45 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 08:45 |
| sdake_ | Jeffrey4l sup dude | 08:45 |
| Jeffrey4l | sdake_, how could we handle the liberty branch now? I saw many backport from master to liberty. | 08:46 |
| Jeffrey4l | sdake_, shouldn't we wait for the `deploy liberty using mitaka`? | 08:47 |
| sdake_ | wait for what | 08:47 |
| openstackgerrit | Merged openstack/kolla: Add nova_ssh in cleanup containers script https://review.openstack.org/301648 | 08:47 |
| Jeffrey4l | sdake_, we will drop the liberty branch and use mitaka branch to deploy liberty code. is this right? | 08:47 |
| sdake_ | we will copy mitaka into liberty branch | 08:48 |
| sdake_ | make liberty deploy liberty instead of mitaka | 08:48 |
| sdake_ | not a great way to handle a stable branch | 08:48 |
| sdake_ | i'd liek o nver do that again | 08:49 |
| Jeffrey4l | sdake_, so we should stop backport code to the liberty branch, right? we need wait for the merge of `copy mitaka into liberty branch` | 08:49 |
| sdake_ | right | 08:49 |
| Jeffrey4l | sdake_, I know the root cause( yea, this is a not good idea.) | 08:49 |
| sdake_ | yes we should ignore liberty for the moemnt | 08:50 |
| Jeffrey4l | sdake_, But I saw may backport from master to liberty. may be we need make some annoucement for this now. | 08:50 |
| Jeffrey4l | s/may/many/ | 08:51 |
| sdake_ | master to mitaka | 08:51 |
| sdake_ | not to liberty | 08:51 |
| sdake_ | if they are in teh queue comment on them | 08:51 |
| *** pbourke has quit IRC | 08:51 | |
| Jeffrey4l | sdake_, https://review.openstack.org/301792 | 08:51 |
| Jeffrey4l | like this. | 08:51 |
| Jeffrey4l | got it. | 08:51 |
| *** pbourke has joined #openstack-kolla | 08:51 | |
| Jeffrey4l | I have make comment for serveral PS. | 08:51 |
| sdake_ | i see | 08:52 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla: Output image statuses to log instead of return value of main function https://review.openstack.org/278853 | 08:53 |
| sdake_ | Jeffrey4l can you test my patch please | 08:53 |
| sdake_ | https://review.openstack.org/302036 | 08:53 |
| sdake_ | you have ubuntu correct? | 08:53 |
| Jeffrey4l | sdake_, no. I only test the centos+source in my env. (A aio env) | 08:54 |
| sdake_ | i mean your 10 node cluster? | 08:54 |
| sdake_ | is centos? | 08:54 |
| Jeffrey4l | sdake_, right. | 08:54 |
| sdake_ | cool can you give it a spin | 08:54 |
| sdake_ | i believe yu ony hae to reuldi the *nova* containers | 08:55 |
| Jeffrey4l | I can not clean that env to do this test. that's a test env, some custom is evaluating it. :( | 08:56 |
| Jeffrey4l | sdake_, ^ | 08:56 |
| sdake_ | i see | 08:56 |
| Jeffrey4l | but, I can make a AIO test. | 08:56 |
| sdake_ | fiar enough | 08:56 |
| sdake_ | i need an uuntu test is what i really need - centoss aio works for me | 08:56 |
| sdake_ | but if you repicate that would make me feel better about it | 08:57 |
| sdake_ | its a significant last minute change | 08:57 |
| sdake_ | not sure how I feel aobut it for mitaka | 08:57 |
| Jeffrey4l | sdake_, you mean the OS of the host machina is Ubuntu rather than using ubuntu+source install? | 08:57 |
| sdake_ | both | 08:58 |
| sdake_ | but if your doing centos + centos/source thats ok too | 08:58 |
| sdake_ | i'll ask vikram o test ubuntu in the morning | 08:58 |
| pmisiak | Hi guys, can I use Ansible 2.0 with Kolla? | 08:58 |
| sdake_ | pmisiak no | 08:58 |
| Jeffrey4l | sdake_, I am cuoristy why we need that change? any issue it raise? especially in the qemu group part. | 08:58 |
| sdake_ | 1.9.4 is what we run with mostly | 08:58 |
| Jeffrey4l | pmisiak, no. | 08:58 |
| pmisiak | ok, thanks :) | 08:59 |
| sdake_ | Jeffrey4l we use qemu 1.5 currently | 08:59 |
| sdake_ | this goes to 2.3.0 | 08:59 |
| Jeffrey4l | sdake_, I know the qemu version part. I have no idea about that why we changed the qemu group. | 09:00 |
| Jeffrey4l | sdake_, I make a comment in you last PS. | 09:00 |
| sdake_ | which part of the patch | 09:00 |
| *** weiyu_ has quit IRC | 09:01 | |
| Jeffrey4l | sdake_, https://review.openstack.org/#/c/302036/3/ansible/roles/nova/templates/qemu.conf.j2 | 09:01 |
| patchbot | Jeffrey4l: patch 302036 - kolla - Use proper CentOS repositories for ceph and qemu | 09:01 |
| sdake_ | we change to group kolla | 09:02 |
| sdake_ | so /var/lib/nova/* can be written to | 09:02 |
| sdake_ | by qemu which runs as user qemu:kolla | 09:02 |
| Jeffrey4l | sdake_, if not, it will raise any issue? | 09:02 |
| *** weiyu_ has joined #openstack-kolla | 09:02 | |
| sdake_ | ya permission error starting vm | 09:03 |
| sdake_ | otherwise libvirt tries to write to /var/lib/nova as the non-nova user | 09:03 |
| sdake_ | previously qemu ran as root | 09:03 |
| Jeffrey4l | sdake_, but I never experience that | 09:03 |
| sdake_ | rather libvirt | 09:04 |
| sdake_ | and tarted instances as root | 09:04 |
| sdake_ | so it alway s had permissions to write everywhere | 09:04 |
| sdake_ | we could change the defualts to root;root in qemu.conf | 09:04 |
| sdake_ | then the permission changes arenot needed | 09:04 |
| sdake_ | but upstream docs recommend agains tthis | 09:05 |
| sdake_ | the defaults changed fro mqemu 1.5 to 2.3 | 09:05 |
| *** Marga_ has joined #openstack-kolla | 09:05 | |
| sdake_ | in 1.5 root:root | 09:05 |
| sdake_ | in 2.3 qemu;qemu | 09:05 |
| Jeffrey4l | Got. thanks for explaination. | 09:05 |
| Jeffrey4l | so ubuntu test is really neccessary. | 09:06 |
| sdake_ | right | 09:06 |
| sdake_ | and ceph since that ws requierd to be changed to work with new kvm | 09:06 |
| Jeffrey4l | ah, a little big change. :D | 09:07 |
| sdake_ | same version different build | 09:07 |
| sdake_ | but ya different nonetheless | 09:07 |
| sdake_ | if i would have known about this 6 months ago we woudl have changed it then | 09:08 |
| sdake_ | sbezverk todl me about it at 4;30 | 09:08 |
| sdake_ | i've been working on it for 8 hours | 09:08 |
| *** Marga_ has quit IRC | 09:10 | |
| Jeffrey4l | in my 10 nodes env(centos + source), i checked the qemu process user, it is qemu not root. | 09:10 |
| Jeffrey4l | sdake_, ^ | 09:11 |
| sdake_ | ok | 09:11 |
| Jeffrey4l | event though in the qemu.conf, the user and group is root with comment out. | 09:11 |
| sdake_ | well it doesn't work with that | 09:11 |
| sdake_ | i tried that first obviously :) | 09:11 |
| *** ccesario_ has joined #openstack-kolla | 09:12 | |
| Jeffrey4l | OK. anyway, i will try to deploy you PS first. thanks. | 09:12 |
| *** ccesario_ has quit IRC | 09:12 | |
| *** coolsvap has quit IRC | 09:13 | |
| *** salv-orlando has joined #openstack-kolla | 09:15 | |
| sdake_ | looks like it is not working | 09:15 |
| sdake_ | i cleaned it up efore i submitted it | 09:15 |
| sdake_ | and must have removed something crucial | 09:15 |
| Jeffrey4l | ok. cool. | 09:15 |
| sdake_ | why dont you try it | 09:16 |
| *** salv-orlando has quit IRC | 09:19 | |
| Jeffrey4l | sdake_, try your PS? I will try that. | 09:20 |
| *** wuhg has joined #openstack-kolla | 09:20 | |
| *** tfukushima has quit IRC | 09:28 | |
| *** mgoddard_ has quit IRC | 09:31 | |
| *** achanda has joined #openstack-kolla | 09:33 | |
| *** andrewvwebber has quit IRC | 09:35 | |
| *** mgoddard has joined #openstack-kolla | 09:37 | |
| *** achanda has quit IRC | 09:38 | |
| sdake_ | might be too big for mitaka | 09:39 |
| sdake_ | i'll have to think about it and consult with the team | 09:39 |
| sdake_ | i am outof steam | 09:39 |
| sdake_ | bedtime | 09:39 |
| sdake_ | night Jeffrey4l | 09:40 |
| sdake_ | report in patchset if yo uwouldn;t; mind your results | 09:40 |
| sdake_ | btw mistral isnt building | 09:40 |
| * sdake_ groans | 09:40 | |
| Jeffrey4l | sdake_, good night | 09:43 |
| sdake_ | i knwo why the patch wasnt working in my environment | 09:50 |
| sdake_ | i yum installed libvirt to check config otpoins | 09:50 |
| sdake_ | which started libvirt... lol | 09:50 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla: Output image statuses to log instead of return value of main function https://review.openstack.org/278853 | 09:52 |
| *** b_bezak has joined #openstack-kolla | 10:02 | |
| *** salv-orlando has joined #openstack-kolla | 10:02 | |
| sdake_ | nihgt all | 10:02 |
| *** sdake_ has quit IRC | 10:02 | |
| *** salv-orlando has quit IRC | 10:03 | |
| *** salv-orlando has joined #openstack-kolla | 10:03 | |
| *** salv-orlando has quit IRC | 10:18 | |
| pbourke | Jeffrey4l: didn't know the same problem was happening on centos | 10:21 |
| pbourke | wrt rabbitmq | 10:21 |
| *** weiyu_ has quit IRC | 10:21 | |
| pbourke | it fails locally also for me, but with a different startup error | 10:22 |
| *** salv-orlando has joined #openstack-kolla | 10:26 | |
| *** salv-orlando has quit IRC | 10:30 | |
| *** weiyu_ has joined #openstack-kolla | 10:32 | |
| *** achanda has joined #openstack-kolla | 10:35 | |
| *** achanda has quit IRC | 10:41 | |
| *** salv-orlando has joined #openstack-kolla | 10:47 | |
| *** salv-orlando has quit IRC | 10:48 | |
| *** ccesario_ has joined #openstack-kolla | 10:52 | |
| *** salv-orlando has joined #openstack-kolla | 10:57 | |
| *** salv-orlando has joined #openstack-kolla | 10:57 | |
| *** daneyon has joined #openstack-kolla | 10:57 | |
| *** salv-orlando has quit IRC | 10:57 | |
| *** salv-orlando has joined #openstack-kolla | 10:58 | |
| *** salv-orlando has quit IRC | 10:58 | |
| *** rhallisey has joined #openstack-kolla | 11:01 | |
| *** ccesario___ has joined #openstack-kolla | 11:01 | |
| *** daneyon has quit IRC | 11:01 | |
| *** ccesario_ has quit IRC | 11:05 | |
| *** weiyu_ has quit IRC | 11:13 | |
| *** Marga_ has joined #openstack-kolla | 11:15 | |
| *** Marga_ has quit IRC | 11:16 | |
| *** Marga_ has joined #openstack-kolla | 11:16 | |
| *** ccesario___ has quit IRC | 11:20 | |
| *** Serlex has joined #openstack-kolla | 11:24 | |
| *** reddy has quit IRC | 11:25 | |
| *** ccesario has quit IRC | 11:30 | |
| *** ccesario___ has joined #openstack-kolla | 11:30 | |
| *** reddy has joined #openstack-kolla | 11:37 | |
| *** Marga_ has quit IRC | 12:07 | |
| *** Serlex has quit IRC | 12:08 | |
| *** Jeffrey4l has quit IRC | 12:09 | |
| *** Marga_ has joined #openstack-kolla | 12:09 | |
| *** salv-orlando has joined #openstack-kolla | 12:10 | |
| *** reddy has quit IRC | 12:15 | |
| *** dwalsh has joined #openstack-kolla | 12:15 | |
| sbezverk | mandre ping | 12:17 |
| sbezverk | good morning | 12:18 |
| *** iceyao has quit IRC | 12:18 | |
| *** ccesario___ has quit IRC | 12:18 | |
| *** SiRiuS has joined #openstack-kolla | 12:18 | |
| *** wuhg has quit IRC | 12:19 | |
| *** ccesario___ has joined #openstack-kolla | 12:25 | |
| *** salv-orlando has quit IRC | 12:26 | |
| *** Liuqing has joined #openstack-kolla | 12:27 | |
| *** unicell has joined #openstack-kolla | 12:27 | |
| *** ccesario___ has quit IRC | 12:35 | |
| *** Marga_ has quit IRC | 12:36 | |
| *** Liuqing has quit IRC | 12:37 | |
| *** Liuqing has joined #openstack-kolla | 12:38 | |
| *** achanda has joined #openstack-kolla | 12:38 | |
| *** mwheckmann has joined #openstack-kolla | 12:39 | |
| *** Liuqing has quit IRC | 12:42 | |
| *** Liuqing has joined #openstack-kolla | 12:43 | |
| *** achanda has quit IRC | 12:43 | |
| *** Liuqing has quit IRC | 12:45 | |
| *** Liuqing has joined #openstack-kolla | 12:49 | |
| *** reddy has joined #openstack-kolla | 12:55 | |
| *** Serlex has joined #openstack-kolla | 12:55 | |
| *** ccesario has joined #openstack-kolla | 12:58 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: [WIP] Add Heka support https://review.openstack.org/286301 | 13:00 |
| *** banix has joined #openstack-kolla | 13:04 | |
| *** salv-orlando has joined #openstack-kolla | 13:09 | |
| *** sdake has joined #openstack-kolla | 13:15 | |
| sdake | morning | 13:15 |
| sbezverk | good morning | 13:15 |
| sbezverk | sdake I really like your qemu patch.. It fixed one issue I have been having. | 13:16 |
| sbezverk | Cannot access storage file '/var/lib/nova/instances/7aa33696-389a-4a90-a3b0-67591e94cb33/disk' (as uid:107, gid:107): Permission denied | 13:16 |
| sbezverk | since you changed group to kolla for user qemu, it works much better. | 13:17 |
| *** salv-orlando has quit IRC | 13:17 | |
| *** sdake_ has joined #openstack-kolla | 13:19 | |
| *** sdake has quit IRC | 13:19 | |
| *** ccesario has quit IRC | 13:22 | |
| *** sdake_ has quit IRC | 13:24 | |
| *** Liuqing has quit IRC | 13:26 | |
| *** sdake has joined #openstack-kolla | 13:27 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Add LVM2/iSCSI documentation https://review.openstack.org/300824 | 13:29 |
| *** Liuqing has joined #openstack-kolla | 13:33 | |
| mandre | hey sbezverk | 13:33 |
| sbezverk | mandre thank you for pinging me back | 13:34 |
| sbezverk | I saw your comment and wanted to explain the situation when this option is required. | 13:34 |
| mandre | yeah, I'm slowly catching up on what was merged recently and stumbled upon your commit | 13:35 |
| sbezverk | mandre when my repository contains multiple versions, it is the only way for me to tell kolla deploy which version to deploy.. I could not find any other way. | 13:35 |
| mandre | right, so nothing actually prevented you to add it in your local globals.yaml before | 13:37 |
| mandre | your change, if I remember correctly is to add back a commented out parameter, correct? | 13:38 |
| mandre | it's documentation only, and considered to be only for advanced users | 13:38 |
| sbezverk | mandre correct | 13:38 |
| mandre | if you read the comments on the patchset I linked in your issue, you'll see the reasons why it was removed | 13:39 |
| *** daneyon has joined #openstack-kolla | 13:39 | |
| *** Jeffrey4l has joined #openstack-kolla | 13:42 | |
| *** diogogmt has joined #openstack-kolla | 13:42 | |
| *** jtriley has joined #openstack-kolla | 13:43 | |
| *** iceyao has joined #openstack-kolla | 13:43 | |
| *** ccesario___ has joined #openstack-kolla | 13:44 | |
| *** daneyon has quit IRC | 13:44 | |
| sbezverk | mandre yep, I saw that but when I chatted with sdake actually it was him who suggested to restore it.. | 13:45 |
| sbezverk | and I agree :-) | 13:45 |
| *** ccesario has joined #openstack-kolla | 13:45 | |
| mandre | ok, I wanted to make sure you knew about the prior discussion | 13:46 |
| sbezverk | mandre sure, thank for brining it up. | 13:46 |
| *** mlima_ has joined #openstack-kolla | 13:47 | |
| *** mlima has quit IRC | 13:50 | |
| sdake | huh | 13:51 |
| sdake | mandre fwiw i think it should stay - so that operators may upstream their alpha streams as they see fit | 13:52 |
| sdake | but ya its docs only | 13:52 |
| *** inc0 has joined #openstack-kolla | 13:52 | |
| sdake | and it was me who requested it | 13:52 |
| sdake | my upstream/my update | 13:52 |
| sdake | holy shit I just used upstream as a veerb | 13:52 |
| sdake | someone shoot me | 13:52 |
| inc0 | my upstream, my update...sounds like title of a political manifesto | 13:53 |
| inc0 | hello | 13:53 |
| sdake | need anestaphine from the medlock! | 13:53 |
| sdake | hey inc0 | 13:53 |
| *** ccesario___ has quit IRC | 13:54 | |
| sdake | i modfieid kolla to use qemu 2.3.0 last night | 13:54 |
| inc0 | mein patch | 13:54 |
| sdake | not sure if we want it for mitaka or not | 13:54 |
| inc0 | probably not | 13:54 |
| sdake | its pianful but my thinking innitially is we ight as well get the pain over with | 13:54 |
| mandre | sdake: apparently Sam had concerns about potential issues during upgrades | 13:54 |
| inc0 | well, upgrade of qemu means restart of vms | 13:54 |
| sdake | mandre was talking about something else inc0 | 13:55 |
| mandre | hey inc0, long time no see :) | 13:55 |
| *** diogogmt has quit IRC | 13:55 | |
| inc0 | yeah, mandre looking at clock...is it France already?:) | 13:55 |
| mandre | inc0: this patch was reverted https://review.openstack.org/#/c/274408/ | 13:55 |
| patchbot | mandre: patch 274408 - kolla - Remove the openstack_release option in the globals... (MERGED) | 13:55 |
| sdake | inc0 ubuntu is on 2.5, centos is on 1.5 | 13:55 |
| sdake | the change moves centos to 2.3 | 13:55 |
| sdake | using official repos | 13:56 |
| mandre | inc0: indeed, I'm fully french now | 13:56 |
| mandre | bonjour tout le monde | 13:56 |
| sdake | to be fully french you have to hate the us people ;) | 13:56 |
| inc0 | sdake, I think british have more reasons to | 13:56 |
| *** diogogmt has joined #openstack-kolla | 13:56 | |
| sdake | they have gotten over the evolution | 13:57 |
| mandre | sdake, like I said :) | 13:57 |
| mandre | jk | 13:57 |
| inc0 | yeah I kinda think that openstack_release should be there | 13:58 |
| sdake | inc0 mandre bikeshed plz | 13:58 |
| inc0 | we are using reno for version and such | 13:58 |
| sdake | https://review.openstack.org/#/c/302036/ | 13:58 |
| patchbot | sdake: patch 302036 - kolla - Use proper CentOS repositories for ceph and qemu | 13:58 |
| sdake | sbezverk your welcome - 2am working on that one | 13:58 |
| mandre | same here, I don't really have objection on the patch itself, I just wanted to make sure sbezverk read the previous discussion about it, which he did | 13:59 |
| *** ccesario has quit IRC | 14:00 | |
| sdake | i didnt' | 14:00 |
| sbezverk | sdake there is one issue though | 14:00 |
| sdake | i guess i should | 14:00 |
| sdake | link? | 14:00 |
| sdake | 7am lazy tired can barely function | 14:00 |
| sdake | i could come up with more excuses if you like :) | 14:00 |
| sbezverk | sdake ImportError: No module named kolla.common | 14:00 |
| sbezverk | sdake but when I do pip install kolla/ | 14:00 |
| sbezverk | it starts working, build.py used to work without pip install kolla/ | 14:01 |
| sdake | git log it | 14:01 |
| sdake | that should work proeprly | 14:01 |
| mandre | sdake, where did the Oracle Linux gate go? | 14:01 |
| sdake | git log biuld.py | 14:01 |
| sdake | mandre there is no oracle gate atm that is running all the time | 14:02 |
| mandre | I remember we had an experimental one not so long ago | 14:02 |
| sdake | to trigger it i think ou leave check experimental | 14:02 |
| mandre | ok, good to know | 14:02 |
| sdake | note i doubt my patch workso n oracle linux | 14:03 |
| sdake | i hit the wall last night | 14:03 |
| sdake | so didn't test it | 14:03 |
| sdake | work to be done | 14:03 |
| mandre | I just triggered the experimental gate | 14:04 |
| mandre | we'll see | 14:04 |
| sdake | it definately wont work | 14:04 |
| sdake | the constraints in versionlock are different | 14:04 |
| sdake | this patch needs testing on ubuntu | 14:04 |
| sdake | anyone up for that | 14:04 |
| sdake | testing being does a vm boot | 14:05 |
| sbezverk | sdake build.py was changed august of last year. | 14:05 |
| sdake | sbezverk ok well you have some other problem then ;) | 14:05 |
| sdake | can only handle about 15 emergencies at once | 14:05 |
| sdake | at capacity ;) | 14:05 |
| sbezverk | sdake np I will dig into it.. | 14:06 |
| sdake | fwiw my main motive for making this change is rhos ships with qemu 2.3, and i want to match with what is going in rhel osp | 14:10 |
| sdake | i dont know i qemu 2.3 is really any better or not | 14:10 |
| sdake | sbezverk had some arguments for it but i forgot them alredy except they involved numa :) | 14:10 |
| sdake | if we decide not to make this change, I think we should make atleast qemu.conf configurable | 14:11 |
| sdake | as it is done now | 14:11 |
| sdake | that is a big wide ass gap in our config system | 14:11 |
| sdake | configurable/templateable | 14:11 |
| sdake | let me sort out an ol solution real quick | 14:12 |
| *** ccesario has joined #openstack-kolla | 14:13 | |
| mwheckmann | @sdake: FWIW: The newer versions of qemu (2.x+) that come from oVirt/RHEV derived CentOS repos are a must | 14:15 |
| sdake | mwheckmann mind adding some argumentatin to that assertion | 14:16 |
| sdake | so I have some ammo to support the change | 14:16 |
| mwheckmann | The main thing for us is advanced HyperV enlightenments for Windows VMs | 14:16 |
| mwheckmann | in particular, the hv_time one | 14:16 |
| mwheckmann | Openstack will take advantage of that automatically since Kilo | 14:16 |
| mwheckmann | we're talking 3x+ performance improvement for some CPU bound workloads on Windows | 14:17 |
| mwheckmann | ones that use a particular Windows perf counter | 14:17 |
| mwheckmann | There were a bunch of patches added to Nova around Kilo to take advantage of this stuff. | 14:18 |
| sdake | peple ue windows in openstack vms? | 14:18 |
| sdake | ;-) | 14:18 |
| mwheckmann | yes :) | 14:18 |
| sdake | ok anything else | 14:18 |
| mwheckmann | Some IO stuff that will benefit Linux VMs too. I don't remember specifically though. | 14:19 |
| mwheckmann | iothreads for one, but that's not turned on by Openstack automatically (yet) | 14:20 |
| mwheckmann | much improved virtio-scsi support I believe too. | 14:20 |
| sdake | virtio-scsi = win | 14:21 |
| mwheckmann | ? | 14:21 |
| sdake | mwheckmann you deploying kolla or just watching its progress? | 14:21 |
| sdake | i mean improving virtio is a good thing | 14:21 |
| *** diogogmt has quit IRC | 14:23 | |
| mwheckmann | @sdake: mostly watching for now. Using it for dev/test. | 14:24 |
| sdake | mwheckmann nice | 14:24 |
| mwheckmann | prod is still on traditionally managed Ansible. | 14:24 |
| sdake | mwheckmann some folks in the commuity want to mke hte dev/tewt cycle faster with kolla by bind mounting source trees optionally | 14:25 |
| sdake | any thoughts on that | 14:25 |
| mwheckmann | no thoughts unfortunatly. | 14:25 |
| sdake | ./build --devtest otheroptions | 14:25 |
| sdake | would build you a devtest enviornment with your source trees external | 14:26 |
| mwheckmann | sure. why not? | 14:26 |
| *** achanda has joined #openstack-kolla | 14:26 | |
| sdake | mwheckmann since you have played with kolla, can you tell me your top 3 pain points at present | 14:26 |
| mwheckmann | I really have no thoughts though. Right we're building packages from upstream repos using delorean | 14:27 |
| sbezverk | mwheckmann 100% agree about qemu 2.3 performance versus 1.5.X, for developers it is no big deal I guess ;-) but for people who uses opensatck for somthing produictive it matters a lot | 14:27 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 14:27 |
| mwheckmann | I barely started. So far everything went really smooth. The only thing was confusion about where to put config changes. I.e /etc/kolla/config/keystone.conf vs. /etc/kolla/keystone/keystone.conf (source vs. dest) | 14:28 |
| mwheckmann | pretty damn confusion | 14:28 |
| *** diogogmt has joined #openstack-kolla | 14:28 | |
| mwheckmann | confusing. | 14:28 |
| mwheckmann | I had to re-read that Ansible task a couple of times to figure out why it wasn't picking up my changes | 14:29 |
| mwheckmann | @sbezverk: Exactly. I'm not sure but I think all the NUMA pinning stuff might require the newer qemu as well. | 14:31 |
| sdake | mwheckmann ya docs issue, we know we need operational docs | 14:31 |
| sdake | anything else? | 14:31 |
| sdake | sbezverk do you have a ceph setup | 14:32 |
| sbezverk | mwheckmann vhost, vhost-user | 14:32 |
| sdake | this patch needs to be tested against a fresh ceph sestup | 14:32 |
| mwheckmann | PS: this is a great high level overview of some of the qemu differences between the vanilla CentOS version and the more advanced one: https://videos.cdn.redhat.com/summit2015/presentations/12752_red-hat-enterprise-virtualization-hypervisor-kvm-now-in-the-future.pdf | 14:32 |
| sdake | to make sure the centos ceph packages dont trigger breakage | 14:32 |
| mwheckmann | @sdake: no not yet, but I'll be sure to bring anything up here :) | 14:33 |
| sbezverk | sdake nope, but if it is not urgent I could work on it.. I need to confirm lvm2/iscsi conditional volumes first | 14:33 |
| mwheckmann | anyway, for Qemu, point is that a lot of the more advanced Openstack features depend on the newer version. | 14:34 |
| mwheckmann | daily stand-up time for me. back later. | 14:34 |
| sdake | mwheckmann thanks for the link thatsh elpful | 14:35 |
| *** reddy has quit IRC | 14:36 | |
| *** ccesario has quit IRC | 14:37 | |
| sdake | brb loggin itn ovpn | 14:38 |
| *** ccesario___ has joined #openstack-kolla | 14:41 | |
| *** sdake_ has joined #openstack-kolla | 14:42 | |
| *** sdake has quit IRC | 14:42 | |
| sdake_ | nothing like an 8am meeting which is cancelled at 7:40 | 14:42 |
| * sdake_ groans | 14:42 | |
| *** sdake_ is now known as sdake | 14:43 | |
| sdake | wtb acks https://review.openstack.org/#/c/300914/ | 14:44 |
| patchbot | sdake: patch 300914 - kolla - Increase max scheduling attempts to 10 | 14:44 |
| *** sdake_ has joined #openstack-kolla | 14:48 | |
| *** ccesario___ has quit IRC | 14:50 | |
| *** ccesario___ has joined #openstack-kolla | 14:50 | |
| *** sdake has quit IRC | 14:51 | |
| *** salv-orlando has joined #openstack-kolla | 14:51 | |
| *** achanda has quit IRC | 14:53 | |
| *** reddy has joined #openstack-kolla | 14:54 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 14:55 |
| sdake_ | mlima_ ping | 14:56 |
| *** pmisiak has quit IRC | 14:57 | |
| *** dougs1 has joined #openstack-kolla | 15:00 | |
| *** Jeffrey4l has quit IRC | 15:01 | |
| *** dougs has quit IRC | 15:01 | |
| *** mbound has quit IRC | 15:01 | |
| *** mbound has joined #openstack-kolla | 15:01 | |
| sdake_ | sbezverk do you have a ceph setup | 15:03 |
| sdake_ | sbezverk can you test that new patch versoin please | 15:03 |
| *** dougs has joined #openstack-kolla | 15:06 | |
| *** b_bezak has quit IRC | 15:06 | |
| *** Jeffrey4l has joined #openstack-kolla | 15:07 | |
| *** mark-casey has joined #openstack-kolla | 15:07 | |
| sdake_ | a cherrypick needing acks: https://review.openstack.org/#/c/301974/1 | 15:08 |
| patchbot | sdake_: patch 301974 - kolla (stable/mitaka) - Change waiting on mariadb.pid | 15:08 |
| *** mbound has quit IRC | 15:09 | |
| *** dougs1 has quit IRC | 15:09 | |
| *** blahRus has joined #openstack-kolla | 15:10 | |
| *** inc0 has quit IRC | 15:10 | |
| *** absubram has joined #openstack-kolla | 15:11 | |
| *** Jeffrey4l has quit IRC | 15:12 | |
| *** vhosakot has joined #openstack-kolla | 15:14 | |
| *** Liuqing has quit IRC | 15:15 | |
| vhosakot | morning! | 15:15 |
| *** dave-mccowan has joined #openstack-kolla | 15:15 | |
| mlima_ | i'm here sdake_ | 15:16 |
| sdake_ | mlima_ do you have a ceph setup available for testing? | 15:16 |
| mlima_ | yes | 15:16 |
| *** Marga_ has joined #openstack-kolla | 15:16 | |
| sdake_ | can you test a patch for me? I dont have my lab back together yet | 15:16 |
| mlima_ | yes, i can | 15:17 |
| sdake_ | https://review.openstack.org/302036 | 15:17 |
| mlima_ | but i have only aio | 15:17 |
| sdake_ | that wfm | 15:17 |
| sdake_ | vhosakot can you test that patch on ubuntu | 15:17 |
| *** daneyon has joined #openstack-kolla | 15:19 | |
| GonZo2000 | Hi on kibana which index patther should i configure ? | 15:19 |
| mlima_ | sdake_, i haven't centos available here | 15:19 |
| GonZo2000 | pattern* | 15:19 |
| vhosakot | sdake_: sure, can we setup loopback devices (losetup) for ceph disks similar to swift ? | 15:19 |
| *** dwalsh has quit IRC | 15:22 | |
| *** salv-orlando has quit IRC | 15:22 | |
| sbezverk | sdake will do the test and let you know.. | 15:23 |
| *** Jeffrey4l has joined #openstack-kolla | 15:24 | |
| sdake_ | mlima_ oh ok | 15:25 |
| sdake_ | GonZo2000 log-* | 15:25 |
| sdake_ | GonZo2000 we are working on a dashboard :) | 15:25 |
| sdake_ | vhosakot i just want yo uto test ubuntu works or not | 15:25 |
| sdake_ | not if ceph works | 15:26 |
| vhosakot | sdake_: ah ok... | 15:26 |
| sdake_ | the patch only afects centos ceph | 15:26 |
| sdake_ | and by works i mean nova boot works | 15:26 |
| sdake_ | and creates an active vm | 15:26 |
| vhosakot | sdake_: got it, will test https://review.openstack.org/302036 and post results in comments.. cool | 15:26 |
| mlima_ | well, i can do a centos deployment to test it sdake_ | 15:27 |
| *** sdake has joined #openstack-kolla | 15:29 | |
| *** dave-mcc_ has joined #openstack-kolla | 15:30 | |
| *** stvnoyes has joined #openstack-kolla | 15:30 | |
| *** sdake_ has quit IRC | 15:30 | |
| vhosakot | meeting in 1 hr | 15:31 |
| *** shardy has quit IRC | 15:31 | |
| *** dave-mccowan has quit IRC | 15:32 | |
| *** ccesario___ has quit IRC | 15:32 | |
| *** haplo37 has joined #openstack-kolla | 15:34 | |
| *** mikelk has quit IRC | 15:36 | |
| *** sdake_ has joined #openstack-kolla | 15:40 | |
| *** daneyon has quit IRC | 15:40 | |
| *** mlima has joined #openstack-kolla | 15:42 | |
| *** banix has quit IRC | 15:42 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 15:42 |
| *** sdake has quit IRC | 15:43 | |
| *** mlima_ has left #openstack-kolla | 15:43 | |
| *** banix has joined #openstack-kolla | 15:44 | |
| *** iceyao has quit IRC | 15:51 | |
| *** salv-orlando has joined #openstack-kolla | 15:53 | |
| *** ccesario has joined #openstack-kolla | 15:53 | |
| *** salv-orlando has quit IRC | 15:56 | |
| *** salv-orlando has joined #openstack-kolla | 15:58 | |
| gmmaha | hi, had a quick Q with ceph.. i deployed iwth ceph enabled, and did a tear down and re-deploy and bootstrap_ceph_mom container fails to start and logs show this. http://paste.openstack.org/show/493191/ | 16:02 |
| gmmaha | is there a special cleanup i have to do ? | 16:02 |
| *** stvnoyes has left #openstack-kolla | 16:03 | |
| mwheckmann | how are config subdirectories handled in Kolla? | 16:04 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Add iscsid/tgtd specific parameters to Kolla infrastructure https://review.openstack.org/300004 | 16:04 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Adds iscsid and tgtd docker containers https://review.openstack.org/291285 | 16:04 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Adds ansible code for iscsid and tgtd containers https://review.openstack.org/299995 | 16:04 |
| mwheckmann | I'm thinking of things like Keystone domain specific drivers: http://docs.openstack.org/developer/keystone/configuration.html#domain-specific-drivers | 16:05 |
| sdake_ | gmmaha ya you need to clean up oproperly | 16:05 |
| sdake_ | gmmaha the cleanup scripts dont do the job | 16:05 |
| gmmaha | sdake_: aaah | 16:05 |
| sdake_ | you need to remove the fstab entry for ceph | 16:05 |
| sdake_ | you need to delete /etc/kolla/*ceph* | 16:05 |
| gmmaha | aah i thought that part was added to the cleanup-host script. (the fstab cleanup) | 16:06 |
| gmmaha | i will check /etc/kolla/*ceph*' | 16:06 |
| mwheckmann | so I would want to have /etc/keystone/domains in my Keystone container | 16:06 |
| mwheckmann | similar for fernet_tokens: "key_repository = /etc/keystone/fernet-keys/" | 16:06 |
| gmmaha | mwheckmann: on the deploy host you can add contents to /etc/kolla/config/<service>.conf and it gets appended to the respective conf file | 16:07 |
| gmmaha | is that what you are loooking for> | 16:07 |
| mwheckmann | @gmmaha: no. I understand that part. | 16:07 |
| mwheckmann | but if I need to add a whole new subdir for config files | 16:07 |
| sdake_ | we hae not solved that scenario | 16:08 |
| mwheckmann | for example, configs for Keystone domain specific drivers need to be in a separate directory | 16:08 |
| sdake_ | what is the use case besides fernet? | 16:08 |
| sdake_ | so plugins is the use case? | 16:08 |
| mwheckmann | domain specific drivers | 16:08 |
| sdake_ | is that like a plugin? | 16:08 |
| mwheckmann | sort of. | 16:08 |
| sdake_ | I can assure you in newton we are going to solve the plugin prolem earlier rather hten later | 16:09 |
| mwheckmann | using different identity backends for different domains | 16:09 |
| sdake_ | it is my primary technical concern at the moemnt | 16:09 |
| mwheckmann | @sdake: is there already an approach in mind? | 16:09 |
| sdake_ | looking to the teeam for answers | 16:09 |
| sdake_ | but we are wrapping up mitaka | 16:10 |
| sdake_ | so not looking immediately :) | 16:10 |
| mwheckmann | ok. fair enough :) | 16:10 |
| *** dwalsh has joined #openstack-kolla | 16:11 | |
| mwheckmann | in the case, all that would be required is the ability to specify source directory (instead of individual files) for service configs | 16:11 |
| mwheckmann | and do a recursive copy. | 16:11 |
| mwheckmann | safely of course | 16:11 |
| dave-mcc_ | i want to set debug to True in nova.conf and update the nova-api container. but, kolla seems to override my config. is there an option i'm missing? | 16:11 |
| gmmaha | sdake_: thanks for the tip.. flushing /etc/kolla/*ceph* on the deployment targets did the job | 16:12 |
| mwheckmann | @dave-mcc_: you need to put your specific change in /etc/kolla/config/nova/nova.conf | 16:13 |
| sdake_ | gmmaha ya file bug plz | 16:13 |
| mwheckmann | it will then merge it with what is in the nova.conf template. | 16:13 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MITAKA BRANCH https://review.openstack.org/299182 | 16:13 |
| gmmaha | sdake_: done.. | 16:14 |
| *** daneyon has joined #openstack-kolla | 16:14 | |
| sdake_ | gmmaha you shoul dgive the qemu 2.3 work a spin | 16:16 |
| sdake_ | if your interested in performance | 16:16 |
| *** banix has quit IRC | 16:16 | |
| gmmaha | sdake_: for sure.. | 16:16 |
| sdake_ | need link? | 16:16 |
| gmmaha | yes please | 16:16 |
| sdake_ | https://review.openstack.org/302036 | 16:16 |
| gmmaha | sdake_: this only affects CentOS / RHEL > | 16:17 |
| gmmaha | ? | 16:17 |
| sdake_ | oh right | 16:17 |
| sdake_ | your using debuntian | 16:17 |
| sdake_ | that has qemu 2.5 already | 16:17 |
| gmmaha | Yup.. :) | 16:17 |
| sdake_ | nm then | 16:18 |
| gmmaha | sweet.. thanks | 16:18 |
| dave-mcc_ | mwheckmann thanks. i missed part of that path. | 16:19 |
| openstackgerrit | Merged openstack/kolla: Enable the kibana access from the external network https://review.openstack.org/302027 | 16:20 |
| gmmaha | sdake_: alright.. odd problem.. probably cause i dont know about ceph.. | 16:25 |
| gmmaha | i flushed /etc/kolla/*ceph*. re-deployed and it installed | 16:25 |
| gmmaha | but now disk space comes up as 0bytes | 16:26 |
| sdake_ | no idea | 16:27 |
| sdake_ | did you relabel? | 16:27 |
| gmmaha | dont think i did | 16:27 |
| gmmaha | aaah haaa | 16:27 |
| gmmaha | didnt realize i have to do that everytime | 16:28 |
| sdake_ | ceph is bust in my patch | 16:29 |
| * sdake_ groans | 16:29 | |
| sdake_ | but feel free to test ubuntu without ceph - i'd like to knwo that works | 16:29 |
| vhosakot | meeting in a bit | 16:29 |
| gmmaha | sdake_: works without ceph.. been doing that.. | 16:31 |
| *** thumpba has joined #openstack-kolla | 16:31 | |
| gmmaha | needed ceph to get more disk space for VMs | 16:31 |
| sdake_ | gmmaha we have our meeting now in openstack-meeting-4 so i'll have to get back to you but your welcome to join in :) | 16:31 |
| gmmaha | sdake_: for sure.. thanks.. | 16:32 |
| * gmmaha goes to join | 16:32 | |
| *** Marga_ has quit IRC | 16:34 | |
| *** Marga_ has joined #openstack-kolla | 16:37 | |
| *** SiRiuS has quit IRC | 16:38 | |
| *** SiRiuS has joined #openstack-kolla | 16:38 | |
| ccesario | a bit question guys... | 16:40 |
| ccesario | to setup an environment var, just put ENV VARNAME varvalu inside docker/base/Dockerfile.j2 and rebuild all images.... right!? | 16:40 |
| ccesario | I meant ... "a environment var in all containers " | 16:42 |
| *** jasonsb has quit IRC | 16:42 | |
| *** Marga_ has quit IRC | 16:44 | |
| *** ccesario___ has joined #openstack-kolla | 16:48 | |
| *** coolsvap has joined #openstack-kolla | 16:48 | |
| *** achanda has joined #openstack-kolla | 16:53 | |
| *** ccesario___ has quit IRC | 16:53 | |
| *** ccesario___ has joined #openstack-kolla | 16:53 | |
| *** Serlex has quit IRC | 16:55 | |
| *** banix has joined #openstack-kolla | 16:56 | |
| *** ayoung has quit IRC | 16:59 | |
| *** mgoddard has quit IRC | 17:03 | |
| *** mgoddard has joined #openstack-kolla | 17:03 | |
| *** banix has quit IRC | 17:06 | |
| mark-casey | I have a bare metal setup that will be "sort of production" (no customers on it, but in the future our devs will have a bad day if it's not working). I'm using MAAS inside Vagrant to PXE boot the target hosts, and will be attempting to run the private registry and the deployment tasks themselved via Vagrant too. I've been documenting this here: https://github.com/ropsoft/kolla_from_vagrant | 17:10 |
| mark-casey | I have no particular plans for this outside of our use but if its useful I'd be happy to contribute it/keep working on it. So far it's ubuntu-only but there are supported CentOS images for MAAS so it seemed like a good fit | 17:11 |
| *** salv-orlando has quit IRC | 17:12 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MITAKA BRANCH https://review.openstack.org/299182 | 17:13 |
| *** daneyon has quit IRC | 17:14 | |
| *** mdnadeem___ has joined #openstack-kolla | 17:14 | |
| *** banix has joined #openstack-kolla | 17:15 | |
| Jeffrey4l | pls stop to backport to liberty branch. Any PS to do this, pls -1 / -2 | 17:15 |
| *** mdnadeem___ has quit IRC | 17:15 | |
| Jeffrey4l | vhosakot, sbezverk let talk about the general_log . | 17:16 |
| vhosakot | sure... let me -2 first :) | 17:16 |
| *** mdnadeem___ has joined #openstack-kolla | 17:17 | |
| Jeffrey4l | so I want to revert that change in the master. what do you think? | 17:17 |
| Jeffrey4l | log every query to the ELK is not a good idea and meaningless in prod env. | 17:18 |
| vhosakot | Jeffrey4l: yes... I was thinking about it | 17:18 |
| *** mdnadeem___ has quit IRC | 17:18 | |
| vhosakot | Jeffrey4l: so, currently, mariadb logs just errros and nothing else.. is this fine ? | 17:18 |
| Jeffrey4l | If someone want to debug the sql query, just enable it and run `reconfigure` is enough. vhosakot | 17:18 |
| mlima | guys, https://review.openstack.org/299534 | 17:18 |
| mlima | review it | 17:19 |
| Jeffrey4l | vhosakot, I do not know that. I do think current log is a little less. | 17:19 |
| Jeffrey4l | But I am sure that log the query sql is not acceptable. | 17:19 |
| *** rajathagasthya has joined #openstack-kolla | 17:20 | |
| sbezverk | vhosakot on a busy opensatck cloud you will chew up disk space very quickly | 17:20 |
| vhosakot | Jeffrey4l: yes, current log is less.. agreed, general_log is bad.. I'm looking if mysql has logs with less verbosity than general_log | 17:20 |
| vhosakot | sbezverk: yes | 17:20 |
| Jeffrey4l | vhosakot, before you have now idea. I think we should revert the change in the master branch. | 17:21 |
| Jeffrey4l | s/now/new/ | 17:21 |
| sbezverk | vhosakot you should match to what regular opensatck default to | 17:21 |
| sdake_ | which change | 17:21 |
| *** sdake_ is now known as sdake | 17:21 | |
| Jeffrey4l | sdake, https://review.openstack.org/290238 | 17:21 |
| vhosakot | https://review.openstack.org/#/c/301975/1 | 17:21 |
| patchbot | vhosakot: patch 301975 - kolla (stable/mitaka) - Add general_log to mariadb container and make heka... | 17:21 |
| vhosakot | sbezverk: what is the default OpenStack setting ? | 17:21 |
| Jeffrey4l | sorry, I paste the wrong link sdake | 17:22 |
| vhosakot | sure, I can revert in master | 17:22 |
| vhosakot | but, is more logging needed ? | 17:22 |
| vhosakot | sdake: this is causing disk space issues - https://review.openstack.org/#/c/299133/ | 17:23 |
| patchbot | vhosakot: patch 299133 - kolla - Add general_log to mariadb container and make heka... (MERGED) | 17:23 |
| Jeffrey4l | vhosakot, cool appreciate that. | 17:23 |
| sdake | yes revert | 17:23 |
| vhosakot | ok cool | 17:23 |
| sdake | dont need a new bug id, just use git revert operation | 17:23 |
| vhosakot | ah ok, will do | 17:24 |
| vhosakot | Jeffrey4l: sdake what about this cehrry pick ? shall I abandon it ? | 17:24 |
| Jeffrey4l | vhosakot, more log can be acceptable, if you can find better solution. But query sql is not. | 17:24 |
| Jeffrey4l | vhosakot, abandon is enough. | 17:24 |
| vhosakot | Jeffrey4l: cool.. | 17:24 |
| sdake | ya abanodn | 17:24 |
| sdake | brb | 17:24 |
| vhosakot | coo.. thanks for the info... | 17:24 |
| vhosakot | Jeffrey4l: sbezverk:, I did not measure disk space .. good catch... I will revert it so that master is good :) | 17:25 |
| Jeffrey4l | vhosakot, cool | 17:26 |
| Jeffrey4l | coolsvap, ping | 17:26 |
| *** achanda has quit IRC | 17:27 | |
| Jeffrey4l | coolsvap, if you are back, please check the comments I made here https://review.openstack.org/297247 | 17:27 |
| sbezverk | vhosakot check this link, it talk about different level of logging in mariadb https://mariadb.com/kb/en/mariadb/log-files/ | 17:27 |
| coolsvap | Jeffrey4l: let me check | 17:28 |
| *** Marga_ has joined #openstack-kolla | 17:28 | |
| vhosakot | sbezverk: thanks, yep, that is where I found general_log :).... but, it is too verbose and noisy and eating up disk space... | 17:28 |
| Jeffrey4l | coolsvap, thanks. | 17:28 |
| vhosakot | I agree with Jeffrey4l.. let mariadb have just error logs, if mariadb issues are found, the operator needs to enable general_log and run mariadb reconfigure | 17:28 |
| Jeffrey4l | vhosakot, sbezverk i think no other log can be enabled except for error-log in the prod env. | 17:28 |
| vhosakot | Jeffrey4l: yes, error-log is good for prod (and gate as well) | 17:29 |
| *** sdake_ has joined #openstack-kolla | 17:29 | |
| *** sdake has quit IRC | 17:29 | |
| coolsvap | Jeffrey4l: i read that in mail notification, yeah we can abandon this change , I will have a look at your change as well | 17:29 |
| Jeffrey4l | coolsvap, thanks. | 17:29 |
| *** mwheckmann has quit IRC | 17:30 | |
| mark-casey | https://mariadb.com/kb/en/mariadb/binary-log-formats/#statement-based | 17:31 |
| mark-casey | I dunno if Maria does, but MySQL rotates binlogs for you... | 17:32 |
| vhosakot | mark-casey: thanks for the link... binray logs need to be decoded right... can a human read it.. say, an operator wants to read binary logs | 17:33 |
| mark-casey | perhaps, but in statement-based mode what you decode is the SQL that was run. In row mode you get little info... perhaps just what table and what rows changed and then a binary diff | 17:34 |
| mark-casey | but... as I put that I'm wondering if Galera would tolerate that | 17:34 |
| vhosakot | mark-casey: good point... I see all mysql setting can be used under the [mysqld] section in my.cnf for galera | 17:35 |
| vhosakot | mark-casey: but, don't know if this too ends up too verbose and eats up the disk | 17:35 |
| vhosakot | mariadb/mysql is one of the busiest process in any cloud... lots of DB operations happen to create just one VM... | 17:36 |
| vhosakot | mark-casey: I will test binary logs and post the results soon | 17:36 |
| Jeffrey4l | dave-mcc_, around? | 17:37 |
| mark-casey | vhosakot: expire_logs_days=1 would help but even that might not be quick enough on the cleanup | 17:37 |
| vhosakot | mark-casey: nice, let me check it | 17:38 |
| Jeffrey4l | about this PS https://review.openstack.org/300852 do you have any info/doc to explain the interface and track_interface? | 17:38 |
| Jeffrey4l | dave-mcc_, ^ | 17:38 |
| mark-casey | and cool. this as discussed should work. Its just that if you set it to statement-based logging and it breaks Galera, then yes... you could do all the same things with the harder to read logs. You get some good info but its nothing like the actual queries | 17:38 |
| Jeffrey4l | mark-casey, are you try to send the log to the ELK? if so, the expire_logs_days is not helpful. | 17:39 |
| dave-mcc_ | Jeffrey4l pong | 17:39 |
| Jeffrey4l | dave-mcc_, about this PS https://review.openstack.org/300852 do you have any info/doc to explain the interface and track_interface? | 17:39 |
| Jeffrey4l | dave-mcc_, pls check the comments i made in it. | 17:39 |
| vhosakot | mark-casey: I see | 17:40 |
| dave-mcc_ | Jeffrey4l yes... sorry, i wrote a reply, but i don't see it. i must have missed a step. | 17:40 |
| mark-casey | Jeffrey4l: oh I don't know. I just saw some logging discussion and I have to deal with that fairly frequently so I chimed in. | 17:40 |
| sbezverk | sdake I just finished testing iscsi with conditional volumes, it works in both scenarios when iscsi configured yes or no, in case NO no iscsi and tgtd containers are started and no volumes they create are mount to nova or cinder. | 17:40 |
| Jeffrey4l | dave-mcc_, LOL | 17:40 |
| dave-mcc_ | Jeffrey4l https://github.com/acassen/keepalived/blob/010cc929aa000b74afd52e2cbcdbf34334701f28/keepalived/include/vrrp.h#L299 | 17:41 |
| dave-mcc_ | there's the code that keepalived uses. it checks the primary first, then goes through the track list. | 17:41 |
| mark-casey | that would be right though. going that expire route would preclude them from going to ELK and still keeping the disk space down | 17:41 |
| sbezverk | sdake I starting to test it on ubuntu | 17:41 |
| *** daneyon has joined #openstack-kolla | 17:42 | |
| *** sdake has joined #openstack-kolla | 17:42 | |
| vhosakot | mark-casey: Jeffrey4l: yes, expired logs will reduce the logs collected by ELK drastically | 17:42 |
| *** SiRiuS has quit IRC | 17:42 | |
| dave-mcc_ | Jeffrey4l i also added the comment to the bug. | 17:43 |
| Jeffrey4l | dave-mcc_, also forgot the C language. need a little time to consume it. | 17:43 |
| dave-mcc_ | Jeffrey4l (at least i think i did) :-) | 17:43 |
| *** daneyon has quit IRC | 17:43 | |
| dave-mcc_ | Jeffrey4l me too. i removed c from my CV. :-) | 17:43 |
| Jeffrey4l | dave-mcc_, but, how a about another case? like me write a cause, wait for a seconds. | 17:43 |
| *** sdake_ has quit IRC | 17:44 | |
| Jeffrey4l | dave-mcc_, check this case http://paste.openstack.org/show/493213/ | 17:46 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 17:47 |
| Jeffrey4l | dave-mcc_, sorry, i am wrong. you are right, no need add the api_interface to the track_interface section. | 17:48 |
| dave-mcc_ | Jeffrey4l thanks for the discussion. hooray for peer reviews! it will be my turn to be wrong next time. ;-) | 17:50 |
| Jeffrey4l | vhosakot, no. the expired logs only reduce the logs on the hard disk. the ELK will keep it, unless you tell the elasticsearch to remove the log explicit. | 17:50 |
| sdake | pbourke ping | 17:50 |
| vhosakot | Jeffrey4l: won't expire_logs_days=1 make ELK see just one day's logs and not all the logs ? | 17:50 |
| Jeffrey4l | dave-mcc_, :D thanks | 17:51 |
| sdake | pbourke does oraclelinux mirror the virtualiation and storage sigs for centos for ole? | 17:51 |
| Jeffrey4l | expire_logs_days=1 is the mariadb parameter, right? vhosakot | 17:51 |
| vhosakot | yes | 17:51 |
| Jeffrey4l | why the elasticsearch need follow this parameter? | 17:52 |
| *** ccesario___ has quit IRC | 17:53 | |
| vhosakot | Jeffrey4l: ELK collects what mariadb writes when central_logging is enabled.. so, if mariadb re-writes logs every 24 hrs, ELK will not see logs older than24 hrs | 17:53 |
| vhosakot | Jeffrey4l: for now, I will revert master.. we can think about improving mariadb logs (if needed) later, and I will implement what is less verbose for mariadb | 17:54 |
| Jeffrey4l | vhosakot, that not true. ELK will log every line log mariadb write. when you remove the log file, the data wrote in the ELK never be removed. unless you tell the elasticsearch to remove the log explicit by using its API. | 17:55 |
| vhosakot | ah i see | 17:56 |
| Jeffrey4l | need go to bed. Bye guys. | 17:59 |
| *** achanda has joined #openstack-kolla | 17:59 | |
| vhosakot | good night Jeffrey4l | 17:59 |
| ccesario | good afternoon guys :) | 18:00 |
| Jeffrey4l | thanks. bye | 18:00 |
| *** Jeffrey4l has quit IRC | 18:05 | |
| *** daneyon has joined #openstack-kolla | 18:11 | |
| *** ccesario___ has joined #openstack-kolla | 18:12 | |
| vhosakot | I now see the mariadb <--> heka <--> elasticsearch connection in code... cool | 18:16 |
| *** daneyon has quit IRC | 18:16 | |
| *** inc0 has joined #openstack-kolla | 18:17 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 18:18 |
| *** thumpba has quit IRC | 18:29 | |
| *** inc0 has quit IRC | 18:29 | |
| *** inc0 has joined #openstack-kolla | 18:31 | |
| *** thumpba has joined #openstack-kolla | 18:31 | |
| *** mlima_ has joined #openstack-kolla | 18:34 | |
| *** coolsvap has quit IRC | 18:37 | |
| *** mwheckmann has joined #openstack-kolla | 18:37 | |
| *** mlima has quit IRC | 18:37 | |
| *** inc0 has quit IRC | 18:38 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Revert "Add general_log to mariadb container and make heka collect it" https://review.openstack.org/302401 | 18:38 |
| *** gfidente has quit IRC | 18:41 | |
| *** daneyon has joined #openstack-kolla | 18:42 | |
| ccesario | guys, could someone pls confirm it ? to setup an environment var, just put ENV VARNAME varvalue inside docker/base/Dockerfile.j2 and rebuild all images.... right!? | 18:43 |
| vhosakot | sbezverk: sdake: could you review the revert in master https://review.openstack.org/302401 | 18:46 |
| vhosakot | ccesario: put it in base if you want to affect all the containers... if not, I'f just put the ENV in the Dockerfile of the container that needs it... | 18:51 |
| vhosakot | ccesario: both the following shoud work | 18:51 |
| vhosakot | ENV key value | 18:51 |
| vhosakot | ENV key=value | 18:51 |
| ccesario | vhosakot, I just would confirm it, because even I doing it, and rebuild my images the ENV config does not show inside container | 18:53 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 18:53 |
| ccesario | vhosakot, thanks by info.... I will try discovery because this does not happen as expected | 18:54 |
| *** rajathagasthya has quit IRC | 18:55 | |
| vhosakot | ccesario: I see the ENV variable inside the container | 18:56 |
| vhosakot | $ docker exec neutron_server printenv | grep KOLLA | 18:56 |
| vhosakot | KOLLA_CONFIG_STRATEGY=COPY_ALWAYS | 18:56 |
| vhosakot | KOLLA_BASE_DISTRO=ubuntu | 18:56 |
| vhosakot | KOLLA_INSTALL_TYPE=source | 18:56 |
| vhosakot | KOLLA_INSTALL_METATYPE=mixed | 18:56 |
| vhosakot | ccesario: refer https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L9-L11 | 18:56 |
| manjeets | sdake i want to make one point here out of my observation | 18:57 |
| manjeets | when i am not using horizon doing everything thing through cli i don't see multiple ports and ip getting created for vms | 18:58 |
| manjeets | I see only single ip and port so far for upto 2000 vms when command is issued through cli | 18:58 |
| sdake | ya horizon creates overload | 18:59 |
| sdake | nova vm client does not | 18:59 |
| sdake | nova doesn't handle overload | 18:59 |
| ccesario | vhosakot, yes, I did exactly it above this line https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L11 I put ENV MYVAR value123 | 18:59 |
| sdake | i wrote write einteh bug what needs to be fixed | 18:59 |
| sdake | some variable or naother needs to be tuned | 18:59 |
| ccesario | vhosakot, then I run docker rmi -f on all images and rebuild the images again | 19:00 |
| manjeets | something might needs to be done how kolla handles horizon with osa horizon does not do this | 19:00 |
| sdake | they hve tuned around it | 19:01 |
| sdake | which is what we are gong to do | 19:01 |
| sdake | as soon as i hae two seconds to come up for breath i'll do the work | 19:01 |
| ccesario | tools/build.py .... and re-deploy ..... the images are deployed without this var..... untouched | 19:01 |
| *** inc0 has joined #openstack-kolla | 19:02 | |
| sdake | manjeets how long will you be around today | 19:06 |
| sdake | if your available now i can spend 2 hours on your problem until i take daughter to dr | 19:07 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 19:07 |
| sdake | manjeets that is an sqlalchemy default | 19:15 |
| *** ayoung has joined #openstack-kolla | 19:16 | |
| sdake | manjeets http://docs.sqlalchemy.org/en/latest/core/pooling.html | 19:18 |
| *** SiRiuS has joined #openstack-kolla | 19:19 | |
| openstackgerrit | Merged openstack/kolla: Start the haproxy by using haproxy-system-wrapper https://review.openstack.org/294701 | 19:23 |
| *** cbits has joined #openstack-kolla | 19:26 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Increase max pool size so conductor doesn't implode https://review.openstack.org/302415 | 19:32 |
| *** daneyon has quit IRC | 19:33 | |
| sdake | manjeets there you go give that a spin | 19:33 |
| dave-mcc_ | sdake I patched 302036 on my code, but i still can boot VMs. is that the only patch needed? | 19:34 |
| sdake | dave-mcc_ it may be broken atm | 19:35 |
| sdake | i am working on it | 19:35 |
| sdake | can only unfortunately do one thing at a time | 19:35 |
| *** salv-orlando has joined #openstack-kolla | 19:37 | |
| sdake | dave-mcc_ i'll lety ou kno whwne 302306 works in my enviornment with ceph | 19:40 |
| *** cbits has left #openstack-kolla | 19:44 | |
| *** salv-orlando has quit IRC | 19:44 | |
| *** rajathagasthya has joined #openstack-kolla | 19:45 | |
| *** thumpba has quit IRC | 19:47 | |
| *** mlima__ has joined #openstack-kolla | 19:50 | |
| *** mlima_ has quit IRC | 19:53 | |
| *** salv-orlando has joined #openstack-kolla | 19:54 | |
| *** salv-orlando has joined #openstack-kolla | 19:55 | |
| *** daneyon has joined #openstack-kolla | 20:07 | |
| vhosakot | sbezverk: ping | 20:11 |
| *** banix has quit IRC | 20:11 | |
| *** daneyon has quit IRC | 20:12 | |
| *** mat128 has joined #openstack-kolla | 20:13 | |
| *** mat128 has left #openstack-kolla | 20:19 | |
| *** SiRiuS has quit IRC | 20:22 | |
| *** mbound has joined #openstack-kolla | 20:25 | |
| *** mbound has quit IRC | 20:25 | |
| *** mbound has joined #openstack-kolla | 20:25 | |
| rajathagasthya | Hi! I can't see any logs in the Kibana dashboard. I'm see this error when I do docker logs heka: http://paste.openstack.org/show/493232/. Any idea how to resolve that? | 20:30 |
| rajathagasthya | FYI, Elasticsearch is reachable from the heka container. curl 172.28.128.254:9200 work fine. | 20:31 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 20:31 |
| sbezverk | vhosakot pong | 20:35 |
| vhosakot | rajathagasthya: those error does not look related to kibana... can you curl the kibana port ? curl -v 172.28.128.254:5601 | 20:36 |
| rajathagasthya | vhosakot: Kibana works fine, I can access the dashboard from browser. Problem is heka isn't sending logs to ES. | 20:37 |
| *** sdake_ has joined #openstack-kolla | 20:37 | |
| vhosakot | sbezverk: log-error is still there, it as just mariadb in file_match since there is log-error=/var/log/kolla/mariadb/mariadb.log in galera.cnf | 20:38 |
| *** sdake has quit IRC | 20:39 | |
| vhosakot | sbezverk: I did "git revert" and this is how it was before https://review.openstack.org/#/c/299133/ was merged | 20:39 |
| patchbot | vhosakot: patch 299133 - kolla - Add general_log to mariadb container and make heka... (MERGED) | 20:39 |
| vhosakot | rajathagasthya: oh ok.. so, you see no logs in Kiabna dashboard ? | 20:40 |
| rajathagasthya | vhosakot: Nope, and that's because heka can't connect to ES. It's strange since I can reach ES from inside Heka container. | 20:40 |
| vhosakot | rajathagasthya: keystone_apache_logstreamer_input is failing in heka | 20:41 |
| vhosakot | rajathagasthya: can you send the full output of "docker logs heka" | 20:41 |
| rajathagasthya | vhosakot: Sure, one moment. | 20:42 |
| sbezverk | vhosakot ok, so you will keep mariadb erro log, right? if it is the case I will change my review right away | 20:42 |
| *** sdake has joined #openstack-kolla | 20:42 | |
| *** ccesario___ has quit IRC | 20:42 | |
| rajathagasthya | vhosakot: http://paste.openstack.org/show/493234/ | 20:43 |
| vhosakot | sbezverk: yep, log-error is not removed... the name of the file is not mariadb-error.log anymore, as the revert made it mariadb.log and it was like mariadb.log before... | 20:43 |
| *** sdake_ has quit IRC | 20:44 | |
| vhosakot | sbezverk: file_match = 'mariadb/mariadb\.log\.?(?P<Seq>\d*)$' was there before https://review.openstack.org/#/c/302401/1 was merged | 20:44 |
| patchbot | vhosakot: patch 302401 - kolla - Revert "Add general_log to mariadb container and m... | 20:44 |
| sbezverk | vhosakot got it | 20:44 |
| vhosakot | sbezverk: are you saying we need name the file mariadb-error.log instead of mariadb.log as they are error logs ? | 20:44 |
| vhosakot | elemoine_ names error logs as mariadb.log as part of central logging | 20:45 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Set db connection retry to infinity https://review.openstack.org/302434 | 20:45 |
| vhosakot | sbezverk: thanks for reverting -1.. should we have mariadb-error.log as the name as it is more approporiate ? | 20:46 |
| vhosakot | sbezverk: ^^ | 20:46 |
| sbezverk | vhosakot no it is fine, I double check my normal openstack and it uses mariadb.log | 20:47 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add etcd Docker container https://review.openstack.org/298451 | 20:49 |
| elemoine_ | rajathagasthya: does the elasticsearch_output error keep repeating in the Heka logs? If no then these occurs just occurred because ES was not yet started when Heka started. If they repeat then it means Heka cannot connect to ES... | 20:50 |
| vhosakot | sbezverk: ah ok... thanks for checking... | 20:50 |
| vhosakot | sbezverk: cool, mariadb.log name is good... the operator will know from galera.cnf that mariadb.log is written from the log-error setting | 20:51 |
| *** salv-orl_ has joined #openstack-kolla | 20:51 | |
| rajathagasthya | elemoine_: This doesn't repeat. But how do I control which container starts first? | 20:51 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add etcd Docker container https://review.openstack.org/298451 | 20:52 |
| *** ccesario___ has joined #openstack-kolla | 20:53 | |
| vhosakot | rajathagasthya: when you say you can reach ES from inside heka container, you mean "docker exec heka curl curl 172.28.128.254:9200" works ? | 20:53 |
| rajathagasthya | vhosakot: Yep, that's what I meant. | 20:54 |
| elemoine_ | rajathagasthya: Heka container is started by the common role, so Ansible will start it before the ES container. Hence the few ES output errors we get when Heka starts. | 20:54 |
| *** salv-orlando has quit IRC | 20:55 | |
| rajathagasthya | elemoine_: So in that case, I will eventually get the logs in Kibana? | 20:55 |
| elemoine_ | rajathagasthya: so it sounds like that Heka does send logs to Elasticsearch | 20:55 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add etcd Docker container https://review.openstack.org/298451 | 20:55 |
| elemoine_ | rajathagasthya: Yes, just select log-* as the log index pattern in Kibana | 20:56 |
| rajathagasthya | elemoine_: Yeah, that's what it looks like to me as well. All the logs are present in heka is /var/log/kolla/ | 20:56 |
| elemoine_ | rajathagasthya: that does not mean Heka collects them and send them to Elasticsearch | 20:57 |
| *** ccesario___ has quit IRC | 20:57 | |
| *** ccesario__ has joined #openstack-kolla | 20:57 | |
| elemoine_ | rajathagasthya: most services write their logs to this directory, and Heka reads them from there | 20:57 |
| elemoine_ | rajathagasthya: so that dir would include logs even if Heka did not work | 20:58 |
| rajathagasthya | elemoine_: Okay, now I get it. I missed the log-* index creation part. :) | 20:59 |
| *** ccesario__ has quit IRC | 20:59 | |
| rajathagasthya | elemoine_: Thanks! | 21:00 |
| elemoine_ | rajathagasthya: cool, we know Kolla needs default Kibana dashboards! | 21:00 |
| rajathagasthya | elemoine_: That's what I'm looking at right now. To have default Kibana dashboard. And I wanted to discuss a couple of things regarding how to do that. | 21:01 |
| elemoine_ | rajathagasthya: please suggest dashboards if you come with nice and useful ones :) | 21:01 |
| rajathagasthya | elemoine_: I thought we were going to do this? https://github.com/openstack/fuel-plugin-elasticsearch-kibana/blob/master/deployment_scripts/puppet/modules/lma_logging_analytics/templates/kibana_dashboards/logs.json | 21:03 |
| rajathagasthya | As a starter, at least. | 21:03 |
| elemoine_ | yes it's a source of inspiration, but those are Kibana 3 dashboards | 21:03 |
| elemoine_ | and we probably don't need more complex dashboards at this point | 21:04 |
| elemoine_ | go for it! | 21:04 |
| elemoine_ | bed time for me, bye | 21:04 |
| rajathagasthya | elemoine_: Sure, goodnight. | 21:05 |
| vhosakot | elemoine_: thanks bye rajathagasthya: what is this log-* index creation part ? is it done in code on the dashboard.. is it documented ? | 21:05 |
| rajathagasthya | vhosakot: No, it's not documented and right now I think we have to do it manually in Kibana. But when we introduce default dashboard, we don't need to do it. It redirects automatically. | 21:06 |
| vhosakot | ah ok, until the default dashboard is implemented, I think we need to document the steps to create the log-* index in the Kibana dashboard.. | 21:08 |
| vhosakot | rajathagasthya: would you mind adding documentation for that in https://github.com/openstack/kolla/blob/master/specs/logging-with-heka.rst... | 21:09 |
| vhosakot | rajathagasthya: if you're busy, could you send the steps, I can add them, and add you as co-author on the patch set | 21:10 |
| rajathagasthya | vhosakot: Sure, I can do that. | 21:11 |
| vhosakot | rajathagasthya: nice! thanks.. so, in https://github.com/openstack/kolla/blob/master/specs/logging-with-heka.rst, please add all the steps to create the log-* index in the Kibana dashboard... | 21:12 |
| vhosakot | rajathagasthya: is there a bug/blueprint that tracks the default Kiabna dashboard you're working on ? | 21:12 |
| *** ccesario__ has joined #openstack-kolla | 21:12 | |
| rajathagasthya | vhosakot: No, there's no bug that tracks it. | 21:13 |
| vhosakot | cool, I will check sdake and keep you psted | 21:13 |
| sdake | file one | 21:13 |
| sdake | blueprint | 21:13 |
| sdake | plz | 21:13 |
| vhosakot | sdake: ok, will do.. thanks | 21:14 |
| rajathagasthya | vhosakot: I think it's better to document those Kibana steps in quickstart.rst (Debugging Kolla section). The file you mentioned is more of a spec file. What do you think? | 21:15 |
| vhosakot | rajathagasthya: yes | 21:15 |
| sdake | we need to fix our docs with sudo | 21:15 |
| sdake | it makes it look like we always run as root | 21:15 |
| sdake | which i sure dont | 21:15 |
| vhosakot | rajathagasthya: add in quiskstart as it is what operators refer than specs | 21:15 |
| sdake | nobody looks at specs after they hit the repository | 21:16 |
| vhosakot | sdake: running docs as root ? what is that ? | 21:16 |
| rajathagasthya | vhosakot: Cool. | 21:16 |
| vhosakot | rajathagasthya: so, after creating the log-* index in the Kibana dashboard, do you see all the logs ? | 21:16 |
| vhosakot | in the Kibana dashboard ? | 21:16 |
| *** ccesario__ has quit IRC | 21:18 | |
| rajathagasthya | vhosakot: Correct. | 21:18 |
| *** ccesario__ has joined #openstack-kolla | 21:19 | |
| vhosakot | rajathagasthya: including keystone logs that threw errors in "docker logs heka" ? | 21:19 |
| *** inc0 has quit IRC | 21:20 | |
| rajathagasthya | vhosakot: No, the error logs were thrown by heka process, not Keystone. I don't know if those are collected as well. | 21:21 |
| vhosakot | rajathagasthya: cool, I will check that.. thanks for all the help! | 21:22 |
| rajathagasthya | vhosakot: No problem! | 21:22 |
| *** jtriley has quit IRC | 21:24 | |
| *** ccesario___ has joined #openstack-kolla | 21:29 | |
| *** ccesario__ has quit IRC | 21:29 | |
| mark-casey | could anyone tell me where to specify the ssh private key so 'kolla-ansible prechecks' will use it? | 21:33 |
| sdake | dave-mcc_ your using --flavor 2 to launch your instances correct? | 21:34 |
| sdake | mark-casey you put it in authorized_keys on every node (the public key ofy our private key) | 21:35 |
| sdake | in the .ssh directory | 21:35 |
| sdake | make sure its mode 600 | 21:35 |
| sdake | and owned by the user | 21:35 |
| sdake | typically root | 21:35 |
| sdake | since kolla-ansible atm unforutnately must run as root | 21:35 |
| dave-mcc_ | sdake i chose "medium" from horizon | 21:35 |
| sdake | dave-mcc_ hmm | 21:35 |
| sdake | i dont use horizon to boot | 21:36 |
| mark-casey | sdake: thx. so it has to be id_rsa or I can give it some other name? | 21:36 |
| sdake | mark_casey you copy your id_rsa.pub to authorized_key or authorized_keys on every node in the .ssh dir | 21:36 |
| sdake | i dont recall which unfortunately | 21:37 |
| *** ccesario___ has quit IRC | 21:37 | |
| mark-casey | right I have that part | 21:37 |
| sdake | google internet for ssh setup there is probaboy a bajilliona rticles | 21:37 |
| dave-mcc_ | sdake i have a tiny cirros image i'm using. should i try a different image or flavor? | 21:37 |
| sdake | if your running under sudo, it needs to be in your root directory as well | 21:37 |
| sdake | dave-mcc_ moment | 21:37 |
| mark-casey | ...bah | 21:38 |
| mark-casey | I see what I did | 21:38 |
| sdake | so the flavors are gone apparently | 21:38 |
| mark-casey | I'm trying to run my deployer inside docker and it dropped me into / not /root | 21:38 |
| sdake | in maste ratleast | 21:38 |
| mark-casey | thx | 21:38 |
| sdake | this was posted to the ml | 21:38 |
| sdake | i didn't think they would do so fast | 21:38 |
| sdake | gotta take daughter to dr bbl | 21:39 |
| *** ayoung has quit IRC | 21:39 | |
| vhosakot | sdake: rajathagasthya: https://blueprints.launchpad.net/kolla/+spec/default-kibana-dashboard | 21:41 |
| vhosakot | rajathagasthya: https://discuss.elastic.co/t/how-to-set-a-dashboard-as-default-in-kibana-4-2/33662 | 21:43 |
| rajathagasthya | vhosakot: This is what we'd want to do: https://github.com/openstack/fuel-plugin-elasticsearch-kibana/blob/master/deployment_scripts/puppet/modules/lma_logging_analytics/manifests/kibana_dashboard.pp#L35-L36 | 21:44 |
| vhosakot | rajathagasthya: ah ok... | 21:46 |
| *** sdake has quit IRC | 21:54 | |
| *** daneyon has joined #openstack-kolla | 21:56 | |
| *** dwalsh has quit IRC | 21:57 | |
| vhosakot | mark-casey: http://docs.ansible.com/ansible/intro_getting_started.html#your-first-commands has steps to setup passwordless SSH for ansible.. basically, add public SSH key of deploy node in authorized_keys on each target host.. | 21:58 |
| vhosakot | mark-casey: ansible also has --private-key option to specify a pem file instead | 21:58 |
| vhosakot | mark-casey: http://docs.ansible.com/ansible/intro_configuration.html#private-key-file | 21:59 |
| *** ccesario___ has joined #openstack-kolla | 22:00 | |
| *** daneyon has quit IRC | 22:01 | |
| mark-casey | vhosakot: thank you. I wasn't seeing an equivalent to ansible-playbook's --private-key option in kolla-ansible, but I | 22:02 |
| mark-casey | *i've just given the key the default id_rsa name and it found it :) | 22:02 |
| vhosakot | mark-casey: kolla-ansible has then -e option to pass Ansible variables https://github.com/openstack/kolla/blob/master/tools/kolla-ansible#L37... dont know if it can be used to pass the private key location thru to Ansible | 22:03 |
| vhosakot | mark-casey: I see it in the output of :ansible-playbook --help" | 22:05 |
| vhosakot | $ ansible-playbook --help | grep private | 22:05 |
| vhosakot | --private-key=PRIVATE_KEY_FILE | 22:05 |
| vhosakot | mark-casey: so, you could try: kolla-ansible deploy -e private-key=~/.ssh/id_rsa | 22:06 |
| mark-casey | vhosakot: cool thx. I will | 22:07 |
| vhosakot | mark-casey: cool, let me know if it works.. may be worth adding documentation for it | 22:08 |
| vhosakot | mark-casey: thanks! | 22:08 |
| *** stvnoyes has joined #openstack-kolla | 22:14 | |
| *** haplo37 has quit IRC | 22:23 | |
| mark-casey | vhosakot: ping | 22:31 |
| vhosakot | mark-casey: pong | 22:31 |
| *** achanda has quit IRC | 22:32 | |
| mark-casey | It didn't like that option for the key, but it is working well enough for now as is. I did have another question... did you all ever solve... "APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/images/create?tag=2.0.0&fromImage=controller%3A4000%2Fkollaglue%2Fubuntu-source-heka',),)]" | 22:32 |
| mark-casey | I got that error and saw it in http://eavesdrop.openstack.org/irclogs/%23kolla/%23kolla.2016-03-18.log.html | 22:32 |
| mark-casey | re: TASK: [common | Starting heka container] ************************************** | 22:33 |
| vhosakot | mark-casey: share the /run mount | 22:33 |
| vhosakot | mount --make-shared /run | 22:33 |
| vhosakot | on UBuntu | 22:33 |
| vhosakot | mark-casey: search "mount" in http://docs.openstack.org/developer/kolla/quickstart.html | 22:33 |
| mark-casey | ahhhh. ok. I must have scrolled past it. I'd made a mental note to come back to that and I guess it got erased :D | 22:34 |
| mark-casey | thanks | 22:34 |
| vhosakot | cool, np | 22:34 |
| vhosakot | mark-casey: I'm curious do you use curl/urlib to search in kolla logs, or manually search | 22:34 |
| vhosakot | I meant in kolla IRC logs | 22:35 |
| openstackgerrit | Rajath Agasthya proposed openstack/kolla: Added Kibana instructions to quickstart.rst https://review.openstack.org/302458 | 22:35 |
| mark-casey | vhosakot: usually google :/ | 22:37 |
| mark-casey | kubernetes site:http://eavesdrop.openstack.org/irclogs/#kolla/ | 22:37 |
| mark-casey | oops, that was supposed to be one whole thing | 22:37 |
| vhosakot | cool | 22:38 |
| *** achanda has joined #openstack-kolla | 22:38 | |
| *** cbits has joined #openstack-kolla | 22:40 | |
| manjeets | sdake vhosakot inc0 i am going to install openstack using linux bridge instead ovs | 22:40 |
| manjeets | should i cleanup registry ? | 22:40 |
| manjeets | or cleanning up images and containers is enough | 22:41 |
| *** daneyon has joined #openstack-kolla | 22:46 | |
| *** vhosakot has quit IRC | 22:46 | |
| sbezverk | manjeets I do not think you need to clean up the registrym you will be using the same images with the exception of neutron for linux bridge | 22:51 |
| sbezverk | just make sure it was built and you can see in the list | 22:51 |
| sbezverk | docker images | grep -i linux | 22:51 |
| manjeets | sbexverk thanks giving it a shot | 22:54 |
| *** mwheckmann has quit IRC | 22:55 | |
| *** pabelanger has joined #openstack-kolla | 22:56 | |
| sbezverk | manjeets good luck! | 22:57 |
| pabelanger | Greetings, wanted to see when kolla is expecting to tag its mitaka release | 22:57 |
| pabelanger | over in openstack-infra we're looking at migrating your devstack-centos7 jobs to centos-7 dibs. Which requires some downtime | 22:58 |
| *** wmiller has quit IRC | 23:05 | |
| *** wmiller has joined #openstack-kolla | 23:06 | |
| *** blahRus has quit IRC | 23:07 | |
| *** cbits has quit IRC | 23:08 | |
| *** ayoung has joined #openstack-kolla | 23:09 | |
| *** mgoddard has quit IRC | 23:09 | |
| *** mgoddard has joined #openstack-kolla | 23:10 | |
| *** achanda has quit IRC | 23:15 | |
| *** cbits has joined #openstack-kolla | 23:18 | |
| *** achanda has joined #openstack-kolla | 23:22 | |
| manjeets | sbezverk what other changes i need to do other than modifying global.yml and all,yml | 23:26 |
| manjeets | i had linuxbridge in those but in deployment i still se neutron ovs agent got started | 23:26 |
| manjeets | sbezverk i just got success how do i make sure linux bridge is done properly | 23:28 |
| *** rajathagasthya has quit IRC | 23:28 | |
| manjeets | http://paste.openstack.org/show/493250/ | 23:29 |
| manjeets | i guess its configured properly | 23:29 |
| *** ccesario___ has quit IRC | 23:34 | |
| *** ccesario___ has joined #openstack-kolla | 23:35 | |
| *** salv-orl_ has quit IRC | 23:36 | |
| *** sdake has joined #openstack-kolla | 23:44 | |
| *** rhallisey has quit IRC | 23:45 | |
| *** sdake_ has joined #openstack-kolla | 23:46 | |
| *** sdake has quit IRC | 23:49 | |
| *** sdake has joined #openstack-kolla | 23:50 | |
| sdake | good news is ceph works single node with my qemu 2.3 patch | 23:51 |
| *** sdake_ has quit IRC | 23:52 | |
| * manjeets just got deploy with lb works | 23:54 | |
| sdake | manjeets did you try that patch | 23:56 |
| sdake | the one i posted for the bug you reported wit hte horizon 2000 vms | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!