Tuesday, 2016-04-12

*** banix has quit IRC00:05
*** mark-casey has quit IRC00:09
*** mark-casey has joined #openstack-kolla00:10
*** mwheckmann has quit IRC00:14
*** ccesario_ has joined #openstack-kolla00:19
ccesario_night!00:19
*** jtriley has joined #openstack-kolla00:26
*** jtriley has quit IRC00:40
*** daneyon_ has joined #openstack-kolla00:41
*** daneyon__ has joined #openstack-kolla00:42
*** rhallisey has quit IRC00:44
*** daneyon has quit IRC00:45
*** daneyon_ has quit IRC00:46
*** jtriley has joined #openstack-kolla00:47
*** mark-casey1 has joined #openstack-kolla00:48
*** mark-casey has quit IRC00:50
*** ccesario has left #openstack-kolla00:54
*** ccesario___ has quit IRC00:59
*** daneyon has joined #openstack-kolla01:00
*** sdake_ has joined #openstack-kolla01:02
*** sdake has quit IRC01:02
*** daneyon__ has quit IRC01:04
*** jtriley has quit IRC01:08
*** sdake_ is now known as sdake01:08
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix the wrong package name in Ubuntu binary build  https://review.openstack.org/30370701:18
*** mark-casey1 has quit IRC01:21
*** Jeffrey4l__ has joined #openstack-kolla01:24
*** jtriley has joined #openstack-kolla01:25
*** stvnoyes has joined #openstack-kolla01:26
*** ccesario_ has quit IRC01:31
*** ccesario___ has joined #openstack-kolla01:33
sdakedamn good thing we brought a towel01:35
*** jtriley has quit IRC01:44
sdakeJeffrey4l__ are you getting sleep dude ;)01:50
Jeffrey4l__sdake, on a train. :D01:51
Jeffrey4l__need to another place.01:51
sdaketake care to get atleast 6 hours a night :)01:52
sdakeJeffrey4l__ openstack-infra said their default firwall rules would prevent leaking of multicat packets01:55
*** ccesario___ has quit IRC01:55
Jeffrey4l__sdake, so bluebox do not stop this? and keepalived connect to each other01:56
sdakewho knows such things01:57
*** salv-orlando has joined #openstack-kolla01:58
Jeffrey4l__sdake, could we make sure with openstack-infra guys? may be they made some mistake in bluebox case.01:58
sdakefeel free to ping them in openstack-infra01:59
sdakethey know who yyou are ;)01:59
sdakethey were pretty clear on it though01:59
sdakebut i'd think we have to drop all our firewall rules in our vm to deploy properly01:59
Jeffrey4l__ok , I will ping them.02:00
sdakenote to all, I've reprioritized kolla 2.0.0 bugs02:01
sdakefeel free to change as you see fit02:01
sdakehttps://launchpad.net/kolla/+milestone/2.0.002:01
openstackgerritSteven Dake proposed openstack/kolla: Fix the neutron reconfigure issue when enable linuxbridge-agent  https://review.openstack.org/30432002:01
Jeffrey4l__no, drop all in our firewall is not good. when we have multi node test env in the gate, drop all will crash everything. So i do not recommend to do that.02:02
Jeffrey4l__cool. i will check the bug list.02:02
sdakei left the implemented ones as is02:02
sdakesince they are done02:02
sdakeand hae no impact on the next week of dev02:02
Jeffrey4l__https://review.openstack.org/303707 please check this . This block the ubuntu-binary in the mitaka branch.02:02
sdakesbezverk after those copy commnds you cn add a trailing backslash and not have to specify the name02:05
sdakeJeffrey4l__ acked02:05
sdakeand rechecked02:06
Jeffrey4l__sdake, wait. when you saying `drop all our firewall rule` you mean drop all the packets? because, in the current image, the iptable should be disabled.02:06
Jeffrey4l__sdake, thanks. the centos-source failed due to the keystone issue.02:06
sdakeopenstack-infra is the right place to ask02:06
sdakewhich keystone issue02:06
*** salv-orlando has quit IRC02:07
sdakeyou mean this http://logs.openstack.org/07/303707/2/check/gate-kolla-dsvm-deploy-centos-source/c3a17e9/console.html#_2016-04-12_01_33_10_61802:07
sdakehard to knwo if we are chasing gate bugs or kolla bugs02:08
sdakethis bug always happened when rabbitmq crashed previously02:08
sdakevhosakot ^^02:08
Jeffrey4l__sdake, yea. this issue is wired too. But i do not think it is related to the rabbitmq issue. Will make some debug later.02:10
*** unicell1 has quit IRC02:10
vhosakotsdake: that too, I think is a gate issue.. like the keepalived VIP bug in gate.... I happened everytime before rabbitmq crashed... but, now, since rabbitmq crash is fixed, that keystone JSON error is still there.. I pritned the output in debug in gate, and the output will be emtpy and cannot be converted into JSSON and hecne fails02:11
*** yuanying has quit IRC02:12
*** yuanying has joined #openstack-kolla02:12
sdakewhy is the output empty02:12
*** kbyrne has quit IRC02:13
vhosakotsdake: 1 sec.. let me send the link to the logs02:13
*** weiyu_ has joined #openstack-kolla02:14
Jeffrey4l__I think there is something wrong exist either in the `keystone-manage bootstrap` or in the our `keystone_bootstrap.sh` scripts.02:15
vhosakotJeffrey4l__: sdake:  here is my debug message that shows empty output in gate  http://logs.openstack.org/86/303686/4/check/gate-kolla-dsvm-deploy-centos-source/17631e5/console.html#_2016-04-09_14_16_26_14002:16
sdakevhosakot i get its empty02:16
sdakewhy is it empty is what I asked ;)02:16
*** pbourke has quit IRC02:16
sdakeso Jeffrey4l__ is spot on02:17
sdakeone of those two things is busted02:17
*** pbourke has joined #openstack-kolla02:17
sdakelets figure out which one02:17
sdakeand fix it :)02:17
vhosakotsdake: yep, as Jeffrey4l__ said, keystone-manage is buggy in gate   https://github.com/openstack/kolla/blob/master/docker/keystone/keystone_bootstrap.sh#L2702:17
sdakeorthe parsing script is buggy02:19
sdakewhich one ?02:19
*** Jeffrey4l__ has quit IRC02:20
sdakeso here is what we know02:20
sdakekeystone-manage output int othe parsing script returns nothing02:20
sdakenext step is determining which of those is busted02:21
sdakelooks more like a legitimate bug rather then a gate problem02:23
sdakebut who knows for sure02:23
sdakewe need debug output at both input and in the script02:23
sdakee.g. all variables passed to the bootstrap script02:23
vhosakotyep, since output is empty, the exit_json function is not hit.. so, somthing in these lines is breaking in gate  https://github.com/openstack/kolla/blob/master/docker/keystone/keystone_bootstrap.sh#L27-L4602:24
*** kbyrne has joined #openstack-kolla02:24
sdakelogging the output of keystone_bootstrap duringa failureand during success would help02:26
sdakeat line 3102:26
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH  https://review.openstack.org/29490602:27
sdakethis is also not executing https://github.com/openstack/kolla/blob/master/docker/keystone/keystone_bootstrap.sh#L2902:27
vhosakotsdake: if we add an echo line in that script, will it be seen in gate logs as it will be run _inside_ the cotainers' shell02:27
sdakeecho > /var/log/kolla/personal_debug02:28
sdakead modify the copy logs script02:28
vhosakotsdake: like that...02:28
vhosakotsdake: ah, that should work, cool02:28
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH  https://review.openstack.org/29490602:28
vhosakotsdake: generally, will echo in a container's starup script be seen in "docker logs" of that cotnainer ?02:29
sdakegood question02:29
sdakedepends on if the heka volume is mounted or not02:29
*** daneyon_ has joined #openstack-kolla02:29
vhosakotsdake: ah ok02:29
sdaketime to feed myself back in 30 mins or so02:30
vhosakotthen, if "docker logs" can see it, we could just add "set -x" and "set -o" at the top of the script02:30
sdakei dont knwo if docker logs ar ecpied out or not02:31
sdakei need to eat, or i'd tell you :)02:31
vhosakotsdake: good to add "set -o pipefail" as well....02:31
vhosakotsdake: cool, eat :)02:31
*** Jeffrey4l__ has joined #openstack-kolla02:33
*** daneyon_ has quit IRC02:34
*** coolsvap has joined #openstack-kolla02:35
*** jtriley has joined #openstack-kolla02:37
*** weiyu_ has quit IRC02:41
*** weiyu_ has joined #openstack-kolla02:43
*** jtriley has quit IRC02:44
*** Jeffrey4l__ has quit IRC02:45
*** yuanying has quit IRC02:52
*** weiyu_ has quit IRC02:52
sdakeok back02:56
openstackgerritMerged openstack/kolla: Fix the neutron reconfigure issue when enable linuxbridge-agent  https://review.openstack.org/30432002:57
*** weiyu_ has joined #openstack-kolla02:57
*** klint has joined #openstack-kolla03:00
*** banix has joined #openstack-kolla03:03
*** weiyu_ has quit IRC03:09
coolsvapvhosakot, i think the ubuntu-binary is only for stable branch03:11
vhosakotcoolsvap: ah ok, thanks for the info!03:12
*** jtriley has joined #openstack-kolla03:12
vhosakotcoolsvap: so, just for mitaka ?03:12
*** sdake_ has joined #openstack-kolla03:12
coolsvap yes, due to the package03:12
coolsvapi dont recall the exact reason, need to search on mailing list i think03:13
*** Marga_ has quit IRC03:14
vhosakotcoolsvap: ah ok... thanks for the info!03:14
*** sdake has quit IRC03:15
coolsvapvhosakot, https://openstack.nimeyo.com/69943/openstack-dev-kolla-adding-ubuntu-liberty-to-kolla-mitaka03:16
*** sdake has joined #openstack-kolla03:18
*** jtriley has quit IRC03:18
vhosakotcoolsvap: thanks for the link!03:20
*** sdake_ has quit IRC03:20
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: TESTING on Mitaka  https://review.openstack.org/30437303:23
coolsvapvhosakot, lets see the results on stable03:23
sdakedave-mccowan around?03:25
kjejlly_Hello everyone, Does anyone know what's wrong with me?  http://paste.openstack.org/show/493709/03:28
sdakenothing is wrong with you03:28
sdake127.0.1.1 is in your /etc/hosts file03:28
sdakewith your hostname used in your deployment03:28
vhosakotcoolsvap: cool03:29
sdakeit hsould be the machines phsyical ip03:29
dave-mccowansdake pong03:29
sdake(in th /etc/hosts file) in place of 127.0.1.103:29
sdakedave-mccowan kibana is supposed to have an haproxxy password03:29
sdakeit does not03:29
sdakering any bells?03:29
dave-mccowani used a password to log into kibana a little while ago.03:30
kjejlly_sdake: Thanks. It works !03:31
dave-mccowani remember talking about using basicauth, instead of requiring shield.03:31
sdake[sdake@minime-03 kolla]$ docker images | grep kibana03:31
sdake192.168.1.103:4000/kollaglue/centos-source-kibana                      2.0.0               7681b7187eae        2 hours ago         490.8 MB03:31
sdakei am not presented with a login prompt03:31
sdakewe can't use shield03:31
sdakeit has a 30 day expiry license03:31
dave-mccowani know.03:32
sdakeyes but you said you were presented with a login prompt03:32
sdakei was not03:32
sdakewhy? :)03:32
dave-mccowani was presented with a login prompt.  what port did you use in your url?03:32
sdake5601 on the box itself03:32
sdakeaoiding haproxy i guess03:32
sdakenm ;)03:32
dave-mccowanso there's a bug03:33
dave-mccowanwant me to fix it?03:34
sdakeno bug03:35
sdakewhat is the kibana username03:35
vhosakotsdake: https://github.com/openstack/kolla/blame/master/doc/quickstart.rst#L649-L66703:36
sdakecool it works03:36
sdakedave-mccowan i was bypasing haproxy to directly acess my ndoes hence no auth03:36
sdakemyw ife wants to make a dash board for kolla i guess ;)03:36
*** banix has quit IRC03:36
dave-mccowanyou should not be allowed to bypass the auth03:37
sdakeif you have access to the management network?03:37
sdakenot sure how we could possibly stop that03:38
*** salv-orlando has joined #openstack-kolla03:38
sdakekibana runs on 192.168.1.103:560103:38
sdakeif i connect directly to that, can bypass auth03:38
sdakeif i use broked.selfip.net:5601 pressented with an auth screen03:39
sdakefrom 1990s03:39
sdakekibana 4 does not  run in apache03:40
dave-mccowanright.  ya.  i was thinking kibana could listen on 127.0.0.1.  but, if it's listening only on management interface, that's good enough.03:40
sdakeit runs in nodejs or some other hpster thing03:40
sdakei dont know if it wild card binds03:41
sdakeif i t does that is something we need to fix03:41
sdake[sdake@minime-03 kolla]$ netstat -nl | grep 560103:41
sdaketcp        0      0 192.168.1.103:5601      0.0.0.0:*               LISTEN03:41
sdaketcp        0      0 192.168.1.149:5601      0.0.0.0:*               LISTEN03:42
sdaketcp        0      0 192.168.1.148:5601      0.0.0.0:*               LISTEN03:42
sdakenot good03:42
dave-mccowani think that's ok.  hard to tell for sure with all three addrs on the same subnet.03:43
sdake0.0.0.0 is bind to all intefaces03:44
sdakethat is the default03:44
dave-mccowanthat column is the remote addr03:44
dave-mccowanfor local addr, it's only listening to host addr, internal vip, and external vip.03:44
*** jtriley has joined #openstack-kolla03:44
sdakeoh you checked it out?03:45
dave-mccowanyep. same time as you did.03:45
sdakeoh right03:46
sdakesomehow i competely missed the host ip binds03:46
* sdake running low on fuel03:46
dave-mccowanlooks like you're running with TLS.  or just two vips?03:47
sdakemoemnt03:47
sdakeyes tls enabled03:47
dave-mccowancool.  glad it's getting extra testing... but looks like a kibana bug.  that session is not over TLS. :-(03:49
*** achanda has joined #openstack-kolla03:49
dave-mccowani thought I brought that up in a code review.  need to add TLS for kibana.03:49
sdakethere is a enable tls bug ffor kibana atm03:49
*** yuanying has joined #openstack-kolla03:50
sdakeserver.host is not being set03:50
sdakewhich is wrong03:50
sdakethe default is 0.0.0.003:50
sdakeso it is binding to all interfaces on my system03:50
*** salv-orlando has quit IRC03:51
sdakehttps://www.elastic.co/guide/en/kibana/current/kibana-server-properties.html03:51
*** jtriley has quit IRC03:51
*** jasonsb has joined #openstack-kolla03:52
dave-mccowanserver.host is set on my system03:53
sdakehttps://bugs.launchpad.net/kolla/+bug/156914803:54
openstackLaunchpad bug 1569148 in kolla mitaka "kibana not setting server.host resulting in wildcard bind" [Critical,Confirmed]03:54
sdakeits not in the .j2 file in the repo03:54
dave-mccowanthe netstat you sent did not show binding to all interfaces.03:54
sdakemaster doesn't define server.host03:54
sdakethe docs ssay that is a 0.0.0.0 bind03:54
dave-mccowanwhat do you have in /etc/kolla/kibana/kibana.yaml?03:55
sdakeya its set there03:56
dave-mccowanin the .j2 file, line 3, there is server.host03:56
sdakeok i guess im a dummy03:56
sdakeparanoid of security problems ;)03:56
dave-mccowanmissing TLS is a problem.  should be a one line fix.03:56
sdakemy wie is developing on my only box atm03:57
dave-mccowanbug is assigned to jeff.03:58
sdakehttps://bugs.launchpad.net/kolla/+bug/156611703:59
openstackLaunchpad bug 1566117 in kolla mitaka "Add TLS to the kibana service" [Critical,Confirmed] - Assigned to Jeffrey Zhang (jeffrey4l)03:59
sdaketbh i dont htink he would mind giving it up04:00
sdakehe has like 15 bugs on his plat atm04:00
sdakehe self assigned04:00
sdakei just brought yet another bug into 2.0.0 :(04:00
dave-mccowani think i brought it up in a code review and he opened the bug instead of fixing it with original commit.04:00
sdakethat is a commoon thing in this project ;)04:01
sdakedave-mccowan if you want to take it over wfm ;)04:02
sdakeif not i'll fix when my wife gets done doing her thiing04:02
coolsvapexcellent ubuntu binary build is green with https://review.openstack.org/#/c/30370704:02
coolsvapno deploy is green sorry04:03
*** Marga_ has joined #openstack-kolla04:04
sdakethat is ubuntu source04:05
sdakewe dont have ubuntu binary in master04:05
*** achanda has quit IRC04:05
sdakei guess we should rethink that decision04:05
sdakerather in the master gates04:06
openstackgerritSteven Dake proposed openstack/kolla: Fix the wrong package name in Ubuntu binary build  https://review.openstack.org/30437304:07
sdakecoolsvap you mean https://review.openstack.org/#/c/304373/04:07
patchbotsdake: patch 304373 - kolla (stable/mitaka) - Fix the wrong package name in Ubuntu binary build04:07
sdakeubuntu binary gate failing for whatever reason04:08
sdakebut deploy is green!!04:08
coolsvapyes04:09
coolsvapwe were just trying out the master fix on the ubuntu-binary gates04:09
openstackgerritMerged openstack/kolla: Fix the wrong package name in Ubuntu binary build  https://review.openstack.org/30370704:09
sdakeyp apolgoies for the double cherrypick04:10
sdakedidn't realize it had already been done04:10
coolsvapnp04:11
coolsvapit was just a test04:11
dave-mccowansdake testing fix now04:11
sdake70 billion dollars of pure destruction04:11
sdakethe md50004:11
*** Marga__ has joined #openstack-kolla04:12
*** banix has joined #openstack-kolla04:14
*** Marga_ has quit IRC04:15
sdakeboy i really like ender's game04:15
sdakei've watched it every day for a week04:15
sdakeneeds ack https://review.openstack.org/#/c/304373/204:19
patchbotsdake: patch 304373 - kolla (stable/mitaka) - Fix the wrong package name in Ubuntu binary build04:19
sdakethis is an epic change: https://github.com/docker/docker/pull/2066204:36
sdakerunc04:36
sdakethis allows containers to be run and docker to be restarted without disrupting the containers or stoppign them04:37
sdakein 1.1104:37
Marga__Hi04:38
*** Marga__ is now known as fragatina04:39
fragatinaHi guys/gals!04:39
fragatinaI'm having some questions with installing kolla in ubuntu04:39
sdakesup fragatina04:40
fragatinasince I haven't successfully installed this yet, this questions will be obvious to many04:40
fragatinaor that I'm doing something wrong04:40
sdakeif its not covered in the QSG its not obvious :)04:40
fragatinaso, background: I have ubuntu 14.04 with 4.2.0-35-generic kernel04:41
fragatinaand I'm running docker as: /usr/bin/docker daemon --insecure-registry 10.0.2.15:4000 --storage-driver=overlay04:42
vhosakotfragatina: yep, 4.2 kernel is good04:42
fragatinaI've removed /var/lib/docker directory, so it doesn't exists04:43
fragatinaand I run:  docker run -d -p 4000:5000 --restart=always --name registry registry:204:43
fragatina(as I see it in the instructions)04:44
sdakedocker will create /var/lib/docker04:44
fragatinaalso, I'm running everything as a root since I found that solves me some issues04:44
openstackgerritMerged openstack/kolla: Fix the wrong package name in Ubuntu binary build  https://review.openstack.org/30437304:44
sdakebut to do that it has to be started04:44
sdakeyay 1 less bug to fix :)04:44
fragatina# ps -ef | grep docker04:45
fragatinaroot       950     1  0 21:28 ?        00:00:00 /usr/bin/docker daemon --insecure-registry 10.0.2.15:4000 --storage-driver=overlay04:45
fragatinaso, docker is running but it didn't create the /var/lib/docker04:45
openstackgerritSteven Dake proposed openstack/kolla: Add quotes for RBMQ nodes definition  https://review.openstack.org/30391904:46
fragatinaso when I try to do the registry it fails with: "docker: open /var/lib/docker/tmp/GetImageBlob590488645: no such file or directory. "04:46
vhosakotfragatina: you see that error when you run "docker run -d -p 4000:5000 --restart=always --name registry registry:2" ?04:46
fragatinayes04:46
fragatinacompletely output:04:47
vhosakotfragatina: can you resatrt the daemon and retry the docker run command04:47
vhosakotsudo service docker restart04:47
fragatina# docker run -d -p 4000:5000 --restart=always --name registry registry:204:47
fragatinaUnable to find image 'registry:2' locally04:47
fragatina2: Pulling from library/registry04:47
fragatinafdd5d7827f33: Downloading [==================================================>] 51.37 MB/51.37 MB04:47
fragatinaa3ed95caeb02: Downloading [==================================================>]     32 B/32 B04:47
fragatinaa79b4a92697e: Downloading [==================================================>] 7.572 MB/7.572 MB04:47
fragatina6cbb75c7cc30: Waiting04:47
fragatina4831699594bc: Downloading [==================================================>]    379 B/379 B04:47
fragatinadocker: open /var/lib/docker/tmp/GetImageBlob590488645: no such file or directory.04:47
fragatinaSee 'docker run --help'.04:47
vhosakotsudo service docker restart04:47
vhosakotthis should recreate /var/lib/docker04:48
fragatinaweird, why it didn't the first time04:48
fragatinanp, thx!04:48
vhosakotwell, why did you delete /var/lib/docker ? :)04:49
sdakebecaue she switched to overlayfs from aufs of course ;)04:50
*** salv-orlando has joined #openstack-kolla04:51
openstackgerritMerged openstack/kolla: Add LVM2/iSCSI documentation  https://review.openstack.org/30082404:51
*** rstarmer has joined #openstack-kolla04:52
fragatinaof course sdake! :-)04:52
*** banix has quit IRC04:56
*** weiyu has joined #openstack-kolla04:57
fragatinathx vhosakot, it seems at least it started building the images now05:02
fragatinathx!05:02
vhosakotfragatina: cool :)05:02
*** salv-orlando has quit IRC05:03
gmmahavhosakot: is this patch just pending someone testing the linuxbridge on a deployment? https://review.openstack.org/#/c/302721/05:03
patchbotgmmaha: patch 302721 - kolla - Fix launch vm failed in neutron-linuxbridge-agent env05:03
fragatina@vhosakot, will let you know in a couple of hours how it went :-)05:04
vhosakotfragatina: cool05:04
vhosakotgmmaha: yeah, I was gonna test it sometime05:05
gmmahavhosakot: aah ok.. i am building it on my setup05:05
gmmahathe latest master as of 10 mins ago with this patch05:05
gmmahawill update the patch if i get it working fine05:05
vhosakotgmmaha: cool, are you planning to test it ?05:05
gmmahavhosakot: yeah05:06
gmmaharunning it for some benchmark testing i am doing05:06
vhosakotsdake: can we have a mechanism to run linuxbridge on-demand in gate... ?05:06
gmmaha1000 VMs05:06
vhosakotgmmaha: nice05:06
sdakevhosakot feel free to add linuxbridge gating :)05:07
sdakeat ummit i am going to trian everyone how to add gates05:07
sdakeand how our gatingworks05:07
vhosakotsdake: then, how we gate OVS ? I was thinking like an ondemand way to run linuxbrdige inn gate just to test linuxbrodge patches05:07
sdakedave-mccowan ping05:09
sdakedave-mccowan can you try this link http://bit.ly/1VkE8hD05:09
sdakeexcept change my ip address with yours05:09
sdakelinuxbridge should be tested all the time05:10
sdakein a sepraet set of gates05:10
sdakeas well as tls05:10
vhosakotsdake: like centos-source-ovs and centos-source-linuxbridge, ...,...,,.. like that ?05:11
sdakeyup05:11
sdakedeploy-ovs05:11
sdakebut yup05:11
sdakeright now our deploy is deploy_ovs05:11
vhosakotsdake: it will double jenkins run time05:12
sdakeor deploy-ovs05:12
sdakeno jenkisn jobs run in prallel05:12
*** daneyon_ has joined #openstack-kolla05:12
vhosakotsdake: ah in parallel.. nice!05:12
dave-mccowansdake i got could not locate dashboard05:12
vhosakotsdake: I will looking into adding linuxbridge gate then... many kolla users use it05:12
*** rbergeron has joined #openstack-kolla05:15
sdakedave-mccowan import this http://paste.fedoraproject.org/354486/4381091/ -> http://paste.fedoraproject.org/354486/0438109105:15
sdakesettings->objects->import05:15
rbergeronoh, you're already there.05:15
rbergeroni was going to say -- https://gist.github.com/robynbergeron/d1fd3634790464064d77f06508c6274005:15
rbergeron(save it as a whateverfilename.json)05:16
sdakemake sur eto make your index first of log-*05:16
*** daneyon_ has quit IRC05:17
sdakevhosakot typically yo uwill wnt for all distro variants05:17
vhosakotsdake: ok..05:17
sdakethe first step is to add a tox target05:17
vhosakotright05:17
sdakethe second  step is to modify deploy_aio so it takes an argument to do the job you want05:17
sdakei think it may already hae this05:18
sdakeor setup_gate05:18
sdakenot sure which05:18
sdaketraining - friday of summit ;)05:18
dave-mccowani'm out of steam for tonight.  i'm looking forward to trying it tomorrow.05:18
*** dave-mccowan has quit IRC05:19
sdakevhosakot do you have elasticsearch enabled05:19
gmmahavhosakot: for that bug, i am not saying if the file /etc/kolla/passwords.yml does not get updated if a key is there and there are no passwords.05:20
*** weiyu has quit IRC05:20
gmmahajust saying that if the file is an old one the script does not add the new items into the file.05:20
vhosakotsdake: no... i have not enabled central_logging..05:20
sdakegmmaha yes it is imperfect05:20
gmmahalet me push up a patch and maybe that can explain what i am tinking05:20
sdakevhosakot do so so you can check out the dashboard my wife made for us ;)05:20
sdakerbergeron can stuff link from one dashboard to another?05:21
vhosakotsdake: cool, so, should I enable central_logging and rebuilf ?05:21
vhosakotrebuild*05:21
sdakenot rebuild05:21
sdakeredeploy05:21
sdakeconnect to ip:560105:21
sdakeindex log-*05:21
vhosakotright right re-deploy05:21
sdakeinstead of logstash-*05:21
vhosakotgmmaha: what "new items" are you talkinga bout in passworks.yml ?05:22
vhosakotgmmaha: kolla-genpwd creates passwords to all fields that are emppty in passwords.yml05:22
sdakevhosakot he is talking about when config files change05:23
sdakeand we hve no way to handle that upgrade atm05:23
sdakegmmaha in newton05:23
sdakewe will deal with that05:23
sdakenot now05:23
gmmahavhosakot: correct.. but lets say today you add a new key neutron-XXX: in the source tree, if i run generate_passwords.py or genpwd.py tjat new keu wil not be adde to my /etc/kolla/passwords.yml05:23
gmmahaunless i add the neutron-XXX: to the file myself and then it will generate a password05:23
gmmahasdake: for sure..05:24
gmmahano rush on it. a very very low priority bug for sure05:24
vhosakotgmmaha: as part of installation, we need to copy etc/kolla in repo to /etc/kolla05:24
sdakeits a fetaure not a bug ;)05:24
sdakevhosakot he is takling about upgrading kolla from mitaka to newton05:24
vhosakotgmmaha: https://github.com/openstack/kolla/blame/master/doc/quickstart.rst#L169-L17405:24
sdakeor master to master05:24
*** coolsvap has quit IRC05:25
vhosakotah upgrade05:25
sdakevhosakot focus - dashboards :)05:25
vhosakotyeah... doing it.. 1 sec05:25
gmmahasdake: vhosakot: thanks and sorry for de-railing05:25
sdakerbergeron it would be cool i the daashboard clicked through05:25
sdakedoes kibana do that?05:26
vhosakotgmmaha: haha.. no need to for any sorry :)05:26
sdakelike click on neutron an dsee a neutron specific dashboard05:26
vhosakotgmmaha: push the PS... i'll see in review05:26
gmmahavhosakot: aye05:26
sdakerbergeron whats with that vertical thing on the dashboard05:28
rbergeronsdake: if you click on openstack.neutron in the total count by logger box05:29
rbergeronyou'll notice that eerything gets filtered by that on the dashboard05:29
rbergeronand you'll see the little green box up towards the top right under the dashboard name05:29
rbergeronthat says "logger openstack.neutron"05:29
sdakehow do you see actual logs :)05:29
rbergeronand then you can click on that to remove the filter, save it, pin it, whatever05:30
openstackgerritMerged openstack/kolla: Add quotes for RBMQ nodes definition  https://review.openstack.org/30391905:30
rbergeronsdake: looking but running out of steam :)05:33
rbergeronit's in the _source field but that isn't indexed but ... not sure if that can be fixed offhand05:33
sdakeya i've been out of steam for months05:33
*** vincent_vdk has quit IRC05:33
sdakewe an reindex05:33
sdakerbergeron i'mg oing to spin up a vm05:35
sdakesee what happens :)05:36
rbergeronuhh05:43
rbergeronokay05:43
*** vincent_vdk has joined #openstack-kolla05:44
*** tfukushima has joined #openstack-kolla05:45
sdakerbergeron i got you some more log to work with05:45
sdakei spun up 3 vms via heat05:45
*** ydeshmukh has joined #openstack-kolla05:46
sdakegetting hang of kibana05:46
sdakeyou create filters in it05:46
sdakevhosakot you ready to roll?05:48
vhosakotde-reployed fine... openining ip:5601.. moment05:49
elemoine_sdake looking at creating default Kibana dashboards? We may look into that pretty soon for our kolla central logging talk. Today maybe.05:49
sdakeelemoine_ check out http://paste.fedoraproject.org/354486/4381091/ -> http://paste.fedoraproject.org/354486/0438109105:50
*** rstarmer has quit IRC05:50
sdakevhosakot when it asks for an index type in log-*05:50
sdakeelemoine_ i think my wife replicated the fuel dashboard in about 45 minutes05:51
vhosakotsdake: ah ok... let me type it05:51
sdakeshe rocks like that :)05:51
elemoine_sdake I believe you!05:51
elemoine_sdake more on that later...05:52
sdakemore on which05:52
rbergeroni don't know about replicated05:53
rbergeronbut poked certainly05:53
sdakemade up her own05:53
sdakewhatever the case wtb actual log messages ;)05:53
* sdake excied :)05:53
rbergeronsdake: patience05:53
sdakepadowan05:54
rbergeronmaybe tomorrow, though tomorrow is supposed to be my day for googlebigquery fun with githubarchive and metrics05:54
sdakedo you hear nothing I say? :)05:54
rbergeronbut05:54
rbergeronmaybe05:54
rbergeronsdake: i think you should be able to curl that json into the kibana index in ES05:55
sdakeyup05:55
rbergeronbecause that is apparently where the dashboards get saved05:55
sdakepretty sure i said that earlier05:55
rbergeronand then kibana is basically looking in that index when you want to open a saved one05:55
rbergeronsdake: it's not entirely supported yet is the thing05:56
rbergeronit used to be the norm05:56
rbergeronthen went away05:56
rbergeronthen they added "import / export" as a button05:56
*** salv-orlando has joined #openstack-kolla05:56
rbergeronthis is all kibana 3 to kibana 4 stuff05:56
sdakehow do i get a box that prints out jus the log  field05:56
vhosakotsdake: after typing in log-* as index pattern, I don't see submit button and see "Unable to fetch mapping" faded out05:57
vhosakotsdake: http://picpaste.com/pics/kibana-zfl8CdoO.1460440537.png05:57
sdakenow click dashboard05:57
vhosakotok05:58
sdakeyou should see a horizontal graph05:58
sdakeclick settings->object->import05:58
sdakeand import that json in the paste above05:59
rbergeronsdake: he has to be able to find the index05:59
rbergeronfor that part to matter :)05:59
vhosakotsdake: should I copy paste the json into kibana ui ?05:59
sdakevhosakot no use import button dude ;)06:00
sdakesettings->object->import button06:00
rbergeronvhosakot: were you able to find the index pattern?06:00
vhosakotsdake: should I save the contents in pastebin as a .json file on disk and then import it ?06:01
sdakeright06:01
sdakehey elemoine_ why didn't you mention logstash-forwarder was deprecated upstream during heka dev06:02
vhosakotsdake: ok I see "Dashboards are Awesomesauce"06:03
sdakethere ya go06:03
vhosakotsdake: in kibana ui.. what next ?06:03
sdakeclick it06:03
openstackgerritMichal Rostecki proposed openstack/kolla: Create and chown /etc/heka directory  https://review.openstack.org/30274206:06
sdakedo ou see a dashboard in black?06:09
sdakewith what appears to be log information?06:09
*** weiyu has joined #openstack-kolla06:10
*** ydeshmukh has quit IRC06:12
sdakevhosakot did you get it rolling06:13
*** unicell has joined #openstack-kolla06:20
*** achanda has joined #openstack-kolla06:23
*** vhosakot has quit IRC06:31
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add Heka support  https://review.openstack.org/28630106:34
*** unicell has quit IRC06:40
*** unicell has joined #openstack-kolla06:40
*** coolsvap has joined #openstack-kolla06:44
openstackgerritMichal Rostecki proposed openstack/kolla: Create /var/cache/swift in source installation  https://review.openstack.org/30384906:46
sdakelatest iteration of dashboard: http://paste.fedoraproject.org/354511/4437821/ -> http://paste.fedoraproject.org/354511/0443782106:49
sdakeelemoine_ you can rpobably start with this06:50
*** salv-orlando has quit IRC06:54
*** weiyu has quit IRC06:57
*** weiyu has joined #openstack-kolla07:00
*** athomas has joined #openstack-kolla07:08
sdakeakwasnie you around07:12
*** mikelk has joined #openstack-kolla07:13
*** salv-orlando has joined #openstack-kolla07:15
*** weiyu has quit IRC07:18
*** sdake_ has joined #openstack-kolla07:24
*** sdake has quit IRC07:26
*** sdake has joined #openstack-kolla07:27
*** sdake_ has quit IRC07:30
sdakeelemoine_ any problems with severity level 4 in the lua code your aware of relating to nova?07:31
sdakeelemoine_ i've got sev level 4 errors but no payload07:32
elemoine_sdake: no07:32
elemoine_could you be more specific?07:32
sdakeelasticsearch when filtering on payloa dlevel 4 doesnot show the payload07:33
sdakeit has no results with level 4 but the graph i've produced does07:33
sdakein nova-api and nova-compute07:33
*** salv-orl_ has joined #openstack-kolla07:34
elemoine_sdake: maybe create a bug report with screenshots demonstrating the problem07:35
sdakepossible07:35
sdakei think that prototype my wife and i produced is a good place for you to start07:35
sdakeit doesnt allow anding of priorities07:36
sdakewhich I'd like to see07:36
sdakenot quite sure how to fix that07:36
sdakestill jerking around with learning how kibana works07:36
*** jmccarthy has joined #openstack-kolla07:37
*** salv-orlando has quit IRC07:37
*** nihilifer has quit IRC07:38
*** unicell has quit IRC07:39
*** unicell has joined #openstack-kolla07:41
*** nihilifer has joined #openstack-kolla07:42
*** tfukushima has quit IRC07:43
*** mbound has joined #openstack-kolla07:47
sdakecool i just found filters07:47
sdakewinning07:47
*** shardy has joined #openstack-kolla07:48
*** tfukushima has joined #openstack-kolla07:50
*** ccesario___ has joined #openstack-kolla07:51
*** ydeshmukh has joined #openstack-kolla07:55
*** ccesario___ has quit IRC08:03
*** ccesario___ has joined #openstack-kolla08:03
ydeshmukhhey sdake08:08
ydeshmukhI still stuck at 'Creating the Glance service and endpoint' when deploying Kolla on 3 Node setup (each control, compute and network).08:08
ydeshmukhNow I'm running with haproxy. I've a question here, which node hosts haproxy and keepalive containers? In my setup they are showing on network node.08:09
sdakethat may be a bug08:09
sdakethey should be hosted on the control nodes08:10
*** fragatina has quit IRC08:11
ydeshmukhsdake ok, I changed 'haproxy:children' network->control in inventory file. now haproxy and keepalive are running on control node.08:13
sdakedid that fix deployment?08:14
ydeshmukhBut still facing issue with 'Creating the Glance service and endpoint'.08:14
ydeshmukhI tried kolla-ansible prechecks earlier and that worked fine.08:15
ydeshmukhCould you guide me how to debug for 'Creating the Glance service and endpoint' issue?08:15
*** weiyu has joined #openstack-kolla08:16
*** weiyu has quit IRC08:17
kjejlly_ydeshmukh: post the error message. Maybe other people will help you08:21
kjejlly_post error message in here. http://paste.openstack.org/08:22
sdakei'd recommend a reboot o all 3 nodes, keepalived still has vip running on wrong network host08:24
sdakeyour vip is an unused ip on your management network, correct?08:25
ydeshmukhkjejlly_ thanks, pasted on http://paste.openstack.org/show/493725/08:25
sdakeelemoine_ what is severity=5?08:26
sdakei think it houd be ok to run haproxy and keepalived on network nodes08:27
elemoine_sdake: https://github.com/openstack/kolla/blob/master/docker/heka/plugins/modules/os_utils.lua#L28-L3708:27
sdakebut it seems odd08:27
ydeshmukhsdake ok, will reboot and check. yeah vip has unused ip and its pingable from all node.08:28
*** Serlex has joined #openstack-kolla08:36
*** nihilifer has quit IRC08:44
sdakesomething is generating alot of femergeny and critical errors08:46
sdakeit hsouldn't respond to ping08:47
sdakeif you reboot every ndoe08:47
*** gfidente has joined #openstack-kolla08:48
*** gfidente has quit IRC08:48
*** gfidente has joined #openstack-kolla08:48
*** weiyu_ has joined #openstack-kolla08:51
*** weiyu_ has quit IRC08:52
*** weiyu has joined #openstack-kolla08:54
ydeshmukhsdake, no luck still same issue.08:55
sdakeafter rebooting the vip responds to ping?08:55
sdakecleanup-containers08:56
sdakereboot all nodes08:56
sdakesee if ip responds to ping08:56
ydeshmukhyeah08:56
sdakeit should not08:56
sdakeif it does, that means your vip is in use08:56
ydeshmukhdo you mean after cleanup-containers?08:56
*** nihilifer has joined #openstack-kolla08:57
*** nihilifer has quit IRC08:57
*** weiyu has quit IRC08:57
*** nihilifer has joined #openstack-kolla08:58
ydeshmukhno vip is not pinable after cleanup-containers08:58
*** nihilifer has joined #openstack-kolla08:58
*** nihilifer has joined #openstack-kolla08:59
ydeshmukhif vip is not pinable or used somewhere else this ' Waiting for MariaDB service to be ready through VIP' step would have failed.09:00
ydeshmukhsdake, right?09:02
sdakeok09:03
sdakewhich distro re you using09:03
sdakecentos binary centos source ubuntu binary ubuntu source?09:03
*** dims has quit IRC09:04
*** weiyu_ has joined #openstack-kolla09:04
ydeshmukhcentos-binary09:04
*** dims has joined #openstack-kolla09:05
sdakeok centos binary is probably bust atm09:08
sdakei'd try centos source09:08
sdakewe are waiting on rdo to release mitaka final09:08
sdakethey are not done yet09:08
sdakemost devs use centos binary or centos source09:09
sdakesorry cenots source or ubuntu source09:09
sdakethats not to say we dont want centos binry and ubuntu binary to work09:09
sdakethey just may be less well tetted atm09:09
sdakeelemoine_ is programname a multifield?09:09
ydeshmukhsdake, ok, can I give try with cenots source?09:11
sdakeyes please do09:11
ydeshmukhok thanks, will update you.09:13
*** weiyu_ has quit IRC09:14
*** openstackgerrit has quit IRC09:17
*** openstackgerrit has joined #openstack-kolla09:17
*** coolsvap has quit IRC09:18
openstackgerritMerged openstack/kolla: Add prechecks to validate kolla_internal_vip_address and kolla_external_vip_address  https://review.openstack.org/30393809:18
openstackgerritMerged openstack/kolla: Identify container through  PS1 shell variable  https://review.openstack.org/29798409:19
*** weiyu_ has joined #openstack-kolla09:23
*** coolsvap has joined #openstack-kolla09:24
openstackgerritMerged openstack/kolla: Remove /etc/kolla configuration folders  https://review.openstack.org/30305509:28
*** jmccarthy has left #openstack-kolla09:29
*** andrewvwebber has joined #openstack-kolla09:30
openstackgerritMerged openstack/kolla: Add note about heat being busted for development  https://review.openstack.org/29027609:32
*** weiyu_ has quit IRC09:35
*** weiyu_ has joined #openstack-kolla09:36
*** daneyon_ has joined #openstack-kolla09:43
*** daneyon_ has quit IRC09:48
elemoine_sdake: what makes you think it's a multifield?09:49
sdakenothign just would be handy if it were09:49
*** shardy has quit IRC09:52
*** weiyu_ has quit IRC09:54
*** weiyu_ has joined #openstack-kolla09:55
*** openstackstatus has quit IRC09:57
*** openstack has joined #openstack-kolla09:58
*** openstackstatus has joined #openstack-kolla09:59
*** ChanServ sets mode: +v openstackstatus09:59
*** shardy has joined #openstack-kolla10:02
*** salv-orl_ has quit IRC10:06
sdakeelemoine_ akwasnie https://drive.google.com/open?id=0B8q6xDPETSkHc1NJV2VxOTZJU1E10:07
*** achanda has quit IRC10:08
sdakeelemoine_ akwasnie time for bed, but heres a start at a dashboard http://paste.fedoraproject.org/354583/14604558/ -> http://paste.fedoraproject.org/354583/1460455810:10
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Add missing base os values to kolla-build.conf  https://review.openstack.org/30449910:11
*** sdake has quit IRC10:11
*** weiyu_ has quit IRC10:12
*** salv-orlando has joined #openstack-kolla10:15
*** ccesario___ has quit IRC10:15
*** weiyu_ has joined #openstack-kolla10:22
*** ccesario___ has joined #openstack-kolla10:23
*** ccesario___ has quit IRC10:32
*** weiyu_ has quit IRC10:33
*** ccesario___ has joined #openstack-kolla10:34
*** achanda has joined #openstack-kolla10:38
*** ccesario___ has quit IRC10:39
*** ccesario___ has joined #openstack-kolla10:39
*** ccesario___ has quit IRC10:42
*** tfukushima has quit IRC10:43
*** achanda has quit IRC10:47
*** ccesario___ has joined #openstack-kolla10:47
*** salv-orlando has quit IRC10:48
*** salv-orlando has joined #openstack-kolla10:48
*** weiyu has joined #openstack-kolla10:52
*** salv-orlando has quit IRC10:53
coolsvapnew cleanup-host should not delete the kolla-build.conf :/10:53
*** ccesario has joined #openstack-kolla10:59
ccesariomorning :)10:59
*** ccesario___ has quit IRC11:00
*** dave-mccowan has joined #openstack-kolla11:01
*** banix has joined #openstack-kolla11:02
*** weiyu has quit IRC11:03
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: cleanup-host should not remove kolla-build.conf  https://review.openstack.org/30451811:03
*** weiyu has joined #openstack-kolla11:05
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Add Mistral Binary Containers for CentOS  https://review.openstack.org/30451911:06
*** achanda has joined #openstack-kolla11:08
*** banix has quit IRC11:10
*** achanda has quit IRC11:12
*** weiyu has quit IRC11:16
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Add Mistral Binary Containers for CentOS  https://review.openstack.org/30451911:17
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Add Mistral Binary Containers  https://review.openstack.org/30451911:21
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Add missing base os values to kolla-build.conf  https://review.openstack.org/30449911:32
mlimacoolsvap, sorry i don't use kolla-build.conf and therefore I did not notice the failure11:36
*** banix has joined #openstack-kolla11:36
coolsvapmlima: np11:37
coolsvapI found out and submitted for review11:38
mlimacoolsvap, i did it https://review.openstack.org/303575 :)11:39
*** rhallisey has joined #openstack-kolla11:39
coolsvapmlima: i am not sure about keeping the scripts in etc/tools11:41
coolsvapbut I will check, im setting up my dev environment again today, will review11:42
mlimacoolsvap, ok :)11:42
mlimabut this role is very usefull11:42
mlima^^11:42
*** banix has quit IRC11:44
coolsvapmlima: agree11:44
*** openstack has quit IRC12:04
*** openstack has joined #openstack-kolla12:05
*** coolsvap is now known as coolsvap|away12:25
*** salv-orlando has joined #openstack-kolla12:39
*** jtriley has joined #openstack-kolla12:47
*** fragatina has joined #openstack-kolla12:51
*** stvnoyes has left #openstack-kolla12:52
*** mwheckmann has joined #openstack-kolla12:54
*** andrewvwebber has quit IRC12:55
*** klint has quit IRC13:01
*** ayoung has quit IRC13:12
nihilifercan someone ack this commit? https://review.openstack.org/#/c/301636/13:18
patchbotnihilifer: patch 301636 - kolla-mesos - RBMQ cluster cant use ip adrr as names, so we reve...13:18
*** daneyon_ has joined #openstack-kolla13:19
*** daneyon_ has quit IRC13:24
openstackgerritMerged openstack/kolla-mesos: RBMQ cluster cant use ip adrr as names, so we reveting this back to hostnames  https://review.openstack.org/30163613:29
*** mgoddard_ has joined #openstack-kolla13:45
*** tfukushima has joined #openstack-kolla13:46
*** mgoddard has quit IRC13:49
*** rstarmer has joined #openstack-kolla13:51
*** allen_gao has quit IRC13:52
*** banix has joined #openstack-kolla13:54
*** jhesketh has left #openstack-kolla13:55
*** andrewvwebber has joined #openstack-kolla13:56
pmisiakhi14:01
pmisiakI have a problem with cinder-volume on centos:source14:01
pmisiakI can't create volume on a LVM14:01
pmisiaklvcreate is waiting on semaphore forever14:02
pmisiakanyone had the same issue here?14:03
*** rstarmer has quit IRC14:03
pmisiakhmm, anyway i see LVM backend will not work14:12
pmisiakbecause i dont see any mechanism for LVM volume attaching to VMs14:12
pmisiakthere is no iscsi target configured or something similar14:13
sbezverkpmisiak the patch for lvm2 support and iscsi has not been yet merged14:13
pmisiakso i should use ceph?14:14
sbezverkif everything goes well, it should be merged today, I just need to convience core to ack last part of PS14:14
sbezverkpmisiak you can cherry pick my patch and try14:15
sbezverkpmisiak or wait for it hitting the master or of course use ceph ;-)14:15
pmisiaki think it wont resolve my issue because cinder-volume can not create LVM volume :/14:22
pmisiakso I have no volumes to attach14:22
pmisiakthis command hangs when run by cinder-volume inside container: /sbin/lvcreate -n cf76107a-183c-474e-aba0-a04d85ea7749 cinder-volumes -L 1g14:23
pmisiakwhen I run this commant directly on host volume is created14:24
pmisiakwhen I run this command directly on host, volume is created14:25
pmisiakmaybe this is because I have centos containers on Ubuntu host?14:26
pmisiaksome udev/kernel mismatches14:26
sbezverkpmisiak believe me I have it working lvm2 as a backend14:27
sbezverkfor cinder and iscsi :-)14:28
pmisiakdo you have the same OS on host and containers?14:28
sbezverkpmisiak your current issue is because by default container runs in a differnet ipc name space than your hos14:28
sbezverkso lvcreate cannot get ack from udev on creation of new device14:29
*** inc0 has joined #openstack-kolla14:29
sbezverkwe explicitely tells containers which needs to create lvm to run in the same ipc name space oas the host14:29
pmisiakin yours patch ?14:30
sbezverkyes14:30
pmisiakok, so I will try14:30
sbezverkit creats all infra you need for lvm2 to be backend of cinder14:30
sbezverkdo not forget to check the doc which is already merged into master14:31
sbezverkit is funny doc is ready but patch not ;-) usually it is in reverse14:31
*** sdake has joined #openstack-kolla14:32
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH  https://review.openstack.org/29490614:34
inc0good morning14:35
*** allen_gao has joined #openstack-kolla14:36
pmisiakso I have to recreate all nova-compute and cinder-volume containers?14:38
sdakemorning14:38
sdakeinc0 i wrote a dashboard dude14:39
sdakedid you see it14:39
sdakeakscram14:39
openstackgerritMerged openstack/kolla: Rabbitmq Upgrade restarts container only when image was changed.  https://review.openstack.org/30385814:39
sdakeakwasnie ^^14:39
inc0sdake, so far I'm trying my kibana to work14:40
inc0but I didn't have much time to debug what's wrong14:40
sdakedid you see my screenshot?14:40
inc0no sir I did not14:41
openstackgerritMerged openstack/kolla: Create and chown /etc/heka directory  https://review.openstack.org/30274214:41
sdakehttps://drive.google.com/open?id=0B8q6xDPETSkHc1NJV2VxOTZJU1E14:41
inc0this is cool dashboard indeed;)14:42
sdakemy wife made the first one in 45 minutes14:42
sdakeit took me about 4 hours to figure out how kibana worke well enough to do that14:42
sdakeat hte bottom is a scroller of the filtered log mesages14:42
*** haplo37 has joined #openstack-kolla14:43
sdakewe have some work to do with heka i think to make the filtered output look a little more usable in kibana14:43
*** mgoddard_ has quit IRC14:43
inc0will you upstream it?14:44
*** mgoddard has joined #openstack-kolla14:44
sdakeupstream is not a verb!14:44
sdakeI was hoping someone would take it over14:44
sdakei have a super full plate atm14:44
sdakehttp://paste.fedoraproject.org/354700/72295146/ -> http://paste.fedoraproject.org/354700/7229514614:45
inc0did you become grammar nazi after ML thread or always was hidden one just waiting for an excuse?14:45
*** cbits has joined #openstack-kolla14:45
sdakesorry upstream and verb are a pet peeve of mine :)14:45
sdakeanywy you can take that paste14:46
sdakeand import it14:46
sdakeif you want ot take over the work14:46
ccesariosdake, cool dashboard!!14:46
sdakeif not I guess I will finish the job after release14:46
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix the gate lying  https://review.openstack.org/30420514:46
sdakeonce imported it can be edited live14:47
inc0after release is good, I still need to figure out how to backport14:47
sdakeright14:48
sdakei will actually beocme free after release more so then i am now14:48
sdakebut i had planned pto 20-23rd14:48
sdakedave-mccowan did you want to try that dashboard and see if an import works? :)14:48
dave-mccowansdake sure.  do i save the json on my laptop or on the controller node?14:49
sdakeon laptop14:50
sdakeclick settings->objects->import14:50
sdakethat shoul do the trick14:50
sdakei want to make sure it works for others before i blow away my containers14:50
sdakewhich i need to do to prep for this webinar i have tomorrow at 10am pt14:50
*** sdake_ has joined #openstack-kolla14:51
dave-mccowanit'll take a few minutes.  redeploying now.14:53
dave-mccowandid you see the note i added to the kibana tls bug?14:54
ccesariosdake, talking  about kibana, what to do  when Unable create a index pattern ?! http://puu.sh/ofC8I/bdfaea1534.png    like describe in kolla  quickstart doc14:54
*** sdake has quit IRC14:55
*** blahRus has joined #openstack-kolla14:58
ccesarioguys does kibana config should not point to IP address,  since enabled http://paste.openstack.org/show/493819/14:58
ccesario?14:59
ccesarios/IP address/VIP address/14:59
*** inc0 has quit IRC15:01
*** dougs1 has joined #openstack-kolla15:02
*** coolsvap|away is now known as coolsvap15:05
*** dougs has quit IRC15:05
dave-mccowansdake_ i imported the dashboard15:07
sdake_now click dashboard15:08
sdake_it should jut work15:08
sdake_or yu may have to load it first15:08
sdake_i'd like to know which on a fresh install15:08
coolsvapmlima: can you tell me what you expect in the review?15:08
coolsvaphttps://review.openstack.org/#/c/30451915:09
sdake_coolsvap if your takingabut mistral15:10
sdake_I just added a review to it15:10
sdake_to clarify mlima's comments15:10
sdake_feel free to read :)15:10
coolsvapsdake_: thanks15:10
dave-mccowani had to setup a default index.  i did that.  then opened the dashboard.  i got a lot of errors "could not locate that visualization"15:11
sdake_dave-mccowan do you have things in your object list?15:11
sdake_let me make sure I sent you the right file15:11
dave-mccowansettings->objects has one dashboard, zero searches, zero visualizations.15:12
*** sdake has joined #openstack-kolla15:12
sdakehad to reconnect15:12
sdakecoolsvap ncie work on mistral bp btw :)15:12
sdakedave-mccowan can you tell me what visualization it says is missing and verify your json file is 25k15:13
sdakehttp://paste.fedoraproject.org/354713/47405214/ -> http://paste.fedoraproject.org/354713/4740521415:14
dave-mccowanmy json file is tiny. 22 lines.15:14
sdakeoh i guess i gave yu wrong one initially15:14
sdakemy bad ;(15:14
sdakejust import that one instead15:14
sdakeit wasn't intentional I have a bunch of files son my hd15:14
sdakeor rather ssd I guess :)15:14
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Add Mistral Binary Containers  https://review.openstack.org/30451915:15
*** sdake_ has quit IRC15:16
dave-mccowana little better.  i have one widget and 3 errors now.  missing Total-Logs-Count-for-Today, Log-Messages-Past-Hour, and Total-Count-by-Severity.15:16
coolsvapsdake: just found the packages on delorean so added the patchset15:16
mlimacoolsvap, sorry i should have explained better :)15:17
coolsvapmlima: np15:17
sdakedave-mccowan i need to export the latest moment15:17
coolsvapactually i found it different from what you said so just wanted to confirm15:17
sdakecoolsvap look at other containers15:17
coolsvapsdake: i did15:17
dave-mccowanthat json has 6 dashboards.  probably i have selected the wrong one. (and there are 5 that should be removed)15:18
*** openstackgerrit has quit IRC15:18
sdakethe idea is to break after the -y \15:18
sdakeoh15:18
sdakeright15:18
sdakekolla diagnotics15:18
*** openstackgerrit has joined #openstack-kolla15:18
sdakeya it was 3am when i finished15:18
mlimasdake, i think that add a CONTRIBUTING.rst can be interesting15:18
sdakeor semi-finished15:18
sdakewe have a contirbuting guide dont 'we?15:18
mlimano, we haven't15:19
*** ayoung has joined #openstack-kolla15:20
dave-mccowansdake yep.  looks nice.15:20
*** vhosakot has joined #openstack-kolla15:20
sdakemlima feel free to make one :)15:20
mlimaok15:20
sdakethat will be some serious pain ;)15:20
sdakedave-mccowan thee box in the lower left is wrong15:21
sdakeits supposed to wor kas a filter15:21
sdakeso you can select two seveities at once15:21
sdakeand ssee all the logs with those severities15:21
sdakei had it working at one point15:21
sdakeand then changed something15:21
sdakeand it stopped working15:21
sdakeupper left box also not reporting in a style I like15:21
openstackgerritMerged openstack/kolla: Create /var/cache/swift in source installation  https://review.openstack.org/30384915:22
sdakethe real quetion is - is it usable15:22
*** salv-orlando has quit IRC15:22
sdakeeach of those things per box acts as a filter15:22
sdakefor the log text15:23
sdakeyou cancreate warnings by going over quota15:23
dave-mccowani clicked around.  i like it.  easy to say show me "warnings" + "neutron".15:23
sdaketeh way I do that is create a flavor that requires 1000gb disk15:23
sdakei think we need changes in something15:23
sdakebecause the programname field isn't always the right data to use15:24
sdakeit would be nice if the dashboard loading could be automated15:24
sdakemy wife siad it can be curled into es15:24
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Aodh dockerfiles for formatting  https://review.openstack.org/30469915:24
sdakeshe wrote the first dashboard which took her 45 minutes15:24
*** achanda has joined #openstack-kolla15:25
sdakedave-mccowan i'd like to get to soething more like this15:29
sdakehttp://fuel-plugin-elasticsearch-kibana.readthedocs.org/en/stable/_images/kibana_logs_dash.png15:29
vhosakotooh, cool UI15:30
sdakevhosakot did you see my go at it15:31
sdakehttps://drive.google.com/open?id=0B8q6xDPETSkHc1NJV2VxOTZJU1E15:31
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Ceilometer dockerfiles for formatting  https://review.openstack.org/30470415:32
vhosakotsdake: nice!15:32
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Aodh dockerfiles for formatting  https://review.openstack.org/30469915:34
*** logan- has quit IRC15:34
*** achanda has quit IRC15:38
*** prithiv has joined #openstack-kolla15:39
ccesarioguys, could someone please, confirm if is it right!?  http://paste.openstack.org/show/493819/15:40
*** logan- has joined #openstack-kolla15:40
*** sdake_ has joined #openstack-kolla15:41
*** sdake has quit IRC15:43
*** mikelk has quit IRC15:43
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update dockerfiles for formatting  https://review.openstack.org/30471615:46
sdake_coolsvap can you do those one docker set at a time please15:46
sdake_as in break apart that review15:46
coolsvapsdake_: its only one merged since gnocchi had only one file but i can split15:47
*** salv-orlando has joined #openstack-kolla15:48
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Cinder dockerfiles for formatting  https://review.openstack.org/30471615:50
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Gnocchi dockerfiles for formatting  https://review.openstack.org/30471715:52
*** tfukushima has quit IRC15:54
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Heat dockerfiles for formatting  https://review.openstack.org/30472015:55
*** sdake_ has quit IRC15:55
*** athomas has quit IRC15:55
*** sdake has joined #openstack-kolla15:55
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Heka dockerfile for formatting  https://review.openstack.org/30472515:58
*** prithiv has quit IRC16:01
*** daneyon_ has joined #openstack-kolla16:01
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Ironic dockerfile for formatting  https://review.openstack.org/30472716:01
sdakesomething isn't quite right with how heka reports programname16:03
sdakeor how i'm using the fields in kibana one of the two ;)16:03
gmmahavhosakot: i tried getting linuxbridge running withthe patch and the contaner stil kept crashing16:03
gmmaha:(16:03
gmmahahave a feeling my setup is busted..16:03
gmmahawont have time to debug that no unfortunately..16:04
gmmahasorry couldnt be of help16:04
*** jmccarthy1 has joined #openstack-kolla16:04
*** jmccarthy1 has left #openstack-kolla16:04
*** prithiv has joined #openstack-kolla16:04
sdakegmmaha its ok dude, thanks for all the help so far!16:05
gmmahasdake: :)16:05
sdakegmmaha feelings good about kolla or badly?16:05
gmmahaohh definitely good16:05
sdakeI recognze this last minute bustage with linuxbridge is subptimal16:05
gmmahaand ntpttr will vouch for that :)16:05
*** daneyon_ has quit IRC16:06
*** prithiv has quit IRC16:06
ntpttryes, certainly a better experience than OSA for us I'd say16:06
gmmahasdake: i think the linuxbridge breakage is a small thing..16:07
sdakewell not if we ship our artifact as such :)16:07
gmmahai am told that manjeet was able to get it working but didnt see huge improvements necessarily16:07
gmmahasdake: :)16:07
sdakei'm glad it was able to be caught16:07
vhosakotgmmaha: linuxbridge container is not starting, or vm is not racing active state ?16:07
vhosakotreaching*16:07
gmmahavhosakot: linuxbridge_agent container is constantly restarting16:07
sdakegot any log of it?16:08
sdakedocke logs linuxbridge_agent16:08
vhosakotgmmaha: are you testing this ? https://review.openstack.org/#/c/302721/16:08
patchbotvhosakot: patch 302721 - kolla - Fix launch vm failed in neutron-linuxbridge-agent env16:08
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Magnum dockerfiles for formatting  https://review.openstack.org/30473216:08
gmmahavhosakot: yeah16:09
gmmahathats the patch16:09
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Manila  dockerfiles for formatting  https://review.openstack.org/30473616:10
*** dwalsh has joined #openstack-kolla16:13
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Marathon dockerfile for formatting  https://review.openstack.org/30474116:13
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Manila  dockerfile for formatting  https://review.openstack.org/30473616:14
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Mariadb dockerfile for formatting  https://review.openstack.org/30474316:15
vhosakotgmmaha: yeah,, docker logs will help16:16
gmmahavhosakot: thre was nothing useful in them16:17
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Memcached dockerfile for formatting  https://review.openstack.org/30474416:17
gmmahavhosakot: it said that its sarting the linuxbridge_agent with the conf file and ini file16:18
gmmahaand then crashed out16:18
*** jasonsb has quit IRC16:18
ccesariogmmaha, are you using multinode ?16:19
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Mesos dockerfiles for formatting  https://review.openstack.org/30474716:19
gmmahaccesario: yes i am16:19
sbezverkgmmaha you do have a fix for missing ebtable binary right?16:19
gmmahasbezverk: yes i pulled the patch16:19
gmmahawas tesing it out16:19
ccesariogmmaha, after apply this ps https://review.openstack.org/#/c/302721/  did you rebuild the images ?16:20
patchbotccesario: patch 302721 - kolla - Fix launch vm failed in neutron-linuxbridge-agent env16:20
gmmahai have a feeling it has to do with my env16:20
gmmahaccesario: yes, removed all images, rebuilt them16:20
ccesariohmmm rigth.... only to check ... :)16:20
gmmahavhosakot: maybe you can test it when uou get a chance and we will know if its the patch or my setup16:20
ccesariolet me try deploy with this patch16:20
gmmahaccesario: awesome.. thanks16:20
ccesariogmmaha, could you pls share your globals ?16:21
vhosakotccesario: thanks! please cleanup first and then re-deploy16:21
vhosakotccesario: also, remove OVS bridges in existing deployment16:21
ccesariovhosakot, yes... and rebuild the images :)16:21
vhosakotgmmaha: did you have ovs bridges from before ?16:21
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Mongodb dockerfile for formatting  https://review.openstack.org/30474816:22
gmmahavhosakot: yeah16:22
gmmahabut i made sure none of those existed in the system, had the rm rebooted16:23
ccesariovhosakot, I dont 't  have ovs brigdes :)16:23
gmmahaccesario: aah changed it to get back to OVS and stuff16:23
gmmahabut yes i can share the globals16:23
sbezverkgmmaha sorry if it is duplicate request, could you share docker logs from linux bridge container16:25
*** achanda has joined #openstack-kolla16:25
ccesarionice  :)16:25
gmmahasbezverk: the docker logs didnt show anything16:25
ccesarioI 'm rebuilding the images now16:25
gmmahait jsut said its starting linuxbridge_agent with options and then the same logs over and over again16:25
gmmahanothing to show errors16:25
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Neutron dockerfiles for formatting  https://review.openstack.org/30475516:25
vhosakotno interface except docker0 should be there.... al those ovs stale bridges, ovs-system, blah blah must be deldted16:27
*** achanda has quit IRC16:30
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Nova dockerfiles for formatting  https://review.openstack.org/30476316:30
openstackgerritMerged openstack/kolla: Put the variables in alphabetical order in all.yml  https://review.openstack.org/30379616:30
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update OpenVSwitch dockerfile for formatting  https://review.openstack.org/30476516:32
pbourkebug ids are still required for all changes hitting master atm right?16:33
*** unicell has quit IRC16:34
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Swift dockerfiles for formatting  https://review.openstack.org/30477516:35
*** ccesario___ has joined #openstack-kolla16:36
pbourkecoolsvap: killing me with the multiple patches :p16:36
coolsvappbourke: sorry i am just putting for each service16:36
coolsvapalmost done16:36
vhosakotcoolsvap: you could have merged into a single PS16:38
vhosakot:)16:38
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Trove dockerfiles for formatting  https://review.openstack.org/30477716:38
openstackgerritMerged openstack/kolla: Configure Neutron to use internalURL for contacting Nova  https://review.openstack.org/30294816:38
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Zaqar dockerfile for formatting  https://review.openstack.org/30477916:40
andrewvwebberubuntu/source 'Failed sending gratuitous ARP to 10.0.0.1 on qr-eb5f9327-f4 in namespace qrouter-923e9b8c-6d8c-440c-bb2d-2da1c9168af2' - anything to be concerned about?16:41
ccesariopbourke,  The bugs fixed on master will be backported/merged on mitaka branch?16:42
*** mark-casey has joined #openstack-kolla16:44
*** ccesario___ has quit IRC16:44
vhosakotandrewvwebber: wow, when do you see this ? when booting a VM ?16:44
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Add  iscsid/tgtd specific parameters to Kolla infrastructure  https://review.openstack.org/30000416:46
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Adds ansible code for iscsid and tgtd containers  https://review.openstack.org/29999516:46
andrewvwebbervhosakot: yes16:47
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update base dockerfile for formatting  https://review.openstack.org/30478316:48
andrewvwebbervhosakot: well technically when creating a router16:49
vhosakotandrewvwebber: is there an IP conflict somewhere ?16:50
andrewvwebbervhosakot: you mean on the external network?16:51
vhosakotandrewvwebber: yep, and also in the qrouter namespace16:52
vhosakotandrewvwebber: also, was mac address of any interface changed manuaaly ?16:53
andrewvwebbervhosakot: is is a fresh machine, wiped. i have not changed any mac address. i will check the external network16:54
*** ydeshmukh has quit IRC16:54
vhosakotandrewvwebber: i'd check neutron l3 agent log16:55
vhosakotsdake: should we link kolla design summit link here ?  https://wiki.openstack.org/wiki/Design_Summit/Newton/Etherpads#Kolla16:56
ccesariogmmaha, globals .. pls16:58
andrewvwebbervhosakot: http://pastebin.com/raw/41kUBG8S16:58
gmmahaccesario: aah apologies..16:58
*** salv-orlando has quit IRC16:58
mlimavhosakot, can you remove -1? https://review.openstack.org/#/c/303575/16:58
patchbotmlima: patch 303575 - kolla - Add a role to cleanup containers after deployment16:58
gmmahagtting itnow16:58
sdakevhosakot when i have the sessions schedueld yes16:59
vhosakotsdake: cool16:59
gmmahaccesario: http://paste.openstack.org/show/493838/16:59
vhosakotmlima: did you test it without sudo ?17:00
ccesariogmmaha,17:00
mlimavhosakot, yes17:00
ccesarioneutron_plugin_agent: "openvswitch"   ?17:00
gmmahaccesario: i had to switch it back17:00
gmmaha:)17:00
vhosakotmlima: cool! so, it works without sudo right ?17:00
ccesarioahhh  ok17:00
gmmahaso replace that with linuxbridge17:00
*** cbits has quit IRC17:00
gmmahai had to get the system up running asap.. tehre was soem testing that needed to happen17:00
mlimavhosakot, test you also to confirm. :P17:00
*** ravig_ has joined #openstack-kolla17:00
vhosakotmlima: yep! :)17:00
ccesariorigth ... give some time until my byuild finish...17:01
vhosakotmlima: did you test it in multinode ?17:02
mlimayes17:03
vhosakotmlima: cool, will the tools directory, validate-docker-execute.sh and all the cleanup scirips be copied to each node (control, network, compute and storage) ?17:04
*** Serlex has quit IRC17:04
mlimayes17:04
mlimajust cleanup-containers and cleanup-host17:05
vhosakotright..17:05
*** rajathagasthya has joined #openstack-kolla17:10
*** mgoddard has quit IRC17:11
*** mgoddard has joined #openstack-kolla17:12
*** dougs has joined #openstack-kolla17:12
*** ravig has joined #openstack-kolla17:14
*** gfidente has quit IRC17:14
sdakedo we have some new bug with linuxbridge?17:16
*** dougs1 has quit IRC17:16
sdakeccesario anything in 2.0.0 that is fixed in master will also be fixed in stable/mitaka17:16
sdakepbourke yes, and nyting that will require a backport requires a bug id for management17:17
andrewvwebbervhosakot: systemd-udevd[12804]: Could not generate persistent MAC address for qg-954a3d7e-22: No such file or directory17:17
sdakepbourke i think we cn relax our rules around bug ids after the 15th (or whenver we tag 2.0.0)17:17
sdakebut anything that needs a backport needs a bug id not ot get lost17:18
sdakeandrewvwebber distro type and base?17:18
andrewvwebberubuntu/source17:19
sdake4.2 kernel?17:20
*** unicell has joined #openstack-kolla17:20
sdakeandrewvwebber 14.04 or some othr version?17:20
andrewvwebber4.5.017:20
sdakeandrewvwebber off the top of my head my gues is the base distro doessn't match the continer image distro17:21
*** ravig has quit IRC17:23
andrewvwebbersdake: does this mean my container build server is too old?17:23
sdakewhich ubuntu distor are you running on your host17:23
sdakehttps://github.com/openstack/kolla/blob/master/kolla/common/config.py#L6217:24
*** salv-orlando has joined #openstack-kolla17:24
andrewvwebberim running coreos, i was successful last week however i decided to rebuild the containers and now external SNATing no longer works. SSH between instances works17:24
sdakeyou see there our ubuntu runtime is ubuntu:latest17:24
sdakenobody tests that case17:25
sdakeyour probably teh first :)17:25
mlimasdake, can you review this patchet https://review.openstack.org/#/c/303575/ ?17:25
patchbotmlima: patch 303575 - kolla - Add a role to cleanup containers after deployment17:25
andrewvwebberim happy to be your coreos tester17:25
sdakei'd try centos source in the short term17:25
andrewvwebberthanks17:26
sdakethe problem is the systemd in the containers is incompatible with the systemd on the system17:26
sdakeccentos is more up to date wrt systemd then the ubuntu 14.04 image ;)17:26
*** ravig_ has quit IRC17:27
andrewvwebberi moved to ubuntu due to a bug that is probably now fixed :)17:27
sdakethe userpsace and kernel need to match for containers to really wor properly17:27
sdakebut coreos does not provide a userspace for contaner runtimes17:27
sdakewe do some compelx things like communicating with the host systemd directly17:27
sdakethat abi must match also17:27
sdakewhich is why we have multidistor support in the first place ;)17:28
sdakeyou were running centos source when it worked?17:28
sdakeor ubuntu source?17:28
andrewvwebberubuntu17:28
sdakewe being libvirt17:28
sdakelibvirtuses the dbus libraries to communicate with systemd i think ;)17:29
sdakethey are supposed to be versioned17:29
sdakebut ymmv17:29
mark-caseysdake: for my understanding is this a requirement that the machine that builds the images is similar to the one that runs them, or is this based on things that change based on the value of --base17:29
sdakemark-casey not builds but  runs17:29
sdakethe container image contains things like /usr/lib/include17:30
sdakeand other libraries17:30
andrewvwebberso i have to be careful of coreos-updates...17:30
sdakeall of which have abis of some type or another17:30
sdakeif they are incompatible, kolla just wont work17:30
*** cbits has joined #openstack-kolla17:30
andrewvwebberunderstood17:30
sdakethis is a problem docker likes to pretend doessn't exist17:30
sdakeit even existss if you use all isolation features of kernel17:31
andrewvwebberthis is probably a case for being able to build the images within a container...?17:31
mark-caseyandrewvwebber: I have done that successfully17:31
sdakeof kernel/of docker17:31
mark-caseywell... Horizon is up. I haven't launched a VM on said images17:31
andrewvwebbermark-casey: legendary...17:31
*** daneyon_ has joined #openstack-kolla17:33
andrewvwebbersdake: do you now recommend centos/rdo(binary) again?17:33
sdakecentos source17:33
andrewvwebberok17:33
sdakerdo binary waiting on rdo package maintainers to finish their job17:33
andrewvwebber:)17:34
*** achanda has joined #openstack-kolla17:34
mlimasdake, mitaka milestone is wrong here? https://bugs.launchpad.net/kolla/mitaka/+bug/156397217:36
openstackLaunchpad bug 1563972 in kolla mitaka "Manila keystone v3 issues for all clients" [Low,Fix committed] - Assigned to Mauricio Lima (mlima) (mauricio-lima)17:36
*** andrewvwebber has quit IRC17:36
*** salv-orl_ has joined #openstack-kolla17:37
*** andrewvwebber has joined #openstack-kolla17:37
sdakefixed17:38
*** salv-orlando has quit IRC17:39
mlima:)17:39
mark-caseyandrewvwebber: I'm running all of my Kolla infra from Vagrant to facilitate tearing down and rebuilding PXE and docker registry. Should it be of use this is the box for image building/the deployer https://github.com/ropsoft/kolla_from_vagrant/blob/master/Vagrantfile#L107-L115 (which just builds https://github.com/ropsoft/kolla_from_vagrant/blob/master/deployer_dockerfile/Dockerfile)17:40
*** sdake_ has joined #openstack-kolla17:40
*** ravig has joined #openstack-kolla17:40
mark-caseyMAAS's PXE would probably work ok for coreos if you weren't planning on using CoreOS's own PXE solution, which IIRC is still in beta17:40
*** daneyon_ has quit IRC17:41
andrewvwebbermark-casey: it sounds like the key factor is the vagrant VM being the same kernel version as the target hosts. so i guess i need a coreos vagrant host..17:42
*** ccesario___ has joined #openstack-kolla17:42
*** sdake has quit IRC17:43
andrewvwebbermark-casey: i am planning on upgrading an existing coreos bare-metal cluster - with no down time :)17:43
mark-caseyandrewvwebber: re: coreos vagrant. ah, yea that could be complex. To do the image build in a container I assume you'd have to tell vagrant to use coreos's box image as the VM to run docker inside17:45
mark-caseyand awesome on the upgrade17:45
*** shardy has quit IRC17:47
mark-caseyI'd love to see some sort of how-to! I'm headed that way myself but probably going with Ubuntu Wily for the time being17:47
*** ccesario___ has quit IRC17:47
andrewvwebbermark-casey: i cant physically live without systemd17:47
*** ccesario___ has joined #openstack-kolla17:47
mark-caseyandrewvwebber: Wily has it. I have to run the CentOS steps for /run17:48
mark-caseybut I still want to move later17:48
andrewvwebbermark-casey: minimal OSs like Atomic and CoreOS are fun and actually work. I would expect to see a Kolla/Atomic blog sometime soon17:50
*** sdake has joined #openstack-kolla17:54
andrewvwebbermark-casey: your Dockerfile is very useful. I will give it a try on CoreOS using a docker.sock volume. thanks17:54
mark-caseysure. the docker run command I used for the kolla-build is in readme17:55
*** andrewvwebber has quit IRC17:57
*** sdake_ has quit IRC17:57
*** ccesario___ has quit IRC17:57
*** ccesario___ has joined #openstack-kolla17:57
*** banix has quit IRC17:58
*** daneyon_ has joined #openstack-kolla18:01
*** ccesario___ has quit IRC18:02
*** ravig_ has joined #openstack-kolla18:05
* coolsvap want to give try out the kibana dashboard18:09
coolsvapbut leaving now, first thing tomorrow morning18:10
*** coolsvap has quit IRC18:10
*** achanda has quit IRC18:23
*** salv-orl_ has quit IRC18:26
*** salv-orlando has joined #openstack-kolla18:27
sbezverkis it s known issue kolla_toolbox is failing to be built on the latest master?18:29
ccesariosbezverk, not for me18:30
*** salv-orlando has quit IRC18:31
sbezverkhm, I will try to rebuild, maybe it was intemittent glitch.18:32
*** daneyon_ has quit IRC18:33
gmmahaccesario: sorry if you had alrady mentioned, were you able to get linuxbridge working on your setup18:36
gmmahaif you are still building things, then i will wait.. :)18:36
*** daneyon_ has joined #openstack-kolla18:37
ccesariogmmaha, finishing the deploy18:40
ccesariosorry for delay18:40
gmmahaccesario: no worries..18:40
gmmahanothing of a rush.. just more curious18:40
ccesarioPLAY RECAP ********************************************************************18:43
ccesario192.168.201.2              : ok=291  changed=61   unreachable=0    failed=018:43
ccesariogmmaha, any specific test ? the deploy it working and the process is up///18:44
ccesariolet me check endpoints18:44
gmmahaccesario: is the linuxbridge_agent container running properly?18:45
sdakeok cats18:45
sdakemitaka is out18:45
sdakei am try8ing to get it to  build18:45
sdakerather mitaka rdo18:45
ccesariogmmaha, yes ...is running18:46
gmmahaccesario: on my tests the deploy was successful, but the linuxbridge_agent container was on a constant reboot pat18:46
gmmahapath*18:46
gmmahaccesario: great.. thanks.. then i guess the patch is fine, but something funky with my setup18:46
ccesariogmmaha, http://paste.openstack.org/show/493862/18:50
gmmahaccesario: awesome.. atleast now we know neutron linuxbrdige is working with the patch18:51
ccesarioe8958739e714        operator.local:4000/kolla/ubuntu-source-neutron-linuxbridge-agent:2.0.0   "kolla_start"           9 minutes ago       Up 9 minutes                            neutron_linuxbridge_agent18:51
gmmahanice18:51
ccesariogmmaha, any more spcific test ?18:51
gmmahanot sure why its on a constant reboot path on my setup.. but its a debug for another day18:51
ccesariobecause I'll destry my env to test other thing :P18:51
gmmahaccesario: i dont have any.. unless vhosakot or sdake have any18:51
ccesariofell free to ask :)18:52
gmmahaccesario: i couldnt get it to this far. :)18:52
*** banix has joined #openstack-kolla18:54
*** daneyon_ has quit IRC18:59
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Add  iscsid/tgtd specific parameters to Kolla infrastructure  https://review.openstack.org/30000419:00
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Adds ansible code for iscsid and tgtd containers  https://review.openstack.org/29999519:00
*** daneyon_ has joined #openstack-kolla19:01
vhosakotccesario: excellent!  thanks a lot for testing! are you able to boot a VM fine witt that linuxbridge path19:01
*** rajathagasthya has quit IRC19:02
*** yuanying has quit IRC19:04
*** achanda has joined #openstack-kolla19:05
*** achanda has quit IRC19:05
ccesariovhosakot, doing it now19:05
vhosakotccesario: cool!19:05
ccesariogive me a momment to complete it....19:06
ccesariovhosakot, $ ip netns exec qrouter-010b0ce0-1dcd-46d6-9105-5907029da365 ifconfig19:16
ccesarioCannot open network namespace "qrouter-010b0ce0-1dcd-46d6-9105-5907029da365": Permission denied19:16
ccesariodenied ?19:16
*** inc0 has joined #openstack-kolla19:26
inc0hey I'm back19:27
sbezverkguys I consistently see kolla_toolbox is failing to built in centos/source19:28
sbezverkon the latest master19:28
*** allen_gao has quit IRC19:29
sbezverkhttp://paste.openstack.org/show/493868/19:30
*** ravig__ has joined #openstack-kolla19:30
*** ravig has quit IRC19:30
*** allen_gao has joined #openstack-kolla19:31
inc0sbezverk, merged your patches:)19:35
inc0if we'll run into problems, we'll fix it as bugs19:35
sbezverkinc0 for sure, I am willing to gets assigned automatically  for iscsi related bugs ;-)19:36
mlimainc0, can you review it https://review.openstack.org/#/c/303575/ ?19:36
patchbotmlima: patch 303575 - kolla - Add a role to cleanup containers after deployment19:36
*** rajathagasthya has joined #openstack-kolla19:37
openstackgerritMerged openstack/kolla: Adds iscsid and tgtd docker containers  https://review.openstack.org/29128519:38
inc0mlima, done19:44
mlimaI disagree inc0 "/etc/ is not a place for code, it's place for configs. Also I'd rather see ansible do cleanup than copy shell and then run ansible. A bit more work but much more reliable, less error prone and error handling will be well..there19:45
mlima"19:45
inc0ok, let's talk then19:47
inc0what aren't you agreeing with specifically?19:47
mlimai left a comment there19:50
mlima:)19:50
fragatinahi!19:50
fragatinaquestion about kolla-build19:50
fragatinaI'm trying to deploy kolla and I get an error because the nova images where not build (I'm using ubuntu/source)19:51
fragatinaI get the following error when I do: kolla-build --base ubuntu --type source nova19:52
fragatinain the following error:19:52
fragatinaINFO:kolla.cmd.build:nova-base:Fetched 3930 kB in 5s (697 kB/s)19:52
fragatinaINFO:kolla.cmd.build:nova-base:E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/libv/libvirt/libvirt0_1.3.1-1ubuntu8~cloud0_amd64.deb  404  Not Found [IP: 91.189.92.152 80]19:52
fragatinaINFO:kolla.cmd.build:nova-base:E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?19:52
fragatinaINFO:kolla.cmd.build:nova-base:19:52
fragatinaINFO:kolla.cmd.build:nova-base:Removing intermediate container c166f0ea751919:53
fragatinaI see that if I can succesfully get a package if I manually do a wget of that link but instead of ubuntu8 I use ubuntu919:54
fragatinabut of course, I cannot change what nova-base pulls down, or I can?19:54
mark-caseymlima inc0    would script module help?   http://docs.ansible.com/ansible/script_module.html19:54
openstackgerritBenedikt Trefzer proposed openstack/kolla: add curlrc file to base image  https://review.openstack.org/30191119:55
*** salv-orlando has joined #openstack-kolla19:55
fragatinaany suggestion of how to proceed with this? I don't have libvirt running on this host19:55
ccesario$ ip netns exec qrouter-010b0ce0-1dcd-46d6-9105-5907029da365 ifconfig    is there any permission needed ?19:55
ccesarioCannot open network namespace "qrouter-010b0ce0-1dcd-46d6-9105-5907029da365": Permission denied19:55
mark-caseyansible -i /usr/local/share/kolla/ansible/inventory/multinode -m script -a '/kolla/tools/cleanup-containers' all19:55
inc0fragatina, is your resolv.conf doing well?19:55
*** vhosakot has quit IRC19:55
inc0I mean it looks like dns issue19:55
inc0what you can do is19:55
inc0docker run -it ubuntu bash19:55
openstackgerritMauricio Lima proposed openstack/kolla: Add a role to cleanup containers after deployment  https://review.openstack.org/30357519:56
inc0and check resolv.conf and such19:56
*** mlima has quit IRC19:57
inc0mark-casey, good idea!19:57
inc0I'd rather have ansible doing logic but for now it's ok19:57
mark-caseycool :)19:58
fragatinathx @inc0, let me see19:59
*** salv-orlando has quit IRC19:59
sbezverkinc0 could you check why two other parts of PS are not merging? especially this one: https://review.openstack.org/#/c/300004/2019:59
patchbotsbezverk: patch 300004 - kolla - Add  iscsid/tgtd specific parameters to Kolla infr...19:59
sbezverkit is all green :-)20:00
inc0sbezverk, https://review.openstack.org/#/c/299995/11 this one has only my +2;)20:01
patchbotinc0: patch 299995 - kolla - Adds ansible code for iscsid and tgtd containers20:01
inc0and another is dependant on this one20:01
*** mark-casey has left #openstack-kolla20:02
sbezverkinc0 got it.. thanks for clarification..20:02
*** vhosakot has joined #openstack-kolla20:03
*** pabelanger has left #openstack-kolla20:04
openstackgerritBenedikt Trefzer proposed openstack/kolla: Use debian repos for debian base docker image  https://review.openstack.org/28893620:04
*** cbits has quit IRC20:06
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make container dind build with debian  https://review.openstack.org/29743420:20
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make heka container build with debian  https://review.openstack.org/29773720:23
*** banix has quit IRC20:24
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make kibana container build with debian  https://review.openstack.org/29720620:24
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make marathon container build with debian  https://review.openstack.org/29737820:25
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make mesos-slave container build with debian  https://review.openstack.org/29738520:27
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make mesosphere-base build with debian  https://review.openstack.org/29735820:28
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make nova-libvirt container build with debian  https://review.openstack.org/29716920:29
ccesariogmmaha, vhosakot  http://puu.sh/ofWXJ/342743d2e3.png http://puu.sh/ofX0W/650d5b3699.png20:30
openstackgerritDave McCowan proposed openstack/kolla: Remove Disable HAProxy option from Quick Start Guide  https://review.openstack.org/30484820:32
*** jtriley has quit IRC20:32
openstackgerritBenedikt Trefzer proposed openstack/kolla: make rabbitmq container build with debian  https://review.openstack.org/29772820:34
fragatinainc0, it doesn't seem a dns issue since I manually pick up the package in: http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/libv/libvirt/libvirt0_1.3.1-1ubuntu8~cloud0_amd64.deb20:34
fragatinabut I cannot get the package in: http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/libv/libvirt/libvirt0_1.3.1-1ubuntu8~cloud0_amd64.deb20:35
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make swift-base container build with debian  https://review.openstack.org/29712120:35
fragatinasorry, typo, the one that I can do succesfully is this one: http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/libv/libvirt/libvirt0_1.3.1-1ubuntu9~cloud0_amd64.deb20:35
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make container kolla-toolbox build with debian  https://review.openstack.org/29710420:36
fragatinaso, kolla-build for nova tries to get libvirt*ubuntu8*.deb package which is not there in ubuntu-cloud but the file name libvirt*ubuntu9*.deb is there in ubuntu-cloud20:36
*** daneyon_ has quit IRC20:36
fragatinaso, is there a way to make kolla pick up the right package from ubuntu cloud?20:37
sbezverkAnybody on centos, please ping back, need sanity check..20:41
*** salv-orlando has joined #openstack-kolla20:41
gmmahaccesario: awesome.. *thumbs_up*20:42
ccesariogmmaha, ;)20:46
*** ccesario___ has joined #openstack-kolla20:50
*** inc0 has quit IRC20:51
*** inc0 has joined #openstack-kolla20:51
*** salv-orlando has quit IRC20:51
*** salv-orlando has joined #openstack-kolla20:52
*** vhosakot has quit IRC21:00
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make container dind unpin old docker relase  https://review.openstack.org/29743421:01
*** sdake has quit IRC21:05
*** daneyon_ has joined #openstack-kolla21:06
*** sdake has joined #openstack-kolla21:08
*** ccesario___ has quit IRC21:09
*** jtriley has joined #openstack-kolla21:10
sdakehttp://paste.fedoraproject.org/354817/14604955/ -> http://paste.fedoraproject.org/354817/1460495521:13
sdakehave a look at that fpaste devs...21:13
sdakethis is how much docker rocks21:13
*** ccesario___ has joined #openstack-kolla21:18
sbezverksdake I am experiencing consisten failure on building kolla_toolbox, the issue is module pytz, the only solution I found to get around it is to install it before it is needed.21:18
*** mark-casey has joined #openstack-kolla21:19
sdakeusing overlay graph driver?21:19
sbezverksdake I saw some notes about removing it and there is rpm -e for this module, notes mentioned your name, do you recall why you wanted to remove pytz?21:19
sdakeyes becaues it doesn't work with source builds21:20
sdakewe have a mix of source and dbinary in the system21:20
sdakewhich is wrong21:20
sdakefor python modules21:20
sdakethis was introduced by the mesos work21:20
sbezverksdake I added it as pip install pytz21:20
sbezverkto Docker's file for kolla_toolbox21:21
sdakeya the above only happens with overlayfs21:22
sdakethe defult graph driver seems to work21:22
sbezverksdake should I file a bug?21:24
sdakeaobut which21:25
sdakewhich graph driver are you using21:25
*** ccesario___ has quit IRC21:25
*** ccesario___ has joined #openstack-kolla21:26
*** ccesario___ has quit IRC21:29
sdakesbezverk ^^^21:31
*** ccesario___ has joined #openstack-kolla21:31
sbezverksdake I am sorry, not sure where graph driver is coming from21:32
sdake--storage-driver= what?21:32
sbezverksdake I am building kolla_toolbox image using build.py, I relly doubt --storage-driver has anything to do here.21:34
*** ccesario___ has quit IRC21:34
sdakethe above paste doesn't happen with default storage driver21:34
*** inc0 has quit IRC21:34
sdakeso yes storage driver has someting to do with it21:35
*** ccesario___ has joined #openstack-kolla21:35
sbezverksdake when you clone master, go into tools and start build.py top build images, where is the storage driver is defined?? Unless it is defined somewhere by default, so I am not touching any default values..21:36
sdakeps -ef | grep docker on your system pleae21:36
sbezverksdake http://paste.openstack.org/show/493877/21:37
sdakeyou are not using the overlayfs graph driver as I was21:38
sdakehow that rpm -e behaves is diferent between he two21:38
sbezverkbb in 30 min21:39
*** daneyon_ has quit IRC21:39
*** ccesario__ has joined #openstack-kolla21:41
*** ccesario___ has quit IRC21:42
*** haplo37 has quit IRC21:42
*** ccesario___ has joined #openstack-kolla21:44
*** ayoung has quit IRC21:46
sdakenice rdo mitaka works21:48
*** ccesario__ has quit IRC21:48
*** ccesario___ has quit IRC21:53
*** sdake_ has joined #openstack-kolla21:53
*** fragatina has quit IRC21:54
*** sdake has quit IRC21:54
*** banix has joined #openstack-kolla21:57
*** salv-orlando has quit IRC22:00
*** salv-orlando has joined #openstack-kolla22:01
openstackgerritSteven Dake proposed openstack/kolla: Port stable/mitaka to use RDO for binary  https://review.openstack.org/30488422:03
*** daneyon_ has joined #openstack-kolla22:04
*** sdake has joined #openstack-kolla22:04
sbezverksdake what is more strange is that yesterday the same operation was working just fine, my building server has not changed, kolla_toolbox's files has not been changed22:05
*** sdake_ has quit IRC22:06
sbezverkthe only difference is the latest master22:06
sdakeand all the bits that make u pteh containers22:06
sdakeincluding the base image and rpms ;)22:06
sdakelots of stuff changes22:06
sbezverksdake I was trying to find somebody with centos/source to test master but could not, would you be able to test it later today?22:13
sdakesbezverk i amgetting a haircut then putting my office together22:14
sdakeso unlikely but the lat time i tested centos source which was yessterday it worked fine22:14
sdakeat some point tonight probably after your asleep i'll be testing my demo and whatnot22:15
sdakemultinode22:15
*** ccesario_ has joined #openstack-kolla22:15
ccesario_night!22:15
*** dwalsh has quit IRC22:16
sbezverksdake ok, I have my workaround so I am ok with my stuff22:16
ccesario_guys, does someone have idea about exec it ?22:17
ccesario_$ ip netns exec qrouter-010b0ce0-1dcd-46d6-9105-5907029da365 ifconfig Cannot open network namespace "qrouter-010b0ce0-1dcd-46d6-9105-5907029da365": Permission denied22:17
ccesario_I 'm running a set of tests and checks, and I remember that ip netns exec it was allowed time ago (maybe neutron conaner it was running as root)22:19
ccesario_another point is that... http://paste.openstack.org/show/493819/22:22
ccesario_since haproxy is enabled, why kibana config does not point to VIP address ?22:22
ccesario_sdake: ^^ any info about these points ?22:23
*** fragatina has joined #openstack-kolla22:24
*** sdake has quit IRC22:25
*** fragatina has quit IRC22:30
*** ayoung has joined #openstack-kolla22:31
*** Marga_ has joined #openstack-kolla22:32
*** daneyon_ has quit IRC22:33
*** Marga_ has quit IRC22:35
*** Marga_ has joined #openstack-kolla22:36
*** asalkeld has quit IRC22:38
*** asalkeld has joined #openstack-kolla22:41
*** jtriley has quit IRC22:49
*** mwheckmann has quit IRC22:52
*** blahRus has quit IRC22:59
*** daneyon_ has joined #openstack-kolla23:03
*** daneyon_ has quit IRC23:07
*** salv-orlando has quit IRC23:08
openstackgerritOpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements  https://review.openstack.org/30489523:09
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-mesos: Updated from global requirements  https://review.openstack.org/30297023:09
*** yuanying has joined #openstack-kolla23:18
*** sdake has joined #openstack-kolla23:18
*** allen_gao has quit IRC23:22
*** daneyon_ has joined #openstack-kolla23:33
*** yuanying has quit IRC23:36
*** allen_gao has joined #openstack-kolla23:37
*** vhosakot has joined #openstack-kolla23:47
*** sdake has quit IRC23:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!