Saturday, 2016-04-09

*** ccesario__ has joined #openstack-kolla00:01
ccesario__good evening :)00:03
*** ccesario___ has quit IRC00:08
*** daneyon_ has joined #openstack-kolla00:11
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Add LVM2/iSCSI documentation  https://review.openstack.org/30082400:13
*** daneyon_ has quit IRC00:15
*** cbits has joined #openstack-kolla00:18
*** cbits has quit IRC00:46
*** ccesario_ has joined #openstack-kolla00:54
*** Marga_ has quit IRC00:56
*** Marga_ has joined #openstack-kolla00:57
*** salv-orl_ has quit IRC00:57
*** stvnoyes has quit IRC01:08
*** jtriley has quit IRC01:09
*** Jeffrey4l_ has joined #openstack-kolla01:13
*** shakamunyi has joined #openstack-kolla01:18
*** unicell has quit IRC01:19
*** achanda has joined #openstack-kolla01:21
*** shakamunyi has quit IRC01:23
*** stvnoyes has joined #openstack-kolla01:23
*** achanda has quit IRC01:30
*** ccesario_ has quit IRC01:31
*** ccesario_ has joined #openstack-kolla01:31
openstackgerritMauricio Lima proposed openstack/kolla: Add a role to cleanup containers after deployment  https://review.openstack.org/30357501:39
openstackgerritMauricio Lima proposed openstack/kolla: Add a role to cleanup containers after deployment  https://review.openstack.org/30357501:40
*** Jeffrey4l_ has quit IRC01:44
*** Kennan has quit IRC01:51
*** Kennan has joined #openstack-kolla01:52
*** cbits has joined #openstack-kolla01:57
*** achanda has joined #openstack-kolla01:59
*** cbits has quit IRC02:03
*** achanda has quit IRC02:07
*** sbezverk has quit IRC02:08
*** dave-mcc_ has joined #openstack-kolla02:09
*** dave-mccowan has quit IRC02:09
*** sbezverk has joined #openstack-kolla02:11
*** mwheckmann has joined #openstack-kolla02:13
*** pbourke has quit IRC02:15
*** pbourke has joined #openstack-kolla02:16
*** salv-orlando has joined #openstack-kolla02:18
*** Kennan has quit IRC02:19
*** banix has joined #openstack-kolla02:21
*** sdake has joined #openstack-kolla02:21
*** Marga_ has quit IRC02:24
*** Marga_ has joined #openstack-kolla02:24
*** salv-orlando has quit IRC02:24
*** wuhg has quit IRC02:29
*** sdake has quit IRC02:29
*** coolsvap has joined #openstack-kolla02:36
*** Kennan has joined #openstack-kolla02:36
*** dave-mcc_ has quit IRC02:43
vhosakotmanjeets: ping02:49
*** mwheckmann has quit IRC02:51
*** klint has joined #openstack-kolla02:52
openstackgerritMerged openstack/kolla: Increase max pool size so conductor doesn't implode  https://review.openstack.org/30241502:57
*** sdake has joined #openstack-kolla03:10
*** mwheckmann has joined #openstack-kolla03:13
*** Kennan has quit IRC03:15
*** Kennan has joined #openstack-kolla03:15
sdakere https://review.openstack.org/#/c/302415/03:16
patchbotsdake: patch 302415 - kolla - Increase max pool size so conductor doesn't implode (MERGED)03:16
sdakedid manjeets confirm it fixed his problem?03:16
openstackgerritSteven Dake proposed openstack/kolla: Increase max pool size so conductor doesn't implode  https://review.openstack.org/30367703:17
*** shakamunyi has joined #openstack-kolla03:18
*** shakamunyi has quit IRC03:23
sdakethis neesd an ack https://review.openstack.org/#/c/303509/03:26
patchbotsdake: patch 303509 - kolla - Fix the qemu has nova permission in nova instances...03:26
*** Kennan has quit IRC03:27
*** Kennan has joined #openstack-kolla03:28
openstackgerritSteven Dake proposed openstack/kolla: Increase max scheduling attempts to 10  https://review.openstack.org/30367803:28
vhosakotsdake: re https://review.openstack.org/#/c/302415/, I gave +2 and workflow as ryan gave +2 and the gate was green.... later, I saw ryan's comment in the reivew that he was waiting for manjeet's confirmation when he gave +1... I pinged manjeets and could not reach him03:29
patchbotvhosakot: patch 302415 - kolla - Increase max pool size so conductor doesn't implode (MERGED)03:29
sdakeok well keep pingin him and get him to veirf yit :)03:30
vhosakotsdake: yes, https://review.openstack.org/#/c/303509/ is great and nova VM in ACTIVE state in gate!03:30
patchbotvhosakot: patch 303509 - kolla - Fix the qemu has nova permission in nova instances...03:30
vhosakotsdake: yes, I will keep findind manjeets :)03:31
sdakewow petty much crashed from noon until 7pm03:32
sdakeatleast sleep is back in order ;)03:32
openstackgerritMerged openstack/kolla: Fix the qemu has nova permission in nova instances folder  https://review.openstack.org/30350903:32
vhosakotsdake: looking at nova.conf's documentation, seeting max_pool_size and max_overflow to high values in nova.conf should not have any side effects to regular loads and in gate.. they will help in scaling for sure...03:33
vhosakotsdake: nice, i got some sleep as well03:33
coolsvaporder or reverse order?03:34
openstackgerritSteven Dake proposed openstack/kolla: Fix the qemu has nova permission in nova instances folder  https://review.openstack.org/30368003:34
vhosakotsdake: https://review.openstack.org/#/c/302036/ just got merge conflict03:35
patchbotvhosakot: patch 302036 - kolla - Use proper CentOS repositories for ceph and qemu03:35
sdakeyes i expected that vhosakot03:35
sdakei was waiting for that patch to mergeto rebase03:35
vhosakotsdake: yep, too many patches touching same line in same file03:35
*** sdake_ has joined #openstack-kolla03:40
*** sdake has quit IRC03:40
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203603:43
openstackgerritSteven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203603:45
*** salv-orlando has joined #openstack-kolla03:48
*** mwheckmann has quit IRC03:50
*** salv-orlando has quit IRC03:52
*** Kennan2 has joined #openstack-kolla03:53
*** Kennan has quit IRC03:55
*** jtriley has joined #openstack-kolla04:06
*** shakamunyi has joined #openstack-kolla04:18
*** shakamunyi has quit IRC04:23
sdake_sbezverk ping04:32
sdake_vhosakot ping04:33
vhosakotsdake_: pong04:34
sdake_vhosakot we have a problem with rabbitmq04:35
sdake_do you know anything bout it04:35
vhosakotsdake_: yes, I have been debugging it for almost a week now.. I think Jeffrey is also looking into it..04:36
sdake_what doy ou know04:36
sdake_wher eis the bug04:36
vhosakotcrash on centos, just in gate, 1 sec04:36
sdake_lets get what you know recorded in the bug if anything04:36
vhosakotsdake_: already recorded https://bugs.launchpad.net/kolla/+bug/156270104:37
openstackLaunchpad bug 1562701 in kolla mitaka "rabbitmq crashed on centos binary deploy" [Critical,Confirmed]04:37
sdake_if it fails in the gate it will fail in produciton04:37
vhosakotsdake_: scroll to the bottom.. I see your reply as well... there are a couple of things I want to try.... related to updating rabbitmq's config file to fix this bug...04:37
vhosakotsdake_: yes, failure in gate is important even if not seen in local env04:38
sdake_i think y pid patch was not rabbitmq related but mysql related04:38
vhosakotsdake_: just search for      Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",econnrefused]     on google... lotf os link04:38
vhosakotsdake_: I made 1 more observation... there is a keystone failure as well everytime this crash is seen04:39
sdake_keystone does not use rabbitmq04:43
vhosakotsdake_: open this logs04:44
vhosakotsdake_: http://logs.openstack.org/34/297434/2/check/gate-kolla-dsvm-deploy-centos-source/2b0a42b/console.html#_2016-04-04_16_13_45_00904:44
*** daneyon_ has joined #openstack-kolla04:44
vhosakotsdake_: search for      Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",econnrefused]04:45
vhosakotsdake_: then, search for            keystone | Creating admin project, user, role, service, and endpoint04:45
sdake_break it down for me - what is the error04:45
sdake_-elazy04:45
vhosakotsdake_: rabbitmq is not starting as epmd is down04:46
sdake_is the failure on one particular provider in the gate?04:46
sdake_you can tell at the top of the gate log by looking t the kernel04:46
vhosakotdevstack-centos7-osic-cloud1 is the gate that failed04:47
sdake_is it always osic04:48
sdake_or is it all over the place04:48
vhosakotlet me check04:49
vhosakotsdake_: bluebox as well... devstack-centos7-bluebox-sjc104:50
sdake_bluebox is flakey04:50
sdake_i see gate failureson bluebox all the time04:50
sdake_i hooked my stereo up today and it worked04:52
* sdake_ yays ;)04:52
*** salv-orlando has joined #openstack-kolla04:52
vhosakotsdake_: always centos, always keystone failure are seen in rabbit crash04:53
vhosakotsdake_: nice!04:53
sdake_what is the keystone failure04:55
sdake_paste ?04:55
*** salv-orlando has quit IRC04:57
vhosakotsdake_: http://paste.openstack.org/show/493556/04:57
vhosakotsdake_: this is seen before every rabbit crash04:58
sdake_is this prior to rabbitmq failing?04:59
sdake_i see jeffrey4l has crshed out :)05:00
*** Kennan2 has quit IRC05:03
sdake_my qemu patch works on centos05:03
sdake_:)05:03
vhosakotnice!05:03
vhosakotsdake_: yes, prior to rabbitmw crash05:04
sdake_ok well that tells me its a systemic problem05:04
*** banix has quit IRC05:04
sdake_not a rabbitmq specific problem05:04
sdake_probably something wrong with /etc/host or the like05:04
sdake_figure out the keystone crash05:05
sdake_via debug05:05
sdake_and the rabbitmq will probablly self-resolve05:05
*** banix has joined #openstack-kolla05:08
sdake_sounds like a gate issue to me05:08
sdake_https://bugs.launchpad.net/kolla/+bug/156804705:09
openstackLaunchpad bug 1568047 in kolla "/etc/hostname has wrong selinux context" [Low,In progress] - Assigned to Paul Belanger (pabelanger)05:09
*** banix has quit IRC05:09
sdake_maybe that will magicaly fix it05:11
*** salv-orlando has joined #openstack-kolla05:12
*** Marga__ has joined #openstack-kolla05:12
*** salv-orlando has quit IRC05:12
*** Marga_ has quit IRC05:15
*** shakamunyi has joined #openstack-kolla05:18
*** Kennan has joined #openstack-kolla05:19
sdake_vhosakot internap gate has this keystone failure as well http://logs.openstack.org/36/302036/25/check/gate-kolla-dsvm-deploy-centos-binary/bde791d/console.html#_2016-04-09_03_58_49_79605:19
vhosakotyep, hopefully /etc/hostname will fix rabbit crash as well05:19
vhosakotsdake_: and the same log has rabbitmw crash as well  http://logs.openstack.org/36/302036/25/check/gate-kolla-dsvm-deploy-centos-binary/bde791d/console.html#_2016-04-09_03_59_08_21805:22
vhosakotsdake_: search for   econnrefused    in the same logs05:22
*** shakamunyi has quit IRC05:23
sdake_the failure is systemtic05:25
sdake_i have an idea05:25
sdake_can you try this05:26
sdake_can you rpint out whatever config files are used for both keystone and rabbbitmq for those two tasks05:26
sdake_and whatever variables are used inteh playbooks05:26
sdake_for those two tasks05:26
sdake_something will be obviously wrong05:26
sdake_ththen get it to fail05:27
sdake_then look at the variables and config files05:27
sdake_and see what isn't set prperly05:27
sdake_once we find wht isn't set prperly05:27
sdake_we can fix it05:27
sdake_can you do that or shall I05:28
vhosakotsdake_: I can do it tomorrow, I think the config files are good, and the bug is not seen on Ubuntu... I think printing just the variables is good...05:31
sdake_the config files re obviously nto good if epmd doens't start05:31
vhosakotsdake_: so, do you think the keystone json error and rabbitmq crash are related... ? keystone error causes rabbit crash ?05:31
sdake_no one doesn ot cause the other05:31
sdake_a systemtic failure causeses both05:31
vhosakotah, single root cause ?05:32
sdake_yes05:32
sdake_well since i just had a 6 hour nap05:35
sdake_i'm going to get started on this05:35
vhosakotok, thanks... I am add more to it if needed... I'm about to go to bed :)05:37
vhosakotam==can05:37
sdake_here are all the variables printed05:39
sdake_http://logs.openstack.org/36/302036/25/check/gate-kolla-dsvm-deploy-centos-binary/bde791d/console.html#_2016-04-09_03_58_49_79305:39
*** Kennan has quit IRC05:39
*** Kennan has joined #openstack-kolla05:40
coolsvapsdake_: let me know if i can help with anything05:43
sdake_here is one whre it works http://logs.openstack.org/09/303509/7/check/gate-kolla-dsvm-deploy-centos-binary/d7a3932/console.html#_2016-04-08_20_10_00_48805:45
vhosakotsdake_: the from_json part in https://github.com/openstack/kolla/blob/master/ansible/roles/keystone/tasks/register.yml#L5 is failing05:45
sdake_see any delta?05:45
vhosakotsdake_: the variable  keystone_bootstrap  must be printed in https://github.com/openstack/kolla/blob/master/ansible/roles/keystone/tasks/register.yml#L5  using -debug05:45
sdake_vhosakot what is the syntax05:46
vhosakotsdake_: 1 sec05:47
* sdake_ still not an ansible rocketscientist05:47
sdake_when you get a syntax setup can you submit a patch05:47
sdake_and i'll rechec kit and add info the the log05:47
vhosakotsdake_: ok, I will submit it...05:49
sdake_the otuput will be extremely helpful05:52
sdake_becaue it will provide another data point05:52
sdake_since they always fail together05:53
*** Kennan has quit IRC05:53
sdake_and i am pretty sure keystone bootstrap does not depend on rabbitmq05:53
*** Kennan has joined #openstack-kolla05:54
vhosakotyep, almost done... pushing PS now05:54
sdake_make sure not to disturb the play tho05:55
sdake_we still want it to run and fail or succeed05:55
vhosakotafter that keystone error, nothing runs, deploy terminates in gate05:58
vhosakotrabbitmq crashes first, keystone error next, deploy terminates, rabbit crash messages are seen in "docker logs --tail all rabbitmq"05:59
sdake_right05:59
sdake_so whti mean is we want the keystone play to run06:00
sdake_what caues the gate to fail is the keystone botstrap play crashes06:00
sdake_not the epmd rabbitmq error06:00
*** Kennan has quit IRC06:07
sdake_my money is on selinux problems06:07
vhosakotah ok06:07
openstackgerritVikram Hosakote proposed openstack/kolla: Debug rabbitmq crash in centos gate  https://review.openstack.org/30368606:09
vhosakotsdake_: ^^^06:09
vhosakotsdake_: just printing keystone_bootstrap06:09
sdake_will that print if the task prior fails?06:10
sdake_we need to knwo the good output and the bad output06:10
sdake_i suspect if ansible crashes which is what is happening now that debug print wont execute06:11
vhosakotah... no I have commented the line that crashed06:11
vhosakotso, the bug line should be hit06:11
sdake_so how will we know when the json would have crashed?06:11
vhosakotah :)06:12
sdake_add a -debug: msg=test {{ kestyone_bootstrap | from_json?)06:12
vhosakotgood point...06:12
vhosakotafter the current debug ?06:12
sdake_right06:12
vhosakotyep06:12
vhosakotlet me push the new PS :)06:12
sdake_can you do that so i dont mess up the syntax06:12
vhosakotyep.. doing it now06:13
sdake_thanks06:13
sdake_good eye catching the keystone thing06:13
*** salv-orlando has joined #openstack-kolla06:13
*** shakamunyi has joined #openstack-kolla06:18
*** daneyon_ has quit IRC06:18
*** Kennan has joined #openstack-kolla06:22
*** shakamunyi has quit IRC06:23
*** Jeffrey4l_ has joined #openstack-kolla06:23
*** salv-orlando has quit IRC06:23
openstackgerritVikram Hosakote proposed openstack/kolla: Debug rabbitmq crash in centos gate  https://review.openstack.org/30368606:27
*** d_code has quit IRC06:27
vhosakotsdake_: ^^^ added second debug to pring json as well06:27
vhosakotcan you recheck to make it hit the gate machine that crashes rabbitmq...06:28
sdake_what does from_json do06:28
sdake_it converts text into a json blob?06:28
vhosakotyep, from raw text stream that comes as the output of a command from the remote machine to json dict so it can be processed it code06:30
sdake_it will be intreting to see what the text is06:31
sdake_i suspect the database is not operational inthe rabbitmq failure either06:32
vhosakotsdake_: this is what is json-ized   https://github.com/openstack/kolla/blob/master/docker/keystone/keystone_bootstrap.sh#L4806:32
sdake_wow i didn't realie we provided that script06:32
vhosakotsdake_: the command that causes the json error is  "docker exec -t keystone kolla_keystone_bootstrap ..... "06:33
sdake_yup i know that06:33
vhosakotsdake_: https://github.com/openstack/kolla/blob/master/docker/keystone/Dockerfile.j2#L6906:33
vhosakotsdake_: keystone_bootstrap.sh  is  kolla_keystone_bootstrap06:34
sdake_whereis project and role coming from06:38
sdake_https://github.com/openstack/kolla/blob/master/docker/keystone/keystone_bootstrap.sh#L2706:38
vhosakotPROJECT=$3  and  ROLE=$4     https://github.com/openstack/kolla/blob/master/docker/keystone/keystone_bootstrap.sh#L10-L1106:38
vhosakotsdake_: {{ openstack_auth.project_name }}     admin     are    project and role repectively06:41
*** daneyon_ has joined #openstack-kolla06:41
sdake_so why is the json parsing wrong?06:41
sdake_I see nothing wrong with that script06:41
vhosakotyep, that script has been working fine both on centos and ubuntu06:43
*** daneyon__ has joined #openstack-kolla06:44
*** daneyon has quit IRC06:44
*** daneyon_ has quit IRC06:47
*** vhosakot has quit IRC06:47
*** daneyon has joined #openstack-kolla06:50
*** daneyon_ has joined #openstack-kolla06:50
*** Kennan has quit IRC06:52
*** SiRiuS_ has joined #openstack-kolla06:53
*** daneyon__ has quit IRC06:53
*** daneyon_ has quit IRC06:56
*** daneyon_ has joined #openstack-kolla06:56
*** daneyon__ has joined #openstack-kolla07:02
*** sdake_ has quit IRC07:04
openstackgerritVikram Hosakote proposed openstack/kolla: Debug rabbitmq crash in centos gate  https://review.openstack.org/30368607:05
*** daneyon_ has quit IRC07:05
*** Kennan has joined #openstack-kolla07:06
*** vincent_vdk has joined #openstack-kolla07:09
*** Kennan2 has joined #openstack-kolla07:11
*** Kennan has quit IRC07:13
*** Kennan2 has quit IRC07:18
*** daneyon has quit IRC07:22
*** Marga__ has quit IRC07:28
*** Kennan has joined #openstack-kolla07:34
*** salv-orlando has joined #openstack-kolla07:43
*** rmart04 has joined #openstack-kolla07:43
*** jmccarthy has joined #openstack-kolla07:43
*** salv-orlando has quit IRC07:45
*** rmart04 has quit IRC07:48
*** rmart04 has joined #openstack-kolla07:49
*** Kennan has quit IRC07:51
*** rmart04 has quit IRC07:55
*** salv-orlando has joined #openstack-kolla08:05
*** reddy has quit IRC08:08
*** Kennan has joined #openstack-kolla08:10
*** salv-orlando has quit IRC08:12
*** daneyon has joined #openstack-kolla08:14
*** daneyon has quit IRC08:19
*** ccesario_ has quit IRC08:19
*** achanda has joined #openstack-kolla08:26
*** aswadr has joined #openstack-kolla08:30
*** ccesario_ has joined #openstack-kolla08:31
*** Kennan has quit IRC08:36
*** Kennan has joined #openstack-kolla08:37
*** subscope has joined #openstack-kolla08:39
*** achanda has quit IRC08:47
*** SiRiuS_ has quit IRC08:48
*** coolsvap has quit IRC09:03
*** salv-orlando has joined #openstack-kolla09:12
*** openstackgerrit has quit IRC09:17
*** openstackgerrit has joined #openstack-kolla09:18
*** shakamunyi has joined #openstack-kolla09:18
*** ssurana has joined #openstack-kolla09:19
*** salv-orlando has quit IRC09:19
*** shakamunyi has quit IRC09:23
*** aswadr has left #openstack-kolla09:34
*** subscope has quit IRC09:39
*** andrewvwebber has joined #openstack-kolla09:42
*** Kennan has quit IRC09:42
*** Kennan has joined #openstack-kolla09:43
*** allen_gao has quit IRC09:51
*** andrewvwebber has quit IRC09:56
*** allen_gao has joined #openstack-kolla09:57
*** Kennan has quit IRC09:59
*** Kennan has joined #openstack-kolla10:00
*** subscope has joined #openstack-kolla10:04
*** shakamunyi has joined #openstack-kolla10:18
*** shakamunyi has quit IRC10:23
*** salv-orlando has joined #openstack-kolla10:42
*** salv-orlando has quit IRC10:47
*** subscope has quit IRC10:57
*** ccesario___ has joined #openstack-kolla11:01
*** ccesario_ has quit IRC11:05
*** salv-orlando has joined #openstack-kolla11:07
*** Kennan has quit IRC11:09
*** salv-orlando has quit IRC11:11
*** ccesario___ has quit IRC11:14
*** subscope has joined #openstack-kolla11:14
*** ccesario___ has joined #openstack-kolla11:14
*** shakamunyi has joined #openstack-kolla11:18
*** ccesario___ has quit IRC11:19
*** shakamunyi has quit IRC11:22
*** sdake has joined #openstack-kolla11:23
*** Marga_ has joined #openstack-kolla11:27
*** Kennan has joined #openstack-kolla11:28
*** ccesario___ has joined #openstack-kolla11:31
*** sdake has quit IRC11:35
*** cbits has joined #openstack-kolla11:42
*** subscope has quit IRC11:55
*** SiRiuS_ has joined #openstack-kolla12:06
*** SiRiuS_ has quit IRC12:11
*** subscope has joined #openstack-kolla12:15
*** shakamunyi has joined #openstack-kolla12:18
*** shakamunyi has quit IRC12:23
*** salv-orlando has joined #openstack-kolla12:24
*** salv-orlando has quit IRC12:28
*** dgonzalez has quit IRC12:30
*** ccesario___ has quit IRC12:34
*** daneyon has joined #openstack-kolla12:37
*** daneyon has quit IRC12:41
*** dgonzalez has joined #openstack-kolla12:42
*** Jeffrey4l_ has quit IRC12:47
*** Jeffrey4l_ has joined #openstack-kolla12:51
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix the wrong package name in Ubuntu binary build  https://review.openstack.org/30370712:58
*** klint has quit IRC13:01
*** subscope has quit IRC13:09
*** SiRiuS_ has joined #openstack-kolla13:11
*** subscope has joined #openstack-kolla13:12
*** shakamunyi has joined #openstack-kolla13:18
*** shakamunyi has quit IRC13:23
*** ccesario___ has joined #openstack-kolla13:36
*** salv-orlando has joined #openstack-kolla13:44
openstackgerritJeffrey Zhang proposed openstack/kolla: Collect the Kolla configs in the gate  https://review.openstack.org/30371013:50
*** vhosakot has joined #openstack-kolla13:53
openstackgerritVikram Hosakote proposed openstack/kolla: [DO NOT MERGE] Debug rabbitmq crash in centos gate  https://review.openstack.org/30368613:56
*** salv-orlando has quit IRC14:01
*** subscope has quit IRC14:17
*** shakamunyi has joined #openstack-kolla14:18
*** vhosakot has quit IRC14:21
*** vhosakot has joined #openstack-kolla14:21
*** shakamunyi has quit IRC14:23
vhosakotJeffrey4l_: ping14:26
*** sdake has joined #openstack-kolla14:26
vhosakotJeffrey4l_: your review  https://review.openstack.org/#/c/294906/  collected erl_crash.dump  in  logs/kolla/  in gate....     http://logs.openstack.org/06/294906/5/check/gate-kolla-dsvm-deploy-centos-source/a8d64f5/logs/kolla/.... awesome!14:26
patchbotvhosakot: patch 294906 - kolla - DO NOT MERGE: TEST MASTER BRANCH14:26
vhosakotJeffrey4l_: we need to decode it..14:27
*** cbits has quit IRC14:27
sdakemorning14:27
vhosakotsdake: morning14:27
Jeffrey4l_vhosakot, sdake morning14:27
vhosakotJeffrey4l_'s review  https://review.openstack.org/#/c/294906/  collected erl_crash.dump  in  logs/kolla/  in gate....     http://logs.openstack.org/06/294906/5/check/gate-kolla-dsvm-deploy-centos-source/a8d64f5/logs/kolla/....14:27
patchbotvhosakot: patch 294906 - kolla - DO NOT MERGE: TEST MASTER BRANCH14:27
vhosakotwe need to decode it now14:28
sdakedid you get any results14:28
sdakei think i passed out at the keyboard ;)14:28
Jeffrey4l_vhosakot, I seem find the root soon. still need more debug .14:28
Jeffrey4l_it should be a network issue, i think now.14:28
Jeffrey4l_but need more time.14:29
Jeffrey4l_vhosakot, so i do not think we need decode the erl_crash.dump file now. :D14:30
vhosakotJeffrey4l_: how will we know the issue then ? :)14:30
vhosakotsdake: search for  keystone_bootstrap.stdout  and   econnrefused   in    http://logs.openstack.org/86/303686/4/check/gate-kolla-dsvm-deploy-centos-source/17631e5/console.html#_2016-04-09_14_16_26_14014:31
Jeffrey4l_vhosakot, i am still checking the log and make some test. I just got some thought( it is caused by some network configuration issue)14:31
sdakeJeffrey4l_ your awaare keystone is failing14:31
Jeffrey4l_sdake, yea.14:31
Jeffrey4l_I think the failed of keystone and rabbitmq are the same now. But I do not have any prove for this now. :(, still checking and debuging this.14:32
vhosakotJeffrey4l_: keystone failed in the same logs  http://logs.openstack.org/06/294906/5/check/gate-kolla-dsvm-deploy-centos-source/a8d64f5/console.html#_2016-04-08_15_58_51_989   that collected  erl_crash.dump  from your review in gate  http://logs.openstack.org/06/294906/5/check/gate-kolla-dsvm-deploy-centos-source/a8d64f5/logs/kolla/14:32
vhosakotJeffrey4l_: cool, I am checking the keystone part...14:33
Jeffrey4l_vhosakot, cool. Let us debug this in different direction. It will be helpful. :D14:34
*** Jeffrey4l_ has left #openstack-kolla14:34
vhosakotyep14:34
*** Jeffrey4l_ has joined #openstack-kolla14:34
vhosakotJeffrey4l_: cool14:35
Jeffrey4l_this bug is very annoying. :( can not reproduce locally.14:35
*** ccesario___ has quit IRC14:35
vhosakotJeffrey4l_: yes, due to network issue probably in centos gate14:36
*** ccesario___ has joined #openstack-kolla14:36
sdakesweet hunger games is on :)14:36
vhosakotJeffrey4l_: i like debugging such cannot-reproduce-locally bugs :)14:36
vhosakotsdake: Jeffrey4l_: when rabbitmq container dies, its logs does not show up in /var/lib/docker/volumes/kolla_logs/_data/rabbitmq/.....14:37
vhosakotsdake: Jeffrey4l_: shouldn;t kolla retain logs of containers that die/crash ?14:38
Jeffrey4l_vhosakot, what i found now is: the failed CentOS eth0 ip address is a public address and the route is especial.14:38
Jeffrey4l_see this http://logs.openstack.org/06/294906/5/check/gate-kolla-dsvm-deploy-centos-source/a8d64f5/console.html#_2016-04-08_15_41_58_83614:38
Jeffrey4l_seem all of them came from `evstack-centos7-ovh-bhs1`14:38
vhosakot158.69.94.147/32 ?14:39
Jeffrey4l_yea.14:39
vhosakotJeffrey4l_: please merge my precheck PS.. it will catch it :)14:39
Jeffrey4l_and the default gateway is  default via 158.69.64.1 dev eth014:39
Jeffrey4l_vhosakot, you PS and this bug is different things.14:39
vhosakotJeffrey4l_: but, it will catch if eth0 is public IP and pingable :)14:40
Jeffrey4l_vhosakot, so we stop to run the jobs and failed the gate?14:40
vhosakotJeffrey4l_: yes.... becasue, when prechecks fail (due to public IP on pingable IP on etho), job will have issue like this... we need to exit the job in gate14:41
vhosakotJeffrey4l_: https://review.openstack.org/#/c/302026/   :)  I have replied to your comments... it works fine and I have tested it :)14:41
patchbotvhosakot: patch 302026 - kolla - Add prechecks to validate kolla_internal_vip_addre...14:41
Jeffrey4l_vhosakot, no. the first thing we need to do is satisfy the gate. Not let the gate satisfy our code.14:42
Jeffrey4l_if you pre-check changes failed the gate. I prefer to fix the gate. If the gate can not be fixed, I prefer to revert your code.14:42
vhosakotJeffrey4l_: no, I'm sure my precheck is important... if it fails, fix the gate, and not revert the PS :)14:43
Jeffrey4l_vhosakot, yes. we are fixing the gate.14:43
vhosakotJeffrey4l_: please see my reply in https://review.openstack.org/#/c/302026/ :)14:43
patchbotvhosakot: patch 302026 - kolla - Add prechecks to validate kolla_internal_vip_addre...14:43
Jeffrey4l_vhosakot, i know your patch.14:43
vhosakotJeffrey4l_: cool, thanks! :)14:43
Jeffrey4l_but it still do not matter the rabbitmq crash issue.14:44
vhosakotJeffrey4l_: agreed, rabbitmq crash is a different matter..14:45
vhosakotJeffrey4l_: we do not run  prechecks  in gate  before "tools/kolla-ansible -vvv deploy".... we need to run prechecks first before depoying...14:46
vhosakotJeffrey4l_: once   https://review.openstack.org/#/c/302026/  is merged, I will submit a PS to add prechecks to gate14:47
patchbotvhosakot: patch 302026 - kolla - Add prechecks to validate kolla_internal_vip_addre...14:47
Jeffrey4l_vhosakot, ok14:47
vhosakotJeffrey4l_: cool14:47
openstackgerritBenedikt Trefzer proposed openstack/kolla: Use debian repos for debian base docker image  https://review.openstack.org/28893614:48
*** subscope has joined #openstack-kolla14:49
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make container dind build with debian  https://review.openstack.org/29743414:49
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make heka container build with debian  https://review.openstack.org/29773714:50
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make kibana container build with debian  https://review.openstack.org/29720614:50
sdakeJeffrey4l_ agree with this: [07:42:32]  <Jeffrey4l_>if you pre-check changes failed the gate. I prefer to fix the gate. If the gate can not be fixed, I prefer to revert your code.14:51
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make marathon container build with debian  https://review.openstack.org/29737814:51
Jeffrey4l_thanks. :D14:51
sdakeso the problem is what with theip address?14:52
sdakeits public or its /32?14:52
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make mesos-slave container build with debian  https://review.openstack.org/29738514:52
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make mesosphere-base build with debian  https://review.openstack.org/29735814:52
Jeffrey4l_sdake, not sure now. I just doubt that. Need more time to check and debug.14:52
openstackgerritVikram Hosakote proposed openstack/kolla: Add prechecks to validate kolla_internal_vip_address and kolla_external_vip_address  https://review.openstack.org/30202614:52
sdakeJeffrey4l_ i got the patch merged you were working on that was bsted14:53
sdakeit ended up being pretty simple14:53
sdakeand myqemu patch worsk on top14:53
vhosakotJeffrey4l_: sdake: is  158.69.94.147/32   right for eth0 in gate ?  o_O14:53
sdakehow would I know ;)14:54
Jeffrey4l_sdake, glad to here that.14:54
Jeffrey4l_vhosakot, rule 1: gate is always right. ( because we can not change that ) :)14:54
vhosakotJeffrey4l_: not necessary... :)  gate is not prod :)14:55
sdakehttp://logs.openstack.org/86/303686/4/check/gate-kolla-dsvm-deploy-centos-source/17631e5/console.html#_2016-04-09_14_16_26_14014:55
sdakewierd the return is not formatted14:55
sdakewith anything14:55
sdakethat shell script is spaghetti coding imo ;)14:57
vhosakotsdake: Jeffrey4l_:  here we set eth0 (that has 158.69.94.147/32) as network_interface in globals.yml in gate....  https://github.com/openstack/kolla/blob/master/tools/deploy_aio.sh#L6114:57
openstackgerritBenedikt Trefzer proposed openstack/kolla: Use debian repos for debian base docker image  https://review.openstack.org/28893614:57
Jeffrey4l_vhosakot, there is only one network_interface.14:57
vhosakotsdake: Jeffrey4l_: but, kolla has a requirement that kolla_internal_vip_address must be in the _same_ network as network_interface (eth0 which is 158.69.94.147/32)14:58
*** diogogmt has quit IRC14:58
sdakeoh yes thisi is probably the bluebox gate problem14:58
sdakevhosakot is it bluebox?14:58
vhosakotsdake: Jeffrey4l_: so, there is not free IP that kolla_internal_vip_address  can get in gate!  as 158.69.94.147/32 is just one address!  and already assigned to eth0!!!14:58
vhosakotsdake: Jeffrey4l_: this is gate issue!!14:58
sdakeits one of gate issues14:59
sdakenot sure if all of them14:59
vhosakottight14:59
vhosakotright14:59
sdakewho is the infra provider14:59
sdakeis it bluebox?14:59
openstackgerritBenedikt Trefzer proposed openstack/kolla: make rabbitmq container build with debian  https://review.openstack.org/29772814:59
Jeffrey4l_i set a almost the same ip address env http://paste.openstack.org/show/493564/ (ignore the eth1, i do not use it), I can deploy the kolla successful.15:00
vhosakotsdake: Jeffrey4l_: well, kolla_internal_vip_address is harcoded to 169.254.169.10 in https://github.com/openstack/kolla/blob/master/tools/deploy_aio.sh#L5915:00
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make swift-base container build with debian  https://review.openstack.org/29712115:00
Jeffrey4l_vhosakot, if you doubt it is the root cause, please try to reproduce it locally.15:00
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make container kolla-toolbox build with debian  https://review.openstack.org/29710415:00
*** salv-orlando has joined #openstack-kolla15:01
vhosakotsdake: Jeffrey4l_: in gate, kolla_internal_vip_address (169.254.169.10) be _not_ in the same network as eth0 (158.69.94.147/32).. isn't this an issue ?15:01
sdakenot if we use eth1 as t he internal network15:02
sdakesomething sepcial sabout 169.254. address range too15:02
sdakei dont know what15:02
sdakewhat i rcommend is creating an deth1 and eth2, and using those for management and neutron networks15:02
sdakeand leave eth0 alone15:03
vhosakotsdake: i dont see eth1 in gate15:04
Jeffrey4l_vhosakot, please try to re-produce this issue locally. After that, you can convince. ( before that, i want to check and debug this issue)15:04
Jeffrey4l_vhosakot, please try to re-produce this issue locally. After that, you can convince me. ( before that, i want to check and debug this issue)15:04
vhosakotJeffrey4l_: I'm trying to reproduce since more than a week :)... I am not able to reprodice it :)15:04
Jeffrey4l_vhosakot, so if is not the issue you talk above "kolla has a requirement that kolla_internal_vip_address must be in the _same_ network as network_interface (eth0 which is 158.69.94.147/32)"15:05
*** subscope has quit IRC15:06
vhosakotJeffrey4l_: right, as sdake said, that requiremtn is not needed and not satisfied in gate15:06
*** salv-orlando has quit IRC15:06
vhosakotJeffrey4l_: gate uses this as kolla_internal_vip_address --> https://github.com/openstack/kolla/blob/master/tools/deploy_aio.sh#L5915:07
Jeffrey4l_vhosakot, we use it all the time. it works all the time.15:07
vhosakotJeffrey4l_: yep, it has been working all the in gate I know.. I am trying to understand how 169.254.169.10 (kolla_internal_vip_address) is in 158.69.94.147/32  :)15:09
vhosakotJeffrey4l_: I think 169.254.* is special range or something15:09
vhosakotyes, creating eth1 is the best idea sounds like.. I will try it locally if it works15:11
*** diogogmt has joined #openstack-kolla15:12
*** salv-orlando has joined #openstack-kolla15:13
vhosakotJeffrey4l_: sdake:  I will set kolla_internal_vip_address as 169.254.169.10, and eth0 (network_interface) as 158.69.94.147/32 and see if I can reproduce the bug locally..15:17
*** shakamunyi has joined #openstack-kolla15:18
*** vhosakot has quit IRC15:19
*** salv-orlando has quit IRC15:20
*** banix has joined #openstack-kolla15:21
*** shakamunyi has quit IRC15:23
SiRiuS_sdake, Jeffrey4l_  https://en.wikipedia.org/wiki/Link-local_address15:27
sdakeyup i knew that was  special15:28
*** salv-orlando has joined #openstack-kolla15:30
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH  https://review.openstack.org/29490615:31
*** vhosakot has joined #openstack-kolla15:32
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make nova-libvirt container build with debian  https://review.openstack.org/29716915:33
SiRiuS_yeah, it usually indicates a network card did not get an IP address automatically and one was not set manually or something failed. Then the OS sets one in that range.15:34
*** salv-orlando has quit IRC15:34
SiRiuS_vhosakot, https://en.wikipedia.org/wiki/Link-local_address   the 169.254.0.0/16 IP addresses15:35
*** banix has quit IRC15:35
vhosakotSiRiuS_: thanks for the link... so, we use link-local address 169.254.169.10 (which is in the range link local range 169.254.0.0/16)... it has been working fine in gate all these days15:36
*** salv-orlando has joined #openstack-kolla15:37
SiRiuS_well, they work fine, as long as you don't sent any packages over a router15:38
SiRiuS_when you don't set an IP address manually and have no DHCP set up, then the OS will give you an address in this range15:39
vhosakotyep15:40
vhosakotbbiab in 30 mins15:41
SiRiuS_it's only unique in a network segment only15:42
SiRiuS_https://en.wikipedia.org/wiki/Network_segment15:42
sdakestill box gates fail on waiting on the link local address we specify15:46
sdakebtw SiRiuS_ the problem booting vms has been solved15:47
SiRiuS_sdake, cool was it present in stable/mitaka as well?15:48
sdakebeen backported15:49
SiRiuS_coll15:50
SiRiuS_cool15:50
sdakenot mergee15:50
sdaked15:50
SiRiuS_I'll cherry pick15:50
SiRiuS_What images should I rebuild?15:50
sdakenot sure - dockerfiles shoudl tell you - think *nova815:51
*** vhosakot has quit IRC15:51
SiRiuS_ok cool, will give it a try :)15:51
*** sdake_ has joined #openstack-kolla15:53
*** jtriley has quit IRC15:54
*** sdake has quit IRC15:55
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH  https://review.openstack.org/29490615:57
*** Marga_ has quit IRC16:05
*** diogogmt has quit IRC16:05
*** subscope has joined #openstack-kolla16:05
*** kjelly_home has joined #openstack-kolla16:05
*** kjelly_home has quit IRC16:07
*** Marga_ has joined #openstack-kolla16:08
*** kjelly_home has joined #openstack-kolla16:10
SiRiuS_sdake_, is this the patch ?16:11
SiRiuS_sdake_, https://review.openstack.org/#/c/303680/16:11
patchbotSiRiuS_: patch 303680 - kolla (stable/mitaka) - Fix the qemu has nova permission in nova instances...16:11
*** vhosakot has joined #openstack-kolla16:16
sdake_SiRiuS_ yes thats the one16:20
*** vhosakot has quit IRC16:21
SiRiuS_awesome, thanks16:21
*** kjelly_home has quit IRC16:21
*** subscope has quit IRC16:24
ccesario__afternoon guys!16:26
sdake_Jeffrey4l_ go to openstack-infra plz16:27
sdake_hey ccesario__16:27
ccesario__hey sdake_ :)16:28
Jeffrey4l_sdake_, i was in that channel.16:28
sdake_well help answer q's elalse :)16:28
Jeffrey4l_sdake_, i do not think that's a issue.16:28
ccesario__guys, could you give me to correct way with this patch ? https://review.openstack.org/#/c/297984/ ... if possible of course...16:29
patchbotccesario__: patch 297984 - kolla - Identify container through  PS1 shell variable16:29
*** ccesario___ has quit IRC16:29
ccesario__I have a bit free time ... and I would like to spend this time with this.... but I need the right direction to do it :)16:30
Jeffrey4l_sdake_, please check ccesario__ PS https://review.openstack.org/#/c/297984/ . two possible direction. could u give some advice? thanks.16:32
patchbotJeffrey4l_: patch 297984 - kolla - Identify container through  PS1 shell variable16:32
sdake_Jeffrey4l_ lets go to opensack-infra and help them help us :)16:33
*** subscope has joined #openstack-kolla16:33
*** allen_gao has quit IRC16:35
*** allen_gao has joined #openstack-kolla16:35
*** unicell has joined #openstack-kolla16:45
*** subscope has quit IRC16:53
*** SiRiuS_ has quit IRC17:04
*** Jeffrey4l_ has quit IRC17:05
ccesario__sdake_: any advice about ps ?17:05
*** daneyon has joined #openstack-kolla17:07
*** daneyon has quit IRC17:12
*** vhosakot has joined #openstack-kolla17:18
sdake_which ps17:20
*** vhosakot has quit IRC17:23
*** ccesario___ has joined #openstack-kolla17:24
sbezverksdake would you have sometime later today to check new version of erlang, which rabbitmq folks suggested to try as a solution for issue I saw in liberty?17:30
*** ccesario___ has quit IRC17:31
sdake_which issue did you see17:39
sdake_it takes a long time to try new versoin of erlang17:39
sdake_like 8 hours of my life17:39
sdake_and that doen't leverage al of the testing we have done on the erlang we currently have17:40
sdake_and how are youseeing issues in liberty, running the stable/liberty branch?17:41
sdake_sbezverk ^^17:41
*** unicell has quit IRC17:45
*** sdake has joined #openstack-kolla17:49
sdakesbezverk ^^17:50
*** sdake_ has quit IRC17:51
*** ssurana has quit IRC17:57
*** sdake_ has joined #openstack-kolla18:05
*** vhosakot has joined #openstack-kolla18:07
*** vhosakot has joined #openstack-kolla18:08
*** sdake has quit IRC18:09
vhosakotsdake_: ping18:09
sdake_shoot18:09
vhosakotsdake_: so, when we join the #openstack-kolla channel, we see this heading banner:18:10
vhosakotTopic: Kolla IRC meetngs on Wednesdays @ 16:30 UTC even weeks, 23:00 UTC odd weeks - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla  http://goo.gl/VKpPzA)18:10
vhosakotrhallisey set the topic at: Mar 31, 2016, 12:24 PM18:10
vhosakotMode: +cnt18:10
vhosakotCreated at: Sep 11, 2014, 1:06 PM18:10
sdake_yes18:10
vhosakotthe logs are still pointing to old location18:10
sdake_yes18:10
sdake_i cant fix it atm because i dont have registration to freenode18:11
vhosakotsdake_: shouldn;t  "IRC channel is *LOGGED* @"  open #openstack-logs IRC18:11
vhosakotsdake_: who should we ask ?18:11
vhosakotsdake_: infra ?18:11
sdake_rhallisey or inc0 can set the channel topic18:11
sdake_i could too if i was registered18:11
vhosakotsdake_: ok, will check with them.. thanks!18:11
*** banix has joined #openstack-kolla18:25
openstackgerritMauricio Lima proposed openstack/kolla: Put the variables in alphabetical order in all.yml  https://review.openstack.org/30340918:25
*** Marga_ has quit IRC18:27
openstackgerritMauricio Lima proposed openstack/kolla: Remove /etc/kolla configuration folders  https://review.openstack.org/30305518:36
*** banix has quit IRC18:36
openstackgerritMauricio Lima proposed openstack/kolla: Add a role to cleanup containers after deployment  https://review.openstack.org/30357518:39
*** Marga_ has joined #openstack-kolla18:48
*** jtriley has joined #openstack-kolla18:51
*** Marga_ has quit IRC18:52
openstackgerritVikram Hosakote proposed openstack/kolla: [DO NOT MERGE] Debug rabbitmq crash in centos gate  https://review.openstack.org/30368619:03
vhosakotsdake_: ping19:10
vhosakotsdake_: what is the jenkins link to check gate status when gate is runnning (before jenkins posts results in the review) ?19:11
vhosakotsdake_: ^^^19:18
sdake_zuul.openstack.org19:19
sdake_type kolla in the search bar19:19
openstackgerritBenedikt Trefzer proposed openstack/kolla: Use debian repos for debian base docker image  https://review.openstack.org/28893619:19
vhosakotsdake_: ah, thanks! :)19:21
vhosakotsdake_: ping19:26
vhosakothttps://jenkins03.openstack.org/job/gate-kolla-dsvm-deploy-centos-binary/611/console19:26
vhosakotsdake_: ^^^ looks like I resolved the rabbitmq crash... no crash on centos binary19:26
sdake_resolved how19:27
sdake_i think oen tesst isn't good enough ;)19:27
sdake_it is intermittent19:27
vhosakotyep, waiting for centos source resukts19:27
vhosakotsdake_: here is the fix - https://review.openstack.org/30368619:27
vhosakotsdake_: ERL_EPMD_ADDRESS needs to be commented out in rabbitmq-env.conf.j2...19:28
vhosakotsdake_: this fix is what I found online many people use19:28
sdake_then epmd binds to 0.0.0.0?19:28
vhosakotsdake_: no rabbitme crash on centos source as well - https://jenkins06.openstack.org/job/gate-kolla-dsvm-deploy-centos-source/603/console19:29
vhosakotsdake_: but, VM not booting in centos... but no crahs19:29
sdake_recheck it19:29
sdake_need to see if it actually fixes it19:29
sdake_what wass the deal with the keystone data output19:29
vhosakotyep,19:29
vhosakotyep19:29
sdake_why did keystone pressent a json parse error19:31
vhosakotsdake_: well, the keystone error was also not seen this time on both centos binary and centos source..19:31
vhosakotsdake_: I will recheck it multiple times19:31
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make nova-libvirt container build with debian  https://review.openstack.org/29716919:31
vhosakotsdake_: to make sure that _both_ rabbitmq crash and keystone json error are not seen19:32
sdake_that doesn't explain why keystone parse error happens19:32
vhosakotsdake_: yes, will debug more19:32
sdake_keystone doesn't use rabbitmq19:32
vhosakotyep19:32
vhosakotsdake_: keystone and ERL_EPMD_ADDRESS ar enot way related :)  ERL_EPMD_ADDRESS is a rabbitmqq config caraible19:33
vhosakotvariable*19:33
sdake_i have yet to see a keystone crash without the rabbitmq crash19:34
vhosakotsdake_: yes... rechecking19:34
vhosakotsdake_: ERL_EPMD_ADDRESS is 10.220.2.147 on centos binary gate, and 149.202.162.18 on centos source gate19:37
sdake_sbezverk coud you specify if this is the issue your seeing pls19:37
vhosakotsbezverk was able to repro the rabbir crash locally ? didn't know that19:38
sdake_vhosakot it is unclear what crash he has reproduced19:39
sdake_because he didn't file a bug or tell me yet :)19:39
vhosakotah ok19:39
vhosakotcool :)19:39
vhosakotsdake_: now I need to see it commenting out ERL_EPMD_ADDRESS does not affect anythign else19:39
vhosakotsdake_: it was used just for centos and not for ubuntu19:39
sdake_i see that19:41
sdake_that is why ubuntu never suffersthis failure19:41
vhosakotsdake_: centos source VM did reach ACTIVE state in gate :)19:45
vhosakotsdake_: 023367b1-4a8d-4281-8e3b-b1d751b83184 | kolla_boot_test | ACTIVE | -          | Running     | demo-net=10.0.0.3 |19:45
vhosakotsdake_: http://logs.openstack.org/86/303686/5/check/gate-kolla-dsvm-deploy-centos-source/8fe9507/console.html#_2016-04-09_19_28_19_79319:45
vhosakotsdake_: rechecking19:45
ccesario__sdake_: this ps.. https://review.openstack.org/#/c/297984/19:47
patchbotccesario__: patch 297984 - kolla - Identify container through  PS1 shell variable19:47
sdake_ccesario__ what woudl you like me to do with that patch19:47
ccesario__[16:23] <Jeffrey4l_> sdake_, please check ccesario__ PS https://review.openstack.org/#/c/297984/ . two possible direction. could u give some advice? thanks.19:50
patchbotccesario__: patch 297984 - kolla - Identify container through  PS1 shell variable19:50
*** daneyon has joined #openstack-kolla19:50
ccesario__sdake_: basicly if is it ok.... or no .... because currently it is working... but Jeffrey4l_ points some things19:51
sdake_zomg patch is big19:52
sdake_not keen on hardcoding environment names19:53
sdake_lots of maintenence there19:53
sdake_why not use a {{ }} jinja2 replacement?19:54
*** dgonzalez has quit IRC19:54
ccesario__yes, but very simple ... as oher vars that exists19:54
*** daneyon has quit IRC19:55
sdake_why not use the docker module to set the hostname of the container?19:55
sdake_i saw that originally in this patch set19:55
*** dgonzalez has joined #openstack-kolla19:55
sdake_ccesario__ i get you have done alot of work to add all these env vars19:55
sdake_but i think there are simpelr ways to achieve same result19:56
*** Marga_ has joined #openstack-kolla19:57
ccesario__sdake_: because the original patch adding -h ... causes a crash in build... and some comments in lauchpad bug point to usage PS1 var19:58
sdake_which launchpad comments19:59
sdake_got a link to the build crash?19:59
ccesario__sdake_: https://bugs.launchpad.net/kolla/+bug/155745419:59
openstackLaunchpad bug 1557454 in kolla "container name in PS1 shell command prompt inside a container" [Wishlist,In progress] - Assigned to Carlos Cesario (ccesario)19:59
openstackgerritVikram Hosakote proposed openstack/kolla: Fix rabbitmq crash in centos gate  https://review.openstack.org/30368619:59
ccesario__sdake_: http://paste.openstack.org/show/492760/20:00
*** Marga_ has quit IRC20:01
sdake_ccesario__ whichversion of docker-py were you using?20:02
sbezverksdake vhosakot I did not see rabbitmq crash, the issue I saw was different. Rabbitmq folks helped to identify it and suggested a fixed version.20:02
sdake_what problem did you see sbezverk20:03
sdake_how about fiiing a bug20:03
sdake_:)20:03
sbezverkI saw this issue on liberty deployed with inc0 patch20:03
sbezverksdake why file a bug if there is a fix already ;-)?20:04
ccesario__sdake_: Name: docker-py Version: 1.7.220:04
ccesario__its support -H20:04
ccesario__-h20:04
sdake_ok well this looks like a docker-py bug or a kolla-docker implemtnation bug20:04
vhosakotsbezverk: cool, thanks for confirming20:05
ccesario__using a simple code using docker-py I get success using -h param20:05
sbezverksdake here is the log I provided to rabbitmq folks http://paste.openstack.org/show/493522/20:05
sdake_ccesario__ Ithink what you need to do is look at the docker engine log where it crashed in your previous -h attempt20:07
sdake_and see what it says20:07
ccesario__sdake_: http://paste.openstack.org/show/493569/20:07
ccesario__it works pefectly20:07
vhosakotccesario__: sbezverk recently updated kolla_docker.py to set --ipc option to "host".... can you refer that as an example to implement hostname param in kolla_docker.py.... here is sbezverk's review  https://review.openstack.org/#/c/293400/20:07
patchbotvhosakot: patch 293400 - kolla - Set --ipc option to "host" for kolla_docker (MERGED)20:07
sdake_ok so the error is not in docker-py20:09
*** Marga_ has joined #openstack-kolla20:09
ccesario__vhosakot: yes, I saw it ...and I did the code look like that20:10
sbezverkguys can you share the error please?20:10
vhosakotsbezverk: ccesario__ is implementing a new param called hostname the exact same way you have successfuly done for --ipc "host"... and seeing some errors.. scroll aabove20:11
ccesario__vhosakot: sdake_ give me a bit time to setup the environment  using my first patch ...20:11
sbezverkvhosakot, maybe a trivial question but does docker supports it? for ipc host we had to confirm that it is actually supported..20:12
sdake_ccesario__ was your first patch just unreliable or did it fail all the time?20:12
vhosakotccesario__: sure... could you refer sbezverk's review in https://review.openstack.org/#/c/293400/ and implement hostname20:12
patchbotvhosakot: patch 29340020:12
vhosakotsbezverk: yes, docker supports it and I have tested it20:12
ccesario__sdake_: yes20:13
sdake_yes what there were two choices :)20:13
sdake_ccesario where is your first hostname patch20:14
ccesario__sdake_: the two choices... hehehe20:15
vhosakotccesario__: instead of --ipc, you'd need to implement --hsotname..... both are strings and both are accepted by "docker run..."20:15
vhosakothttps://docs.docker.com/engine/reference/commandline/run/20:15
ccesario__vhosakot: the docker --ipc it is supported  by build_host_config function20:16
ccesario__the --hostname nned to be used on build_container_options20:18
*** shakamunyi has joined #openstack-kolla20:18
ccesario__sdake_: https://review.openstack.org/gitweb?p=openstack/kolla.git;a=commitdiff;h=1f6840f3cc7b6560e3f456f803c6929439225fc820:18
ccesario__so simple20:19
vhosakotccesario__: does this work ?20:19
ccesario__vhosakot: http://paste.openstack.org/show/492760/20:20
ccesario__:P20:20
vhosakotyep, I saw that mariadb error you sent earlier...20:21
vhosakotccesario__: I will dig the code and let you know what should be updated in kolla_docker.py for --hostname20:22
vhosakotccesario__: is your /run shared ?20:22
sbezverkccesario__ are you sure it is your change related? I saw similar error when I had an issue with rights on my ubuntu test bed20:22
sdake_i suspect hostname has to go after name in the ansible call20:23
*** shakamunyi has quit IRC20:23
sdake_the hostname is not part of the call20:23
sdake_and there is a wird ,)20:23
sdake_so something is not formatting the options correclty20:23
sbezverkhas anybody else repoduced the same issue in different env?20:23
ccesario__vhosakot: yes /run is shared20:24
sbezverkbased on my past troubleshooting experience if the issue looks too wierd then you are looking at the wrong place ;-)20:24
ccesario__sbezverk: yes.... withou it no errors20:24
sbezverkbut my question was if somebody else repro the same issue with your change20:24
sbezverk?20:25
sdake_ccesario__ where is the gerrit patch if you would like me to debug this hostname thing20:25
vhosakotsbezverk: the hostname value is coming in kolla_docker.py correctly but not used in kolla_docker.py properly...20:25
sdake_right20:25
ccesario__sbezverk: I don't  know... nobody reported20:25
sdake_vhosakot thats right non20:25
sbezverkccesario__ ok let me try to bring up my centos TB and see if I hit the same issue20:26
ccesario__sdake_: https://review.openstack.org/#/c/297984/320:26
patchbotccesario__: patch 297984 - kolla - Identify container through  PS1 shell variable20:26
ccesario__sbezverk: thanks20:26
*** sdake has joined #openstack-kolla20:28
*** cmceniry has joined #openstack-kolla20:29
vhosakotsdake_: recheck did not see rabbitmq crash and nova VM reach active state in centos gate20:30
sdakekeep at it rechecking plz20:30
vhosakotyep20:30
vhosakotaa6b3771-a2b6-4b46-adbe-6b5f1c399ba6 | kolla_boot_test | ACTIVE | -          | Running     | demo-net=10.0.0.3 |20:30
*** sdake_ has quit IRC20:30
sdakeccesario i've pulled your patch looking into it now20:32
sbezverksdake which version of PS you took 3 or 4?20:33
sdake320:33
sbezverkso 4 is no good?20:33
vhosakotyep, 1,2,3 have docker.py chnages... 4 has the second approach... adding env var in dockerfile20:35
sdakei am not a fan of approach #420:36
sbezverkccesario__ probably it is not matter but you do not have default: None20:38
sbezverkdo you mind to add it and try?20:38
sdakeTASK: [haproxy | Starting haproxy container] **********************************20:39
sdakefailed: [localhost] => {"changed": true, "failed": true}20:39
sdakemsg: APIError(HTTPError(u'409 Client Error: Conflict for url: http+docker://localunixsocket/v1.22/containers/create?name=haproxy',),)20:39
sdakehttp://paste.fedoraproject.org/352970/46023436/ -> http://paste.fedoraproject.org/352970/4602343620:39
sdaketime to learn how kolla_docker.py works ;)20:39
sbezverksdake since you already have it up, could you please add Default: None20:40
sdakehttp://paste.fedoraproject.org/352971/34453146/ -> http://paste.fedoraproject.org/352971/3445314620:41
sdakethat is the command as seen in the docker engine20:41
sbezverkhttp://paste.fedoraproject.org/352972/23451514/20:42
sdakelol sb20:43
sbezverksdake in that command you pasted, both hostname and name are specified and they are identical, wondering if it might create a conflict20:45
*** salv-orl_ has joined #openstack-kolla20:45
vhosakotsdake: we need to make the gate show red FAILED when nova VM does not go to ACTIVE state20:45
vhosakotsdake: now, it shows green SUCCESS even when nova VM does not go to ACTIVE state, and it is misleasing (false positive).. I will try to fix it20:46
sdakesbezverk i dont see why20:46
sdakebut i'll try a different hostname20:47
sdakesbezverk thats not it20:48
sdakelet me try your default idea20:48
*** salv-orlando has quit IRC20:48
sdakebulid_host_config lacking hostname20:50
sbezverkyep, I am looking at the same spot20:52
sbezverkbut build_container_options has it20:52
*** jasonsb has joined #openstack-kolla21:00
sdakeApr 09 17:06:24 minime-03 docker[1250]: time="2016-04-09T17:06:24.805347868-04:00" level=error msg="Handler for POST /v1.22/containers/create returned error: Conflicting options: hostname and the network mode"21:10
sdakeyup21:10
sdakethats the prolem21:10
sdakefirst off, hostname is a magic ansible variable21:10
sdakeso it has to be called soething else21:10
sdakein this place i used hostnamemagic21:10
sdakeapparently you cant specify hostname and network mode at the same time21:10
sdakehttps://github.com/docker/compose/issues/246021:11
sdakeccesario ^^21:11
sbezverksdake what if we generate hostname file and insert it to /etc/ during container baking?21:12
sbezverkI am positive it will work on centos/rhel , not sure about ubuntu though21:12
ccesario__hey guys... I had t solve some problem here21:13
sdakeccesario i root caused why your patch no beuno21:14
ccesario__sdake:  I saw that you got the same error21:14
sdakeyes but you never said its because we use --net=host ;)21:14
ccesario__I did not now about this conflict :P21:16
ccesario__I only read abut docker-p documentation21:16
sbezverksdake how about /etc/hostname idea?21:16
sdake[sdake@minime-03 tools]$ docker run -it --net=host --hostname=bust ubuntu21:16
sdakedocker: Error response from daemon: Conflicting options: hostname and the network mode.21:16
sdakeSee 'docker run --help'.21:16
sdakeit appears docker-py doesne't know about this conflict either ;)21:17
ccesario__docker-py api does not mean it21:17
vhosakotsdake: 3rd recheck saw no rabbitmq crash and nova VM reached ACTIVE state on centos in gate....21:17
vhosakot624c2c44-a9e7-4d51-aef8-ff6a4d14c3a6 | kolla_boot_test | ACTIVE | -          | Running     | demo-net=10.0.0.3 |21:17
ccesario__yes.... sorry by missing this info21:18
vhosakotsdake: doing 4th recheck now...21:18
*** shakamunyi has joined #openstack-kolla21:18
vhosakotsdake: https://bugs.launchpad.net/kolla/+bug/156838921:20
openstackLaunchpad bug 1568389 in kolla "Gate must show FAILURE when nova VM does not reach ACTIVE state" [Undecided,New] - Assigned to Vikram Hosakote (vhosakot)21:20
ccesario__sdake: based in this info.. what is the best way to do solve it into kolla ?21:21
ccesario__what is your suggestion ?21:21
sdakei dont know but it doesn't involve a kolla-docker.py chane21:22
sdakeinstead of hardcoding the envrionment in the docker fiels why not put in ansible environment?21:22
*** shakamunyi has quit IRC21:23
sbezverkccesario__ sdake we can create a  file in /etc called hostname and insert there any name which will become container's hostname21:23
sdakesbezverk i dont have issue with ccesario__ 's bashrc chaneg21:23
sdakeits the modifying 20 million docker files21:24
ccesario__sdake, sbezverk : I have any idea21:24
sdakeccesario__ set the PS1 environment variable to the container name21:25
ccesario__I can usage the PS1 and make it dynamic based iin container name21:25
sdakewhen docker runs it21:25
ccesario__yep21:25
sdakelet me show you e xampel21:26
ccesario__sdake: I have it done21:26
ccesario__let me show you21:26
ccesario__sdake: http://pastebin.com/wRzBk3e321:29
ccesario__it works21:29
ccesario__it inject the var when the container run/start21:30
sdakeon line 11 where is KOLLA_SERVICE_NAME come from?21:30
sdakeis/does21:30
sdakehttp://paste.fedoraproject.org/352991/23757414/ -> http://paste.fedoraproject.org/352991/2375741421:33
ccesario__self.params.get('name').replace('_','-')     https://github.com/openstack/kolla/blob/master/ansible/library/kolla_docker.py#L49121:33
ccesario__the param passed on container build21:33
sdaketry the paste abovve21:34
sdakesee if it works21:34
sdakethat is a low pain way to  do the job21:34
ccesario__let me test your suggestion21:36
*** daneyon has joined #openstack-kolla21:38
sdakehow do you get in a container as bash?21:43
*** daneyon has quit IRC21:43
sdakedocker exec containername bash returns immediately for me21:43
sdake[sdake@minime-03 tools]$ docker exec -it haproxy bash21:44
sdakesteak_for_dinner_haproxy21:44
sdakehttp://paste.fedoraproject.org/352993/14602382/ -> http://paste.fedoraproject.org/352993/1460238221:44
sdakewfm21:44
sdakeif you want to have more control over what goes in PS1, just encode the cointainer name in the environment in ansible, not in the docker files21:45
sdakeccesario__ enjoy ;)21:45
ccesario__sdake: hmmmm let me test it on my ubuntu21:47
sbezverksdake hav you seen this on the latest master with #3 of ccesario__  http://paste.openstack.org/show/493575/21:49
*** cmceniry has quit IRC21:53
sdakeno21:54
sdakelet me go on21:55
sdakelike i blister in the sun21:55
sbezverkretrying to see if it reproducible or not21:55
ccesario__sdake: on ubuntu the PS1 env is overwrited by bashrc file21:55
sdakebig hands i knwo our the one21:55
sdakeok well overwrit the bashrc file21:55
sdakeand passs in an environment variable "CONTAINER_NAME"21:55
*** jtriley has quit IRC21:56
sdakeand set ps1 to that environment variable21:56
sdakeagain my issueisn' with th ebashrc but with the modifying 90 docker files21:56
ccesario__sdake: yes yes21:56
sdakeit would be nice if container name included the bare metal hostname in it too ;)21:57
*** jtriley has joined #openstack-kolla21:57
sdakeas in [192.168.1.103:haproxy ~]:21:57
sdakeor whatever21:57
ccesario__can I do it on base image file ?21:58
ccesario__http://pastebin.com/LHdTuLRf21:58
sdakethat looks solid21:59
sdakeone file to change21:59
*** vhosakot has quit IRC21:59
sdakedont ask permission btw ;)21:59
*** vhosakot has joined #openstack-kolla21:59
* sdake will break people of that habit eventually ;)21:59
sdakejust do what you think is  right, the reviewers will sort it out if its wrong22:00
vhosakotsdake: 4th recheck saw no rabbitmq crash and nova VM reached ACTIVE state in gate22:01
vhosakot19f99786-fc45-4d79-9dd8-ec1708f2f2ac | kolla_boot_test | ACTIVE | -          | Running     | demo-net=10.0.0.3 |22:01
vhosakotsdake: doing 5th recheck22:01
*** sdake_ has joined #openstack-kolla22:01
vhosakotsdake: I mean, nova VM on centos gate reach active state22:01
*** jtriley has quit IRC22:02
sdake_binary gat working well now?22:03
ccesario__sdake: thanks a lot by suggestions and directions22:03
sdake_ccesario__ your welcome ;)22:04
ccesario__I'll provide a new patch set22:04
ccesario__vhosakot: sbezverk thanks by help too!!!22:04
*** sdake has quit IRC22:04
vhosakotccesario__: haha, i did not help... but, I like your PS a lot.. very useful when we see hostname inside container!22:04
vhosakotsdake_: centos binary gate does not see rabbit crash and no keystone json error, but nova vm does not reach active state... it looks like a new issue, and we'll a new bug to track it (nova VM not reaching ACTIVE state in centos binary gate).... looks like a nova-api issue in centos binary gate... search for Traceback   in  http://logs.openstack.org/86/303686/6/check/gate-kolla-dsvm-deploy-centos-binary/a0ac497/logs/kolla/no22:07
vhosakotva/nova-api.log22:07
vhosakotsdake_: looks like a nova-api issue in centos binary gate... search for Traceback   in  http://logs.openstack.org/86/303686/6/check/gate-kolla-dsvm-deploy-centos-binary/a0ac497/logs/kolla/nova/nova-api.log22:07
vhosakotsdake_: could you ack  https://review.openstack.org/#/c/302026/..... updated prechecks and addresses Jeffrey's comments22:10
patchbotvhosakot: patch 302026 - kolla - Add prechecks to validate kolla_internal_vip_addre...22:10
vhosakotbbiab 20 mins22:10
sbezverksdake if you check docker exec heka ls -al  /var/log/kolla/rabbitmq/rabbit.log on any test bed you will see that log has size 0.22:15
sbezverkto address it newer version of erlang is required22:16
sbezverkbasically we do not collect any rabbitmq logs22:16
*** cmceniry has joined #openstack-kolla22:17
*** shakamunyi has joined #openstack-kolla22:18
sbezverkfor tracking purposes https://bugs.launchpad.net/kolla/+bug/156840822:22
openstackLaunchpad bug 1568408 in kolla "rabbitmq logs are not collected" [Undecided,New]22:22
*** shakamunyi has quit IRC22:23
*** vhosakot has quit IRC22:25
*** cmceniry has quit IRC22:30
*** cmceniry has joined #openstack-kolla22:32
*** vhosakot has joined #openstack-kolla22:32
ccesario__inside ansible/roles/haproxy/tasks/start.yml  is there any var that return vale of property "name"  in this case name: "haproxy"22:33
*** vhosakot_ has joined #openstack-kolla22:36
*** vhosakot has quit IRC22:38
*** banix has joined #openstack-kolla22:55
*** Marga_ is now known as marga_23:14
*** marga_ is now known as fragatina23:15
*** fragatina has quit IRC23:16
*** shakamunyi has joined #openstack-kolla23:18
*** allen_gao has quit IRC23:18
*** allen_gao has joined #openstack-kolla23:19
*** fragatina has joined #openstack-kolla23:21
*** shakamunyi has quit IRC23:23
*** fragatina has quit IRC23:24
*** sdake has joined #openstack-kolla23:27
*** sdake_ has quit IRC23:29
*** ccesario__ has quit IRC23:35
*** fragatina has joined #openstack-kolla23:36
*** sdake has quit IRC23:36
*** ccesario_ has joined #openstack-kolla23:37
*** fragatina has quit IRC23:41

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!