| *** ccesario__ has joined #openstack-kolla | 00:01 | |
| ccesario__ | good evening :) | 00:03 |
|---|---|---|
| *** ccesario___ has quit IRC | 00:08 | |
| *** daneyon_ has joined #openstack-kolla | 00:11 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Add LVM2/iSCSI documentation https://review.openstack.org/300824 | 00:13 |
| *** daneyon_ has quit IRC | 00:15 | |
| *** cbits has joined #openstack-kolla | 00:18 | |
| *** cbits has quit IRC | 00:46 | |
| *** ccesario_ has joined #openstack-kolla | 00:54 | |
| *** Marga_ has quit IRC | 00:56 | |
| *** Marga_ has joined #openstack-kolla | 00:57 | |
| *** salv-orl_ has quit IRC | 00:57 | |
| *** stvnoyes has quit IRC | 01:08 | |
| *** jtriley has quit IRC | 01:09 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 01:13 | |
| *** shakamunyi has joined #openstack-kolla | 01:18 | |
| *** unicell has quit IRC | 01:19 | |
| *** achanda has joined #openstack-kolla | 01:21 | |
| *** shakamunyi has quit IRC | 01:23 | |
| *** stvnoyes has joined #openstack-kolla | 01:23 | |
| *** achanda has quit IRC | 01:30 | |
| *** ccesario_ has quit IRC | 01:31 | |
| *** ccesario_ has joined #openstack-kolla | 01:31 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add a role to cleanup containers after deployment https://review.openstack.org/303575 | 01:39 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add a role to cleanup containers after deployment https://review.openstack.org/303575 | 01:40 |
| *** Jeffrey4l_ has quit IRC | 01:44 | |
| *** Kennan has quit IRC | 01:51 | |
| *** Kennan has joined #openstack-kolla | 01:52 | |
| *** cbits has joined #openstack-kolla | 01:57 | |
| *** achanda has joined #openstack-kolla | 01:59 | |
| *** cbits has quit IRC | 02:03 | |
| *** achanda has quit IRC | 02:07 | |
| *** sbezverk has quit IRC | 02:08 | |
| *** dave-mcc_ has joined #openstack-kolla | 02:09 | |
| *** dave-mccowan has quit IRC | 02:09 | |
| *** sbezverk has joined #openstack-kolla | 02:11 | |
| *** mwheckmann has joined #openstack-kolla | 02:13 | |
| *** pbourke has quit IRC | 02:15 | |
| *** pbourke has joined #openstack-kolla | 02:16 | |
| *** salv-orlando has joined #openstack-kolla | 02:18 | |
| *** Kennan has quit IRC | 02:19 | |
| *** banix has joined #openstack-kolla | 02:21 | |
| *** sdake has joined #openstack-kolla | 02:21 | |
| *** Marga_ has quit IRC | 02:24 | |
| *** Marga_ has joined #openstack-kolla | 02:24 | |
| *** salv-orlando has quit IRC | 02:24 | |
| *** wuhg has quit IRC | 02:29 | |
| *** sdake has quit IRC | 02:29 | |
| *** coolsvap has joined #openstack-kolla | 02:36 | |
| *** Kennan has joined #openstack-kolla | 02:36 | |
| *** dave-mcc_ has quit IRC | 02:43 | |
| vhosakot | manjeets: ping | 02:49 |
| *** mwheckmann has quit IRC | 02:51 | |
| *** klint has joined #openstack-kolla | 02:52 | |
| openstackgerrit | Merged openstack/kolla: Increase max pool size so conductor doesn't implode https://review.openstack.org/302415 | 02:57 |
| *** sdake has joined #openstack-kolla | 03:10 | |
| *** mwheckmann has joined #openstack-kolla | 03:13 | |
| *** Kennan has quit IRC | 03:15 | |
| *** Kennan has joined #openstack-kolla | 03:15 | |
| sdake | re https://review.openstack.org/#/c/302415/ | 03:16 |
| patchbot | sdake: patch 302415 - kolla - Increase max pool size so conductor doesn't implode (MERGED) | 03:16 |
| sdake | did manjeets confirm it fixed his problem? | 03:16 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Increase max pool size so conductor doesn't implode https://review.openstack.org/303677 | 03:17 |
| *** shakamunyi has joined #openstack-kolla | 03:18 | |
| *** shakamunyi has quit IRC | 03:23 | |
| sdake | this neesd an ack https://review.openstack.org/#/c/303509/ | 03:26 |
| patchbot | sdake: patch 303509 - kolla - Fix the qemu has nova permission in nova instances... | 03:26 |
| *** Kennan has quit IRC | 03:27 | |
| *** Kennan has joined #openstack-kolla | 03:28 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Increase max scheduling attempts to 10 https://review.openstack.org/303678 | 03:28 |
| vhosakot | sdake: re https://review.openstack.org/#/c/302415/, I gave +2 and workflow as ryan gave +2 and the gate was green.... later, I saw ryan's comment in the reivew that he was waiting for manjeet's confirmation when he gave +1... I pinged manjeets and could not reach him | 03:29 |
| patchbot | vhosakot: patch 302415 - kolla - Increase max pool size so conductor doesn't implode (MERGED) | 03:29 |
| sdake | ok well keep pingin him and get him to veirf yit :) | 03:30 |
| vhosakot | sdake: yes, https://review.openstack.org/#/c/303509/ is great and nova VM in ACTIVE state in gate! | 03:30 |
| patchbot | vhosakot: patch 303509 - kolla - Fix the qemu has nova permission in nova instances... | 03:30 |
| vhosakot | sdake: yes, I will keep findind manjeets :) | 03:31 |
| sdake | wow petty much crashed from noon until 7pm | 03:32 |
| sdake | atleast sleep is back in order ;) | 03:32 |
| openstackgerrit | Merged openstack/kolla: Fix the qemu has nova permission in nova instances folder https://review.openstack.org/303509 | 03:32 |
| vhosakot | sdake: looking at nova.conf's documentation, seeting max_pool_size and max_overflow to high values in nova.conf should not have any side effects to regular loads and in gate.. they will help in scaling for sure... | 03:33 |
| vhosakot | sdake: nice, i got some sleep as well | 03:33 |
| coolsvap | order or reverse order? | 03:34 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix the qemu has nova permission in nova instances folder https://review.openstack.org/303680 | 03:34 |
| vhosakot | sdake: https://review.openstack.org/#/c/302036/ just got merge conflict | 03:35 |
| patchbot | vhosakot: patch 302036 - kolla - Use proper CentOS repositories for ceph and qemu | 03:35 |
| sdake | yes i expected that vhosakot | 03:35 |
| sdake | i was waiting for that patch to mergeto rebase | 03:35 |
| vhosakot | sdake: yep, too many patches touching same line in same file | 03:35 |
| *** sdake_ has joined #openstack-kolla | 03:40 | |
| *** sdake has quit IRC | 03:40 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 03:43 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 03:45 |
| *** salv-orlando has joined #openstack-kolla | 03:48 | |
| *** mwheckmann has quit IRC | 03:50 | |
| *** salv-orlando has quit IRC | 03:52 | |
| *** Kennan2 has joined #openstack-kolla | 03:53 | |
| *** Kennan has quit IRC | 03:55 | |
| *** jtriley has joined #openstack-kolla | 04:06 | |
| *** shakamunyi has joined #openstack-kolla | 04:18 | |
| *** shakamunyi has quit IRC | 04:23 | |
| sdake_ | sbezverk ping | 04:32 |
| sdake_ | vhosakot ping | 04:33 |
| vhosakot | sdake_: pong | 04:34 |
| sdake_ | vhosakot we have a problem with rabbitmq | 04:35 |
| sdake_ | do you know anything bout it | 04:35 |
| vhosakot | sdake_: yes, I have been debugging it for almost a week now.. I think Jeffrey is also looking into it.. | 04:36 |
| sdake_ | what doy ou know | 04:36 |
| sdake_ | wher eis the bug | 04:36 |
| vhosakot | crash on centos, just in gate, 1 sec | 04:36 |
| sdake_ | lets get what you know recorded in the bug if anything | 04:36 |
| vhosakot | sdake_: already recorded https://bugs.launchpad.net/kolla/+bug/1562701 | 04:37 |
| openstack | Launchpad bug 1562701 in kolla mitaka "rabbitmq crashed on centos binary deploy" [Critical,Confirmed] | 04:37 |
| sdake_ | if it fails in the gate it will fail in produciton | 04:37 |
| vhosakot | sdake_: scroll to the bottom.. I see your reply as well... there are a couple of things I want to try.... related to updating rabbitmq's config file to fix this bug... | 04:37 |
| vhosakot | sdake_: yes, failure in gate is important even if not seen in local env | 04:38 |
| sdake_ | i think y pid patch was not rabbitmq related but mysql related | 04:38 |
| vhosakot | sdake_: just search for Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",econnrefused] on google... lotf os link | 04:38 |
| vhosakot | sdake_: I made 1 more observation... there is a keystone failure as well everytime this crash is seen | 04:39 |
| sdake_ | keystone does not use rabbitmq | 04:43 |
| vhosakot | sdake_: open this logs | 04:44 |
| vhosakot | sdake_: http://logs.openstack.org/34/297434/2/check/gate-kolla-dsvm-deploy-centos-source/2b0a42b/console.html#_2016-04-04_16_13_45_009 | 04:44 |
| *** daneyon_ has joined #openstack-kolla | 04:44 | |
| vhosakot | sdake_: search for Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",econnrefused] | 04:45 |
| vhosakot | sdake_: then, search for keystone | Creating admin project, user, role, service, and endpoint | 04:45 |
| sdake_ | break it down for me - what is the error | 04:45 |
| sdake_ | -elazy | 04:45 |
| vhosakot | sdake_: rabbitmq is not starting as epmd is down | 04:46 |
| sdake_ | is the failure on one particular provider in the gate? | 04:46 |
| sdake_ | you can tell at the top of the gate log by looking t the kernel | 04:46 |
| vhosakot | devstack-centos7-osic-cloud1 is the gate that failed | 04:47 |
| sdake_ | is it always osic | 04:48 |
| sdake_ | or is it all over the place | 04:48 |
| vhosakot | let me check | 04:49 |
| vhosakot | sdake_: bluebox as well... devstack-centos7-bluebox-sjc1 | 04:50 |
| sdake_ | bluebox is flakey | 04:50 |
| sdake_ | i see gate failureson bluebox all the time | 04:50 |
| sdake_ | i hooked my stereo up today and it worked | 04:52 |
| * sdake_ yays ;) | 04:52 | |
| *** salv-orlando has joined #openstack-kolla | 04:52 | |
| vhosakot | sdake_: always centos, always keystone failure are seen in rabbit crash | 04:53 |
| vhosakot | sdake_: nice! | 04:53 |
| sdake_ | what is the keystone failure | 04:55 |
| sdake_ | paste ? | 04:55 |
| *** salv-orlando has quit IRC | 04:57 | |
| vhosakot | sdake_: http://paste.openstack.org/show/493556/ | 04:57 |
| vhosakot | sdake_: this is seen before every rabbit crash | 04:58 |
| sdake_ | is this prior to rabbitmq failing? | 04:59 |
| sdake_ | i see jeffrey4l has crshed out :) | 05:00 |
| *** Kennan2 has quit IRC | 05:03 | |
| sdake_ | my qemu patch works on centos | 05:03 |
| sdake_ | :) | 05:03 |
| vhosakot | nice! | 05:03 |
| vhosakot | sdake_: yes, prior to rabbitmw crash | 05:04 |
| sdake_ | ok well that tells me its a systemic problem | 05:04 |
| *** banix has quit IRC | 05:04 | |
| sdake_ | not a rabbitmq specific problem | 05:04 |
| sdake_ | probably something wrong with /etc/host or the like | 05:04 |
| sdake_ | figure out the keystone crash | 05:05 |
| sdake_ | via debug | 05:05 |
| sdake_ | and the rabbitmq will probablly self-resolve | 05:05 |
| *** banix has joined #openstack-kolla | 05:08 | |
| sdake_ | sounds like a gate issue to me | 05:08 |
| sdake_ | https://bugs.launchpad.net/kolla/+bug/1568047 | 05:09 |
| openstack | Launchpad bug 1568047 in kolla "/etc/hostname has wrong selinux context" [Low,In progress] - Assigned to Paul Belanger (pabelanger) | 05:09 |
| *** banix has quit IRC | 05:09 | |
| sdake_ | maybe that will magicaly fix it | 05:11 |
| *** salv-orlando has joined #openstack-kolla | 05:12 | |
| *** Marga__ has joined #openstack-kolla | 05:12 | |
| *** salv-orlando has quit IRC | 05:12 | |
| *** Marga_ has quit IRC | 05:15 | |
| *** shakamunyi has joined #openstack-kolla | 05:18 | |
| *** Kennan has joined #openstack-kolla | 05:19 | |
| sdake_ | vhosakot internap gate has this keystone failure as well http://logs.openstack.org/36/302036/25/check/gate-kolla-dsvm-deploy-centos-binary/bde791d/console.html#_2016-04-09_03_58_49_796 | 05:19 |
| vhosakot | yep, hopefully /etc/hostname will fix rabbit crash as well | 05:19 |
| vhosakot | sdake_: and the same log has rabbitmw crash as well http://logs.openstack.org/36/302036/25/check/gate-kolla-dsvm-deploy-centos-binary/bde791d/console.html#_2016-04-09_03_59_08_218 | 05:22 |
| vhosakot | sdake_: search for econnrefused in the same logs | 05:22 |
| *** shakamunyi has quit IRC | 05:23 | |
| sdake_ | the failure is systemtic | 05:25 |
| sdake_ | i have an idea | 05:25 |
| sdake_ | can you try this | 05:26 |
| sdake_ | can you rpint out whatever config files are used for both keystone and rabbbitmq for those two tasks | 05:26 |
| sdake_ | and whatever variables are used inteh playbooks | 05:26 |
| sdake_ | for those two tasks | 05:26 |
| sdake_ | something will be obviously wrong | 05:26 |
| sdake_ | ththen get it to fail | 05:27 |
| sdake_ | then look at the variables and config files | 05:27 |
| sdake_ | and see what isn't set prperly | 05:27 |
| sdake_ | once we find wht isn't set prperly | 05:27 |
| sdake_ | we can fix it | 05:27 |
| sdake_ | can you do that or shall I | 05:28 |
| vhosakot | sdake_: I can do it tomorrow, I think the config files are good, and the bug is not seen on Ubuntu... I think printing just the variables is good... | 05:31 |
| sdake_ | the config files re obviously nto good if epmd doens't start | 05:31 |
| vhosakot | sdake_: so, do you think the keystone json error and rabbitmq crash are related... ? keystone error causes rabbit crash ? | 05:31 |
| sdake_ | no one doesn ot cause the other | 05:31 |
| sdake_ | a systemtic failure causeses both | 05:31 |
| vhosakot | ah, single root cause ? | 05:32 |
| sdake_ | yes | 05:32 |
| sdake_ | well since i just had a 6 hour nap | 05:35 |
| sdake_ | i'm going to get started on this | 05:35 |
| vhosakot | ok, thanks... I am add more to it if needed... I'm about to go to bed :) | 05:37 |
| vhosakot | am==can | 05:37 |
| sdake_ | here are all the variables printed | 05:39 |
| sdake_ | http://logs.openstack.org/36/302036/25/check/gate-kolla-dsvm-deploy-centos-binary/bde791d/console.html#_2016-04-09_03_58_49_793 | 05:39 |
| *** Kennan has quit IRC | 05:39 | |
| *** Kennan has joined #openstack-kolla | 05:40 | |
| coolsvap | sdake_: let me know if i can help with anything | 05:43 |
| sdake_ | here is one whre it works http://logs.openstack.org/09/303509/7/check/gate-kolla-dsvm-deploy-centos-binary/d7a3932/console.html#_2016-04-08_20_10_00_488 | 05:45 |
| vhosakot | sdake_: the from_json part in https://github.com/openstack/kolla/blob/master/ansible/roles/keystone/tasks/register.yml#L5 is failing | 05:45 |
| sdake_ | see any delta? | 05:45 |
| vhosakot | sdake_: the variable keystone_bootstrap must be printed in https://github.com/openstack/kolla/blob/master/ansible/roles/keystone/tasks/register.yml#L5 using -debug | 05:45 |
| sdake_ | vhosakot what is the syntax | 05:46 |
| vhosakot | sdake_: 1 sec | 05:47 |
| * sdake_ still not an ansible rocketscientist | 05:47 | |
| sdake_ | when you get a syntax setup can you submit a patch | 05:47 |
| sdake_ | and i'll rechec kit and add info the the log | 05:47 |
| vhosakot | sdake_: ok, I will submit it... | 05:49 |
| sdake_ | the otuput will be extremely helpful | 05:52 |
| sdake_ | becaue it will provide another data point | 05:52 |
| sdake_ | since they always fail together | 05:53 |
| *** Kennan has quit IRC | 05:53 | |
| sdake_ | and i am pretty sure keystone bootstrap does not depend on rabbitmq | 05:53 |
| *** Kennan has joined #openstack-kolla | 05:54 | |
| vhosakot | yep, almost done... pushing PS now | 05:54 |
| sdake_ | make sure not to disturb the play tho | 05:55 |
| sdake_ | we still want it to run and fail or succeed | 05:55 |
| vhosakot | after that keystone error, nothing runs, deploy terminates in gate | 05:58 |
| vhosakot | rabbitmq crashes first, keystone error next, deploy terminates, rabbit crash messages are seen in "docker logs --tail all rabbitmq" | 05:59 |
| sdake_ | right | 05:59 |
| sdake_ | so whti mean is we want the keystone play to run | 06:00 |
| sdake_ | what caues the gate to fail is the keystone botstrap play crashes | 06:00 |
| sdake_ | not the epmd rabbitmq error | 06:00 |
| *** Kennan has quit IRC | 06:07 | |
| sdake_ | my money is on selinux problems | 06:07 |
| vhosakot | ah ok | 06:07 |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Debug rabbitmq crash in centos gate https://review.openstack.org/303686 | 06:09 |
| vhosakot | sdake_: ^^^ | 06:09 |
| vhosakot | sdake_: just printing keystone_bootstrap | 06:09 |
| sdake_ | will that print if the task prior fails? | 06:10 |
| sdake_ | we need to knwo the good output and the bad output | 06:10 |
| sdake_ | i suspect if ansible crashes which is what is happening now that debug print wont execute | 06:11 |
| vhosakot | ah... no I have commented the line that crashed | 06:11 |
| vhosakot | so, the bug line should be hit | 06:11 |
| sdake_ | so how will we know when the json would have crashed? | 06:11 |
| vhosakot | ah :) | 06:12 |
| sdake_ | add a -debug: msg=test {{ kestyone_bootstrap | from_json?) | 06:12 |
| vhosakot | good point... | 06:12 |
| vhosakot | after the current debug ? | 06:12 |
| sdake_ | right | 06:12 |
| vhosakot | yep | 06:12 |
| vhosakot | let me push the new PS :) | 06:12 |
| sdake_ | can you do that so i dont mess up the syntax | 06:12 |
| vhosakot | yep.. doing it now | 06:13 |
| sdake_ | thanks | 06:13 |
| sdake_ | good eye catching the keystone thing | 06:13 |
| *** salv-orlando has joined #openstack-kolla | 06:13 | |
| *** shakamunyi has joined #openstack-kolla | 06:18 | |
| *** daneyon_ has quit IRC | 06:18 | |
| *** Kennan has joined #openstack-kolla | 06:22 | |
| *** shakamunyi has quit IRC | 06:23 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 06:23 | |
| *** salv-orlando has quit IRC | 06:23 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Debug rabbitmq crash in centos gate https://review.openstack.org/303686 | 06:27 |
| *** d_code has quit IRC | 06:27 | |
| vhosakot | sdake_: ^^^ added second debug to pring json as well | 06:27 |
| vhosakot | can you recheck to make it hit the gate machine that crashes rabbitmq... | 06:28 |
| sdake_ | what does from_json do | 06:28 |
| sdake_ | it converts text into a json blob? | 06:28 |
| vhosakot | yep, from raw text stream that comes as the output of a command from the remote machine to json dict so it can be processed it code | 06:30 |
| sdake_ | it will be intreting to see what the text is | 06:31 |
| sdake_ | i suspect the database is not operational inthe rabbitmq failure either | 06:32 |
| vhosakot | sdake_: this is what is json-ized https://github.com/openstack/kolla/blob/master/docker/keystone/keystone_bootstrap.sh#L48 | 06:32 |
| sdake_ | wow i didn't realie we provided that script | 06:32 |
| vhosakot | sdake_: the command that causes the json error is "docker exec -t keystone kolla_keystone_bootstrap ..... " | 06:33 |
| sdake_ | yup i know that | 06:33 |
| vhosakot | sdake_: https://github.com/openstack/kolla/blob/master/docker/keystone/Dockerfile.j2#L69 | 06:33 |
| vhosakot | sdake_: keystone_bootstrap.sh is kolla_keystone_bootstrap | 06:34 |
| sdake_ | whereis project and role coming from | 06:38 |
| sdake_ | https://github.com/openstack/kolla/blob/master/docker/keystone/keystone_bootstrap.sh#L27 | 06:38 |
| vhosakot | PROJECT=$3 and ROLE=$4 https://github.com/openstack/kolla/blob/master/docker/keystone/keystone_bootstrap.sh#L10-L11 | 06:38 |
| vhosakot | sdake_: {{ openstack_auth.project_name }} admin are project and role repectively | 06:41 |
| *** daneyon_ has joined #openstack-kolla | 06:41 | |
| sdake_ | so why is the json parsing wrong? | 06:41 |
| sdake_ | I see nothing wrong with that script | 06:41 |
| vhosakot | yep, that script has been working fine both on centos and ubuntu | 06:43 |
| *** daneyon__ has joined #openstack-kolla | 06:44 | |
| *** daneyon has quit IRC | 06:44 | |
| *** daneyon_ has quit IRC | 06:47 | |
| *** vhosakot has quit IRC | 06:47 | |
| *** daneyon has joined #openstack-kolla | 06:50 | |
| *** daneyon_ has joined #openstack-kolla | 06:50 | |
| *** Kennan has quit IRC | 06:52 | |
| *** SiRiuS_ has joined #openstack-kolla | 06:53 | |
| *** daneyon__ has quit IRC | 06:53 | |
| *** daneyon_ has quit IRC | 06:56 | |
| *** daneyon_ has joined #openstack-kolla | 06:56 | |
| *** daneyon__ has joined #openstack-kolla | 07:02 | |
| *** sdake_ has quit IRC | 07:04 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Debug rabbitmq crash in centos gate https://review.openstack.org/303686 | 07:05 |
| *** daneyon_ has quit IRC | 07:05 | |
| *** Kennan has joined #openstack-kolla | 07:06 | |
| *** vincent_vdk has joined #openstack-kolla | 07:09 | |
| *** Kennan2 has joined #openstack-kolla | 07:11 | |
| *** Kennan has quit IRC | 07:13 | |
| *** Kennan2 has quit IRC | 07:18 | |
| *** daneyon has quit IRC | 07:22 | |
| *** Marga__ has quit IRC | 07:28 | |
| *** Kennan has joined #openstack-kolla | 07:34 | |
| *** salv-orlando has joined #openstack-kolla | 07:43 | |
| *** rmart04 has joined #openstack-kolla | 07:43 | |
| *** jmccarthy has joined #openstack-kolla | 07:43 | |
| *** salv-orlando has quit IRC | 07:45 | |
| *** rmart04 has quit IRC | 07:48 | |
| *** rmart04 has joined #openstack-kolla | 07:49 | |
| *** Kennan has quit IRC | 07:51 | |
| *** rmart04 has quit IRC | 07:55 | |
| *** salv-orlando has joined #openstack-kolla | 08:05 | |
| *** reddy has quit IRC | 08:08 | |
| *** Kennan has joined #openstack-kolla | 08:10 | |
| *** salv-orlando has quit IRC | 08:12 | |
| *** daneyon has joined #openstack-kolla | 08:14 | |
| *** daneyon has quit IRC | 08:19 | |
| *** ccesario_ has quit IRC | 08:19 | |
| *** achanda has joined #openstack-kolla | 08:26 | |
| *** aswadr has joined #openstack-kolla | 08:30 | |
| *** ccesario_ has joined #openstack-kolla | 08:31 | |
| *** Kennan has quit IRC | 08:36 | |
| *** Kennan has joined #openstack-kolla | 08:37 | |
| *** subscope has joined #openstack-kolla | 08:39 | |
| *** achanda has quit IRC | 08:47 | |
| *** SiRiuS_ has quit IRC | 08:48 | |
| *** coolsvap has quit IRC | 09:03 | |
| *** salv-orlando has joined #openstack-kolla | 09:12 | |
| *** openstackgerrit has quit IRC | 09:17 | |
| *** openstackgerrit has joined #openstack-kolla | 09:18 | |
| *** shakamunyi has joined #openstack-kolla | 09:18 | |
| *** ssurana has joined #openstack-kolla | 09:19 | |
| *** salv-orlando has quit IRC | 09:19 | |
| *** shakamunyi has quit IRC | 09:23 | |
| *** aswadr has left #openstack-kolla | 09:34 | |
| *** subscope has quit IRC | 09:39 | |
| *** andrewvwebber has joined #openstack-kolla | 09:42 | |
| *** Kennan has quit IRC | 09:42 | |
| *** Kennan has joined #openstack-kolla | 09:43 | |
| *** allen_gao has quit IRC | 09:51 | |
| *** andrewvwebber has quit IRC | 09:56 | |
| *** allen_gao has joined #openstack-kolla | 09:57 | |
| *** Kennan has quit IRC | 09:59 | |
| *** Kennan has joined #openstack-kolla | 10:00 | |
| *** subscope has joined #openstack-kolla | 10:04 | |
| *** shakamunyi has joined #openstack-kolla | 10:18 | |
| *** shakamunyi has quit IRC | 10:23 | |
| *** salv-orlando has joined #openstack-kolla | 10:42 | |
| *** salv-orlando has quit IRC | 10:47 | |
| *** subscope has quit IRC | 10:57 | |
| *** ccesario___ has joined #openstack-kolla | 11:01 | |
| *** ccesario_ has quit IRC | 11:05 | |
| *** salv-orlando has joined #openstack-kolla | 11:07 | |
| *** Kennan has quit IRC | 11:09 | |
| *** salv-orlando has quit IRC | 11:11 | |
| *** ccesario___ has quit IRC | 11:14 | |
| *** subscope has joined #openstack-kolla | 11:14 | |
| *** ccesario___ has joined #openstack-kolla | 11:14 | |
| *** shakamunyi has joined #openstack-kolla | 11:18 | |
| *** ccesario___ has quit IRC | 11:19 | |
| *** shakamunyi has quit IRC | 11:22 | |
| *** sdake has joined #openstack-kolla | 11:23 | |
| *** Marga_ has joined #openstack-kolla | 11:27 | |
| *** Kennan has joined #openstack-kolla | 11:28 | |
| *** ccesario___ has joined #openstack-kolla | 11:31 | |
| *** sdake has quit IRC | 11:35 | |
| *** cbits has joined #openstack-kolla | 11:42 | |
| *** subscope has quit IRC | 11:55 | |
| *** SiRiuS_ has joined #openstack-kolla | 12:06 | |
| *** SiRiuS_ has quit IRC | 12:11 | |
| *** subscope has joined #openstack-kolla | 12:15 | |
| *** shakamunyi has joined #openstack-kolla | 12:18 | |
| *** shakamunyi has quit IRC | 12:23 | |
| *** salv-orlando has joined #openstack-kolla | 12:24 | |
| *** salv-orlando has quit IRC | 12:28 | |
| *** dgonzalez has quit IRC | 12:30 | |
| *** ccesario___ has quit IRC | 12:34 | |
| *** daneyon has joined #openstack-kolla | 12:37 | |
| *** daneyon has quit IRC | 12:41 | |
| *** dgonzalez has joined #openstack-kolla | 12:42 | |
| *** Jeffrey4l_ has quit IRC | 12:47 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 12:51 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the wrong package name in Ubuntu binary build https://review.openstack.org/303707 | 12:58 |
| *** klint has quit IRC | 13:01 | |
| *** subscope has quit IRC | 13:09 | |
| *** SiRiuS_ has joined #openstack-kolla | 13:11 | |
| *** subscope has joined #openstack-kolla | 13:12 | |
| *** shakamunyi has joined #openstack-kolla | 13:18 | |
| *** shakamunyi has quit IRC | 13:23 | |
| *** ccesario___ has joined #openstack-kolla | 13:36 | |
| *** salv-orlando has joined #openstack-kolla | 13:44 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Collect the Kolla configs in the gate https://review.openstack.org/303710 | 13:50 |
| *** vhosakot has joined #openstack-kolla | 13:53 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: [DO NOT MERGE] Debug rabbitmq crash in centos gate https://review.openstack.org/303686 | 13:56 |
| *** salv-orlando has quit IRC | 14:01 | |
| *** subscope has quit IRC | 14:17 | |
| *** shakamunyi has joined #openstack-kolla | 14:18 | |
| *** vhosakot has quit IRC | 14:21 | |
| *** vhosakot has joined #openstack-kolla | 14:21 | |
| *** shakamunyi has quit IRC | 14:23 | |
| vhosakot | Jeffrey4l_: ping | 14:26 |
| *** sdake has joined #openstack-kolla | 14:26 | |
| vhosakot | Jeffrey4l_: your review https://review.openstack.org/#/c/294906/ collected erl_crash.dump in logs/kolla/ in gate.... http://logs.openstack.org/06/294906/5/check/gate-kolla-dsvm-deploy-centos-source/a8d64f5/logs/kolla/.... awesome! | 14:26 |
| patchbot | vhosakot: patch 294906 - kolla - DO NOT MERGE: TEST MASTER BRANCH | 14:26 |
| vhosakot | Jeffrey4l_: we need to decode it.. | 14:27 |
| *** cbits has quit IRC | 14:27 | |
| sdake | morning | 14:27 |
| vhosakot | sdake: morning | 14:27 |
| Jeffrey4l_ | vhosakot, sdake morning | 14:27 |
| vhosakot | Jeffrey4l_'s review https://review.openstack.org/#/c/294906/ collected erl_crash.dump in logs/kolla/ in gate.... http://logs.openstack.org/06/294906/5/check/gate-kolla-dsvm-deploy-centos-source/a8d64f5/logs/kolla/.... | 14:27 |
| patchbot | vhosakot: patch 294906 - kolla - DO NOT MERGE: TEST MASTER BRANCH | 14:27 |
| vhosakot | we need to decode it now | 14:28 |
| sdake | did you get any results | 14:28 |
| sdake | i think i passed out at the keyboard ;) | 14:28 |
| Jeffrey4l_ | vhosakot, I seem find the root soon. still need more debug . | 14:28 |
| Jeffrey4l_ | it should be a network issue, i think now. | 14:28 |
| Jeffrey4l_ | but need more time. | 14:29 |
| Jeffrey4l_ | vhosakot, so i do not think we need decode the erl_crash.dump file now. :D | 14:30 |
| vhosakot | Jeffrey4l_: how will we know the issue then ? :) | 14:30 |
| vhosakot | sdake: search for keystone_bootstrap.stdout and econnrefused in http://logs.openstack.org/86/303686/4/check/gate-kolla-dsvm-deploy-centos-source/17631e5/console.html#_2016-04-09_14_16_26_140 | 14:31 |
| Jeffrey4l_ | vhosakot, i am still checking the log and make some test. I just got some thought( it is caused by some network configuration issue) | 14:31 |
| sdake | Jeffrey4l_ your awaare keystone is failing | 14:31 |
| Jeffrey4l_ | sdake, yea. | 14:31 |
| Jeffrey4l_ | I think the failed of keystone and rabbitmq are the same now. But I do not have any prove for this now. :(, still checking and debuging this. | 14:32 |
| vhosakot | Jeffrey4l_: keystone failed in the same logs http://logs.openstack.org/06/294906/5/check/gate-kolla-dsvm-deploy-centos-source/a8d64f5/console.html#_2016-04-08_15_58_51_989 that collected erl_crash.dump from your review in gate http://logs.openstack.org/06/294906/5/check/gate-kolla-dsvm-deploy-centos-source/a8d64f5/logs/kolla/ | 14:32 |
| vhosakot | Jeffrey4l_: cool, I am checking the keystone part... | 14:33 |
| Jeffrey4l_ | vhosakot, cool. Let us debug this in different direction. It will be helpful. :D | 14:34 |
| *** Jeffrey4l_ has left #openstack-kolla | 14:34 | |
| vhosakot | yep | 14:34 |
| *** Jeffrey4l_ has joined #openstack-kolla | 14:34 | |
| vhosakot | Jeffrey4l_: cool | 14:35 |
| Jeffrey4l_ | this bug is very annoying. :( can not reproduce locally. | 14:35 |
| *** ccesario___ has quit IRC | 14:35 | |
| vhosakot | Jeffrey4l_: yes, due to network issue probably in centos gate | 14:36 |
| *** ccesario___ has joined #openstack-kolla | 14:36 | |
| sdake | sweet hunger games is on :) | 14:36 |
| vhosakot | Jeffrey4l_: i like debugging such cannot-reproduce-locally bugs :) | 14:36 |
| vhosakot | sdake: Jeffrey4l_: when rabbitmq container dies, its logs does not show up in /var/lib/docker/volumes/kolla_logs/_data/rabbitmq/..... | 14:37 |
| vhosakot | sdake: Jeffrey4l_: shouldn;t kolla retain logs of containers that die/crash ? | 14:38 |
| Jeffrey4l_ | vhosakot, what i found now is: the failed CentOS eth0 ip address is a public address and the route is especial. | 14:38 |
| Jeffrey4l_ | see this http://logs.openstack.org/06/294906/5/check/gate-kolla-dsvm-deploy-centos-source/a8d64f5/console.html#_2016-04-08_15_41_58_836 | 14:38 |
| Jeffrey4l_ | seem all of them came from `evstack-centos7-ovh-bhs1` | 14:38 |
| vhosakot | 158.69.94.147/32 ? | 14:39 |
| Jeffrey4l_ | yea. | 14:39 |
| vhosakot | Jeffrey4l_: please merge my precheck PS.. it will catch it :) | 14:39 |
| Jeffrey4l_ | and the default gateway is default via 158.69.64.1 dev eth0 | 14:39 |
| Jeffrey4l_ | vhosakot, you PS and this bug is different things. | 14:39 |
| vhosakot | Jeffrey4l_: but, it will catch if eth0 is public IP and pingable :) | 14:40 |
| Jeffrey4l_ | vhosakot, so we stop to run the jobs and failed the gate? | 14:40 |
| vhosakot | Jeffrey4l_: yes.... becasue, when prechecks fail (due to public IP on pingable IP on etho), job will have issue like this... we need to exit the job in gate | 14:41 |
| vhosakot | Jeffrey4l_: https://review.openstack.org/#/c/302026/ :) I have replied to your comments... it works fine and I have tested it :) | 14:41 |
| patchbot | vhosakot: patch 302026 - kolla - Add prechecks to validate kolla_internal_vip_addre... | 14:41 |
| Jeffrey4l_ | vhosakot, no. the first thing we need to do is satisfy the gate. Not let the gate satisfy our code. | 14:42 |
| Jeffrey4l_ | if you pre-check changes failed the gate. I prefer to fix the gate. If the gate can not be fixed, I prefer to revert your code. | 14:42 |
| vhosakot | Jeffrey4l_: no, I'm sure my precheck is important... if it fails, fix the gate, and not revert the PS :) | 14:43 |
| Jeffrey4l_ | vhosakot, yes. we are fixing the gate. | 14:43 |
| vhosakot | Jeffrey4l_: please see my reply in https://review.openstack.org/#/c/302026/ :) | 14:43 |
| patchbot | vhosakot: patch 302026 - kolla - Add prechecks to validate kolla_internal_vip_addre... | 14:43 |
| Jeffrey4l_ | vhosakot, i know your patch. | 14:43 |
| vhosakot | Jeffrey4l_: cool, thanks! :) | 14:43 |
| Jeffrey4l_ | but it still do not matter the rabbitmq crash issue. | 14:44 |
| vhosakot | Jeffrey4l_: agreed, rabbitmq crash is a different matter.. | 14:45 |
| vhosakot | Jeffrey4l_: we do not run prechecks in gate before "tools/kolla-ansible -vvv deploy".... we need to run prechecks first before depoying... | 14:46 |
| vhosakot | Jeffrey4l_: once https://review.openstack.org/#/c/302026/ is merged, I will submit a PS to add prechecks to gate | 14:47 |
| patchbot | vhosakot: patch 302026 - kolla - Add prechecks to validate kolla_internal_vip_addre... | 14:47 |
| Jeffrey4l_ | vhosakot, ok | 14:47 |
| vhosakot | Jeffrey4l_: cool | 14:47 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Use debian repos for debian base docker image https://review.openstack.org/288936 | 14:48 |
| *** subscope has joined #openstack-kolla | 14:49 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make container dind build with debian https://review.openstack.org/297434 | 14:49 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make heka container build with debian https://review.openstack.org/297737 | 14:50 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make kibana container build with debian https://review.openstack.org/297206 | 14:50 |
| sdake | Jeffrey4l_ agree with this: [07:42:32] <Jeffrey4l_>if you pre-check changes failed the gate. I prefer to fix the gate. If the gate can not be fixed, I prefer to revert your code. | 14:51 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make marathon container build with debian https://review.openstack.org/297378 | 14:51 |
| Jeffrey4l_ | thanks. :D | 14:51 |
| sdake | so the problem is what with theip address? | 14:52 |
| sdake | its public or its /32? | 14:52 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make mesos-slave container build with debian https://review.openstack.org/297385 | 14:52 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make mesosphere-base build with debian https://review.openstack.org/297358 | 14:52 |
| Jeffrey4l_ | sdake, not sure now. I just doubt that. Need more time to check and debug. | 14:52 |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Add prechecks to validate kolla_internal_vip_address and kolla_external_vip_address https://review.openstack.org/302026 | 14:52 |
| sdake | Jeffrey4l_ i got the patch merged you were working on that was bsted | 14:53 |
| sdake | it ended up being pretty simple | 14:53 |
| sdake | and myqemu patch worsk on top | 14:53 |
| vhosakot | Jeffrey4l_: sdake: is 158.69.94.147/32 right for eth0 in gate ? o_O | 14:53 |
| sdake | how would I know ;) | 14:54 |
| Jeffrey4l_ | sdake, glad to here that. | 14:54 |
| Jeffrey4l_ | vhosakot, rule 1: gate is always right. ( because we can not change that ) :) | 14:54 |
| vhosakot | Jeffrey4l_: not necessary... :) gate is not prod :) | 14:55 |
| sdake | http://logs.openstack.org/86/303686/4/check/gate-kolla-dsvm-deploy-centos-source/17631e5/console.html#_2016-04-09_14_16_26_140 | 14:55 |
| sdake | wierd the return is not formatted | 14:55 |
| sdake | with anything | 14:55 |
| sdake | that shell script is spaghetti coding imo ;) | 14:57 |
| vhosakot | sdake: Jeffrey4l_: here we set eth0 (that has 158.69.94.147/32) as network_interface in globals.yml in gate.... https://github.com/openstack/kolla/blob/master/tools/deploy_aio.sh#L61 | 14:57 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Use debian repos for debian base docker image https://review.openstack.org/288936 | 14:57 |
| Jeffrey4l_ | vhosakot, there is only one network_interface. | 14:57 |
| vhosakot | sdake: Jeffrey4l_: but, kolla has a requirement that kolla_internal_vip_address must be in the _same_ network as network_interface (eth0 which is 158.69.94.147/32) | 14:58 |
| *** diogogmt has quit IRC | 14:58 | |
| sdake | oh yes thisi is probably the bluebox gate problem | 14:58 |
| sdake | vhosakot is it bluebox? | 14:58 |
| vhosakot | sdake: Jeffrey4l_: so, there is not free IP that kolla_internal_vip_address can get in gate! as 158.69.94.147/32 is just one address! and already assigned to eth0!!! | 14:58 |
| vhosakot | sdake: Jeffrey4l_: this is gate issue!! | 14:58 |
| sdake | its one of gate issues | 14:59 |
| sdake | not sure if all of them | 14:59 |
| vhosakot | tight | 14:59 |
| vhosakot | right | 14:59 |
| sdake | who is the infra provider | 14:59 |
| sdake | is it bluebox? | 14:59 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: make rabbitmq container build with debian https://review.openstack.org/297728 | 14:59 |
| Jeffrey4l_ | i set a almost the same ip address env http://paste.openstack.org/show/493564/ (ignore the eth1, i do not use it), I can deploy the kolla successful. | 15:00 |
| vhosakot | sdake: Jeffrey4l_: well, kolla_internal_vip_address is harcoded to 169.254.169.10 in https://github.com/openstack/kolla/blob/master/tools/deploy_aio.sh#L59 | 15:00 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make swift-base container build with debian https://review.openstack.org/297121 | 15:00 |
| Jeffrey4l_ | vhosakot, if you doubt it is the root cause, please try to reproduce it locally. | 15:00 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make container kolla-toolbox build with debian https://review.openstack.org/297104 | 15:00 |
| *** salv-orlando has joined #openstack-kolla | 15:01 | |
| vhosakot | sdake: Jeffrey4l_: in gate, kolla_internal_vip_address (169.254.169.10) be _not_ in the same network as eth0 (158.69.94.147/32).. isn't this an issue ? | 15:01 |
| sdake | not if we use eth1 as t he internal network | 15:02 |
| sdake | something sepcial sabout 169.254. address range too | 15:02 |
| sdake | i dont know what | 15:02 |
| sdake | what i rcommend is creating an deth1 and eth2, and using those for management and neutron networks | 15:02 |
| sdake | and leave eth0 alone | 15:03 |
| vhosakot | sdake: i dont see eth1 in gate | 15:04 |
| Jeffrey4l_ | vhosakot, please try to re-produce this issue locally. After that, you can convince. ( before that, i want to check and debug this issue) | 15:04 |
| Jeffrey4l_ | vhosakot, please try to re-produce this issue locally. After that, you can convince me. ( before that, i want to check and debug this issue) | 15:04 |
| vhosakot | Jeffrey4l_: I'm trying to reproduce since more than a week :)... I am not able to reprodice it :) | 15:04 |
| Jeffrey4l_ | vhosakot, so if is not the issue you talk above "kolla has a requirement that kolla_internal_vip_address must be in the _same_ network as network_interface (eth0 which is 158.69.94.147/32)" | 15:05 |
| *** subscope has quit IRC | 15:06 | |
| vhosakot | Jeffrey4l_: right, as sdake said, that requiremtn is not needed and not satisfied in gate | 15:06 |
| *** salv-orlando has quit IRC | 15:06 | |
| vhosakot | Jeffrey4l_: gate uses this as kolla_internal_vip_address --> https://github.com/openstack/kolla/blob/master/tools/deploy_aio.sh#L59 | 15:07 |
| Jeffrey4l_ | vhosakot, we use it all the time. it works all the time. | 15:07 |
| vhosakot | Jeffrey4l_: yep, it has been working all the in gate I know.. I am trying to understand how 169.254.169.10 (kolla_internal_vip_address) is in 158.69.94.147/32 :) | 15:09 |
| vhosakot | Jeffrey4l_: I think 169.254.* is special range or something | 15:09 |
| vhosakot | yes, creating eth1 is the best idea sounds like.. I will try it locally if it works | 15:11 |
| *** diogogmt has joined #openstack-kolla | 15:12 | |
| *** salv-orlando has joined #openstack-kolla | 15:13 | |
| vhosakot | Jeffrey4l_: sdake: I will set kolla_internal_vip_address as 169.254.169.10, and eth0 (network_interface) as 158.69.94.147/32 and see if I can reproduce the bug locally.. | 15:17 |
| *** shakamunyi has joined #openstack-kolla | 15:18 | |
| *** vhosakot has quit IRC | 15:19 | |
| *** salv-orlando has quit IRC | 15:20 | |
| *** banix has joined #openstack-kolla | 15:21 | |
| *** shakamunyi has quit IRC | 15:23 | |
| SiRiuS_ | sdake, Jeffrey4l_ https://en.wikipedia.org/wiki/Link-local_address | 15:27 |
| sdake | yup i knew that was special | 15:28 |
| *** salv-orlando has joined #openstack-kolla | 15:30 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH https://review.openstack.org/294906 | 15:31 |
| *** vhosakot has joined #openstack-kolla | 15:32 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make nova-libvirt container build with debian https://review.openstack.org/297169 | 15:33 |
| SiRiuS_ | yeah, it usually indicates a network card did not get an IP address automatically and one was not set manually or something failed. Then the OS sets one in that range. | 15:34 |
| *** salv-orlando has quit IRC | 15:34 | |
| SiRiuS_ | vhosakot, https://en.wikipedia.org/wiki/Link-local_address the 169.254.0.0/16 IP addresses | 15:35 |
| *** banix has quit IRC | 15:35 | |
| vhosakot | SiRiuS_: thanks for the link... so, we use link-local address 169.254.169.10 (which is in the range link local range 169.254.0.0/16)... it has been working fine in gate all these days | 15:36 |
| *** salv-orlando has joined #openstack-kolla | 15:37 | |
| SiRiuS_ | well, they work fine, as long as you don't sent any packages over a router | 15:38 |
| SiRiuS_ | when you don't set an IP address manually and have no DHCP set up, then the OS will give you an address in this range | 15:39 |
| vhosakot | yep | 15:40 |
| vhosakot | bbiab in 30 mins | 15:41 |
| SiRiuS_ | it's only unique in a network segment only | 15:42 |
| SiRiuS_ | https://en.wikipedia.org/wiki/Network_segment | 15:42 |
| sdake | still box gates fail on waiting on the link local address we specify | 15:46 |
| sdake | btw SiRiuS_ the problem booting vms has been solved | 15:47 |
| SiRiuS_ | sdake, cool was it present in stable/mitaka as well? | 15:48 |
| sdake | been backported | 15:49 |
| SiRiuS_ | coll | 15:50 |
| SiRiuS_ | cool | 15:50 |
| sdake | not mergee | 15:50 |
| sdake | d | 15:50 |
| SiRiuS_ | I'll cherry pick | 15:50 |
| SiRiuS_ | What images should I rebuild? | 15:50 |
| sdake | not sure - dockerfiles shoudl tell you - think *nova8 | 15:51 |
| *** vhosakot has quit IRC | 15:51 | |
| SiRiuS_ | ok cool, will give it a try :) | 15:51 |
| *** sdake_ has joined #openstack-kolla | 15:53 | |
| *** jtriley has quit IRC | 15:54 | |
| *** sdake has quit IRC | 15:55 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH https://review.openstack.org/294906 | 15:57 |
| *** Marga_ has quit IRC | 16:05 | |
| *** diogogmt has quit IRC | 16:05 | |
| *** subscope has joined #openstack-kolla | 16:05 | |
| *** kjelly_home has joined #openstack-kolla | 16:05 | |
| *** kjelly_home has quit IRC | 16:07 | |
| *** Marga_ has joined #openstack-kolla | 16:08 | |
| *** kjelly_home has joined #openstack-kolla | 16:10 | |
| SiRiuS_ | sdake_, is this the patch ? | 16:11 |
| SiRiuS_ | sdake_, https://review.openstack.org/#/c/303680/ | 16:11 |
| patchbot | SiRiuS_: patch 303680 - kolla (stable/mitaka) - Fix the qemu has nova permission in nova instances... | 16:11 |
| *** vhosakot has joined #openstack-kolla | 16:16 | |
| sdake_ | SiRiuS_ yes thats the one | 16:20 |
| *** vhosakot has quit IRC | 16:21 | |
| SiRiuS_ | awesome, thanks | 16:21 |
| *** kjelly_home has quit IRC | 16:21 | |
| *** subscope has quit IRC | 16:24 | |
| ccesario__ | afternoon guys! | 16:26 |
| sdake_ | Jeffrey4l_ go to openstack-infra plz | 16:27 |
| sdake_ | hey ccesario__ | 16:27 |
| ccesario__ | hey sdake_ :) | 16:28 |
| Jeffrey4l_ | sdake_, i was in that channel. | 16:28 |
| sdake_ | well help answer q's elalse :) | 16:28 |
| Jeffrey4l_ | sdake_, i do not think that's a issue. | 16:28 |
| ccesario__ | guys, could you give me to correct way with this patch ? https://review.openstack.org/#/c/297984/ ... if possible of course... | 16:29 |
| patchbot | ccesario__: patch 297984 - kolla - Identify container through PS1 shell variable | 16:29 |
| *** ccesario___ has quit IRC | 16:29 | |
| ccesario__ | I have a bit free time ... and I would like to spend this time with this.... but I need the right direction to do it :) | 16:30 |
| Jeffrey4l_ | sdake_, please check ccesario__ PS https://review.openstack.org/#/c/297984/ . two possible direction. could u give some advice? thanks. | 16:32 |
| patchbot | Jeffrey4l_: patch 297984 - kolla - Identify container through PS1 shell variable | 16:32 |
| sdake_ | Jeffrey4l_ lets go to opensack-infra and help them help us :) | 16:33 |
| *** subscope has joined #openstack-kolla | 16:33 | |
| *** allen_gao has quit IRC | 16:35 | |
| *** allen_gao has joined #openstack-kolla | 16:35 | |
| *** unicell has joined #openstack-kolla | 16:45 | |
| *** subscope has quit IRC | 16:53 | |
| *** SiRiuS_ has quit IRC | 17:04 | |
| *** Jeffrey4l_ has quit IRC | 17:05 | |
| ccesario__ | sdake_: any advice about ps ? | 17:05 |
| *** daneyon has joined #openstack-kolla | 17:07 | |
| *** daneyon has quit IRC | 17:12 | |
| *** vhosakot has joined #openstack-kolla | 17:18 | |
| sdake_ | which ps | 17:20 |
| *** vhosakot has quit IRC | 17:23 | |
| *** ccesario___ has joined #openstack-kolla | 17:24 | |
| sbezverk | sdake would you have sometime later today to check new version of erlang, which rabbitmq folks suggested to try as a solution for issue I saw in liberty? | 17:30 |
| *** ccesario___ has quit IRC | 17:31 | |
| sdake_ | which issue did you see | 17:39 |
| sdake_ | it takes a long time to try new versoin of erlang | 17:39 |
| sdake_ | like 8 hours of my life | 17:39 |
| sdake_ | and that doen't leverage al of the testing we have done on the erlang we currently have | 17:40 |
| sdake_ | and how are youseeing issues in liberty, running the stable/liberty branch? | 17:41 |
| sdake_ | sbezverk ^^ | 17:41 |
| *** unicell has quit IRC | 17:45 | |
| *** sdake has joined #openstack-kolla | 17:49 | |
| sdake | sbezverk ^^ | 17:50 |
| *** sdake_ has quit IRC | 17:51 | |
| *** ssurana has quit IRC | 17:57 | |
| *** sdake_ has joined #openstack-kolla | 18:05 | |
| *** vhosakot has joined #openstack-kolla | 18:07 | |
| *** vhosakot has joined #openstack-kolla | 18:08 | |
| *** sdake has quit IRC | 18:09 | |
| vhosakot | sdake_: ping | 18:09 |
| sdake_ | shoot | 18:09 |
| vhosakot | sdake_: so, when we join the #openstack-kolla channel, we see this heading banner: | 18:10 |
| vhosakot | Topic: Kolla IRC meetngs on Wednesdays @ 16:30 UTC even weeks, 23:00 UTC odd weeks - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA) | 18:10 |
| vhosakot | rhallisey set the topic at: Mar 31, 2016, 12:24 PM | 18:10 |
| vhosakot | Mode: +cnt | 18:10 |
| vhosakot | Created at: Sep 11, 2014, 1:06 PM | 18:10 |
| sdake_ | yes | 18:10 |
| vhosakot | the logs are still pointing to old location | 18:10 |
| sdake_ | yes | 18:10 |
| sdake_ | i cant fix it atm because i dont have registration to freenode | 18:11 |
| vhosakot | sdake_: shouldn;t "IRC channel is *LOGGED* @" open #openstack-logs IRC | 18:11 |
| vhosakot | sdake_: who should we ask ? | 18:11 |
| vhosakot | sdake_: infra ? | 18:11 |
| sdake_ | rhallisey or inc0 can set the channel topic | 18:11 |
| sdake_ | i could too if i was registered | 18:11 |
| vhosakot | sdake_: ok, will check with them.. thanks! | 18:11 |
| *** banix has joined #openstack-kolla | 18:25 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Put the variables in alphabetical order in all.yml https://review.openstack.org/303409 | 18:25 |
| *** Marga_ has quit IRC | 18:27 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Remove /etc/kolla configuration folders https://review.openstack.org/303055 | 18:36 |
| *** banix has quit IRC | 18:36 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add a role to cleanup containers after deployment https://review.openstack.org/303575 | 18:39 |
| *** Marga_ has joined #openstack-kolla | 18:48 | |
| *** jtriley has joined #openstack-kolla | 18:51 | |
| *** Marga_ has quit IRC | 18:52 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: [DO NOT MERGE] Debug rabbitmq crash in centos gate https://review.openstack.org/303686 | 19:03 |
| vhosakot | sdake_: ping | 19:10 |
| vhosakot | sdake_: what is the jenkins link to check gate status when gate is runnning (before jenkins posts results in the review) ? | 19:11 |
| vhosakot | sdake_: ^^^ | 19:18 |
| sdake_ | zuul.openstack.org | 19:19 |
| sdake_ | type kolla in the search bar | 19:19 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Use debian repos for debian base docker image https://review.openstack.org/288936 | 19:19 |
| vhosakot | sdake_: ah, thanks! :) | 19:21 |
| vhosakot | sdake_: ping | 19:26 |
| vhosakot | https://jenkins03.openstack.org/job/gate-kolla-dsvm-deploy-centos-binary/611/console | 19:26 |
| vhosakot | sdake_: ^^^ looks like I resolved the rabbitmq crash... no crash on centos binary | 19:26 |
| sdake_ | resolved how | 19:27 |
| sdake_ | i think oen tesst isn't good enough ;) | 19:27 |
| sdake_ | it is intermittent | 19:27 |
| vhosakot | yep, waiting for centos source resukts | 19:27 |
| vhosakot | sdake_: here is the fix - https://review.openstack.org/303686 | 19:27 |
| vhosakot | sdake_: ERL_EPMD_ADDRESS needs to be commented out in rabbitmq-env.conf.j2... | 19:28 |
| vhosakot | sdake_: this fix is what I found online many people use | 19:28 |
| sdake_ | then epmd binds to 0.0.0.0? | 19:28 |
| vhosakot | sdake_: no rabbitme crash on centos source as well - https://jenkins06.openstack.org/job/gate-kolla-dsvm-deploy-centos-source/603/console | 19:29 |
| vhosakot | sdake_: but, VM not booting in centos... but no crahs | 19:29 |
| sdake_ | recheck it | 19:29 |
| sdake_ | need to see if it actually fixes it | 19:29 |
| sdake_ | what wass the deal with the keystone data output | 19:29 |
| vhosakot | yep, | 19:29 |
| vhosakot | yep | 19:29 |
| sdake_ | why did keystone pressent a json parse error | 19:31 |
| vhosakot | sdake_: well, the keystone error was also not seen this time on both centos binary and centos source.. | 19:31 |
| vhosakot | sdake_: I will recheck it multiple times | 19:31 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make nova-libvirt container build with debian https://review.openstack.org/297169 | 19:31 |
| vhosakot | sdake_: to make sure that _both_ rabbitmq crash and keystone json error are not seen | 19:32 |
| sdake_ | that doesn't explain why keystone parse error happens | 19:32 |
| vhosakot | sdake_: yes, will debug more | 19:32 |
| sdake_ | keystone doesn't use rabbitmq | 19:32 |
| vhosakot | yep | 19:32 |
| vhosakot | sdake_: keystone and ERL_EPMD_ADDRESS ar enot way related :) ERL_EPMD_ADDRESS is a rabbitmqq config caraible | 19:33 |
| vhosakot | variable* | 19:33 |
| sdake_ | i have yet to see a keystone crash without the rabbitmq crash | 19:34 |
| vhosakot | sdake_: yes... rechecking | 19:34 |
| vhosakot | sdake_: ERL_EPMD_ADDRESS is 10.220.2.147 on centos binary gate, and 149.202.162.18 on centos source gate | 19:37 |
| sdake_ | sbezverk coud you specify if this is the issue your seeing pls | 19:37 |
| vhosakot | sbezverk was able to repro the rabbir crash locally ? didn't know that | 19:38 |
| sdake_ | vhosakot it is unclear what crash he has reproduced | 19:39 |
| sdake_ | because he didn't file a bug or tell me yet :) | 19:39 |
| vhosakot | ah ok | 19:39 |
| vhosakot | cool :) | 19:39 |
| vhosakot | sdake_: now I need to see it commenting out ERL_EPMD_ADDRESS does not affect anythign else | 19:39 |
| vhosakot | sdake_: it was used just for centos and not for ubuntu | 19:39 |
| sdake_ | i see that | 19:41 |
| sdake_ | that is why ubuntu never suffersthis failure | 19:41 |
| vhosakot | sdake_: centos source VM did reach ACTIVE state in gate :) | 19:45 |
| vhosakot | sdake_: 023367b1-4a8d-4281-8e3b-b1d751b83184 | kolla_boot_test | ACTIVE | - | Running | demo-net=10.0.0.3 | | 19:45 |
| vhosakot | sdake_: http://logs.openstack.org/86/303686/5/check/gate-kolla-dsvm-deploy-centos-source/8fe9507/console.html#_2016-04-09_19_28_19_793 | 19:45 |
| vhosakot | sdake_: rechecking | 19:45 |
| ccesario__ | sdake_: this ps.. https://review.openstack.org/#/c/297984/ | 19:47 |
| patchbot | ccesario__: patch 297984 - kolla - Identify container through PS1 shell variable | 19:47 |
| sdake_ | ccesario__ what woudl you like me to do with that patch | 19:47 |
| ccesario__ | [16:23] <Jeffrey4l_> sdake_, please check ccesario__ PS https://review.openstack.org/#/c/297984/ . two possible direction. could u give some advice? thanks. | 19:50 |
| patchbot | ccesario__: patch 297984 - kolla - Identify container through PS1 shell variable | 19:50 |
| *** daneyon has joined #openstack-kolla | 19:50 | |
| ccesario__ | sdake_: basicly if is it ok.... or no .... because currently it is working... but Jeffrey4l_ points some things | 19:51 |
| sdake_ | zomg patch is big | 19:52 |
| sdake_ | not keen on hardcoding environment names | 19:53 |
| sdake_ | lots of maintenence there | 19:53 |
| sdake_ | why not use a {{ }} jinja2 replacement? | 19:54 |
| *** dgonzalez has quit IRC | 19:54 | |
| ccesario__ | yes, but very simple ... as oher vars that exists | 19:54 |
| *** daneyon has quit IRC | 19:55 | |
| sdake_ | why not use the docker module to set the hostname of the container? | 19:55 |
| sdake_ | i saw that originally in this patch set | 19:55 |
| *** dgonzalez has joined #openstack-kolla | 19:55 | |
| sdake_ | ccesario__ i get you have done alot of work to add all these env vars | 19:55 |
| sdake_ | but i think there are simpelr ways to achieve same result | 19:56 |
| *** Marga_ has joined #openstack-kolla | 19:57 | |
| ccesario__ | sdake_: because the original patch adding -h ... causes a crash in build... and some comments in lauchpad bug point to usage PS1 var | 19:58 |
| sdake_ | which launchpad comments | 19:59 |
| sdake_ | got a link to the build crash? | 19:59 |
| ccesario__ | sdake_: https://bugs.launchpad.net/kolla/+bug/1557454 | 19:59 |
| openstack | Launchpad bug 1557454 in kolla "container name in PS1 shell command prompt inside a container" [Wishlist,In progress] - Assigned to Carlos Cesario (ccesario) | 19:59 |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Fix rabbitmq crash in centos gate https://review.openstack.org/303686 | 19:59 |
| ccesario__ | sdake_: http://paste.openstack.org/show/492760/ | 20:00 |
| *** Marga_ has quit IRC | 20:01 | |
| sdake_ | ccesario__ whichversion of docker-py were you using? | 20:02 |
| sbezverk | sdake vhosakot I did not see rabbitmq crash, the issue I saw was different. Rabbitmq folks helped to identify it and suggested a fixed version. | 20:02 |
| sdake_ | what problem did you see sbezverk | 20:03 |
| sdake_ | how about fiiing a bug | 20:03 |
| sdake_ | :) | 20:03 |
| sbezverk | I saw this issue on liberty deployed with inc0 patch | 20:03 |
| sbezverk | sdake why file a bug if there is a fix already ;-)? | 20:04 |
| ccesario__ | sdake_: Name: docker-py Version: 1.7.2 | 20:04 |
| ccesario__ | its support -H | 20:04 |
| ccesario__ | -h | 20:04 |
| sdake_ | ok well this looks like a docker-py bug or a kolla-docker implemtnation bug | 20:04 |
| vhosakot | sbezverk: cool, thanks for confirming | 20:05 |
| ccesario__ | using a simple code using docker-py I get success using -h param | 20:05 |
| sbezverk | sdake here is the log I provided to rabbitmq folks http://paste.openstack.org/show/493522/ | 20:05 |
| sdake_ | ccesario__ Ithink what you need to do is look at the docker engine log where it crashed in your previous -h attempt | 20:07 |
| sdake_ | and see what it says | 20:07 |
| ccesario__ | sdake_: http://paste.openstack.org/show/493569/ | 20:07 |
| ccesario__ | it works pefectly | 20:07 |
| vhosakot | ccesario__: sbezverk recently updated kolla_docker.py to set --ipc option to "host".... can you refer that as an example to implement hostname param in kolla_docker.py.... here is sbezverk's review https://review.openstack.org/#/c/293400/ | 20:07 |
| patchbot | vhosakot: patch 293400 - kolla - Set --ipc option to "host" for kolla_docker (MERGED) | 20:07 |
| sdake_ | ok so the error is not in docker-py | 20:09 |
| *** Marga_ has joined #openstack-kolla | 20:09 | |
| ccesario__ | vhosakot: yes, I saw it ...and I did the code look like that | 20:10 |
| sbezverk | guys can you share the error please? | 20:10 |
| vhosakot | sbezverk: ccesario__ is implementing a new param called hostname the exact same way you have successfuly done for --ipc "host"... and seeing some errors.. scroll aabove | 20:11 |
| ccesario__ | vhosakot: sdake_ give me a bit time to setup the environment using my first patch ... | 20:11 |
| sbezverk | vhosakot, maybe a trivial question but does docker supports it? for ipc host we had to confirm that it is actually supported.. | 20:12 |
| sdake_ | ccesario__ was your first patch just unreliable or did it fail all the time? | 20:12 |
| vhosakot | ccesario__: sure... could you refer sbezverk's review in https://review.openstack.org/#/c/293400/ and implement hostname | 20:12 |
| patchbot | vhosakot: patch 293400 | 20:12 |
| vhosakot | sbezverk: yes, docker supports it and I have tested it | 20:12 |
| ccesario__ | sdake_: yes | 20:13 |
| sdake_ | yes what there were two choices :) | 20:13 |
| sdake_ | ccesario where is your first hostname patch | 20:14 |
| ccesario__ | sdake_: the two choices... hehehe | 20:15 |
| vhosakot | ccesario__: instead of --ipc, you'd need to implement --hsotname..... both are strings and both are accepted by "docker run..." | 20:15 |
| vhosakot | https://docs.docker.com/engine/reference/commandline/run/ | 20:15 |
| ccesario__ | vhosakot: the docker --ipc it is supported by build_host_config function | 20:16 |
| ccesario__ | the --hostname nned to be used on build_container_options | 20:18 |
| *** shakamunyi has joined #openstack-kolla | 20:18 | |
| ccesario__ | sdake_: https://review.openstack.org/gitweb?p=openstack/kolla.git;a=commitdiff;h=1f6840f3cc7b6560e3f456f803c6929439225fc8 | 20:18 |
| ccesario__ | so simple | 20:19 |
| vhosakot | ccesario__: does this work ? | 20:19 |
| ccesario__ | vhosakot: http://paste.openstack.org/show/492760/ | 20:20 |
| ccesario__ | :P | 20:20 |
| vhosakot | yep, I saw that mariadb error you sent earlier... | 20:21 |
| vhosakot | ccesario__: I will dig the code and let you know what should be updated in kolla_docker.py for --hostname | 20:22 |
| vhosakot | ccesario__: is your /run shared ? | 20:22 |
| sbezverk | ccesario__ are you sure it is your change related? I saw similar error when I had an issue with rights on my ubuntu test bed | 20:22 |
| sdake_ | i suspect hostname has to go after name in the ansible call | 20:23 |
| *** shakamunyi has quit IRC | 20:23 | |
| sdake_ | the hostname is not part of the call | 20:23 |
| sdake_ | and there is a wird ,) | 20:23 |
| sdake_ | so something is not formatting the options correclty | 20:23 |
| sbezverk | has anybody else repoduced the same issue in different env? | 20:23 |
| ccesario__ | vhosakot: yes /run is shared | 20:24 |
| sbezverk | based on my past troubleshooting experience if the issue looks too wierd then you are looking at the wrong place ;-) | 20:24 |
| ccesario__ | sbezverk: yes.... withou it no errors | 20:24 |
| sbezverk | but my question was if somebody else repro the same issue with your change | 20:24 |
| sbezverk | ? | 20:25 |
| sdake_ | ccesario__ where is the gerrit patch if you would like me to debug this hostname thing | 20:25 |
| vhosakot | sbezverk: the hostname value is coming in kolla_docker.py correctly but not used in kolla_docker.py properly... | 20:25 |
| sdake_ | right | 20:25 |
| ccesario__ | sbezverk: I don't know... nobody reported | 20:25 |
| sdake_ | vhosakot thats right non | 20:25 |
| sbezverk | ccesario__ ok let me try to bring up my centos TB and see if I hit the same issue | 20:26 |
| ccesario__ | sdake_: https://review.openstack.org/#/c/297984/3 | 20:26 |
| patchbot | ccesario__: patch 297984 - kolla - Identify container through PS1 shell variable | 20:26 |
| ccesario__ | sbezverk: thanks | 20:26 |
| *** sdake has joined #openstack-kolla | 20:28 | |
| *** cmceniry has joined #openstack-kolla | 20:29 | |
| vhosakot | sdake_: recheck did not see rabbitmq crash and nova VM reach active state in centos gate | 20:30 |
| sdake | keep at it rechecking plz | 20:30 |
| vhosakot | yep | 20:30 |
| vhosakot | aa6b3771-a2b6-4b46-adbe-6b5f1c399ba6 | kolla_boot_test | ACTIVE | - | Running | demo-net=10.0.0.3 | | 20:30 |
| *** sdake_ has quit IRC | 20:30 | |
| sdake | ccesario i've pulled your patch looking into it now | 20:32 |
| sbezverk | sdake which version of PS you took 3 or 4? | 20:33 |
| sdake | 3 | 20:33 |
| sbezverk | so 4 is no good? | 20:33 |
| vhosakot | yep, 1,2,3 have docker.py chnages... 4 has the second approach... adding env var in dockerfile | 20:35 |
| sdake | i am not a fan of approach #4 | 20:36 |
| sbezverk | ccesario__ probably it is not matter but you do not have default: None | 20:38 |
| sbezverk | do you mind to add it and try? | 20:38 |
| sdake | TASK: [haproxy | Starting haproxy container] ********************************** | 20:39 |
| sdake | failed: [localhost] => {"changed": true, "failed": true} | 20:39 |
| sdake | msg: APIError(HTTPError(u'409 Client Error: Conflict for url: http+docker://localunixsocket/v1.22/containers/create?name=haproxy',),) | 20:39 |
| sdake | http://paste.fedoraproject.org/352970/46023436/ -> http://paste.fedoraproject.org/352970/46023436 | 20:39 |
| sdake | time to learn how kolla_docker.py works ;) | 20:39 |
| sbezverk | sdake since you already have it up, could you please add Default: None | 20:40 |
| sdake | http://paste.fedoraproject.org/352971/34453146/ -> http://paste.fedoraproject.org/352971/34453146 | 20:41 |
| sdake | that is the command as seen in the docker engine | 20:41 |
| sbezverk | http://paste.fedoraproject.org/352972/23451514/ | 20:42 |
| sdake | lol sb | 20:43 |
| sbezverk | sdake in that command you pasted, both hostname and name are specified and they are identical, wondering if it might create a conflict | 20:45 |
| *** salv-orl_ has joined #openstack-kolla | 20:45 | |
| vhosakot | sdake: we need to make the gate show red FAILED when nova VM does not go to ACTIVE state | 20:45 |
| vhosakot | sdake: now, it shows green SUCCESS even when nova VM does not go to ACTIVE state, and it is misleasing (false positive).. I will try to fix it | 20:46 |
| sdake | sbezverk i dont see why | 20:46 |
| sdake | but i'll try a different hostname | 20:47 |
| sdake | sbezverk thats not it | 20:48 |
| sdake | let me try your default idea | 20:48 |
| *** salv-orlando has quit IRC | 20:48 | |
| sdake | bulid_host_config lacking hostname | 20:50 |
| sbezverk | yep, I am looking at the same spot | 20:52 |
| sbezverk | but build_container_options has it | 20:52 |
| *** jasonsb has joined #openstack-kolla | 21:00 | |
| sdake | Apr 09 17:06:24 minime-03 docker[1250]: time="2016-04-09T17:06:24.805347868-04:00" level=error msg="Handler for POST /v1.22/containers/create returned error: Conflicting options: hostname and the network mode" | 21:10 |
| sdake | yup | 21:10 |
| sdake | thats the prolem | 21:10 |
| sdake | first off, hostname is a magic ansible variable | 21:10 |
| sdake | so it has to be called soething else | 21:10 |
| sdake | in this place i used hostnamemagic | 21:10 |
| sdake | apparently you cant specify hostname and network mode at the same time | 21:10 |
| sdake | https://github.com/docker/compose/issues/2460 | 21:11 |
| sdake | ccesario ^^ | 21:11 |
| sbezverk | sdake what if we generate hostname file and insert it to /etc/ during container baking? | 21:12 |
| sbezverk | I am positive it will work on centos/rhel , not sure about ubuntu though | 21:12 |
| ccesario__ | hey guys... I had t solve some problem here | 21:13 |
| sdake | ccesario i root caused why your patch no beuno | 21:14 |
| ccesario__ | sdake: I saw that you got the same error | 21:14 |
| sdake | yes but you never said its because we use --net=host ;) | 21:14 |
| ccesario__ | I did not now about this conflict :P | 21:16 |
| ccesario__ | I only read abut docker-p documentation | 21:16 |
| sbezverk | sdake how about /etc/hostname idea? | 21:16 |
| sdake | [sdake@minime-03 tools]$ docker run -it --net=host --hostname=bust ubuntu | 21:16 |
| sdake | docker: Error response from daemon: Conflicting options: hostname and the network mode. | 21:16 |
| sdake | See 'docker run --help'. | 21:16 |
| sdake | it appears docker-py doesne't know about this conflict either ;) | 21:17 |
| ccesario__ | docker-py api does not mean it | 21:17 |
| vhosakot | sdake: 3rd recheck saw no rabbitmq crash and nova VM reached ACTIVE state on centos in gate.... | 21:17 |
| vhosakot | 624c2c44-a9e7-4d51-aef8-ff6a4d14c3a6 | kolla_boot_test | ACTIVE | - | Running | demo-net=10.0.0.3 | | 21:17 |
| ccesario__ | yes.... sorry by missing this info | 21:18 |
| vhosakot | sdake: doing 4th recheck now... | 21:18 |
| *** shakamunyi has joined #openstack-kolla | 21:18 | |
| vhosakot | sdake: https://bugs.launchpad.net/kolla/+bug/1568389 | 21:20 |
| openstack | Launchpad bug 1568389 in kolla "Gate must show FAILURE when nova VM does not reach ACTIVE state" [Undecided,New] - Assigned to Vikram Hosakote (vhosakot) | 21:20 |
| ccesario__ | sdake: based in this info.. what is the best way to do solve it into kolla ? | 21:21 |
| ccesario__ | what is your suggestion ? | 21:21 |
| sdake | i dont know but it doesn't involve a kolla-docker.py chane | 21:22 |
| sdake | instead of hardcoding the envrionment in the docker fiels why not put in ansible environment? | 21:22 |
| *** shakamunyi has quit IRC | 21:23 | |
| sbezverk | ccesario__ sdake we can create a file in /etc called hostname and insert there any name which will become container's hostname | 21:23 |
| sdake | sbezverk i dont have issue with ccesario__ 's bashrc chaneg | 21:23 |
| sdake | its the modifying 20 million docker files | 21:24 |
| ccesario__ | sdake, sbezverk : I have any idea | 21:24 |
| sdake | ccesario__ set the PS1 environment variable to the container name | 21:25 |
| ccesario__ | I can usage the PS1 and make it dynamic based iin container name | 21:25 |
| sdake | when docker runs it | 21:25 |
| ccesario__ | yep | 21:25 |
| sdake | let me show you e xampel | 21:26 |
| ccesario__ | sdake: I have it done | 21:26 |
| ccesario__ | let me show you | 21:26 |
| ccesario__ | sdake: http://pastebin.com/wRzBk3e3 | 21:29 |
| ccesario__ | it works | 21:29 |
| ccesario__ | it inject the var when the container run/start | 21:30 |
| sdake | on line 11 where is KOLLA_SERVICE_NAME come from? | 21:30 |
| sdake | is/does | 21:30 |
| sdake | http://paste.fedoraproject.org/352991/23757414/ -> http://paste.fedoraproject.org/352991/23757414 | 21:33 |
| ccesario__ | self.params.get('name').replace('_','-') https://github.com/openstack/kolla/blob/master/ansible/library/kolla_docker.py#L491 | 21:33 |
| ccesario__ | the param passed on container build | 21:33 |
| sdake | try the paste abovve | 21:34 |
| sdake | see if it works | 21:34 |
| sdake | that is a low pain way to do the job | 21:34 |
| ccesario__ | let me test your suggestion | 21:36 |
| *** daneyon has joined #openstack-kolla | 21:38 | |
| sdake | how do you get in a container as bash? | 21:43 |
| *** daneyon has quit IRC | 21:43 | |
| sdake | docker exec containername bash returns immediately for me | 21:43 |
| sdake | [sdake@minime-03 tools]$ docker exec -it haproxy bash | 21:44 |
| sdake | steak_for_dinner_haproxy | 21:44 |
| sdake | http://paste.fedoraproject.org/352993/14602382/ -> http://paste.fedoraproject.org/352993/14602382 | 21:44 |
| sdake | wfm | 21:44 |
| sdake | if you want to have more control over what goes in PS1, just encode the cointainer name in the environment in ansible, not in the docker files | 21:45 |
| sdake | ccesario__ enjoy ;) | 21:45 |
| ccesario__ | sdake: hmmmm let me test it on my ubuntu | 21:47 |
| sbezverk | sdake hav you seen this on the latest master with #3 of ccesario__ http://paste.openstack.org/show/493575/ | 21:49 |
| *** cmceniry has quit IRC | 21:53 | |
| sdake | no | 21:54 |
| sdake | let me go on | 21:55 |
| sdake | like i blister in the sun | 21:55 |
| sbezverk | retrying to see if it reproducible or not | 21:55 |
| ccesario__ | sdake: on ubuntu the PS1 env is overwrited by bashrc file | 21:55 |
| sdake | big hands i knwo our the one | 21:55 |
| sdake | ok well overwrit the bashrc file | 21:55 |
| sdake | and passs in an environment variable "CONTAINER_NAME" | 21:55 |
| *** jtriley has quit IRC | 21:56 | |
| sdake | and set ps1 to that environment variable | 21:56 |
| sdake | again my issueisn' with th ebashrc but with the modifying 90 docker files | 21:56 |
| ccesario__ | sdake: yes yes | 21:56 |
| sdake | it would be nice if container name included the bare metal hostname in it too ;) | 21:57 |
| *** jtriley has joined #openstack-kolla | 21:57 | |
| sdake | as in [192.168.1.103:haproxy ~]: | 21:57 |
| sdake | or whatever | 21:57 |
| ccesario__ | can I do it on base image file ? | 21:58 |
| ccesario__ | http://pastebin.com/LHdTuLRf | 21:58 |
| sdake | that looks solid | 21:59 |
| sdake | one file to change | 21:59 |
| *** vhosakot has quit IRC | 21:59 | |
| sdake | dont ask permission btw ;) | 21:59 |
| *** vhosakot has joined #openstack-kolla | 21:59 | |
| * sdake will break people of that habit eventually ;) | 21:59 | |
| sdake | just do what you think is right, the reviewers will sort it out if its wrong | 22:00 |
| vhosakot | sdake: 4th recheck saw no rabbitmq crash and nova VM reached ACTIVE state in gate | 22:01 |
| vhosakot | 19f99786-fc45-4d79-9dd8-ec1708f2f2ac | kolla_boot_test | ACTIVE | - | Running | demo-net=10.0.0.3 | | 22:01 |
| vhosakot | sdake: doing 5th recheck | 22:01 |
| *** sdake_ has joined #openstack-kolla | 22:01 | |
| vhosakot | sdake: I mean, nova VM on centos gate reach active state | 22:01 |
| *** jtriley has quit IRC | 22:02 | |
| sdake_ | binary gat working well now? | 22:03 |
| ccesario__ | sdake: thanks a lot by suggestions and directions | 22:03 |
| sdake_ | ccesario__ your welcome ;) | 22:04 |
| ccesario__ | I'll provide a new patch set | 22:04 |
| ccesario__ | vhosakot: sbezverk thanks by help too!!! | 22:04 |
| *** sdake has quit IRC | 22:04 | |
| vhosakot | ccesario__: haha, i did not help... but, I like your PS a lot.. very useful when we see hostname inside container! | 22:04 |
| vhosakot | sdake_: centos binary gate does not see rabbit crash and no keystone json error, but nova vm does not reach active state... it looks like a new issue, and we'll a new bug to track it (nova VM not reaching ACTIVE state in centos binary gate).... looks like a nova-api issue in centos binary gate... search for Traceback in http://logs.openstack.org/86/303686/6/check/gate-kolla-dsvm-deploy-centos-binary/a0ac497/logs/kolla/no | 22:07 |
| vhosakot | va/nova-api.log | 22:07 |
| vhosakot | sdake_: looks like a nova-api issue in centos binary gate... search for Traceback in http://logs.openstack.org/86/303686/6/check/gate-kolla-dsvm-deploy-centos-binary/a0ac497/logs/kolla/nova/nova-api.log | 22:07 |
| vhosakot | sdake_: could you ack https://review.openstack.org/#/c/302026/..... updated prechecks and addresses Jeffrey's comments | 22:10 |
| patchbot | vhosakot: patch 302026 - kolla - Add prechecks to validate kolla_internal_vip_addre... | 22:10 |
| vhosakot | bbiab 20 mins | 22:10 |
| sbezverk | sdake if you check docker exec heka ls -al /var/log/kolla/rabbitmq/rabbit.log on any test bed you will see that log has size 0. | 22:15 |
| sbezverk | to address it newer version of erlang is required | 22:16 |
| sbezverk | basically we do not collect any rabbitmq logs | 22:16 |
| *** cmceniry has joined #openstack-kolla | 22:17 | |
| *** shakamunyi has joined #openstack-kolla | 22:18 | |
| sbezverk | for tracking purposes https://bugs.launchpad.net/kolla/+bug/1568408 | 22:22 |
| openstack | Launchpad bug 1568408 in kolla "rabbitmq logs are not collected" [Undecided,New] | 22:22 |
| *** shakamunyi has quit IRC | 22:23 | |
| *** vhosakot has quit IRC | 22:25 | |
| *** cmceniry has quit IRC | 22:30 | |
| *** cmceniry has joined #openstack-kolla | 22:32 | |
| *** vhosakot has joined #openstack-kolla | 22:32 | |
| ccesario__ | inside ansible/roles/haproxy/tasks/start.yml is there any var that return vale of property "name" in this case name: "haproxy" | 22:33 |
| *** vhosakot_ has joined #openstack-kolla | 22:36 | |
| *** vhosakot has quit IRC | 22:38 | |
| *** banix has joined #openstack-kolla | 22:55 | |
| *** Marga_ is now known as marga_ | 23:14 | |
| *** marga_ is now known as fragatina | 23:15 | |
| *** fragatina has quit IRC | 23:16 | |
| *** shakamunyi has joined #openstack-kolla | 23:18 | |
| *** allen_gao has quit IRC | 23:18 | |
| *** allen_gao has joined #openstack-kolla | 23:19 | |
| *** fragatina has joined #openstack-kolla | 23:21 | |
| *** shakamunyi has quit IRC | 23:23 | |
| *** fragatina has quit IRC | 23:24 | |
| *** sdake has joined #openstack-kolla | 23:27 | |
| *** sdake_ has quit IRC | 23:29 | |
| *** ccesario__ has quit IRC | 23:35 | |
| *** fragatina has joined #openstack-kolla | 23:36 | |
| *** sdake has quit IRC | 23:36 | |
| *** ccesario_ has joined #openstack-kolla | 23:37 | |
| *** fragatina has quit IRC | 23:41 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!