Sunday, 2016-04-10

« Saturday, 2016-04-09 Index Monday, 2016-04-11 »
*** Jeffrey4l_ has joined #openstack-kolla00:17
*** shakamunyi has joined #openstack-kolla00:18
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix the neutron reconfigure issue when enable linuxbridge-agent  https://review.openstack.org/30378300:20
*** shakamunyi has quit IRC00:23
*** daneyon has joined #openstack-kolla00:40
*** jtriley has joined #openstack-kolla00:52
*** daneyon has quit IRC00:54
*** ssurana has joined #openstack-kolla01:16
*** shakamunyi has joined #openstack-kolla01:18
*** stvnoyes has quit IRC01:23
*** shakamunyi has quit IRC01:23
*** stvnoyes has joined #openstack-kolla01:24
*** daneyon has joined #openstack-kolla01:45
*** ssurana has quit IRC01:48
*** daneyon has quit IRC01:50
*** ssurana has joined #openstack-kolla01:50
*** ssurana has quit IRC01:55
*** banix has quit IRC02:00
*** pbourke has quit IRC02:16
*** pbourke has joined #openstack-kolla02:16
*** shakamunyi has joined #openstack-kolla02:18
*** shakamunyi has quit IRC02:23
*** salv-orlando has joined #openstack-kolla02:45
*** klint has joined #openstack-kolla02:46
*** salv-orl_ has quit IRC02:48
*** vhosakot_ has quit IRC02:55
*** vhosakot has joined #openstack-kolla02:56
*** banix has joined #openstack-kolla02:57
*** weiyu has joined #openstack-kolla03:00
*** ccesario___ has joined #openstack-kolla03:02
*** shakamunyi has joined #openstack-kolla03:18
*** shakamunyi has quit IRC03:23
openstackgerritMerged openstack/kolla: Add quotes for RBMQ nodes definition  https://review.openstack.org/28841503:25
openstackgerritVikram Hosakote proposed openstack/kolla: make rabbitmq container build with debian  https://review.openstack.org/29772803:32
*** ccesario___ has quit IRC03:44
*** ccesario___ has joined #openstack-kolla03:45
*** banix has quit IRC03:49
openstackgerritHui Kang proposed openstack/kolla: Add etcd Docker container  https://review.openstack.org/29845103:54
openstackgerritVikram Hosakote proposed openstack/kolla: Make kibana container build with debian  https://review.openstack.org/29720603:56
*** weiyu has quit IRC04:01
*** cmceniry has quit IRC04:02
*** daneyon has joined #openstack-kolla04:03
*** cmceniry has joined #openstack-kolla04:18
*** shakamunyi has joined #openstack-kolla04:18
*** sdake has joined #openstack-kolla04:22
*** shakamunyi has quit IRC04:23
*** sdake has quit IRC04:24
openstackgerritHui Kang proposed openstack/kolla: Fix missing libvirt python module in ceilometer-compute  https://review.openstack.org/30065004:31
*** achanda has joined #openstack-kolla04:39
openstackgerritMerged openstack/kolla: Put the variables in alphabetical order in all.yml  https://review.openstack.org/30340904:42
*** achanda has quit IRC04:42
*** achanda has joined #openstack-kolla04:45
openstackgerritMauricio Lima proposed openstack/kolla: Put the variables in alphabetical order in all.yml  https://review.openstack.org/30379604:46
*** achanda has quit IRC04:47
*** achanda has joined #openstack-kolla04:52
*** coolsvap has joined #openstack-kolla05:00
openstackgerritMerged openstack/kolla: Spec: Deploy Kolla images using Kubernetes  https://review.openstack.org/25545005:05
openstackgerritVikram Hosakote proposed openstack/kolla: add unit tests for build script  https://review.openstack.org/29042305:09
*** shakamunyi has joined #openstack-kolla05:18
*** shakamunyi has quit IRC05:23
openstackgerritVikram Hosakote proposed openstack/kolla: Add consul Docker container  https://review.openstack.org/29800105:23
*** cmceniry has quit IRC05:25
*** vhosakot has quit IRC05:42
*** achanda has quit IRC05:49
*** achanda has joined #openstack-kolla05:50
*** achanda has quit IRC05:54
*** achanda has joined #openstack-kolla05:56
*** achanda has quit IRC06:00
*** shakamunyi has joined #openstack-kolla06:18
elemoine_sbezverk: I guess the problem with the RabbitMQ logs was introduced recently, with a RabbitMQ update or something. I also observed it. Have you created a bug for this?06:18
*** shakamunyi has quit IRC06:23
*** achanda has joined #openstack-kolla06:23
*** achanda has quit IRC06:26
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make container dind build with debian  https://review.openstack.org/29743407:03
*** subscope has joined #openstack-kolla07:15
*** brad[] has quit IRC07:15
*** brad[] has joined #openstack-kolla07:16
*** cmceniry has joined #openstack-kolla07:22
*** coolsvap has quit IRC07:24
*** cmceniry has quit IRC07:28
*** sdake has joined #openstack-kolla07:29
*** sdake_ has joined #openstack-kolla07:32
*** fragatina has joined #openstack-kolla07:33
*** sdake has quit IRC07:34
*** sdake_ has quit IRC07:34
*** fragatina has quit IRC07:37
*** achanda has joined #openstack-kolla07:41
*** jmccarthy has quit IRC07:43
*** jmccarthy has joined #openstack-kolla07:44
*** fragatina has joined #openstack-kolla07:46
*** subscope has quit IRC07:59
*** subscope has joined #openstack-kolla08:00
*** fragatina has quit IRC08:01
*** achanda has quit IRC08:02
*** daneyon has quit IRC08:09
*** daneyon has joined #openstack-kolla08:20
*** daneyon has quit IRC08:22
*** cmceniry has joined #openstack-kolla08:24
*** cmceniry has quit IRC08:29
*** salv-orl_ has joined #openstack-kolla08:44
*** salv-orlando has quit IRC08:47
*** daneyon has joined #openstack-kolla08:59
*** fragatina has joined #openstack-kolla09:02
*** daneyon has quit IRC09:04
*** fragatina has quit IRC09:07
*** subscope has quit IRC09:41
*** subscope has joined #openstack-kolla09:42
*** subscope has quit IRC09:50
*** ssurana has joined #openstack-kolla09:50
*** subscope has joined #openstack-kolla09:53
*** cmceniry has joined #openstack-kolla10:26
*** cmceniry has quit IRC10:30
*** ssurana has quit IRC10:42
*** achanda has joined #openstack-kolla10:57
*** achanda has quit IRC11:03
*** fragatina has joined #openstack-kolla11:04
*** fragatina has quit IRC11:08
*** achanda has joined #openstack-kolla11:19
*** achanda has quit IRC11:20
*** jtriley has quit IRC11:23
*** daneyon has joined #openstack-kolla11:42
*** daneyon has quit IRC11:46
sbezverkelemoine_ elemoine_11:48
sbezverkelemoine_ https://bugs.launchpad.net/kolla/+bug/156840811:48
openstackLaunchpad bug 1568408 in kolla "rabbitmq logs are not collected" [Undecided,New]11:48
*** fragatina has joined #openstack-kolla12:05
*** fragatina has quit IRC12:10
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Add  iscsid/tgtd specific parameters to Kolla infrastructure  https://review.openstack.org/30000412:19
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Adds iscsid and tgtd docker containers  https://review.openstack.org/29128512:19
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Adds ansible code for iscsid and tgtd containers  https://review.openstack.org/29999512:19
*** cmceniry has joined #openstack-kolla12:27
*** cmceniry has quit IRC12:32
*** banix has joined #openstack-kolla12:35
*** subscope has quit IRC12:40
*** banix has quit IRC12:51
*** klint has quit IRC13:00
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Add  iscsid/tgtd specific parameters to Kolla infrastructure  https://review.openstack.org/30000413:19
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Adds iscsid and tgtd docker containers  https://review.openstack.org/29128513:19
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Adds ansible code for iscsid and tgtd containers  https://review.openstack.org/29999513:19
*** achanda has joined #openstack-kolla13:20
*** achanda has quit IRC13:21
*** sdake has joined #openstack-kolla13:23
sdakemorning13:34
sbezverkgood morning13:35
*** jtriley has joined #openstack-kolla13:53
*** sdake has quit IRC14:02
*** sdake has joined #openstack-kolla14:04
sdakehey sbezverk14:05
*** fragatina has joined #openstack-kolla14:06
*** rhallisey has joined #openstack-kolla14:06
sbezverksdake what can we do about adding new repo for erlang to test rabbitmq fix?14:09
*** jtriley has quit IRC14:10
*** fragatina has quit IRC14:11
sbezverkI tried all sort of things, this erlang rpm has 30-40 dependencies sitting in the same repo, but yum does not search for these rpms in that repo..14:12
sdakecouple options14:18
sdakeyou can versionlock the erlang deps14:18
sdakewhere is the repo file14:18
sdakelet me have a look around14:18
sdakeother  thing is you can set priority on that repo - it will lock out other repos from being pulled in when there are conflicts14:19
sdakepersonally I'd go for versionlock although its more onerous14:19
sdaketo use it, type yum versionlock reponame14:19
sbezverksdake it is not even considering it14:19
sdakesorry14:19
sdakeyum versionlock depname14:19
sdakeyou hae a patch for me to look at14:19
sdakenote i am tring to take care of some housekeeping today14:20
sdakemy grass has grown long in the last month14:20
sdakefiguratively14:20
sbezverkhttps://jeckersb.fedorapeople.org/erlang-R16B-03.17.el7/erlang-R16B-03.17.el7.repo14:20
sbezverksame here14:21
sdakenot keen on adding someone's personal repository to kolla14:22
sdakeunless it is one of the core reviewers14:22
sdakethis issue you hve, is it 100% reproducible?14:22
sbezverkyep, other folks elemoine_ for example saw it too14:24
sdake100% reproduciblee?14:24
sdakeor intermittent?14:24
*** daneyon has joined #openstack-kolla14:24
sdakeso if I spend the time to pull this repo in, you can verify it indeed solves the problem is what i'm getting at14:24
sdakejust a heads up we cna't merge this as is14:25
sdakeI know jeckersb, he has a good reputation14:25
sdakebut still we cna't just add someone's random repo that isn't atleast hosted on coprs or somewhere other then a fedorapeople page14:25
sdakeI will need the src.rpm for this package repo set14:25
sbezverksdake I understand, all I want is PoC14:26
sdakeok well lets get your poc working14:26
sdakeplease, git dif or git review whatever poc work you have in progresss so I don't have to waste time getting  there14:26
sdakethe issue, intermittent or 100%?14:27
*** cmceniry has joined #openstack-kolla14:29
*** daneyon has quit IRC14:29
sbezverksdake 100%, it might be a day one issue as people not often check rabbitmg.log14:29
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH  https://review.openstack.org/29490614:29
elemoine_I guess the problem is related a recent update of Rabbit14:29
sdakeok sbezverk post your dif of the repo14:33
sdakeor do ou not have one14:33
sdakewe could and should be versionlocking rabbitmq14:33
sdakebrb connecting to personal lab14:34
*** sdake_ has joined #openstack-kolla14:36
sdake_rabbitmq-server can't be versionlocked on centos - it comes from the delorean repo14:36
sdake_sbezverk yuor platform is ubuntu or centos?14:36
*** sdake has quit IRC14:38
sbezverksdake centos14:40
elemoine_sdake it was an voluntary update I think (I don't have the cr handy right now)14:41
sbezverksdake http://paste.openstack.org/show/493583/ I modified config.py to add erlang repo14:43
*** salv-orlando has joined #openstack-kolla14:45
sbezverkand then in rabbitmq's Docker file I tried to install new version of erlang rpm before main rabbitmq binaries are installed.14:45
*** cmceniry has quit IRC14:46
openstackgerritSteven Dake proposed openstack/kolla: [DO NOT MERGE] Test for sbezverk  https://review.openstack.org/30383714:47
sdake_enjoy14:47
sdake_note we can't merge it like that14:47
sdake_if you want it merged, I need the src.rpm14:48
*** salv-orl_ has quit IRC14:48
openstackgerritSteven Dake proposed openstack/kolla: [DO NOT MERGE] Test for sbezverk  https://review.openstack.org/30383714:50
sdake_sbezverk file a bug please if you have a 100% failure so it can be triaged14:50
sdake_or even if its an intermittent failure14:51
sbezverksdake_ already done14:51
sbezverkhttps://bugs.launchpad.net/kolla/+bug/156840814:52
openstackLaunchpad bug 1568408 in kolla "rabbitmq logs are not collected" [Undecided,New]14:52
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH  https://review.openstack.org/29490614:53
sdake_groan 12 new bugs14:54
*** ccesario___ has quit IRC14:59
sdake_hey Jeffrey4l_15:02
Jeffrey4l_sdake_, morning15:02
sdake_Jeffrey4l_ note vhosakot fixed the gate by removing EPMD_URL or whatever that variable is15:04
sdake_it looks buggy in general on all distros15:04
sdake_but it only applies on centos distros  so we dont see same failure in ubuntu15:05
sdake_ipv6, ipv4 and bugs in erlang are all mixed up in this mess15:05
Jeffrey4l_sdake_, noticed that. But I still curiosity why.15:05
Jeffrey4l_yea.15:05
sdake_sbezverk when you have test results let me know so I can get it fixed in the upstream repos15:06
sdake_jeckersb needs karma on his change15:06
Jeffrey4l_do not see error log. can not re-produce this bug locally. I do not let this. :(15:06
*** jtriley has joined #openstack-kolla15:06
Jeffrey4l_sdake_, so sbezverk know the root cause? ( sorry, i am not familiar with elang/rabbitmq)15:07
sdake_Jeffrey4l_ different bug sbezverk is sorting out15:07
*** fragatina has joined #openstack-kolla15:07
sdake_rabbitmq is not generating log output15:08
Jeffrey4l_OK. cool15:08
sdake_because its log handler crashes15:08
sdake_itss an upstream bug not a kolla problem15:08
Jeffrey4l_yes. I just saw the rabbitmq in `docker logs rabbitmq`15:08
Jeffrey4l_roger15:08
sdake_the EPMD is an upsstream bug too15:08
Jeffrey4l_sdake_, so you know the bug number? where is it?15:09
sdake_which could possibly be fixed by this 17 release\15:09
sbezverkJeffrey4l_ it is not that simple with rabbitmq, I see some other collateral, example otjher services are timing out because of rabbitmq issue15:09
Jeffrey4l_sbezverk, are you saying the rabbitmq log issue? or epmd issue?15:11
sbezverkJeffrey4l_ log issue15:11
Jeffrey4l_OK15:12
*** fragatina has quit IRC15:12
*** jtriley has quit IRC15:12
sdake_sbezverk if you like that log change please ack it here by loggign in and giving it karma (voting) https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-e1035fad9015:16
sbezverksdake_ same approach you took in your PS I tried yesterday with the same result: see this paste:http://paste.openstack.org/show/493584/15:21
sdake_dont try to install erlang15:22
sdake_jut try to install rabbitmq-server15:22
sdake_your doing the install wrong ;)15:22
sdake_trust the tools they will do the righ thing15:22
sdake_they are highly stable at this point15:22
sdake_minus docker of course ;)15:22
sbezverksdake_ hm, ok I will let it finish and check which version it will pick up15:23
sdake_fpaste is broken or i'd show you my build output15:24
sbezverksdake ok I believe you :-) but let me test it ;-)15:25
sdake_http://paste.fedoraproject.org/353295/3018141/ -> http://paste.fedoraproject.org/353295/0301814115:26
sbezverksdake_ well you talk about apple, I talk about orranges, paste you gave me is for local yum install which is wokring fine, I was talking about yum install during rabbitmq contianer, and there it still picks up old erlang..15:32
sbezverksdake_ http://paste.fedoraproject.org/353299/60302485/15:35
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH  https://review.openstack.org/29490615:36
*** salv-orlando has quit IRC15:37
*** salv-orlando has joined #openstack-kolla15:37
*** jtriley has joined #openstack-kolla15:38
sdake_ahh but i have delorean repos installed ;)15:38
sdake_sbezverk so whie its outside container is not apple/orange, its the same hting15:40
sdake_rather while its not  apple/orae its similar thing15:40
sdake_the problem with erlang not being installed is repository priority protections15:40
sdake_check the gate logs on that review15:40
sdake_sbezverk http://logs.openstack.org/37/303837/2/check/gate-kolla-dsvm-build-centos-binary/cff9f5c/console.html#_2016-04-10_14_57_33_21215:42
sdake_look its pulling .17-el715:42
*** cmceniry has joined #openstack-kolla15:43
sdake_ok cats15:44
sdake_we need an epmd core dump15:44
sdake_when the gate fails15:44
sdake_when epmd crashes15:44
sdake_Jeffrey4l_ can you figure out how to get me one15:44
sdake_and I will get it fixed15:44
sdake_epmd is written in C15:45
sdake_with debug info, the core dump  can be used to pinpoint the defective code15:45
sdake_i hae found coredumps are written outside the cntainer not inside it15:45
Jeffrey4l_sdake_, see here http://logs.openstack.org/06/294906/9/check/gate-kolla-dsvm-deploy-centos-binary/e983c42/logs/kolla/15:45
Jeffrey4l_My test PS collect the core dump file.15:46
Jeffrey4l_sdake_, moreover, i catch this log in syslog:15:46
Jeffrey4l_<3>2016-04-10T15:11:36Z node1 epmd[0]: epmd: failed to bind socket15:46
sdake_Jeffrey4l_ that is not a core dump i dont think15:46
sdake_thats a crash dump15:46
Jeffrey4l_sdake_, hmm, how to get the core dump ?15:47
Jeffrey4l_sorry do not know that.15:47
sdake_Jeffrey4l_ its a bit of a challenge to find the core dump15:47
*** cmceniry has quit IRC15:47
sdake_create a coredump of keepalived on your system and see where it gets put15:47
Jeffrey4l_sdake_, is this helpful? empd failed to start due to : epmd[0]: epmd: failed to bind socket15:47
sdake_you may hae to enable core dumps15:47
sdake_yes is tht 100% of the time or only on the crash?15:48
sdake_to work around that use ffff:ipv4address15:48
sdake_epmd either owrks with ipv6, or ipv4, but not both15:49
sdake_if you hae ipv4 network, you can address it via ffff:15:49
sdake_I believe15:49
openstackgerritHui Kang proposed openstack/kolla: Add etcd Docker container  https://review.openstack.org/29845115:49
sdake_the epmd cannot bind error is 100%?15:49
Jeffrey4l_should only happen on crash. it seem related to this issue https://bugs.launchpad.net/ubuntu/+source/erlang/+bug/137410915:50
openstackLaunchpad bug 1374109 in erlang (Ubuntu) "epmd does not support binding to an IPv4 address anymore" [High,Confirmed] - Assigned to Dariusz Gadomski (dgadomski)15:50
sdake_see comment #6 for workaroudn15:50
Jeffrey4l_sdake_, yes. But i can not re-produce this locally. :(15:51
sdake_here is the issue as i see it15:51
sdake_moment brain just had it15:51
sdake_the gate machines don't have ivp4 addresses assigned to their ipv6 addresses15:52
sdake_we turn of ipv615:52
sdake_is that a possibility?15:52
Jeffrey4l_i do not know the ipv6 stuff. why you say "the gate machines don't have ivp4 addresses assigned to their ipv6 addresses"15:53
*** cmceniry has joined #openstack-kolla15:53
Jeffrey4l_proof?15:53
openstackgerritHui Kang proposed openstack/kolla: Add Kuryr ansible role  https://review.openstack.org/29889415:54
Jeffrey4l_sdake_, see this, it the gate has ipv4 address http://logs.openstack.org/06/294906/9/check/gate-kolla-dsvm-deploy-centos-binary/e983c42/console.html#_2016-04-10_14_54_12_21715:54
sdake_ok throw my theory out15:55
sdake_but somehow its relted ;)15:55
sdake_did thatgate job fail later?15:55
Jeffrey4l_i do think it is relate to the ipv6 stuff, too.15:56
Jeffrey4l_yea, agreed.15:56
sdake_sbezverk is this the error you have: http://logs.openstack.org/06/294906/9/check/gate-kolla-dsvm-deploy-centos-binary/e983c42/console.html#_2016-04-10_15_12_45_46215:56
Jeffrey4l_which gate?15:56
sdake_Jeffrey4l_ nevermind i validted that atleast error_logger crashed15:57
sdake_which I thnk is EPMD15:57
Jeffrey4l_yes. epmd failed to start.15:57
sdake_wht ththe hell does epmd do15:58
Jeffrey4l_empd say: epmd[0]: epmd: failed to bind socket15:58
Jeffrey4l_sdake_, i also catch the syslog in that test. http://logs.openstack.org/06/294906/9/check/gate-kolla-dsvm-deploy-centos-binary/e983c42/logs/kolla/default.log15:58
sdake_i understand but what is its purpose in the system15:58
Jeffrey4l_you can get it from there.15:59
sbezverksdake no15:59
Jeffrey4l_sbezverk, about the epmd issue, I think it is relate to this: https://bugs.launchpad.net/ubuntu/+source/erlang/+bug/1374109.16:00
openstackLaunchpad bug 1374109 in erlang (Ubuntu) "epmd does not support binding to an IPv4 address anymore" [High,Confirmed] - Assigned to Dariusz Gadomski (dgadomski)16:00
sdake_Jeffrey4l_ can you capture dmesg16:00
Jeffrey4l_because we bind it to a ipv4 address.16:00
Jeffrey4l_when remove the bind, it works fine.16:01
sdake_Jeffrey4l_ capture the core file and dmesg plz :)16:01
Jeffrey4l_sdake_, ok16:01
sdake_where is the epmd message in teh logs16:01
sdake_-16 was supposed ot fix an ipv6 issue16:02
sdake_that is why we see this problem now16:02
Jeffrey4l_epmd write its own log to /dev/log16:02
sdake_recently epel introduced -16 version of erlang16:03
sdake_this made erlang have same problem as ubuntu package16:03
sdake_which is epmd can't bind to an ipv6 address16:03
Jeffrey4l_cool. I think you are right.16:03
sdake_rather ipv4 address16:03
openstackgerritHui Kang proposed openstack/kolla: Implement ceilometer ansible role  https://review.openstack.org/30057416:03
*** rhallisey has quit IRC16:03
sdake_why it works on some gates and not others is a mystery16:03
sdake_so lets use the workaround in 1616:04
sdake_rather in comment #616:04
sbezverksdake_ then this verion .17 should resolve both issues16:04
sdake_it looks solid16:04
sdake_sbezverk no it does not16:04
sdake_according to jeckersb it only fixes the log handler failure16:04
sdake_here is the root of the problelm16:05
sdake_epmd, a c program, cannot handle binding to either an ipv4 and ipv6 address in its configuration16:05
sdake_the workaround when epmd is compiled with ipv6 is to use ::ffff:ipv4address16:05
sdake_https://bugs.launchpad.net/ubuntu/+source/erlang/+bug/137410916:06
openstackLaunchpad bug 1374109 in erlang (Ubuntu) "epmd does not support binding to an IPv4 address anymore" [High,Confirmed] - Assigned to Dariusz Gadomski (dgadomski)16:06
sdake_coment #616:06
sbezverksdake as I was suspecting rabbitmq container is still using .16 version16:06
sdake_sbezverk look at the build logs dude16:06
Jeffrey4l_sdake_, question: why we need bind to empd to a specified ip address? In default, it bind to 127.0.0.1.16:06
Jeffrey4l_that should be works in cluster.16:06
sdake_Jeffrey4l_ ya tht should actuallybe fine - assuming there is a loopback address16:07
sdake_and someone hasn't done something stupid like put their cluster nme in the loopback ip address :)16:07
sbezverksdake_ I ran docker exec rabbitmq rpm -qa | grep erlang16:07
Jeffrey4l_so that's why vhosakete's PS fix the gate. https://review.openstack.org/30368616:07
Jeffrey4l_sdake_, what do you mean by "and someone hasn't done something stupid like put their cluster nme in the loopback ip address :)"?16:08
sdake_http://logs.openstack.org/37/303837/2/check/gate-kolla-dsvm-deploy-centos-binary/0a404a9/console.html#_2016-04-10_14_57_07_26016:09
sdake_sbezverk did you docker pull?16:09
sdake_rather kolla-ansible pull16:09
sdake_sbezverk the -17 is definately being instlaled in the gate16:09
Jeffrey4l_in my understanding, in cluster case, only the rabbitmq-server need to communicate each. the empd not. So empd listen on a loopback address is fine. am i right?16:10
sdake_http://logs.openstack.org/37/303837/2/check/gate-kolla-dsvm-deploy-centos-binary/0a404a9/console.html#_2016-04-10_14_57_08_36316:10
sdake_i dont know what the purpose of epmd is in the system16:10
sdake_but here is the issue16:10
Jeffrey4l_ah, I do not know either. :D16:11
sdake_epmd today binds to 0.0.0.0 (wildcard)16:11
sdake_we definately dont want that16:11
sdake_that is why epmd was made to bind to a specific address16:11
sdake_if epmd has no auth it shoudl only bind to 127.0.0.116:11
sdake_http://erlang.org/doc/man/epmd.html16:12
Jeffrey4l_sdake_, yes. you are right. sorry.16:12
sdake_it ooks like it ha saccess restrictions16:13
sdake_Jeffrey4l_ this may fix it: https://review.openstack.org/#/c/303837/216:14
patchbotsdake_: patch 303837 - kolla - [DO NOT MERGE] Test for sbezverk16:14
sdake_atleast for centos16:15
sdake_first gate passed successfully16:15
sdake_sbezverk did you do kolla-ansible pull before docker exec?16:15
*** kragniz has left #openstack-kolla16:15
*** achanda has joined #openstack-kolla16:16
Jeffrey4l_cool.16:16
sdake_ya 16 busted us16:16
sdake_17 possibly unbusts us16:16
Jeffrey4l_i am trying to use ::ffff:ipv4address solution and try to see what will happend.16:16
sdake_jeckersb said 16 was full of bust16:17
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH  https://review.openstack.org/29490616:17
*** achanda has quit IRC16:18
Jeffrey4l_(but, is it really ok to use a personal repo?)16:18
sbezverksdake I built completely new set of images with different tag, then I deployed it, still see .16 used by rabbitmq16:18
*** achanda has joined #openstack-kolla16:19
*** achanda has quit IRC16:23
*** achanda has joined #openstack-kolla16:25
sdake_sbezverk can you acknowlege you see gate job16:27
sdake_Jeffrey4l_ the plan is to gt this merged into epel16:28
Jeffrey4l_cool16:28
sdake_if it fixes the problem16:28
sdake_sbezverk ./bulid --no-cache16:28
sdake_why was --no-cache deprecatedbtw...16:28
sbezverksdake_ I always use --no-cache16:29
sdake_we muck with how the cache oeprtes in kolla by changing timestamps16:29
sdake_sbezverk let me actually bulid it and see if it builds properly16:29
sbezverkhere is command I used: ./build.py --type source --base centos --registry 192.168.80.240:4000 --push --no-cache --tag 2.0.916:29
sdake_sbezverk you see the gate builds it properly16:30
sdake_or not?16:31
sbezverksdake_ sorry what do you mean gate builds??16:32
sbezverksdake_ on my deployment station building of the image is  completed fine16:32
*** athomas has joined #openstack-kolla16:32
sbezverkif it is "gate" in your terminology then yes16:32
sdake_sbezverk thegate is the thing that tests our softwre16:33
sdake_you can see your patch logs here16:33
sdake_http://logs.openstack.org/37/303837/2/check/gate-kolla-dsvm-deploy-centos-binary/0a404a9/console.html#_2016-04-10_14_57_08_36316:33
sdake_sbezverk click the link - see -17 installed16:34
*** achanda has quit IRC16:34
sdake_sbezverk apples/apples http://paste.fedoraproject.org/353328/30613714/ -> http://paste.fedoraproject.org/353328/3061371416:35
sdake_sbezverk i'm not sure wht your doing wrong, perhaps you dont have my 2nd update ofmy atch applied16:36
sdake_the first one had a typo16:36
sbezverksdake_ I see in your case it gets installed, but not in mine. one difference is I am installing it on top of inc0 patch16:37
sbezverkfor deploying liberty16:37
*** sdake has joined #openstack-kolla16:38
sbezverkI will clean everything get fresh master and try it16:38
sdakesbezverk i dropped off16:39
sdakedid you see my apples to apples comparison16:39
sdakehttp://paste.fedoraproject.org/353328/30613714/ -> http://paste.fedoraproject.org/353328/3061371416:39
sdakeopenstack puppet guys dont set erl_EPMD_ADDRESS and see no crashes with 16 rabbitmq16:40
*** sdake_ has quit IRC16:41
sdakeJeffrey4l_ if you want to test out ffff you will have to rebase on top of that sbezverk test patch16:41
sdakei suspect16:41
sdakebut am not certain16:41
Jeffrey4l_i will16:42
sdakewould be nice if this 17 version fixed it properly without further patches16:44
sdakethen its an upstream bug for ubuntu which we have already workekd around16:44
sdakeJeffrey4l_ first gate and first recheck successful16:45
Jeffrey4l_"first gate and first recheck successful"   -> what?16:46
sdakeof https://review.openstack.org/#/c/303837/216:46
patchbotsdake: patch 303837 - kolla - [DO NOT MERGE] Test for sbezverk16:46
sdakehow often does gate typically fail, 50% of the time?16:46
Jeffrey4l_sdake, the gate do not tell the truth. the rabbitmq still failed. http://logs.openstack.org/37/303837/2/check/gate-kolla-dsvm-deploy-centos-binary/6808dd8/console.html#_2016-04-10_16_38_36_52116:48
Jeffrey4l_the rabbitmq is failed 100% i think .16:48
Jeffrey4l_now.16:48
sdakeJeffrey4l_ you can click a line16:49
sdakeand show me a speciic link16:49
sdakeoh i see what you mean now moment16:49
sdakewell we need to fix that for sure16:51
sdakethe gate lieing16:51
Jeffrey4l_yes.16:51
Jeffrey4l_about my test for ffff fix: here is the result.16:51
Jeffrey4l_configure file: http://logs.openstack.org/06/294906/11/check/gate-kolla-dsvm-deploy-centos-binary/c2699b8/logs/kolla_configs/kolla/rabbitmq/rabbitmq-env.conf16:52
sdakehere is what happened16:52
Jeffrey4l_listen result: http://logs.openstack.org/06/294906/11/check/gate-kolla-dsvm-deploy-centos-binary/c2699b8/console.html#_2016-04-10_16_41_24_67516:52
sdake-16 enabled ipv616:52
Jeffrey4l_after add: export ERL_EPMD_ADDRESS= ::ffff:10.220.0.120, empd listen on 0.0.0.016:53
Jeffrey4l_tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      11335/epmd16:53
Jeffrey4l_is this a bug too?16:53
Jeffrey4l_-16 is really bust. :(16:53
Jeffrey4l_sdake, sbezverk ^^16:54
openstackgerritSteven Dake proposed openstack/kolla: RabbitMQ -16 introduced ipv6 breakage  https://review.openstack.org/30383716:55
Jeffrey4l_sdake, any chance could we disable the ipv6 support in empd?16:55
sdakeJeffrey4l_ requires recopmile16:55
sbezverkJeffrey4l_ I am surprised that it was just discovered16:55
sdakesbezverk we hae been discvering it over the last several weeks16:55
sdakeour gates have been flakey as fuck16:56
sdakeand this is the reason!16:56
sdakeit takes time to find the patterns16:56
sdakeand people actually to hae to look at the gate logs and notice a pattern16:56
Jeffrey4l_sdake, i will fix the gate and let him tell the truth.16:57
sdakeJeffrey4l_ we need to file bug for tht epmd listen problem16:57
Jeffrey4l_for epmd?16:57
sdakeyes file it at red hat16:57
Jeffrey4l_ok16:57
sdakethey ctually have people that will fix it16:57
sdakefile it against rdo16:57
sdakeprobably want to link the ubuntu bug16:58
Jeffrey4l_but i can not re-product the ipv6 issue. :(16:58
sdakeand include a core dump16:58
sdakewhich ipv6 issue16:58
Jeffrey4l_epmd can not listen to a ipv4 address, when ipv6 enabled.16:59
sdakethere are two issues here16:59
sdakeepmd can not listne to ipv4 address when ipv6 is inabled16:59
sdakeit depends on some type of configuration option in the system16:59
sdakethis is where a coredump would be invaluable16:59
Jeffrey4l_btw, how to get the coredump? you said it is not the crash dump. no idea about this.17:00
sdakethe second bug is when epmd is used to bind ipv6 addressses in ipv4 mode it binds to wildcard17:00
sdakeJeffrey4l_ do this17:00
sdakesomehow get keepalived to crash (on your local system)17:00
Jeffrey4l_clearly.17:00
sdakeit will save a keepalived.conre file somewhre on yoru system17:00
sdakefigure out where it saves it17:01
Jeffrey4l_like kill -9 keepalived?17:01
sdaketo generate a keepalived crash send it a SIGSEGV17:01
sdakeno not kill -9 keepalived17:01
sdakekill SIGSEGV17:01
sdake keepalived17:01
sdakethe same can be done with rabbitmq17:01
sdakethen copy it out17:02
Jeffrey4l_got.17:02
sdakethe coredump will contain the state of the c stcak at crash which probablhy around binding17:02
sdakei can debug this personally17:02
sdakeor the red hat guys can17:02
sdakebut its busted epmd code17:02
Jeffrey4l_but the rabbitmq is failed to start, so how could we seed a SIGSEGV signal?17:02
sdakeepmd should never segfault17:03
sdakeoh wait,you said epmd didn't segfuat, but said it coudlnt bind17:03
sdakeJeffrey4l_ lets be clear on this point17:03
sdakein every gate failure that occurs, epmd fails to bind is that correct?17:03
Jeffrey4l_right17:03
Jeffrey4l_then epmd failed to start.17:04
Jeffrey4l_then rabbitmq can not connect to the epmd, so he failed to start too.17:04
sdakewell what i'd do since you can not reproduce on your local machine is generate your own epmd binary with a null dereference17:04
sdakethis will trigger a segv17:04
sdakeand save the state of the system17:05
sdakewould you like me to generate you a repo?17:05
Jeffrey4l_sorry, do not following you.17:05
sdakewe need to see the state of epmd's process when it fails to bind17:05
sdakewhat its stack looks like17:06
Jeffrey4l_right.17:06
sdakebecause on my gear it always binds just grandly17:06
sdakei hae ipv6 and ipv4 on on my gear17:06
sdakeso we can put a null dereference to cause epmd to segfault17:06
*** daneyon has joined #openstack-kolla17:06
sdakethis will give us a full coredump in the gate which can be analyzed17:07
Jeffrey4l_(so you can not re-product this bug, right)17:07
sdakecan not reproduce17:07
Jeffrey4l_cool.17:07
sdakebut if it hapepns in the gate it will happen in the field17:07
Jeffrey4l_so you have the code with the modification?17:07
sdakeno it will take me several hours to do the job17:07
Jeffrey4l_add a num ref?17:07
sdakenull reference17:08
Jeffrey4l_s/num/null17:08
sdakedereference to hte c code17:08
sdakefirst before i do that, see if you can geta core file from the gate17:08
sdakeany core file will do17:08
*** fragatina has joined #openstack-kolla17:09
Jeffrey4l_do you want a proof that we can collect the core dump. right?17:09
sdakeright17:09
sbezverksdake is there a way to ssh to the gate?17:09
sdakebefoer i spend half my day making a repo17:09
sdakesbezverk no17:09
sdakei want to make sure we can get a core file17:09
Jeffrey4l_got. I will try to do that.17:10
sdakenot try do ;)17:10
sdakeif you ever hear me say try smack me hardly :)17:11
*** daneyon has quit IRC17:11
sdakesbezverk what exactly did -17 change?17:11
sdakewhere was the koji for it again psl17:11
Jeffrey4l_hmm... what the `smack` meaning here. :(, but i can guess what your means.17:12
sdakeok folks on this bug - if its not fixed by tuesday -we are just permitting EPMD to bind to all interfaces17:13
sdakeits sloppy and could possibly interfere with neutron17:13
sdakebut red hat products don't bind epmd and they haven't had neutron compalints much17:13
*** mwheckmann has joined #openstack-kolla17:13
sbezverk sdake it was supposed to address access to rammitmq.log17:14
*** fragatina has quit IRC17:14
sdakesbezverk point me at the koji so i can read the logs17:14
sdakerad the source patch i eman17:14
sbezverkI am not aware of other changes that went into it. I suspect there are some17:15
sdakekoji plz :)17:15
sbezverkhttp://koji.fedoraproject.org/koji/buildinfo?buildID=75206517:15
sbezverkbb in 30 min17:16
*** cmceniry has quit IRC17:17
sdakesbezverk * Thu Apr  7 2016 John Eckersberg <eck@redhat.com> - R16B-03.1717:22
sdake- Revert "Enable error_logger depth fine tuning" (rhbz#1324922)17:22
sdake* Sun Feb 28 2016 Peter Lemenkov <lemenkov@gmail.com> - R16B-03.1617:25
sdake- Fixed issue with nodes registration over IPv617:25
sdakeso since march we have been dealing with this bullshit17:26
sdakemake it harder plz :)17:26
*** mwheckmann has quit IRC17:27
Jeffrey4l_sdake, need to sleep. will file the bug tomorrow.17:27
sdakeJeffrey4l_ sleep well17:27
sdakesee you tmorrow :)17:27
Jeffrey4l_thanks. see you.17:27
*** cmceniry has joined #openstack-kolla17:30
*** Jeffrey4l_ has quit IRC17:32
openstackgerritMichal Rostecki proposed openstack/kolla: [WIP] Fix swift on ubuntu source  https://review.openstack.org/30384917:35
sdakesbezverk log in with your fas accoutn and leave feedback on their patch here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-e1035fad9017:42
sdake-3 kills the push17:42
*** sdake_ has joined #openstack-kolla17:45
*** sdake has quit IRC17:48
*** sdake has joined #openstack-kolla17:50
*** cmceniry has quit IRC17:52
*** sdake_ has quit IRC17:53
sbezverksdake what have you found in the source?17:53
sdakei see zero signs of a revert i the src.rpm17:54
*** fragatina has joined #openstack-kolla17:56
*** fragatina has quit IRC17:56
*** fragatina has joined #openstack-kolla17:57
sbezverksdake I thought when you test gate with .17 it was all green, have I missed something?18:05
sdakethe gate was green but it failed18:05
sdakeour gate returns false positives it appears18:05
sdakebut via inspection it is easy to see the error18:06
sdakeit definately "gets further"18:06
sbezverksdake so in summary .16 was bad but .17 made things worse?18:06
sdakethats the summary18:07
sdaketry for yourself18:07
sdake4% power left18:08
sbezverkyep, going to in 5 minutes, waiting for target server to come up18:08
sbezverksdake inet_tcp",econnrefused and rabbitmq is crashing18:13
*** jtriley has quit IRC18:26
openstackgerritSteven Dake proposed openstack/kolla: RabbitMQ -16 introduced ipv6 breakage  https://review.openstack.org/30383718:26
openstackgerritRyan Hallisey proposed openstack/kolla: Set db connection retry to infinity  https://review.openstack.org/30243418:27
*** rhallisey has joined #openstack-kolla18:45
*** sdake has quit IRC18:45
*** rhallisey has quit IRC19:18
*** fragatina has quit IRC19:19
*** jtriley has joined #openstack-kolla19:22
*** jtriley has quit IRC19:27
openstackgerritHui Kang proposed openstack/kolla: Add etcd Docker container  https://review.openstack.org/29845119:52
openstackgerritHui Kang proposed openstack/kolla: Add Kuryr ansible role  https://review.openstack.org/29889419:56
openstackgerritHui Kang proposed openstack/kolla: Implement ceilometer ansible role  https://review.openstack.org/30057420:05
*** vhosakot has joined #openstack-kolla20:09
*** vhosakot has quit IRC20:09
*** vhosakot has joined #openstack-kolla20:16
*** vhosakot has quit IRC20:18
*** jtriley has joined #openstack-kolla20:23
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Rabbitmq Upgrade restarts container only when image was changed.  https://review.openstack.org/30385820:23
*** jtriley has quit IRC20:29
*** salv-orl_ has joined #openstack-kolla20:45
*** salv-orlando has quit IRC20:48
*** daneyon has joined #openstack-kolla21:37
*** daneyon has quit IRC21:42
*** daneyon__ has quit IRC21:46
*** daneyon has joined #openstack-kolla21:47
*** asalkeld_ has joined #openstack-kolla22:11
*** ayoung has quit IRC22:12
*** jtriley has joined #openstack-kolla22:25
*** jtriley has quit IRC22:30
*** ayoung has joined #openstack-kolla22:47
*** ayoung has quit IRC23:37
*** fragatina has joined #openstack-kolla23:53
« Saturday, 2016-04-09 Index Monday, 2016-04-11 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!