Monday, 2016-04-11

*** vhosakot has joined #openstack-kolla00:07
*** wuhg has joined #openstack-kolla00:10
*** daneyon_ has joined #openstack-kolla00:19
*** daneyon_ has quit IRC00:21
*** daneyon_ has joined #openstack-kolla00:23
*** jtriley has joined #openstack-kolla00:25
*** fragatina has quit IRC00:29
*** jtriley has quit IRC00:31
*** ayoung has joined #openstack-kolla00:32
openstackgerritCarlos Cesario proposed openstack/kolla: Identify container through  PS1 shell variable  https://review.openstack.org/29798400:36
openstackgerritCarlos Cesario proposed openstack/kolla: Identify container through  PS1 shell variable  https://review.openstack.org/29798400:38
ccesario_vhosakot: ^^00:39
*** unicell has joined #openstack-kolla00:42
*** banix has joined #openstack-kolla00:45
*** banix has quit IRC00:46
*** daneyon_ has quit IRC00:54
*** ccesario_ has quit IRC01:01
*** ccesario___ has joined #openstack-kolla01:02
*** daneyon_ has joined #openstack-kolla01:05
*** ccesario_ has joined #openstack-kolla01:16
*** ccesario___ has quit IRC01:16
*** daneyon_ has quit IRC01:19
openstackgerritMauricio Lima proposed openstack/kolla: Put hosts in alphabetical order in site.yml  https://review.openstack.org/30387301:19
*** stvnoyes has quit IRC01:24
*** fragatina has joined #openstack-kolla01:25
*** stvnoyes has joined #openstack-kolla01:25
*** banix has joined #openstack-kolla01:40
*** bmotz has quit IRC01:42
*** bmotz has joined #openstack-kolla01:42
*** achanda has joined #openstack-kolla01:45
*** banix has quit IRC01:46
*** weiyu has joined #openstack-kolla01:52
*** jtriley has joined #openstack-kolla01:52
*** daneyon_ has joined #openstack-kolla01:54
*** jtriley has quit IRC01:58
*** ccesario___ has joined #openstack-kolla01:58
*** ccesario_ has quit IRC01:58
*** Jeffrey4l_ has joined #openstack-kolla02:02
*** Jeffrey4l_ has quit IRC02:08
*** daneyon_ has quit IRC02:16
*** pbourke has quit IRC02:16
*** pbourke has joined #openstack-kolla02:17
*** banix has joined #openstack-kolla02:19
*** Jeffrey4l_ has joined #openstack-kolla02:22
*** sdake has joined #openstack-kolla02:24
*** sdake_ has joined #openstack-kolla02:27
*** sdake has quit IRC02:28
openstackgerritVikram Hosakote proposed openstack/kolla: Add prechecks to validate kolla_internal_vip_address and kolla_external_vip_address  https://review.openstack.org/30202602:38
*** weiyu has quit IRC02:38
*** banix has quit IRC02:38
*** sdake has joined #openstack-kolla02:39
*** banix has joined #openstack-kolla02:41
*** sdake_ has quit IRC02:43
*** salv-orlando has joined #openstack-kolla02:45
vhosakotJeffrey4l_: sdake:  https://review.openstack.org/#/c/302026/02:45
patchbotvhosakot: patch 302026 - kolla - Add prechecks to validate kolla_internal_vip_addre...02:45
*** yuanying has quit IRC02:46
*** salv-orl_ has quit IRC02:47
*** wuhg has quit IRC02:49
*** daneyon_ has joined #openstack-kolla02:50
*** klint has joined #openstack-kolla02:51
*** daneyon_ has quit IRC02:55
*** weiyu has joined #openstack-kolla02:58
*** achanda has quit IRC02:59
*** jasonsb has quit IRC03:10
sdakevhosakot i think i sorted out the gate issue03:22
vhosakotsdake: cool... which issue ? rabbitmq crash ?03:24
sdakeyup03:24
sdakethe crashes started happening when rdo upgraded their repos to -16 version of rabbitmq03:24
sdakethis enabled ipv603:25
sdakeipv6 and ipv4 don't work  with epmd03:25
sdakeso if we disable the EPMD address setting, it will work03:25
sdakenote this review:03:25
vhosakotah ok... well, fix I put in https://review.openstack.org/#/c/303686/ solved the crash issue in centos gate as well :)03:26
patchbotvhosakot: patch 303686 - kolla - Fix rabbitmq crash in centos gate03:26
sdakehttps://review.openstack.org/30383703:26
vhosakotsdake: I removed the ERL_EPMD_ADDRESS variable in rabbitmq-env.conf in the fix03:26
sdakeread my review03:26
vhosakotsdake: yep, I reviewed and left a comment03:27
sdakewell what i just explained is the root cause03:27
vhosakotah ok03:27
sdakedid you catch the part about heisenbug03:27
sdakebinding to 0.0.0.0 is not ideal03:28
*** banix has quit IRC03:28
vhosakotsdake: ah, reading you comment in https://review.openstack.org/#/c/303837/4/ansible/roles/rabbitmq/templates/rabbitmq-env.conf.j203:29
patchbotvhosakot: patch 303837 - kolla - RabbitMQ -16 introduced ipv6 breakage03:29
vhosakotsdake: I see ERL_EPMD_ADDRESS is commented, isn;t it needed03:31
sdakewell yo uwould think so03:32
sdakebut erlang is completely busted wrt empd03:33
sdakewhich is the logging daeon iiuc03:33
vhosakotsdake: ah, so, rabbitmq-env.conf.j2 will have ERL_EPMD_ADDRESS commented or removed ?03:33
sdakecommented for the moment03:33
sdakewith a  warning03:33
sdakefeel freet to update your change to that affect03:33
sdakethe problem is binding to an ipv4 address using ipv4 addressing scheme i.e. ::ffff:ipv4address binds to 0.0.0.003:34
sdakeso that workaround is no bueno either03:34
vhosakotsdake: well, both our change resolve the same.. so, does it make sense to abandon my review and add me as co-author on your review and merge it I found a bit about removing ERL_EPMD_ADDRESS as well :)03:34
sdakei think i will abandon my change03:35
sdakeand you can just integrate my comments03:35
sdakeand leave me off the coauthor line is fine03:35
vhosakotsdake: then, should I add erlang-R16B-03.17.el7.repo in my fix ?03:35
sdakeNO03:35
vhosakotsdake: yes, I will add you as co-author03:35
sdakethat was a tes for sbezverk03:35
vhosakotsdake: ah ok03:35
sdakei really dont think you need to do so03:36
sdakebut its up to you :)03:36
vhosakotsdake: I will add you as co-author, but what should I change in https://review.openstack.org/#/c/303686/ ?  add your comment about Heisenbugs (about how epmd is broken with ipv6) ?03:37
patchbotvhosakot: patch 303686 - kolla - Fix rabbitmq crash in centos gate03:37
vhosakotsdake: let me know if this what I need to do.. yep, will add you as co-author.. :)03:38
*** achanda has joined #openstack-kolla03:44
*** achanda has quit IRC03:46
sdakeyes add the comment03:46
openstackgerritMerged openstack/kolla: Fix the qemu has nova permission in nova instances folder  https://review.openstack.org/30368003:48
*** yuanying has joined #openstack-kolla03:49
vhosakotsdake: ok, I will add your comment.. cool03:50
vhosakotsdake: so, will https://review.openstack.org/#/c/303837/ be abandoned then ?03:50
patchbotvhosakot: patch 303837 - kolla - RabbitMQ -16 introduced ipv6 breakage03:50
*** coolsvap has joined #openstack-kolla03:51
*** jtriley has joined #openstack-kolla03:53
openstackgerritVikram Hosakote proposed openstack/kolla: Revert "Add general_log to mariadb container and make heka collect it"  https://review.openstack.org/30240103:54
openstackgerritMerged openstack/kolla: Increase max scheduling attempts to 10  https://review.openstack.org/30367803:57
*** jtriley has quit IRC03:58
*** haplo37 has joined #openstack-kolla04:02
sdakevhosakot this has been what I've been doing all weekend04:04
sdakehttps://drive.google.com/file/d/0B8q6xDPETSkHZDgyOTFMLTVPbm8/view?usp=sharing04:04
vhosakotsdake: ah :)04:04
sdakethose were galant desks with damaged tops04:04
vhosakotsdake: where is your home server ?04:04
sdakeikea discontinued the furniture04:04
sdakein that room04:04
sdakenot yet plugged in04:04
sdakeatm its in my daughters room04:05
sdakeplugged in through hacky networking techinuqes ;)04:05
vhosakotsdake: ah ok04:05
vhosakotnice!04:05
sdakeso anyway the furniture got discontinued and we needed new tops04:05
sdakeso we bought counter tops for kitchens04:05
sdake100 bucks a crack04:05
sdakeand i mounted them no the hardware from the ikea galant desks04:05
sdakeone more to go04:05
vhosakotyep, looks great04:05
sdakeand then i can gear up04:05
vhosakotsdake: you glued it or screwed it ?04:06
sdakewe are putting  atable coming out into the room 57 inches04:06
sdakescrewed04:06
vhosakotright04:06
sdake3/4 #12 screws04:06
vhosakotcool04:06
sdakecountertops shoud hold up well04:06
sdakethey are completely plastic04:06
vhosakotis that a gramaphone ? :)04:06
sdakei dont even know what that is ;)04:07
sdakemy stereo is modern :)04:07
sdakemy dac and digital music play are yet  to be hooked up04:07
vhosakotI see something like a vinly disc player04:07
coolsvapsdake: looks good04:07
sdakeyup thats my feikert turntable04:08
vhosakotcool04:08
sdakehttp://www.feickert.de/index.php?id=1004:08
sdakemine is slightly different staylin but same model04:08
sdakethat is the non-us model04:08
vhosakotnice.. turntable!04:09
sdakein the us they make them with differnet materials04:09
sdakeought to be04:09
sdakethanks coolsvap04:09
sdakewe aren't done yet04:09
sdakewe have one more tablee to go ;)04:09
sdakebut my remodel is officially over04:10
sdakenow its a matter of  putting the house back together04:10
sdakeremodeling the desks cost about 450 bucks :(04:10
coolsvap:)04:10
*** weiyu has quit IRC04:10
sdakebut a double workstation with professional grade gear is 6-8k04:10
sdakeso we are holding off on that for vestmas :)04:10
vhosakotwow... 45004:11
sdake450 is cheap for 3 tables04:11
sdakethat never wear out no matter how abusive your children are to your gear04:11
vhosakotright04:12
sdakei have 40 amps o power along the desk wall as well04:12
sdakeon two seprate circuits04:12
sdakehttp://www.amazon.com/Tripp-Lite-PDUMV40-Metered-Vertical/dp/B000MT4HQM?ie=UTF8&psc=1&redirect=true&ref_=oh_aui_search_detailpage04:13
sdakepower solution ^04:13
sdakewe planned 17k for our remodel and it was about 18.5k04:14
sdake10% over budget groan04:14
sdakebut i wasn't' planning 40 amps of power which was expensive04:14
sdakeor new closets for kids bedroom04:15
vhosakothome improvemnt projects always go over planned time and budget04:15
sdakeelectriian cost = 700 bucks04:15
sdakewhich was a super deal04:15
sdakei expected to be done early feb04:15
sdakebut whe nyou live in teh place04:15
vhosakotcool vertical power strip in amazon04:15
sdakeits hard to actually deliver on that04:15
sdakeya we have that running horizontally04:16
sdakethe black is one circuit white is the second 20 amp circuit04:16
vhosakotcool04:16
sdakeso basically we have mroe poewr then we can possibly use04:16
sdakethat was because my daughter microwaves and used to blow the circuit daily04:16
sdakeso we put the office on its own circuit04:16
sdakei also have a 3rd 20 amp circuit dedicated to my stereo04:17
*** achanda has joined #openstack-kolla04:18
sdakewith 8 gauge wire and furutech gtx-d rhodium power plug ;)04:18
vhosakotnice.. yep, our microwave circuit is also separated from most of the houce circuites...04:18
sdakehttp://www.enjoythemusic.com/magazine/equipment/0114/audiophile_ac_outlets.htm04:18
sdakethe problem we have is our microwave is connected to our entire front of our hosuse04:19
sdakelike morons the electricians didn't run a separate circuit for the microwae04:19
*** weiyu_ has joined #openstack-kolla04:19
vhosakotour microwave trips sometimes...04:20
sdakemy stereo is pretty off the hook04:26
sdakedefinately sounds different with the tile the then carpet04:26
sdakeneed to get some rugs down or something04:26
sdakedifferent as in not better04:26
*** weiyu_ has joined #openstack-kolla04:27
sdakewell i hate hitler04:30
sdakebut heres a docker on hitler thing that is making the rounds04:30
sdakehttps://m.youtube.com/watch?feature=youtu.be&v=PivpCKEiQOQ04:30
vhosakothaha.. yeah.. I 've seen it..!04:32
sdakethe part about the proxy server is so hilarious and so true04:35
vhosakothaha... yeah04:36
openstackgerritVikram Hosakote proposed openstack/kolla: Drop root for HAProxy  https://review.openstack.org/24501304:36
*** weiyu_ has quit IRC04:40
sdakesudo curl | bash04:41
sdakebad habit to get people into04:41
sdakehttps://helm.sh/04:43
sdakehttp://helm.readthedocs.org/en/latest/modeling_services/04:47
sdakeusing labels for wait conditions04:47
sdakeso sloppy04:47
vhosakotwow, I never pipe file I curl from a public site to bash without reading the contents of the file..04:47
vhosakotcurl | bash is very dangeruos04:48
vhosakotsdake: we have it in our qsg as well http://docs.openstack.org/developer/kolla/quickstart.html04:48
vhosakotsdake: curl -sSL https://get.docker.io | bash04:48
vhosakotthe userland proxy being slow is very true...04:48
sdakei know its there04:49
sdakelook t git blame on that line04:49
*** daneyon_ has joined #openstack-kolla04:50
vhosakotlet me check git blame04:50
vhosakotwow https://github.com/openstack/kolla/blame/master/doc/quickstart.rst#L8704:50
sdakehmm i thought i put that in there not sam ;)04:51
vhosakot:)04:52
sdakewhere i was going to point out the oxymoronism that i am such a security freak yet would propose such an insecure behavior04:52
sdakewe need to sudoize our documentation04:53
sdakei guess i'll do that after releases04:53
sdakesam always ran as root so he was very negative about having sudo in the dos04:54
sdakedocs04:54
vhosakotah ok04:55
sdakepossibly why selinux never worked for him04:55
sdakerunning as root breaks selinux pretty easily04:55
sdakei was shocked thoug hto find our gate was runnign seinux04:56
vhosakotisn't is enfored by default in Centos ?04:56
sdakeyes04:56
vhosakotenforced*04:56
sdakethe gate images used t o turn it off by deffault04:57
sdakeas most cloud providers do04:57
nihiliferguys, one question, if anyone here is good at ubuntu05:12
nihiliferwhat's the best way to make "mount --make-shared /run" permanent?05:13
nihilifer/etc/fstab? something in upstart script?05:13
openstackgerritVikram Hosakote proposed openstack/kolla: Drop root for HAProxy  https://review.openstack.org/24501305:16
sdakeok catss05:17
sdakewe need kolla upgrades and reconfigure totally working for this05:17
sdakehttp://www.midokura.com/Webinars/05:17
sdake100% reliability pzl :)05:18
sdakenihilifer vhosakot is your man05:18
sdakeprobably rc.local woud lbe my recommendtaion though05:18
sdakejust from general unix experience05:18
sdakebut there may be a better / more intuitve way05:19
nihiliferok, thx05:21
*** sdake_ has joined #openstack-kolla05:23
vhosakotnihilifer: yes, adding an entry in /etc/fstab should make it persist thru reboot... I have not added it as I run the command after every reboot..05:23
vhosakotnihilifer: you you see how /run is by doing     cat /proc/self/mountinfo | grep run05:23
*** sdake has quit IRC05:24
*** weiyu_ has joined #openstack-kolla05:26
*** daneyon_ has quit IRC05:30
vhosakotsdake_: what is this midokura kolla webinar ?05:31
sdake_something midokura setup05:34
sdake_any chance you could register at zoom.com so i can see what its like to video conference with it?05:34
sdake_would like to know the tool so I have some idea of what i've gotten myself into ;)05:35
vhosakotsdake_: I cannot access zoom.com05:37
sdake_try off the vpn05:38
sdake_its probably firewalled or something05:38
sdake_sorry zoom.us05:38
*** dimsum__ has quit IRC05:40
nihilifervhosakot: ok, another stupid question from me05:41
nihiliferi have the following entry in mountinfo05:41
nihilifer21 22 0:16 / /run rw,nosuid,noexec,relatime shared:1 - tmpfs tmpfs rw,size=256604k,mode=75505:41
vhosakothaha there is no sypid Q..05:41
nihiliferhow to pass the "shared:1" parameter to fstab?05:41
nihiliferadding it as an option after comma doesn't work for me05:42
nihilifermy fstab entry i've tried is05:42
vhosakotyep, I dont know that part.. I tried many things and did not work... that is why I have added it to /etc/fstab...05:43
*** achanda has quit IRC05:43
nihilifertmpfs /run tmpfs rw,nosuid,noexec,relatime,shared:1 0 005:43
vhosakotnihilifer: what is /run filesystem type in your environmeny ?   df /run05:43
vhosakottmpfs is good.. cool05:43
sdake_lets get that in the docs05:44
sdake_plz ;005:44
vhosakotyes..05:44
sdake_we get this question all the time05:44
sdake_nihilifer will you be at summit?05:45
nihilifersdake_: i will not, unfortunately05:45
* sdake_ whines05:45
*** sdake_ is now known as sdake05:45
sdakesame story with pbourke05:45
*** weiyu_ has quit IRC05:46
vhosakotoh, was hoping to meet everyone05:46
sdakenihilifer since your on our security team05:46
sdakei will need your help prepring architecture documenttion for a ta session05:46
sdakewe will be using a tool called gliffy05:46
sdakeplesae register :)05:46
*** dims has joined #openstack-kolla05:47
sdakeyo dims05:47
nihiliferi'm registered, so no worries05:47
nihilifervhosakot: here is the result of "df /run"05:47
nihiliferhttp://paste.openstack.org/show/493599/ - when running the mount command by hand05:48
nihiliferhttp://paste.openstack.org/show/493600/ - after my failed attempt with fstab05:48
vhosakotnihilifer: same output as mine...05:48
vhosakotnihilifer: the thing I have observed it the <options> column in the /etc/fstab file does not accept the "shared" value... http://man7.org/linux/man-pages/man8/mount.8.html#FILESYSTEM-INDEPENDENT_MOUNT%20OPTIONS05:49
vhosakotnihilifer: I have not been able to figure it out either, and hence run the mount command manually after eveyr reboot05:49
nihilifer:(05:50
* nihilifer waits for 16.04 LTS and systemd05:50
vhosakotor move to Centos05:51
nihiliferi cannot05:51
nihilifermy company forces me to use ubuntu, lol05:51
vhosakotso, after updating /etc/fstab ? did you reboot or re-mount the mountpoints using "mount -a" ?05:51
nihiliferi really wish i could use centos05:51
*** haplo37 has quit IRC05:52
nihiliferi did a reboot05:53
vhosakotcool...05:53
*** jtriley has joined #openstack-kolla05:54
sdakeyou aare forced to use ubuntu?05:56
sdakehow can such a thing be forced?05:56
nihiliferi'm joking with "forceD"05:57
nihilifer;)05:57
sdakeyou always hae a choice ;)05:57
nihiliferwe just care about kolla working on ubuntu05:57
nihiliferwithout dirty hacks05:57
sdakeroger and we intend to deliver on that05:57
sdakeand thus far i think we have done a good job05:57
sdakeas well as centos ole debian and lastly fedora ;)05:57
sdakei'd like suse support if any suse cats came to implement it05:58
sdakei reached out to them but snakeeyes05:58
*** jtriley has quit IRC05:59
sdakenihilifer jeffrey4l and I fixed the qemu bug on mesos06:02
sdakeas well as everywhere else it was broken06:02
*** karthiks has quit IRC06:03
sdakeif you stil lsee that kicking around let me know06:03
sdakethis is our final week until release06:03
sdakeif there is any more doa let me know plz :)06:03
*** achanda has joined #openstack-kolla06:15
openstackgerritSteven Dake proposed openstack/kolla: Add note about heat being busted for development  https://review.openstack.org/29027606:18
*** weiyu has joined #openstack-kolla06:18
*** achanda has quit IRC06:20
*** daneyon_ has joined #openstack-kolla06:24
*** tfukushima has joined #openstack-kolla06:26
*** daneyon_ has quit IRC06:30
*** dims has quit IRC06:31
*** Serlex has joined #openstack-kolla06:33
openstackgerritVikram Hosakote proposed openstack/kolla: Drop root for HAProxy  https://review.openstack.org/24501306:33
openstackgerritSteven Dake proposed openstack/kolla: Add quotes for RBMQ nodes definition  https://review.openstack.org/30391906:34
vhosakotsdake: I see you abadoned your rabbitmq crash review... I just need to add your comments in rabbitmq-env.conf.j2 in my PS right ?   https://review.openstack.org/#/c/303837/4/ansible/roles/rabbitmq/templates/rabbitmq-env.conf.j206:42
patchbotvhosakot: patch 303837 - kolla - RabbitMQ -16 introduced ipv6 breakage (ABANDONED)06:42
*** coolsvap has quit IRC06:43
*** achanda has joined #openstack-kolla06:44
sdakeya just add my note06:44
sdakeand comment out the entire section so we don't lose it06:44
sdakevhosakot ^^06:44
vhosakotsdake: ok.. cool... should I add ::ffff: for ERL_EPMD_ADDRESS as well ?06:45
vhosakotsdake: ^^06:47
sdakei wouldn't06:47
sdakeor add both lines06:47
sdakesay one is an option recommended in the launchpad bug above in comment #606:48
sdakebut it results in a bind to all interfaces.06:48
sdakewhich is the same as not specifying anything here06:48
*** dims has joined #openstack-kolla06:54
vhosakotsdake: cool.. I wont add the ::ffff: for ERL_EPMD_ADDRESS then06:55
*** coolsvap has joined #openstack-kolla06:57
*** dims has quit IRC06:58
openstackgerritVikram Hosakote proposed openstack/kolla: Drop root for HAProxy  https://review.openstack.org/24501307:00
*** weiyu has quit IRC07:03
*** weiyu has joined #openstack-kolla07:04
*** mikelk has joined #openstack-kolla07:11
sdakevhosakot can you sort out that gate change now07:12
sdakeso I can ack it before bedtime07:12
vhosakotsure... doing it now07:12
sdakegetting red hat to revert ipv6 support will be very difficult07:13
sdakeand getting them to fix epmd to work with ipv4 and ipv6 also difficult07:13
sdakethe epmd binding code is an absolute charlie foxtrot07:13
sdakei've looekd at it, I'd have to rewrite the entire binding to work correctly07:13
sdakeand upstream may have already solved it07:13
sdakebut I dont know07:14
sdakeipv4 and ipv6 binding code in corosync was over 2500 loc07:14
sdakeits very difficult to get right07:14
sdakeespecially in a one-off port07:14
*** jmccarthy has left #openstack-kolla07:14
sdakeas in something thatdoesn't go into master07:14
openstackgerritSteven Dake proposed openstack/kolla: Add note about heat being busted for development  https://review.openstack.org/29027607:16
*** weiyu has quit IRC07:17
coolsvapsdake: s/Hea/Heat07:18
sdakecoolsvap line # pleae07:18
coolsvap#2207:19
coolsvap2107:19
openstackgerritMerged openstack/kolla: Revert "Add general_log to mariadb container and make heka collect it"  https://review.openstack.org/30240107:19
coolsvapsdake: line 2107:19
*** weiyu has joined #openstack-kolla07:20
openstackgerritVikram Hosakote proposed openstack/kolla: Fix rabbitmq crash in centos gate  https://review.openstack.org/30368607:20
vhosakotsdake: ^^07:20
sdaketry that on07:20
openstackgerritSteven Dake proposed openstack/kolla: Add note about heat being busted for development  https://review.openstack.org/29027607:20
vhosakotJeffrey4l_: sdake  https://review.openstack.org/#/c/303686/07:20
patchbotvhosakot: patch 303686 - kolla - Fix rabbitmq crash in centos gate07:20
openstackgerritMerged openstack/kolla: Add prechecks to validate kolla_internal_vip_address and kolla_external_vip_address  https://review.openstack.org/30202607:21
coolsvapsdake: should i correct it, i have a feeling you need to get to bed asap07:21
sdakebeuno is spelled bueno07:21
sdakecoolsvap i'm good07:21
sdakei'm up for a couple more hours07:21
sdakejust tyring to get some adminstrative tasks done07:22
coolsvaps/beliee/believe07:22
coolsvapsame line07:22
coolsvapline 2107:22
coolsvapsdake: ^^07:22
openstackgerritSteven Dake proposed openstack/kolla: Add note about heat being busted for development  https://review.openstack.org/29027607:22
vhosakotJeffrey4l_: sdake coolsvap:  https://review.openstack.org/#/c/303686/  is updated07:24
patchbotvhosakot: patch 303686 - kolla - Fix rabbitmq crash in centos gate07:24
sdakebeno/bueno07:25
sdakeplease change ;)07:25
sdakelets atleast spell it correctly07:25
sdakevhosakot ^^07:25
vhosakothaha... sdake ok07:25
openstackgerritVikram Hosakote proposed openstack/kolla: Fix rabbitmq crash in centos gate  https://review.openstack.org/30368607:28
vhosakotsdake: bueno pushed ^^07:28
sdakecoolsvap or nihilifer can you review https://review.openstack.org/#/c/303686/807:30
patchbotsdake: patch 303686 - kolla - Fix rabbitmq crash in centos gate07:30
sdakecoolsvap will you be at summit?07:30
coolsvapsdake: yes07:30
sdakesweet07:30
sdakeso only 3 people i think from the cor eteam wont make it07:30
vhosakotsdake: get us all beer ;)07:30
sdake3/4 is better then 0/4ths :)07:31
sdakewhat do I look like - an atm? :)07:31
vhosakotJeffrey4l_: ping07:31
sdakevhosakot agree with jeffrey4l07:31
vhosakotthe rabbitmq crash is not seen on Ubuntu07:31
sdakethat is because the cndition is never executed07:31
vhosakotah, right...07:32
vhosakotlet me remove it07:32
sdakeso the variable is never set07:32
coolsvapatb (brewer)07:32
openstackgerritVikram Hosakote proposed openstack/kolla: Fix rabbitmq crash in centos gate  https://review.openstack.org/30368607:32
vhosakotsdake: Jeffrey4l_ ^^^07:32
sdakeafter this merges i am doing a massive recheck of all our outstanding patches07:33
vhosakotsdake: Jeffrey4l_ if condition is removed07:33
vhosakothahah coolsvap  atb (brewer) :)07:33
*** shardy has joined #openstack-kolla07:34
vhosakotif Jeffrey4l_ coolsvap acks it, our gates will be good.... and no rabbotmq crash07:35
sdakethat wil lspeed up dev quite a bit ;)07:35
vhosakotsdake: we need a bug to track the nova vm issue not reaching active state in centos binary gate... new issue, not related to rabbitmq crash07:35
sdake5 rechecks perpatch is annoing07:35
sdakevhosakot agree can youf ile one07:36
sdakeand file a tech debt bug as requested for this particular EPMD issue07:36
vhosakotyep, I have debugeed it almost...a nova issue07:36
vhosakotah ok07:36
sdakeso its tracked as moer then just a TODO07:36
vhosakot2 bugs07:36
vhosakotthanks for the +2 coolsvap07:37
coolsvapvhosakot: np!07:37
* coolsvap away for lunch07:37
vhosakotsdake: so, how will the EPDM IPv6 issue be solved ? do we need to upgrade rabbit-server ?07:37
sdakehell if I know07:37
*** jmccarthy has joined #openstack-kolla07:37
sdakeI dont have a month to rewrite RHELs version of epmd binding07:38
sdakewich is far different from upstream07:38
sdakemodel is usptream first, then backport07:38
sdakeI think we will jut have to live with it07:38
sdakemaybe we can get red hat to fix it since its their port07:38
sdakebut who knows07:38
sdakeit will affect their customer base severely07:38
vhosakotsdake: cool agreed07:39
sdakeand also our users will be negatively impacted by this problem07:39
*** weiyu has quit IRC07:39
sdakethe issue is a wildcard bind on whatever port epmd runs on will trigger software that binds on that port from not being able to bind - theoretically07:39
sdakeand also may have some other negative influence07:39
*** dims has joined #openstack-kolla07:45
*** reddy has joined #openstack-kolla07:51
openstackgerritVikram Hosakote proposed openstack/kolla: Add prechecks to validate kolla_internal_vip_address and kolla_external_vip_address  https://review.openstack.org/30393807:52
*** vhosakot has quit IRC07:53
*** jtriley has joined #openstack-kolla07:55
*** achanda has quit IRC07:56
*** mbound has joined #openstack-kolla07:57
*** jtriley has quit IRC08:00
sdakeJeffrey4l_ or mandre can i get an ack on https://review.openstack.org/#/c/290276/08:00
patchbotsdake: patch 290276 - kolla - Add note about heat being busted for development08:00
*** gfidente has joined #openstack-kolla08:08
openstackgerritMerged openstack/kolla: Fix rabbitmq crash in centos gate  https://review.openstack.org/30368608:08
openstackgerritSteven Dake proposed openstack/kolla: Fix rabbitmq crash in centos gate  https://review.openstack.org/30395108:11
sdakeneed an ack on this cherrypick please ^^08:12
sdakeasalkeld nihilifer mandre Jeffrey4l_ ^^08:12
*** weiyu has joined #openstack-kolla08:15
mandresdake: I'm not sure I understand the resolution of https://bugs.launchpad.net/kolla/+bug/156270108:18
openstackLaunchpad bug 1562701 in kolla mitaka "rabbitmq crashed on centos binary deploy" [Critical,Confirmed] - Assigned to Vikram Hosakote (vhosakot)08:18
*** gfidente has quit IRC08:19
sdakedo you rreally want me to explain it to you08:20
sdakei am happy to do so08:20
sdakebut it is time consuming and complicated :)08:21
mandresdake, yes please :)08:21
sdakeepmd which is some part of erlang is busted when compiled with ipv6 support08:21
sdakeit wont bind properly to an ipv4 address08:21
sdakein some conditions08:21
sdakein others it will08:21
sdakeI am not quite sure what the conditions are08:22
mandrecan you first explain the comment you added?08:22
sdakei'm getting there08:22
*** achanda has joined #openstack-kolla08:22
mandreaccording to your comment, we should bind to a known IPv4 address and it will fix it for all cases08:22
sdakebecause erlang is broken and wont bind to ipv4, and kolla doesn't really work well with ipv6, the only option available to us is to remove the bind option08:22
sdakeno, according to my comment we should be binding08:23
sdakebut we cnanot ecause erlang is defective08:23
mandrethat's not what your comment say08:23
sdakedisagree08:23
mandreyour comment says we should be binding, then the bind is commented out08:23
sdakei you read the bug log and the note it seems clear08:23
mandrethat's what I don't understand08:23
sdakeyes we should be binding08:24
sdakebut we cant because as documented in the bug, doing so makes erlang bomb out08:24
sdakeits a cherrypick, too late now to debate the language08:24
sdakeif you want to clarify it submit a follow up patch and we can cherrypick that ;)08:24
mandrejust trying to understand the resolution now...08:24
*** weiyu has quit IRC08:25
*** gfidente has joined #openstack-kolla08:25
sdakeessentially its a problem in erlang08:25
sdakeand erlang needs to fix it08:25
sdakeand red hat needs to backport it into centos08:25
sdakeand ubuntu the same08:26
mandreyou're saying that binding with ERL_EPMD_ADDRESS is broken, correct?08:26
sdakeyes precisely08:26
sdakethe act o fbinding is correct08:26
sdake the implementation of erlang's binding is incorrect08:26
mandreok, so I think we need to clarify your comment :)08:27
sdakeok i can do or you can - use same bug08:27
sdakewhats it to be ;)08:27
mandreI'll clarify the comment, you'll tell me if I understood correctly in the patch review08:28
sdakecool but do it in master08:28
sdakein the meantime letes get the cherrypick merged08:28
sdakeso i can start a massive recheck on all our patches08:29
sdakeread the bug log too please08:29
sdakeit has key information08:29
sdakewhat triggered this ws the introduction of ipv6 compile flags to erlang in -16 of erlang08:29
sdaket-2 weeks ago08:29
*** weiyu has joined #openstack-kolla08:29
sdakeubuntu never had the problem in the gate08:30
sdakeeven though ubuntu is equally broken08:30
mandrei see, nice find08:30
sdakebecause it had a conditional around it08:30
sdakeit wsn't until we tried sbezverk's -17 patch that it all came together forme08:30
sdakeand jeffrey's discovery of the syslog message epmd: failed to bind08:31
sdakewhen epmd fails to bind, erlang crashes08:31
sdakethis causes anything running in erlang (rabbitmq for example) to damage the gate ;(08:31
mandredo you have a bug for broken binding with ERL_EPMD_ADDRESS?08:32
sdakei asked vhosakot to file one in kolla as a tracking bug08:32
sdakebut i dont know if he did tonight or planned to tomorrow08:32
sdakebut the bug info we know bout is in the bug in the cherrypick that comes from master08:33
mandreit should be an erlang bug not a kolla one08:33
sdakei dont know the erlang bug08:33
sdakebut they are aware of it apparently08:33
mandrei'll look for it08:33
sdakemaking ipv4 and ipv6 bind properly is very difficult08:33
sdakeand epmd has itwrong08:34
sdakethe crux of the situation we are in now is that we are binding to all interfaces08:34
sdakewhich if controller code is collocated08:34
sdakecould possibly cause disruptions of network services runnign on the same port as epmd08:34
sdakei suspect, but i am not sure08:34
sdakethey have the wildcard bind working08:35
sdakethey dont have specific interface binding woring08:35
sdaketo make specific interface binding work, it is necessary to scan all interfaces, find the interface number, and pass it as an ioctl in a special bind oeration08:35
sdakethis is done in corosync correctly but it took us literally 3-4 months to get right08:36
sdakeepmd needs the same attention08:36
sdakeits complicated by the fact that red hat's erlang is months/years behind upstream08:36
openstackgerritSteven Dake proposed openstack/kolla: Fix rabbitmq crash in centos gate  https://review.openstack.org/30395108:37
*** fragatina has quit IRC08:43
*** salv-orl_ has joined #openstack-kolla08:44
*** salv-orlando has quit IRC08:47
*** dims has quit IRC08:48
openstackgerritMartin André proposed openstack/kolla: Clarify comment about binding erlang to IPv4  https://review.openstack.org/30396908:50
mandresdake: ^08:53
sdakesome suggestions in that doc mandre08:53
sdakeratherin the review08:53
sdaketo clarify further08:53
sdakeyour totally right the comment was unclear08:54
sdakelets make it crystal clear :)08:54
*** dims has joined #openstack-kolla08:55
sdakemandre you might also include a link to my abaonded review08:58
sdakehttps://review.openstack.org/#/c/303837/08:59
patchbotsdake: patch 303837 - kolla - RabbitMQ -16 introduced ipv6 breakage (ABANDONED)08:59
mandresdake: I think this is information that should go in our issue, not in the code09:01
mandreI don't want to make the comment too verbose, it's already quite long09:01
sdakei hate to see someone uncomment for no good reason other then they think its fixed09:01
sdakei want them to be sure ;)09:02
mandrethat's the reason why we're putting a link to the issue09:02
sdakechange of topic09:02
sdakebut the issue is closed and fixed09:03
openstackgerritMartin André proposed openstack/kolla: Clarify comment about binding erlang to IPv4  https://review.openstack.org/30396909:03
sdakepeople dont typically click through anyway :)09:03
mandreit doesn't mean we can't comment on it to provide more info09:03
sdakeok09:04
sdakethere is also a red hat bug on this issue09:04
sdakehttps://bugzilla.redhat.com/show_bug.cgi?id=132492209:05
openstackbugzilla.redhat.com bug 1324922 in erlang "Log handler repeatedly crashes" [Urgent,On_qa] - Assigned to jeckersb09:05
*** daneyon_ has joined #openstack-kolla09:07
sdakemandre just for completeness can you add a link to that bug tracker09:11
sdakesince that is our upstream for this bug09:11
*** daneyon_ has quit IRC09:11
mandreare you sure it's the same bug? I've seen you commented on it, but the trace looks somewhat different from what I remember09:12
sdakesame bug09:14
sdakebut your right different presentation09:14
sdakeit revolves around epmd binding09:14
sdakewhatever the case, -17 is totally broken09:14
sdakeand I expect red hat will track it in that bug or provide links to other bugs where they sovlve the ipv6 problem with epmd :)09:15
sdakeepmd is written in c09:16
sdakei've looked it over09:16
sdakethe binding code is plain wrong09:16
openstackgerritMartin André proposed openstack/kolla: Clarify comment about binding erlang to IPv4  https://review.openstack.org/30396909:16
*** jmccarthy has left #openstack-kolla09:16
mandresdake: should be good now09:17
sdakeagree09:17
sdakecoolsvap asalkeld nihilifer anyone else still awake can we get an ack on that09:17
sdakeJeffrey4l_ ?09:18
coolsvapdone09:18
openstackgerritMerged openstack/kolla: Fix rabbitmq crash in centos gate  https://review.openstack.org/30395109:20
*** weiyu has quit IRC09:23
*** ccesario___ has quit IRC09:37
*** Kimmo_ has joined #openstack-kolla09:48
*** weiyu has joined #openstack-kolla09:49
*** allen_gao has quit IRC09:50
*** allen_gao has joined #openstack-kolla09:55
*** jtriley has joined #openstack-kolla09:55
*** jtriley has quit IRC10:00
*** weiyu has quit IRC10:10
*** coolsvap has quit IRC10:15
openstackgerritSteven Dake proposed openstack/kolla: Revert "Spec: Deploy Kolla images using Kubernetes"  https://review.openstack.org/30401010:17
*** weiyu has joined #openstack-kolla10:19
*** reddy has quit IRC10:22
Jeffrey4l_sdake, back.10:23
*** reddy has joined #openstack-kolla10:25
sdakehey Jeffrey4l_10:25
sdakeJeffrey4l_ can you ack the above patch10:25
sdakespec workflow process was not followed10:26
Jeffrey4l_sdake, i am checking it.10:26
sdakewe dont really have a specworkflow process10:26
sdakebut historically we require majority vote on any specs10:26
sdakeinstead there was only 2 votes10:27
*** mgoddard_ has joined #openstack-kolla10:27
sdakeand I would have voted -1 had I known it was hiding around in the review queue10:27
sdakesince it is incomplete10:27
*** andrewvwebber has joined #openstack-kolla10:28
*** mgoddard has quit IRC10:30
*** wmiller has quit IRC10:30
*** wmiller has joined #openstack-kolla10:31
*** weiyu has quit IRC10:32
*** weiyu has joined #openstack-kolla10:38
*** wmiller has quit IRC10:41
*** sdake_ has joined #openstack-kolla10:45
*** ccesario___ has joined #openstack-kolla10:47
*** sdake has quit IRC10:48
*** weiyu has quit IRC10:49
sdake_if ryan really wants to do kolla-kubernetes that wfm, but i'd prefer a well thought out specification on what the work is10:50
sdake_kolla-kubernetes is a major undertaking, and just like kolla-mesos requires majority approval imo :)10:51
*** weiyu has joined #openstack-kolla10:55
*** rhallisey has joined #openstack-kolla11:06
ccesariomorning guys!!11:12
sdake_here is another gate failure that is semi-common11:13
sdake_https://review.openstack.org/#/c/303969/311:13
patchbotsdake_: patch 303969 - kolla - Clarify comment about binding erlang to IPv4 (MERGED)11:13
sdake_happens only on bluebox from what i can tell11:14
sdake_hi ccesario11:14
sdake_rhallisey can you ack that revert patch at the top of the queue11:14
rhalliseyalready did11:14
sdake_and submit a new patch once the revert hits the repo11:14
ccesariosdake_, hello....11:14
sdake_thanks11:14
rhalliseyshouldn't have been merged..11:14
sdake_sorry about that totaly my fault11:14
sdake_i didn't tell vikram about our workflow for specs11:14
rhalliseysdake_, nah I should've -111:14
rhalliseyd it11:14
ccesariosdake_, I saw your and Jeffrey4l_  comments....11:15
sdake_still my fault ;)11:15
rhalliseyit was still WIP.  Didn't make it clear11:15
rhalliseywhatever it's fine11:15
*** weiyu has quit IRC11:15
Jeffrey4l_ccesario, cool. what do you think?11:15
*** tfukushima has quit IRC11:15
sdake_when yu get your spec resubmitted we can get action on the reviews going11:15
rhalliseyya agreed11:15
sdake_so you can actually get a well written spec ready :)11:15
rhalliseyya that one was stale11:16
ccesariowell, to clarify for me....  So, the ideia is make is dynamic using  kolla_docker.py !?11:16
sdake_i would hae done thta already but i didn't notie it11:16
sdake_ccesario right11:16
sdake_ccesario i dont know precisely how to implement it11:16
ccesarioJeffrey4l_, I agree with you11:16
Jeffrey4l_ccesario, correct. just check the build_container function part.11:16
ccesariosdake_, I know11:16
sdake_but i think Jeffrey4l_ has a good solution11:16
ccesarioJeffrey4l_, based on container name ?11:16
Jeffrey4l_ccesario, yes. it will be enough, with less confused.11:17
Jeffrey4l_when `docker exec -it nova_api bash`, show the container name is helpful. show other name will be confused.11:18
ccesarioJeffrey4l_, sdake_ well... I have this running in my env....   please look this and check if you agree..   http://paste.openstack.org/show/493631/11:18
sdake_Jeffrey4l_ even author agrees spec was prematurely merged11:18
sdake_https://review.openstack.org/#/c/304010/11:18
patchbotsdake_: patch 304010 - kolla - Revert "Spec: Deploy Kolla images using Kubernetes"11:18
sdake_please lets get this fixed asap11:19
sbezverksdake have you seen my reply to your comment? is it sufficient to change your mind from +1 to +2 ?? ;-)11:19
sdake_sbezverk didn't see it11:19
sdake_link?11:19
sdake_sbezverk i've been tyring to cleanup the review queue and get the backports going all evening11:20
sdake_hey look its 4:2011:20
sbezverkhttps://review.openstack.org/#/c/300004/1811:20
rhalliseysdake_, when does the sun come up in arizona these days?11:20
patchbotsbezverk: patch 300004 - kolla - Add  iscsid/tgtd specific parameters to Kolla infr...11:20
sdake_no idea11:20
sdake_burning midnight oil today11:20
Jeffrey4l_ccesario, your code has bug. the environment variable is ignored.11:21
sdake_sbezverk where is it duplicated?11:21
sdake_github link?11:21
openstackgerritRyan Hallisey proposed openstack/kolla: Set db connection retry to infinity  https://review.openstack.org/30243411:22
ccesarioJeffrey4l_, well... look this11:22
ccesariohttp://paste.openstack.org/show/493632/11:22
ccesarioline 13811:22
sbezverksdake iscsi uses port 326011:22
sdake_looks good rhallisey  :)11:23
rhalliseycool11:23
ccesarioJeffrey4l_, I have deployed and all container have the KOLLA_SERVICE_NAME env var11:23
sbezverkif you check the right side11:23
sbezverkwe check already for the right group using config variable11:23
sdake_line 138 of what sbezverk11:23
sdake_line 138 on the right side is checking keystone11:24
Jeffrey4l_sorry, i make a mistake. Your code works. bug this line is useless env = self.params.get('environment')11:24
Jeffrey4l_sorry..11:24
Jeffrey4l_i am wrong.11:24
sbezverksdake line 15411:24
ccesarioJeffrey4l_, no replace _ to - ?11:24
sdake_sbezverk thts to check if the port is free11:24
ccesariokeep original name ?11:24
sdake_got it11:25
sdake_issed that11:25
Jeffrey4l_it is a diff file, i just aware.11:25
sdake_missed that11:25
Jeffrey4l_OK. the only question: should we request replate _ to -?11:25
Jeffrey4l_ccesario, ^11:26
sdake_sbezverk ok all set11:26
sdake_sbezverk btw i added you to kolla group on fedorapeopole.org11:26
ccesarioJeffrey4l_, I don't know ... I jus replace to see better in PS1 var11:26
sdake_or whereever the right url is11:26
*** mlima has joined #openstack-kolla11:28
ccesarioJeffrey4l_,  may I send the patch set based in this code!?11:28
Jeffrey4l_ccesario, cool. btw, your PS is very helpful for operators. thanks.11:29
ccesarioJeffrey4l_, thanks.... I 'll provide it in few minutes11:29
Jeffrey4l_OK.11:29
ccesariothanks a lot guys by infos11:29
sdake_Jeffrey4l_ now that we have fixed the epmd + erlang + rabbitmq crash a common gate fialure occurs only on bluebox11:30
sdake_http://logs.openstack.org/40/299540/1/check/gate-kolla-dsvm-deploy-centos-source/6c69e15/console.html#_2016-04-11_09_06_37_45411:30
sdake_i have seen this a whoel lot today11:30
sdake_every failed job is bluebox + link local waiting for keepalived11:30
openstackgerritMerged openstack/kolla: Revert "Spec: Deploy Kolla images using Kubernetes"  https://review.openstack.org/30401011:30
sdake_i'm thinking multicast is turned off possibly?11:31
Jeffrey4l_sdake_, i am confused. let me checking the log more carefully.11:32
sdake_keepalived uses multicast11:34
sdake_if multicast is disabled i  suppose its possible for keepalived not to assign a link local address11:34
Jeffrey4l_sdake_, first of all, this should not related to the fix of rabbitmq. The rabbitmq even not be ran11:34
Jeffrey4l_sdake_, maybe. may be need some debug/test for this issue in the gate.11:35
sdake_i agree this is a different issue11:35
sdake_our gates have been terribly flakey11:35
*** andrewvwebber has quit IRC11:35
sdake_they revolve around the epmd crash11:35
sdake_and the bluebox provider11:35
sdake_the bluebox provider blows up on waiting for vip11:35
*** Yogesh_Deshmukh has joined #openstack-kolla11:36
Jeffrey4l_yea. so many test nodes without the same configuration, which is annoying.11:37
sdake_Jeffrey4l_ http://logs.openstack.org/40/299540/1/check/gate-kolla-dsvm-deploy-centos-source/6c69e15/logs/kolla/keepalived/keepalived.log11:38
sdake_check out the bogus vrrp11:39
Jeffrey4l_invalid password ? interesting.11:40
sdake_yup11:41
sdake_bluebox only11:41
sdake_i have never seen it elsewhere11:41
sdake_i'll file bug for that11:42
sdake_moment11:42
Jeffrey4l_maybe, there are two kolla tests running on bluebox. they can communicate. But we have different keepalived password for different test( gernerated randomly). So one keepalived received a vrrp package with wrong password?11:42
Jeffrey4l_is this possible?11:42
*** jtriley has joined #openstack-kolla11:43
Jeffrey4l_sdake_, when only one keepalived instance. it should work even the vrrp packet is blocked.11:45
sdake_wackamole: https://bugs.launchpad.net/kolla/+bug/156881811:46
openstackLaunchpad bug 1568818 in kolla mitaka "bluebox gate generates invalid password in keepalived logs" [High,Confirmed]11:46
sdake_Jeffrey4l_ this is possible11:47
Jeffrey4l_sdake_, notice this line (bogus VRRP packet received on eth0 !!!). I am a little sure for that.11:47
Yogesh_DeshmukhHi, I'm deploying kolla on 3 node setup i.e. 1 control, compute and network node.11:47
Yogesh_Deshmukhwithout haproxy, I'm giving control node first interfaces IP address in kolla_internal_vip_address in globals.yml.11:47
sdake_hey Yogesh_Deshmukh11:47
Yogesh_DeshmukhIs this correct?11:48
sdake_why not run with haproxy?11:48
*** jtriley has quit IRC11:48
sdake_running without haproxy really only works well on AIO deploys11:48
sdake_i wouldnt recommend running without haproxy at all personally11:48
Yogesh_DeshmukhFor single control node do we need haproxy?11:48
sdake_I never run kolla that way and neither do a majority of the developers11:48
sdake_haproxy sets up your internal and external VIPs11:49
sdake_it could probablybe made to work without haproxy but why bother?11:49
sdake_worried about performance?11:49
*** daneyon_ has joined #openstack-kolla11:50
sdake_i am petty sure but not certian you cannot do mutinode of any type without haproxy11:50
sdake_the person to talk to on this subject is inc011:50
sdake_or dave-mccowan11:50
Yogesh_DeshmukhYeah, but I'm running for testing purpose.11:50
sdake_ya test with haproxy then ? :)11:50
Yogesh_DeshmukhOk, will run with haproxy.11:50
Yogesh_Deshmukhwhat will be in kolla_internal_vip_address?11:51
Yogesh_Deshmukhany random non exist IP?11:51
sdake_yes but it must be routable from your management network interface11:52
*** vivek__ has joined #openstack-kolla11:53
*** daneyon_ has quit IRC11:54
*** wmiller has joined #openstack-kolla11:57
Jeffrey4l_sdake_, let me try to use a random `keepalived_virtual_router_id` in the gate test.11:59
Jeffrey4l_I will this should fix this issue.11:59
*** sdake_ is now known as sdake11:59
Jeffrey4l_s/will/think/11:59
*** mgoddard_ has quit IRC12:00
sdakeinteresitng idea Jeffrey4l_12:01
*** allen_gao has quit IRC12:04
sdakeJeffrey4l_ you mind assigning that bug to yourself then since your looking into it?12:04
sdakei'd say priortiize it after the 15th12:04
*** allen_gao has joined #openstack-kolla12:04
sdakebut dontworry about it prior12:05
Jeffrey4l_sdake, I have already done that. :D12:06
sdakecool12:06
sdakepretty sure this is our last gate failure12:07
sdakei am not sure i bluebox always fails12:07
sdakeor if there are colocated jobs in bluebox12:07
Jeffrey4l_how about the keystone issue in the gate. is it fixed?12:09
Jeffrey4l_lets push the fix for keepalived and see what will happen.12:09
*** sdake_ has joined #openstack-kolla12:10
sbezverksdake if you have time please check this: https://review.openstack.org/#/c/303858/12:11
patchbotsbezverk: patch 303858 - kolla - Rabbitmq Upgrade restarts container only when imag...12:11
*** sdake has quit IRC12:12
openstackgerritJeffrey Zhang proposed openstack/kolla: Use a random virtual_router_id in the keepalived  https://review.openstack.org/30408012:13
*** dave-mccowan has joined #openstack-kolla12:14
openstackgerritJeffrey Zhang proposed openstack/kolla: Use a random virtual_router_id in the keepalived  https://review.openstack.org/30408012:14
*** sdake_ has quit IRC12:18
*** achanda has quit IRC12:18
openstackgerritMartin André proposed openstack/kolla: Clarify comment about binding erlang to IPv4  https://review.openstack.org/30409112:26
Yogesh_Deshmukhsdake, now I'm running with haproxy but it is giving error on 'Waiting for MariaDB service to be ready through VIP'12:32
Yogesh_DeshmukhCan't connect to MySQL server on '192.168.20.205' (113 "No route to host")12:32
Yogesh_Deshmukhthe management IP is 192.168.20.202. i.e. 205 is on same network of management. I'm deplyoing Kolla on VMs.12:34
*** achanda has joined #openstack-kolla12:35
*** achanda has quit IRC12:35
*** salv-orl_ has quit IRC12:36
sbezverkYogesh_Deshmukh check if you can ping 192.168.20.205, check if keepalived is running.12:36
sbezverkYogesh_Deshmukh also check for haproxy container12:37
Yogesh_Deshmukhok12:39
*** mgoddard has joined #openstack-kolla12:40
*** achanda has joined #openstack-kolla12:44
*** jtriley has joined #openstack-kolla12:44
sbezverkrhallisey if you have a minute could please you check this: https://review.openstack.org/#/c/300004/1812:47
patchbotsbezverk: patch 300004 - kolla - Add  iscsid/tgtd specific parameters to Kolla infr...12:47
*** mwheckmann has joined #openstack-kolla12:47
rhalliseysbezverk, looks great12:50
sbezverkrhallisey thank you.. There are two other parts which also need reviews, but I do not want to be pushy ;-)12:51
rhalliseyya I'm lookin12:51
rhalliseyjust a minute12:51
openstackgerritCarlos Cesario proposed openstack/kolla: Identify container through  PS1 shell variable  https://review.openstack.org/29798412:52
ccesarioJeffrey4l_, ^^12:53
*** ByPasS has joined #openstack-kolla12:54
rhalliseysbezverk, do you use ubuntu or centos?12:55
rhalliseywhich have you been testing on12:55
rhalliseyas a base OS not a base container os12:56
sbezverkrhallisey tested both12:56
rhalliseydid you have any issue with /dev/pts/ptmx ?12:56
rhalliseycentos by default sets that to 00012:57
sbezverkrhallisey nope. never seen it, the only issue we hit on ubuntu, configfs has not been mounted by default12:57
*** inc0 has joined #openstack-kolla12:57
*** klint has quit IRC12:57
rhalliseyhttps://www.kernel.org/doc/Documentation/filesystems/devpts.txt12:58
rhalliseysbezverk, I think it be worth mentioning in docs somewhere to set this12:59
rhalliseyI'd imagine someone will run into this12:59
sbezverkrhallisey from the first read I do not see how it is applicable to our case, but I will read it again..13:01
rhalliseysbezverk, I'm just saying that I have seen ptmx be 00013:04
sbezverkrhallisey got it..13:04
rhalliseyand it will cause iscsi to fail to connect13:04
sbezverkrhallisey which was flavor/version and kernel version you have seen this condition?13:06
rhalliseyI saw it in centos-atomic from ~month ago13:06
rhalliseyhad the same issue when I did cinder way back13:06
Yogesh_Deshmukhsbezverk, I checked, 192.168.20.205 is pinable. The haproxy and keepalived containers are present only on network node. Is that correct?13:07
sbezverkYogesh_Deshmukh ah that is where I think your problem is, haproxy and keepalived should also be on controller node13:07
sbezverkbasically their purpose to protect your control plane from a failure13:08
sbezverkrhallisey Interesting, never played with atomic flavor.. will keep that in mind. I have PS that adds documentation for iscsi, I will add a link to the page you gave me..13:09
*** ccesario___ has quit IRC13:09
rhalliseysbezverk, yes please13:10
*** stvnoyes has left #openstack-kolla13:11
*** ccesario___ has joined #openstack-kolla13:11
Yogesh_Deshmukhsbezverk ahh, I also thought the same issue but in inventory file for multinode, [haproxy:children] network.13:12
openstackgerritCarlos Cesario proposed openstack/kolla: Identify container through  PS1 shell variable  https://review.openstack.org/29798413:13
Yogesh_DeshmukhSo haproxy and keepalived  are only deployed on network. I'm confused here.13:13
Yogesh_Deshmukhthe haproxy deployment is conditional include: config.yml when: inventory_hostname in groups['haproxy'] in ansible/roles/haproxy/tasks/deploy.yml13:15
Yogesh_DeshmukhHow haproxy is also deployed on control node?13:15
sbezverkYogesh_Deshmukh yes13:16
sbezverkrhallisey I just checked my target server and it does have /dev/pts/ptmx  : c---------. 1 root root 5, 2 Apr 11 09:59 /dev/pts/ptmx13:18
rhalliseysbezverk, so then it worked with that13:18
rhalliseyhm13:18
inc0good morning13:19
sbezverkrhallisey yep13:19
sbezverkit is not centos, it is rhel 7.2 but it should be the same thing..13:20
sbezverkgood morning inc013:21
rhalliseymaybe scsi isn't create a session in this case.. idk13:21
inc0https://www.youtube.com/watch?v=PivpCKEiQOQ&feature=youtu.be I hope all of you have seen it?13:22
rhalliseysbezverk, did you try multinode?13:22
rhalliseyidk if that would make a difference13:23
rhalliseysbezverk, if you haven't I'll give it a try on centos and see13:23
*** ccesario___ has quit IRC13:23
*** ccesario___ has joined #openstack-kolla13:24
sbezverkrhallisey nope, do not have h/w yet13:25
sbezverkif you could test, that would be fantastic13:26
rhalliseysbezverk, sure13:26
rhalliseysbezverk, I actually don't have the h/w either. I cheat :)13:26
rhalliseywhen I finish the heat template I got I'll share them for dev purposes13:27
sbezverkrhallisey how ? please share? VMs?13:27
rhalliseycould also use vagrant13:27
rhalliseysbezverk, I use virt on virt with tripleo13:28
rhalliseydeploy nodes with ironic13:28
rhalliseybut they are 'vms'13:28
rhalliseyworks great13:28
rhalliseyI'm not familiar with vagrant so this works for me13:29
*** banix has joined #openstack-kolla13:30
sbezverkrhallisey Interesting, will look into virt with tripleo, thank you for info..13:32
rhalliseysbezverk, if you do look into it. https://github.com/rthallisey/kolla-heat-templates13:33
rhalliseysbezverk, I'll try and come out with docs after I smooth eveything out in the templates13:34
*** salv-orlando has joined #openstack-kolla13:37
*** alyson_ has joined #openstack-kolla13:38
*** daneyon_ has joined #openstack-kolla13:38
*** daneyon_ has quit IRC13:42
*** salv-orlando has quit IRC13:43
openstackgerritCarlos Cesario proposed openstack/kolla: Identify container through  PS1 shell variable  https://review.openstack.org/29798413:45
*** Hui has joined #openstack-kolla13:46
*** achanda has quit IRC13:50
*** achanda has joined #openstack-kolla13:58
ccesariorhallisey, Kolla Triple-O ?! O_o13:58
rhalliseyccesario, I use it for multinode13:58
rhalliseysince I don't have the hardware13:58
rhalliseyI deploy stock undercloud and use my own heat-template to deploy nodes13:59
ccesariorhallisey, ohhh nice to know .. :)13:59
rhalliseyfeed those ips into kolla and done, multinode13:59
ccesarioI don't have hardwarde too.... but I usage other method :P13:59
rhalliseyyou just leave the undercloud around and keep making new stacks13:59
pabelangermorning, we'd like to land https://review.openstack.org/#/c/303414/ if possible today. Its a requirement for openstack-infra to migrate kolla from devstack-centos7 to centos-713:59
patchbotpabelanger: patch 303414 - kolla - Make sure /etc/hostname is valid for SELinux13:59
rhalliseyccesario, ya this method is heavy I'd say14:00
pabelangersee https://review.openstack.org/#/c/301320/ for the openstack-infra change14:00
patchbotpabelanger: patch 301320 - openstack-infra/project-config - Migrate kolla jobs to centos-7 dib14:00
rhalliseyccesario, allows you to use ironic which is sweet14:00
ccesariorhallisey, VMware -> Virtual Machine - > 4 KVM machines to play Kolla :)14:01
*** ccesario___ has quit IRC14:01
ccesariorhallisey, nice...14:01
rhalliseyccesario, cool14:01
ccesariorhallisey, nested over nested :P14:01
rhalliseyccesario, I know :)14:02
ccesario:)14:03
*** achanda has quit IRC14:07
*** mgoddard has quit IRC14:07
*** mgoddard has joined #openstack-kolla14:08
*** bmotz has left #openstack-kolla14:18
*** bmotz has joined #openstack-kolla14:18
*** dwalsh has joined #openstack-kolla14:24
*** reddy has quit IRC14:24
*** ccesario___ has joined #openstack-kolla14:30
*** haplo37 has joined #openstack-kolla14:35
*** vhosakot has joined #openstack-kolla14:37
*** tpeoplesvacation is now known as tpeoples14:40
*** Jeffrey4l_ has quit IRC14:51
*** inc0 has quit IRC14:59
*** mgoddard_ has joined #openstack-kolla15:01
*** bmotz_ has joined #openstack-kolla15:02
*** Serlex has quit IRC15:02
*** salv-orlando has joined #openstack-kolla15:03
*** salv-orlando has quit IRC15:03
*** salv-orlando has joined #openstack-kolla15:04
*** mgoddard has quit IRC15:04
*** wmiller has quit IRC15:04
*** wmiller has joined #openstack-kolla15:05
*** bmotz has quit IRC15:05
openstackgerritRyan Hallisey proposed openstack/kolla: Spec: Deploy Kolla images using Kubernetes  https://review.openstack.org/30418215:13
*** Jeffrey4l_ has joined #openstack-kolla15:15
*** achanda has joined #openstack-kolla15:15
*** inc0 has joined #openstack-kolla15:16
*** openstackgerrit has quit IRC15:18
*** openstackgerrit has joined #openstack-kolla15:18
*** The_Ball has quit IRC15:20
*** daneyon_ has joined #openstack-kolla15:26
mlimavhosakot, can you remove the -1 these patchsets https://review.openstack.org/#/c/303055/ and https://review.openstack.org/#/c/303575/ ?15:33
patchbotmlima: patch 303055 - kolla - Remove /etc/kolla configuration folders15:33
patchbotmlima: patch 303575 - kolla - Add a role to cleanup containers after deployment15:33
vhosakotmlima: yep, reviewing them now... few ming15:34
vhosakotmins15:34
*** Yogesh_Deshmukh has quit IRC15:34
mlimavhosakot, thank you :)15:34
vhosakotcool, np15:34
openstackgerritPaul Belanger proposed openstack/kolla: Make sure /etc/hostname is valid for SELinux  https://review.openstack.org/30341415:38
pabelangerplease review ^ per your requests15:39
vhosakotmlima: np!15:41
*** mimizone has joined #openstack-kolla15:41
*** yogesh_deshmukh has joined #openstack-kolla15:45
gmmahavhosakot: mlima: quick Q about https://bugs.launchpad.net/kolla/+bug/156653815:45
openstackLaunchpad bug 1566538 in kolla "generate_password.yml does not add new keys added to deployment" [Undecided,Incomplete] - Assigned to Mauricio Lima (mlima) (mauricio-lima)15:45
gmmahaYou have mentioned that it works in the latest master, but i havent seen any changes in the generate file since the last time i tested.15:46
mlimayes gmmaha15:46
gmmahamaybe i am not explaining the scenario properly.15:46
gmmahalets say i have an old install of kolla and there are new variables being added to the passwords.yml file. if i dont get the new values into /etc/kolla/passwords.yml, they will never get injected into the file15:47
gmmahalet me add the explanation to the bug.. I am sure you guys are busy with the release. maybe that might clarify15:47
*** dwalsh has quit IRC15:47
mlimaI think we need to define a template to report bugs15:48
mlima:)15:48
gmmahamlima: or i have to get a better hold on my english :(15:48
mlimano no gmmaha15:48
mlimaI mean that many bugs does not have a step by step to test and confirm the bug.15:49
mlimaEnvironment15:49
gmmahamlima: true that. :)15:49
mlimaUbuntu/Source or Centos/binary15:49
mlimathis is very useful15:50
gmmahamlima: aah yes.. agreed..15:51
gmmahathanks.. will add them all15:51
vhosakotgmmaha: cool thanks!15:52
*** dwalsh has joined #openstack-kolla15:53
gmmahavhosakot: mlima: added to the bug. thanks15:57
mlimaok15:57
vhosakotgmmaha: ok, thanks!15:57
*** mikelk has quit IRC15:58
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix the gate lying  https://review.openstack.org/30420515:59
mlimagmmaha, can you test using the master branch and instead of using "tools / generate_passwords.py" use "kolla-genpwd"16:01
gmmahamlima: sure.. i can test master shortly16:01
gmmahabut i thought generate_passwords.yml was a symlink to kolla-genpwd16:01
gmmahabut yes, will definitely test it today and let you know if its different16:02
*** inc0_ has joined #openstack-kolla16:08
pmisiakhi guys16:09
*** openstackstatus has joined #openstack-kolla16:09
*** ChanServ sets mode: +v openstackstatus16:09
pmisiakthere are still problems with nova/instances access rights16:09
pmisiakcentos/source16:10
pmisiakqemu user is in nova group16:10
pmisiakbut /var/lib/nova and /var/lib/nova/instances has wrong rights16:10
pmisiakfolder /var/lib/nova hos no rights for group16:11
*** inc0 has quit IRC16:11
pmisiakfolder /var/lib/nova/instances nas no write rights for group16:11
pmisiakand it is after applying this commit: https://github.com/openstack/kolla/commit/ad1461f46d730ed0393ffcb16b866ed2e0f3494316:12
*** dwalsh has quit IRC16:12
pmisiakI had to do: chmod g+rx /var/lib/nova and chmod -R g+w /var/lib/nova/instances to be able to spawn a VM16:13
*** mbound has quit IRC16:13
*** mbound has joined #openstack-kolla16:14
openstackgerritMerged openstack/kolla: Make sure /etc/hostname is valid for SELinux  https://review.openstack.org/30341416:14
*** mbound has quit IRC16:18
*** mark-casey has joined #openstack-kolla16:19
*** ccesario___ has quit IRC16:21
yogesh_deshmukhHi, I stuck at 'Creating the Glance service and endpoint' when deploying Kolla on 3 Node setup (each control, compute and network).16:22
yogesh_deshmukhI looked for ansible.log from kolla_toolbox container it is showing 'Could not determine a suitable URL for the plugin' errror message.16:22
yogesh_deshmukhAny idea?16:23
*** dwalsh has joined #openstack-kolla16:24
rhalliseyyogesh_deshmukh, that sounds like a configuration issue16:27
rhalliseysomething with you vips not be available on all nodes16:27
rhalliseyor something16:27
openstackgerritMerged openstack/kolla: Increase max pool size so conductor doesn't implode  https://review.openstack.org/30367716:27
yogesh_deshmukhrhallisey, how to check vip is working fine?16:29
*** unicell has quit IRC16:30
rhalliseyyogesh_deshmukh, ping the addres you specified16:30
rhalliseyfor each of your hosts16:30
*** ccesario___ has joined #openstack-kolla16:30
yogesh_deshmukhrhallisey, I pinged vip from all i.e. 3 nodes that is working fine.16:32
yogesh_deshmukhhaproxy and keepalived containers are deployed only on control node, is that correct or do we need them on all nodes?16:34
rhalliseythat's fine16:34
*** Hui has quit IRC16:40
*** Marga_ has joined #openstack-kolla16:42
*** inc0_ has quit IRC16:44
*** ccesario___ has quit IRC16:44
*** Marga_ has quit IRC16:51
*** Marga_ has joined #openstack-kolla16:52
*** Marga_ has quit IRC16:53
*** yogesh_deshmukh has quit IRC16:53
*** unicell has joined #openstack-kolla16:58
*** rajathagasthya has joined #openstack-kolla17:00
*** Marga_ has joined #openstack-kolla17:00
*** Marga_ has quit IRC17:00
*** Marga_ has joined #openstack-kolla17:01
*** daneyon_ has quit IRC17:01
*** inc0 has joined #openstack-kolla17:04
ccesarioJeffrey4l_, just reply your questions17:06
*** ccesario___ has joined #openstack-kolla17:11
*** unicell has quit IRC17:15
*** mgoddard has joined #openstack-kolla17:16
*** mgoddard_ has quit IRC17:17
*** unicell has joined #openstack-kolla17:17
*** ccesario___ has quit IRC17:20
*** Marga_ has quit IRC17:21
*** ccesario___ has joined #openstack-kolla17:21
*** Marga_ has joined #openstack-kolla17:23
*** Marga_ has quit IRC17:28
*** jasonsb has joined #openstack-kolla17:30
*** daneyon_ has joined #openstack-kolla17:31
*** neophy has joined #openstack-kolla17:40
*** ccesario___ has quit IRC17:41
*** shardy has quit IRC17:49
vhosakotyogesh_deshmukh: did you run prechecks ?17:49
*** JoseMello has joined #openstack-kolla17:50
vhosakotyep haproxy and keepalived on control nodes is correct17:51
*** shardy has joined #openstack-kolla17:53
*** achanda has quit IRC17:53
*** achanda has joined #openstack-kolla17:56
*** ccesario___ has joined #openstack-kolla17:56
*** daneyon_ has quit IRC17:58
*** Marga_ has joined #openstack-kolla18:00
*** achanda_ has joined #openstack-kolla18:04
*** achanda has quit IRC18:04
*** daneyon_ has joined #openstack-kolla18:04
*** daneyon_ has quit IRC18:08
*** inc0 has quit IRC18:11
*** sdake has joined #openstack-kolla18:18
*** shardy is now known as shardy_afk18:20
*** jasonsb has quit IRC18:21
*** rajathagasthya has quit IRC18:21
*** achanda_ has quit IRC18:21
*** achanda has joined #openstack-kolla18:21
sdakehey folks18:27
*** achanda has quit IRC18:27
*** achanda has joined #openstack-kolla18:30
openstackgerritMerged openstack/kolla: Clarify comment about binding erlang to IPv4  https://review.openstack.org/30409118:33
*** vhosakot has quit IRC18:36
*** vhosakot has joined #openstack-kolla18:36
*** achanda has quit IRC18:36
openstackgerritMerged openstack/kolla: Fix the neutron reconfigure issue when enable linuxbridge-agent  https://review.openstack.org/30378318:38
*** vhosakot has quit IRC18:39
*** vhosakot has joined #openstack-kolla18:39
*** vhosakot has quit IRC18:59
*** vhosakot has joined #openstack-kolla18:59
mlimaheu sdake19:05
mlimahey*19:05
sdakehey mlima19:05
mlima:)19:05
mlimasdake, i think that this bug https://bugs.launchpad.net/kolla/+bug/1534547 was fixed.19:07
openstackLaunchpad bug 1534547 in kolla mitaka "compare_environment compares incorrect" [High,Triaged]19:07
sdakemlima thanks do you have a review to point at?19:08
daneyonvhosakot sdake my builds are still failing. Here is the latest https://bugs.launchpad.net/kolla/+bug/156765719:08
openstackLaunchpad bug 1567657 in kolla "Build Fails Using Tunk" [Undecided,New]19:08
mlimaone sec19:08
sdakedaneyon are you using https proxy properly ?19:09
sdakedave-mccowan can help you19:09
mlimasdake, https://review.openstack.org/#/c/283887/19:09
patchbotmlima: patch 283887 - kolla - Switch ceph to use storage network (MERGED)19:09
sdakemlima studyign patch19:11
daneyonsdake thx, i'll sync with dave-mccowan19:11
vhosakotdaneyon: i have seen that error too rarely on Ubuntu... I just rebuild and it works... did you rebuild ? may the mirror/repo was down19:12
vhosakotdaneyon: HTTPConnectionPool(host='tarballs.openstack.org', port=80): Read timed out.19:12
daneyonvhosakot that error is about 2 hours old19:13
vhosakotdaneyon: ah ok19:13
vhosakotdaneyon: what container threw this error ?19:14
daneyonvhosakot do i need to wipe out my old docker images when i do a rebuild?19:14
vhosakotdaneyon: just rebuidl what failed19:14
daneyonvhosakot error occured right after INFO:kolla.cmd.build:neutron-base:Pushed successfully19:15
vhosakotdaneyon: this is that I do... kolla-build --base ubuntu --type source --registry <registry IP>:4000 <container name that failed>19:15
vhosakotdaneyon: can you send the command you ran ? did you use --no-cache ?19:16
daneyonvhosakot this is what i do > # kolla-build --type source --registry 10.30.118.91:4000 --push19:16
mlimaok sdake19:17
daneyonvhosakot I just updated the bug, so you can see a bit more info19:19
vhosakotdaneyon: --base is centos ?19:19
daneyonvhosakot i believe so, the docs state cent7 is def19:19
daneyonlooks like the error is related to building the magnum image19:20
vhosakotyep.. cent is def19:20
vhosakotdaneyon: can you try this ?19:20
vhosakotdaneyon:    kolla-build --type source --registry 10.30.118.91:4000 --push magnum19:20
*** salv-orlando has quit IRC19:21
daneyonvhosakot doing it now19:23
vhosakotdaneyon: also, send entire output of kolla-build19:27
daneyonvhosakot: looks like magnum images built https://gist.github.com/danehans/c1a6d7178c820dc35af5dcabd5d3da8119:27
vhosakotyes, all three were built - magnum-api, magnum-base and magnum-conductor.. cool19:28
vhosakotdaneyon: now, can you rerun     kolla-build --type source --registry 10.30.118.91:4000 --push19:28
daneyonvhosakot: in progress kolla-build --type source --registry 10.30.118.91:4000 --push19:29
vhosakotdaneyon: passed the point where you saw failure last time ?19:30
daneyonvhosakot looks like it19:30
daneyondave-mccowan fyi ^ looks like vhosakot helped me get past my latest build issue19:31
vhosakotdaneyon: cool, BTW, the "Make sure Docker is running and that you have the correct privileges to run Docker (root)" in the bug may be due to permission issue in local env19:31
*** Jeffrey4l__ has joined #openstack-kolla19:32
vhosakotdaneyon: let the build finish... some images fail due to intermittent mirror/repo issues from where they are pulled19:33
*** neophy has quit IRC19:34
vhosakotdaneyon: usually, a retry works fine19:34
*** Jeffrey4l_ has quit IRC19:34
daneyonvhosakot: OK, I'll start doing that. Thanks again for all the help.19:34
vhosakotcool, np...19:34
vhosakotdaneyon: by default, kolla-buila retried three times... you can control that by passing the --retries to kolla-build19:35
vhosakotretires*19:35
sdake mllima got toallly sidetracked19:36
*** mwheckmann has quit IRC19:41
vhosakotdaneyon: when you see the error, should you pree CTRL+C to exit ?19:43
sdakemlima i am not confident the bug you linked is the same issue reported19:48
sdakerather the review you linked matche the bug reported19:49
mlimahmm, "But second tasks has no environment variables, and compare_environment make decision that environment is not different."19:49
mlimait is the bug summary ^19:50
sdakedidnt 'caatch that19:50
sdakejust woke up19:50
sdakegive me a minute to look again19:50
sdakemlima so your saying that review addresses the second task?19:51
sdakeits really hard for me to tell19:51
sdakefrom the review19:51
sdakei see it now19:51
sdakemlima good spot!!19:51
mlimathe bug was reported in 2016-01-15 and this patchset https://review.openstack.org/#/c/283887/ was merged in 2016-01-2419:52
patchbotmlima: patch 283887 - kolla - Switch ceph to use storage network (MERGED)19:52
mlima2016-02-24**19:52
mlimawell, the environment variables are there sdake19:53
sdakeya that patch fixes multiple things19:53
sdaketrivialfix abuse at it finest ;)19:53
mlimahahaha19:53
*** salv-orlando has joined #openstack-kolla19:54
mlimasuper patchset19:54
mlima:)19:54
sdakemlima i yuo find more already fixed please do let me know ;)19:54
mlimaok,  I was looking for a bug to fix and i found it19:55
mlima-1 High bug to fix sdake :)19:57
openstackgerritCarlos Cesario proposed openstack/kolla: Identify container through  PS1 shell variable  https://review.openstack.org/29798419:58
sdakeagreed19:58
sdakeevery thing counts at this point19:58
sdake1 week to go19:58
mlima:)19:58
sdakealso openstack-inra is blocked on us releasing19:58
sdakeso we are prety hard committed to 4/15 at this point19:59
sdakeso their work doesn't bleed over into summit19:59
mlimakolla team is working hard :)19:59
sdakei knwo everyone is19:59
*** achanda has joined #openstack-kolla19:59
sdakejust pointing out there will be no more slips :)19:59
sdakepabelanger ^^19:59
sbezverksdake how about this one: it was marked as a critical? https://review.openstack.org/#/c/303858/20:01
patchbotsbezverk: patch 303858 - kolla - Rabbitmq Upgrade restarts container only when imag...20:01
sdakei really want inc0 to review that one20:02
sbezverksdake ok got it20:02
sdakei'd like inc0 to take lead on upgrade review changes20:02
sdakerather reviewing upgrade changes20:02
sdaketo be more clear20:02
sdakeit is also failing the gate sbezverk  - i jsut triggered a recheck on it20:03
sbezverksdake I am sorry but fixing haproxy and VIP failing at the gate is out of scope of this bug ;-)20:05
sdakesbezverk was it bluebox?20:05
*** openstackgerrit has quit IRC20:05
sdakesbezverk agree - that is why we recheck until gate is green20:05
sdakewe are working on the keepalived bug separately20:05
*** inc0 has joined #openstack-kolla20:05
sdaketo recheck a failed patch just type recheck in the review and click reply20:05
sdakeinc0 sbezverk has a patch for you to review20:06
sdakeand i'd really like if you take first to review on upgrade patches in this last week of our releaes20:06
sbezverksdake how do I know if it is bluebox or red box or whatever color that box is ;-) ?20:06
sdakeit says right at the top of the log20:06
sdakebluebox is a cloud provider20:06
sdakeof which there are about 6 i think20:06
sdakethey all have slightly different environments20:07
sbezverksdake that is the problem :-) always go straight to the bottom where the error is ..20:07
inc0ah, issue with gate?20:07
inc0sbezverk, review about iscsi?20:08
ccesariogerrit is it down!>?20:08
inc0sbezverk, I'll take a look later today20:08
inc0I need to run few errands first20:08
sbezverkinc0 thank you..20:08
inc0(buying a new car...)20:08
sdakeyes gerrit down for awhile20:08
sdakeinc0 grats!20:09
inc0or rather chosing one;)20:09
sdakewhat type20:09
sbezverkinc0 I hope it is mazerati20:09
sdakenew car smell rocks ;)20:09
inc02016 mazda miata is my fav now20:09
sdakesbezverk when we bought our panamera i reaally wanted a mazarati20:09
sdakebut wife wouldn't have anything to do with it20:09
sdakemazda makes nice vehicles20:09
inc0yeah20:09
inc0and this one is so much fun20:10
*** banix has quit IRC20:10
inc0I already have new subaru crosstrek, but that will be my wifes car20:11
sdakeo folks20:11
inc0so I'm buying something for myself;) and it doesn't have to be practical20:11
sdakegerrit is down until 21:0020:11
inc0ok20:11
sdakeour gerrit bot updates were not coming because of the channel rename20:11
inc0so I'll review stuff later20:11
sdakebut that has been merged20:11
sdakeso if there is any more gerrrit slip on schedule it should update our channel topic20:11
inc0ok, afk20:12
*** inc0 has quit IRC20:16
*** vhosakot has quit IRC20:17
*** shardy_afk has quit IRC20:18
*** neophy has joined #openstack-kolla20:21
*** achanda has quit IRC20:22
*** dwalsh has quit IRC20:24
*** banix has joined #openstack-kolla20:26
*** vhosakot has joined #openstack-kolla20:27
*** openstackgerrit has joined #openstack-kolla20:27
*** jtriley has quit IRC20:37
*** rstarmer has joined #openstack-kolla20:38
*** unicell has quit IRC20:38
*** openstackgerrit has quit IRC20:41
*** neophy has quit IRC20:53
*** openstackgerrit has joined #openstack-kolla20:55
*** rhallisey has quit IRC20:58
*** banix has quit IRC20:58
*** unicell1 has joined #openstack-kolla21:02
*** daneyon_ has joined #openstack-kolla21:02
*** ccesario___ has quit IRC21:03
*** ccesario___ has joined #openstack-kolla21:03
*** ccesario___ has quit IRC21:06
*** haplo37 has quit IRC21:07
manjeetssdake ping21:07
manjeetsi guess I reported a bug of linux bridge getting out of synce with ml2 both with kolla and neutron which i don't see any more21:08
*** dwalsh has joined #openstack-kolla21:31
sdakemanjeets yesthat is because we merged aa fix for  that i think ;)21:42
sdakedave-mccowan docker exec heka ls /var/log/kolla21:42
manjeetsno sdake that was separate bug21:42
manjeetslinux bridge was working when built from source21:42
manjeetsbut some times ml2 and linux bridge were getting out of sync21:43
manjeetslike 300 vms getting error out of 1000 because21:43
manjeetsone bug was linux bridge was not working totally when built from source  ( thats fixed I guess )21:43
sdakeya Jeffrey4l__ fixed that21:44
manjeetsI reported this bug last friday21:44
sdakei can't tell yu the reviewbecause gerrit is offline21:44
*** ccesario___ has joined #openstack-kolla21:44
manjeetsgerrit is up again and I know that change21:44
vhosakotmanjeets: yes, Jeffrey4l__ fixed the neutron linuxbridge reconfigure bug... https://review.openstack.org/#/c/303783/21:45
patchbotvhosakot: patch 303783 - kolla - Fix the neutron reconfigure issue when enable linu... (MERGED)21:45
openstackgerritSteven Dake proposed openstack/kolla: Fix the neutron reconfigure issue when enable linuxbridge-agent  https://review.openstack.org/30432021:47
*** asalkeld has left #openstack-kolla21:48
manjeetsvhosakot it was something i reported friday21:48
manjeetsI added neutron and kolla both21:48
manjeetsto bug ticket21:48
vhosakotmanjeets: https://review.openstack.org/#/c/302415/ is merged too21:48
patchbotvhosakot: patch 302415 - kolla - Increase max pool size so conductor doesn't implode (MERGED)21:48
sdakemanjeets question21:48
vhosakotmanjeets: can you send the bug ID pls21:48
sdakeis your deployment workingflawlessly now?21:48
manjeetsit was sdake21:49
manjeetsbut linuxbridge was getting ut of sync with ml221:49
manjeetswhile spining 1000 vms21:49
manjeetssome times it worked 1000 vm successfully but after few runs I was having issues21:50
sdakemanjeets is your deployment working flawlessly or are there still problems?21:50
manjeetssdake with ovs it did21:50
manjeetswith lb there were issues21:50
sdakeeven with master?21:51
*** asalkeld_ has quit IRC21:51
sdakeif so do you have logs21:51
*** asalkeld has joined #openstack-kolla21:51
sdakeand a bug id to go on?21:51
manjeetssdake  I am trying to find the bug ticket but not able to find it seems like somebody marked it invalid or duplicate21:51
manjeetsbut it was a different error21:52
sdakecheck your email for the bug id21:52
manjeetssdake it was you who marked that as duplicate21:54
*** ccesario___ has quit IRC21:54
sdakeok well i cna undo that21:54
sdakegive me a bug link21:54
manjeetshttps://bugs.launchpad.net/kolla/+bug/156820221:55
openstackLaunchpad bug 1567406 in kolla "duplicate for #1568202 neutron linux bridge agent is buggy and can not boot instance" [Critical,In progress] - Assigned to Jeffrey Zhang (jeffrey4l)21:55
sdakejust to be clear, this bug exists in master, only for linuxbridge, but not ovs, when spinning 1000 vms21:55
manjeetsspining 1000 vms was working with lb for initial runs21:55
manjeetsbut after two or three runs it lb was getting out of sync with ml221:55
manjeetsand when i was spining 1000 vms for 4th time 300 out of 1000 were ending errir21:56
manjeetserror21:56
*** ccesario___ has joined #openstack-kolla21:56
sdakemanjeets you tried this review: https://review.openstack.org/#/c/302721/21:56
patchbotsdake: patch 302721 - kolla - Fix launch vm failed in neutron-linuxbridge-agent env21:56
sdakein your logs brctl was missing21:56
sdaketht ptch adds brctl21:56
*** ccesario___ has quit IRC21:57
manjeetsbut 1000 vms worked for 2 or 3 runs21:57
manjeetsnot sure why they worked without brctl package21:58
sdakebut your saying its fixed?21:59
sdakeor still busted21:59
sdakewith master21:59
*** ccesario___ has joined #openstack-kolla21:59
manjeetshav not tried current master22:01
*** ccesario___ has quit IRC22:01
*** ccesario___ has joined #openstack-kolla22:01
sdakecould you try and report in that bug22:01
sdakewe hve made alot of fixes22:01
sdakei'd like to make sure we aren't investigating solved problems ;)22:02
sdakenote not everything has been backported in master to stable/mitaka hat needs to be yet22:02
*** ccesario___ has quit IRC22:03
*** ccesario___ has joined #openstack-kolla22:04
*** manjeets is now known as manjeetsingh22:05
dave-mccowansdake if i have multi node, does a heka container run on every node?22:05
sdakeyup22:05
*** ccesario___ has quit IRC22:05
sdakewith relevant logs for that node22:05
*** ccesario___ has joined #openstack-kolla22:05
*** manjeetsingh is now known as manjeets22:06
dave-mccowancool.  which node(s) are the aggregate logs on?22:06
sdakecontrol nodes22:06
sdakeis where elasticsearch runs22:06
sdakeakwasnie or elemoine_ is elastic search in its current implementtion in kolla ha?22:13
*** vhosakot has quit IRC22:14
*** ccesario___ has quit IRC22:17
*** ccesario___ has joined #openstack-kolla22:21
*** dwalsh has quit IRC22:22
*** daneyon_ has quit IRC22:22
*** gfidente has quit IRC22:27
*** rstarmer has quit IRC22:29
*** ccesario___ has quit IRC22:29
*** rstarmer has joined #openstack-kolla22:38
*** rstarmer has quit IRC22:39
*** ccesario___ has joined #openstack-kolla22:39
*** vhosakot has joined #openstack-kolla22:43
*** daneyon_ has joined #openstack-kolla22:43
vhosakotdave-mccowan: http://paste.openstack.org/show/493701/22:47
*** rstarmer has joined #openstack-kolla22:49
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Drop root for swift  https://review.openstack.org/25937422:50
*** sdake_ has joined #openstack-kolla22:53
*** sdake has quit IRC22:55
*** JoseMello has quit IRC23:09
*** rstarmer has quit IRC23:09
*** rhallisey has joined #openstack-kolla23:18
*** salv-orlando has quit IRC23:21
*** banix has joined #openstack-kolla23:21
*** Jeffrey4l__ has quit IRC23:42
*** mwheckmann has joined #openstack-kolla23:47
*** sdake has joined #openstack-kolla23:52
*** sdake_ has quit IRC23:53
*** daneyon_ has quit IRC23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!