| *** vhosakot has joined #openstack-kolla | 00:07 | |
| *** wuhg has joined #openstack-kolla | 00:10 | |
| *** daneyon_ has joined #openstack-kolla | 00:19 | |
| *** daneyon_ has quit IRC | 00:21 | |
| *** daneyon_ has joined #openstack-kolla | 00:23 | |
| *** jtriley has joined #openstack-kolla | 00:25 | |
| *** fragatina has quit IRC | 00:29 | |
| *** jtriley has quit IRC | 00:31 | |
| *** ayoung has joined #openstack-kolla | 00:32 | |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Identify container through PS1 shell variable https://review.openstack.org/297984 | 00:36 |
|---|---|---|
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Identify container through PS1 shell variable https://review.openstack.org/297984 | 00:38 |
| ccesario_ | vhosakot: ^^ | 00:39 |
| *** unicell has joined #openstack-kolla | 00:42 | |
| *** banix has joined #openstack-kolla | 00:45 | |
| *** banix has quit IRC | 00:46 | |
| *** daneyon_ has quit IRC | 00:54 | |
| *** ccesario_ has quit IRC | 01:01 | |
| *** ccesario___ has joined #openstack-kolla | 01:02 | |
| *** daneyon_ has joined #openstack-kolla | 01:05 | |
| *** ccesario_ has joined #openstack-kolla | 01:16 | |
| *** ccesario___ has quit IRC | 01:16 | |
| *** daneyon_ has quit IRC | 01:19 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Put hosts in alphabetical order in site.yml https://review.openstack.org/303873 | 01:19 |
| *** stvnoyes has quit IRC | 01:24 | |
| *** fragatina has joined #openstack-kolla | 01:25 | |
| *** stvnoyes has joined #openstack-kolla | 01:25 | |
| *** banix has joined #openstack-kolla | 01:40 | |
| *** bmotz has quit IRC | 01:42 | |
| *** bmotz has joined #openstack-kolla | 01:42 | |
| *** achanda has joined #openstack-kolla | 01:45 | |
| *** banix has quit IRC | 01:46 | |
| *** weiyu has joined #openstack-kolla | 01:52 | |
| *** jtriley has joined #openstack-kolla | 01:52 | |
| *** daneyon_ has joined #openstack-kolla | 01:54 | |
| *** jtriley has quit IRC | 01:58 | |
| *** ccesario___ has joined #openstack-kolla | 01:58 | |
| *** ccesario_ has quit IRC | 01:58 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 02:02 | |
| *** Jeffrey4l_ has quit IRC | 02:08 | |
| *** daneyon_ has quit IRC | 02:16 | |
| *** pbourke has quit IRC | 02:16 | |
| *** pbourke has joined #openstack-kolla | 02:17 | |
| *** banix has joined #openstack-kolla | 02:19 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 02:22 | |
| *** sdake has joined #openstack-kolla | 02:24 | |
| *** sdake_ has joined #openstack-kolla | 02:27 | |
| *** sdake has quit IRC | 02:28 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Add prechecks to validate kolla_internal_vip_address and kolla_external_vip_address https://review.openstack.org/302026 | 02:38 |
| *** weiyu has quit IRC | 02:38 | |
| *** banix has quit IRC | 02:38 | |
| *** sdake has joined #openstack-kolla | 02:39 | |
| *** banix has joined #openstack-kolla | 02:41 | |
| *** sdake_ has quit IRC | 02:43 | |
| *** salv-orlando has joined #openstack-kolla | 02:45 | |
| vhosakot | Jeffrey4l_: sdake: https://review.openstack.org/#/c/302026/ | 02:45 |
| patchbot | vhosakot: patch 302026 - kolla - Add prechecks to validate kolla_internal_vip_addre... | 02:45 |
| *** yuanying has quit IRC | 02:46 | |
| *** salv-orl_ has quit IRC | 02:47 | |
| *** wuhg has quit IRC | 02:49 | |
| *** daneyon_ has joined #openstack-kolla | 02:50 | |
| *** klint has joined #openstack-kolla | 02:51 | |
| *** daneyon_ has quit IRC | 02:55 | |
| *** weiyu has joined #openstack-kolla | 02:58 | |
| *** achanda has quit IRC | 02:59 | |
| *** jasonsb has quit IRC | 03:10 | |
| sdake | vhosakot i think i sorted out the gate issue | 03:22 |
| vhosakot | sdake: cool... which issue ? rabbitmq crash ? | 03:24 |
| sdake | yup | 03:24 |
| sdake | the crashes started happening when rdo upgraded their repos to -16 version of rabbitmq | 03:24 |
| sdake | this enabled ipv6 | 03:25 |
| sdake | ipv6 and ipv4 don't work with epmd | 03:25 |
| sdake | so if we disable the EPMD address setting, it will work | 03:25 |
| sdake | note this review: | 03:25 |
| vhosakot | ah ok... well, fix I put in https://review.openstack.org/#/c/303686/ solved the crash issue in centos gate as well :) | 03:26 |
| patchbot | vhosakot: patch 303686 - kolla - Fix rabbitmq crash in centos gate | 03:26 |
| sdake | https://review.openstack.org/303837 | 03:26 |
| vhosakot | sdake: I removed the ERL_EPMD_ADDRESS variable in rabbitmq-env.conf in the fix | 03:26 |
| sdake | read my review | 03:26 |
| vhosakot | sdake: yep, I reviewed and left a comment | 03:27 |
| sdake | well what i just explained is the root cause | 03:27 |
| vhosakot | ah ok | 03:27 |
| sdake | did you catch the part about heisenbug | 03:27 |
| sdake | binding to 0.0.0.0 is not ideal | 03:28 |
| *** banix has quit IRC | 03:28 | |
| vhosakot | sdake: ah, reading you comment in https://review.openstack.org/#/c/303837/4/ansible/roles/rabbitmq/templates/rabbitmq-env.conf.j2 | 03:29 |
| patchbot | vhosakot: patch 303837 - kolla - RabbitMQ -16 introduced ipv6 breakage | 03:29 |
| vhosakot | sdake: I see ERL_EPMD_ADDRESS is commented, isn;t it needed | 03:31 |
| sdake | well yo uwould think so | 03:32 |
| sdake | but erlang is completely busted wrt empd | 03:33 |
| sdake | which is the logging daeon iiuc | 03:33 |
| vhosakot | sdake: ah, so, rabbitmq-env.conf.j2 will have ERL_EPMD_ADDRESS commented or removed ? | 03:33 |
| sdake | commented for the moment | 03:33 |
| sdake | with a warning | 03:33 |
| sdake | feel freet to update your change to that affect | 03:33 |
| sdake | the problem is binding to an ipv4 address using ipv4 addressing scheme i.e. ::ffff:ipv4address binds to 0.0.0.0 | 03:34 |
| sdake | so that workaround is no bueno either | 03:34 |
| vhosakot | sdake: well, both our change resolve the same.. so, does it make sense to abandon my review and add me as co-author on your review and merge it I found a bit about removing ERL_EPMD_ADDRESS as well :) | 03:34 |
| sdake | i think i will abandon my change | 03:35 |
| sdake | and you can just integrate my comments | 03:35 |
| sdake | and leave me off the coauthor line is fine | 03:35 |
| vhosakot | sdake: then, should I add erlang-R16B-03.17.el7.repo in my fix ? | 03:35 |
| sdake | NO | 03:35 |
| vhosakot | sdake: yes, I will add you as co-author | 03:35 |
| sdake | that was a tes for sbezverk | 03:35 |
| vhosakot | sdake: ah ok | 03:35 |
| sdake | i really dont think you need to do so | 03:36 |
| sdake | but its up to you :) | 03:36 |
| vhosakot | sdake: I will add you as co-author, but what should I change in https://review.openstack.org/#/c/303686/ ? add your comment about Heisenbugs (about how epmd is broken with ipv6) ? | 03:37 |
| patchbot | vhosakot: patch 303686 - kolla - Fix rabbitmq crash in centos gate | 03:37 |
| vhosakot | sdake: let me know if this what I need to do.. yep, will add you as co-author.. :) | 03:38 |
| *** achanda has joined #openstack-kolla | 03:44 | |
| *** achanda has quit IRC | 03:46 | |
| sdake | yes add the comment | 03:46 |
| openstackgerrit | Merged openstack/kolla: Fix the qemu has nova permission in nova instances folder https://review.openstack.org/303680 | 03:48 |
| *** yuanying has joined #openstack-kolla | 03:49 | |
| vhosakot | sdake: ok, I will add your comment.. cool | 03:50 |
| vhosakot | sdake: so, will https://review.openstack.org/#/c/303837/ be abandoned then ? | 03:50 |
| patchbot | vhosakot: patch 303837 - kolla - RabbitMQ -16 introduced ipv6 breakage | 03:50 |
| *** coolsvap has joined #openstack-kolla | 03:51 | |
| *** jtriley has joined #openstack-kolla | 03:53 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Revert "Add general_log to mariadb container and make heka collect it" https://review.openstack.org/302401 | 03:54 |
| openstackgerrit | Merged openstack/kolla: Increase max scheduling attempts to 10 https://review.openstack.org/303678 | 03:57 |
| *** jtriley has quit IRC | 03:58 | |
| *** haplo37 has joined #openstack-kolla | 04:02 | |
| sdake | vhosakot this has been what I've been doing all weekend | 04:04 |
| sdake | https://drive.google.com/file/d/0B8q6xDPETSkHZDgyOTFMLTVPbm8/view?usp=sharing | 04:04 |
| vhosakot | sdake: ah :) | 04:04 |
| sdake | those were galant desks with damaged tops | 04:04 |
| vhosakot | sdake: where is your home server ? | 04:04 |
| sdake | ikea discontinued the furniture | 04:04 |
| sdake | in that room | 04:04 |
| sdake | not yet plugged in | 04:04 |
| sdake | atm its in my daughters room | 04:05 |
| sdake | plugged in through hacky networking techinuqes ;) | 04:05 |
| vhosakot | sdake: ah ok | 04:05 |
| vhosakot | nice! | 04:05 |
| sdake | so anyway the furniture got discontinued and we needed new tops | 04:05 |
| sdake | so we bought counter tops for kitchens | 04:05 |
| sdake | 100 bucks a crack | 04:05 |
| sdake | and i mounted them no the hardware from the ikea galant desks | 04:05 |
| sdake | one more to go | 04:05 |
| vhosakot | yep, looks great | 04:05 |
| sdake | and then i can gear up | 04:05 |
| vhosakot | sdake: you glued it or screwed it ? | 04:06 |
| sdake | we are putting atable coming out into the room 57 inches | 04:06 |
| sdake | screwed | 04:06 |
| vhosakot | right | 04:06 |
| sdake | 3/4 #12 screws | 04:06 |
| vhosakot | cool | 04:06 |
| sdake | countertops shoud hold up well | 04:06 |
| sdake | they are completely plastic | 04:06 |
| vhosakot | is that a gramaphone ? :) | 04:06 |
| sdake | i dont even know what that is ;) | 04:07 |
| sdake | my stereo is modern :) | 04:07 |
| sdake | my dac and digital music play are yet to be hooked up | 04:07 |
| vhosakot | I see something like a vinly disc player | 04:07 |
| coolsvap | sdake: looks good | 04:07 |
| sdake | yup thats my feikert turntable | 04:08 |
| vhosakot | cool | 04:08 |
| sdake | http://www.feickert.de/index.php?id=10 | 04:08 |
| sdake | mine is slightly different staylin but same model | 04:08 |
| sdake | that is the non-us model | 04:08 |
| vhosakot | nice.. turntable! | 04:09 |
| sdake | in the us they make them with differnet materials | 04:09 |
| sdake | ought to be | 04:09 |
| sdake | thanks coolsvap | 04:09 |
| sdake | we aren't done yet | 04:09 |
| sdake | we have one more tablee to go ;) | 04:09 |
| sdake | but my remodel is officially over | 04:10 |
| sdake | now its a matter of putting the house back together | 04:10 |
| sdake | remodeling the desks cost about 450 bucks :( | 04:10 |
| coolsvap | :) | 04:10 |
| *** weiyu has quit IRC | 04:10 | |
| sdake | but a double workstation with professional grade gear is 6-8k | 04:10 |
| sdake | so we are holding off on that for vestmas :) | 04:10 |
| vhosakot | wow... 450 | 04:11 |
| sdake | 450 is cheap for 3 tables | 04:11 |
| sdake | that never wear out no matter how abusive your children are to your gear | 04:11 |
| vhosakot | right | 04:12 |
| sdake | i have 40 amps o power along the desk wall as well | 04:12 |
| sdake | on two seprate circuits | 04:12 |
| sdake | http://www.amazon.com/Tripp-Lite-PDUMV40-Metered-Vertical/dp/B000MT4HQM?ie=UTF8&psc=1&redirect=true&ref_=oh_aui_search_detailpage | 04:13 |
| sdake | power solution ^ | 04:13 |
| sdake | we planned 17k for our remodel and it was about 18.5k | 04:14 |
| sdake | 10% over budget groan | 04:14 |
| sdake | but i wasn't' planning 40 amps of power which was expensive | 04:14 |
| sdake | or new closets for kids bedroom | 04:15 |
| vhosakot | home improvemnt projects always go over planned time and budget | 04:15 |
| sdake | electriian cost = 700 bucks | 04:15 |
| sdake | which was a super deal | 04:15 |
| sdake | i expected to be done early feb | 04:15 |
| sdake | but whe nyou live in teh place | 04:15 |
| vhosakot | cool vertical power strip in amazon | 04:15 |
| sdake | its hard to actually deliver on that | 04:15 |
| sdake | ya we have that running horizontally | 04:16 |
| sdake | the black is one circuit white is the second 20 amp circuit | 04:16 |
| vhosakot | cool | 04:16 |
| sdake | so basically we have mroe poewr then we can possibly use | 04:16 |
| sdake | that was because my daughter microwaves and used to blow the circuit daily | 04:16 |
| sdake | so we put the office on its own circuit | 04:16 |
| sdake | i also have a 3rd 20 amp circuit dedicated to my stereo | 04:17 |
| *** achanda has joined #openstack-kolla | 04:18 | |
| sdake | with 8 gauge wire and furutech gtx-d rhodium power plug ;) | 04:18 |
| vhosakot | nice.. yep, our microwave circuit is also separated from most of the houce circuites... | 04:18 |
| sdake | http://www.enjoythemusic.com/magazine/equipment/0114/audiophile_ac_outlets.htm | 04:18 |
| sdake | the problem we have is our microwave is connected to our entire front of our hosuse | 04:19 |
| sdake | like morons the electricians didn't run a separate circuit for the microwae | 04:19 |
| *** weiyu_ has joined #openstack-kolla | 04:19 | |
| vhosakot | our microwave trips sometimes... | 04:20 |
| sdake | my stereo is pretty off the hook | 04:26 |
| sdake | definately sounds different with the tile the then carpet | 04:26 |
| sdake | need to get some rugs down or something | 04:26 |
| sdake | different as in not better | 04:26 |
| *** weiyu_ has joined #openstack-kolla | 04:27 | |
| sdake | well i hate hitler | 04:30 |
| sdake | but heres a docker on hitler thing that is making the rounds | 04:30 |
| sdake | https://m.youtube.com/watch?feature=youtu.be&v=PivpCKEiQOQ | 04:30 |
| vhosakot | haha.. yeah.. I 've seen it..! | 04:32 |
| sdake | the part about the proxy server is so hilarious and so true | 04:35 |
| vhosakot | haha... yeah | 04:36 |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Drop root for HAProxy https://review.openstack.org/245013 | 04:36 |
| *** weiyu_ has quit IRC | 04:40 | |
| sdake | sudo curl | bash | 04:41 |
| sdake | bad habit to get people into | 04:41 |
| sdake | https://helm.sh/ | 04:43 |
| sdake | http://helm.readthedocs.org/en/latest/modeling_services/ | 04:47 |
| sdake | using labels for wait conditions | 04:47 |
| sdake | so sloppy | 04:47 |
| vhosakot | wow, I never pipe file I curl from a public site to bash without reading the contents of the file.. | 04:47 |
| vhosakot | curl | bash is very dangeruos | 04:48 |
| vhosakot | sdake: we have it in our qsg as well http://docs.openstack.org/developer/kolla/quickstart.html | 04:48 |
| vhosakot | sdake: curl -sSL https://get.docker.io | bash | 04:48 |
| vhosakot | the userland proxy being slow is very true... | 04:48 |
| sdake | i know its there | 04:49 |
| sdake | look t git blame on that line | 04:49 |
| *** daneyon_ has joined #openstack-kolla | 04:50 | |
| vhosakot | let me check git blame | 04:50 |
| vhosakot | wow https://github.com/openstack/kolla/blame/master/doc/quickstart.rst#L87 | 04:50 |
| sdake | hmm i thought i put that in there not sam ;) | 04:51 |
| vhosakot | :) | 04:52 |
| sdake | where i was going to point out the oxymoronism that i am such a security freak yet would propose such an insecure behavior | 04:52 |
| sdake | we need to sudoize our documentation | 04:53 |
| sdake | i guess i'll do that after releases | 04:53 |
| sdake | sam always ran as root so he was very negative about having sudo in the dos | 04:54 |
| sdake | docs | 04:54 |
| vhosakot | ah ok | 04:55 |
| sdake | possibly why selinux never worked for him | 04:55 |
| sdake | running as root breaks selinux pretty easily | 04:55 |
| sdake | i was shocked thoug hto find our gate was runnign seinux | 04:56 |
| vhosakot | isn't is enfored by default in Centos ? | 04:56 |
| sdake | yes | 04:56 |
| vhosakot | enforced* | 04:56 |
| sdake | the gate images used t o turn it off by deffault | 04:57 |
| sdake | as most cloud providers do | 04:57 |
| nihilifer | guys, one question, if anyone here is good at ubuntu | 05:12 |
| nihilifer | what's the best way to make "mount --make-shared /run" permanent? | 05:13 |
| nihilifer | /etc/fstab? something in upstart script? | 05:13 |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Drop root for HAProxy https://review.openstack.org/245013 | 05:16 |
| sdake | ok catss | 05:17 |
| sdake | we need kolla upgrades and reconfigure totally working for this | 05:17 |
| sdake | http://www.midokura.com/Webinars/ | 05:17 |
| sdake | 100% reliability pzl :) | 05:18 |
| sdake | nihilifer vhosakot is your man | 05:18 |
| sdake | probably rc.local woud lbe my recommendtaion though | 05:18 |
| sdake | just from general unix experience | 05:18 |
| sdake | but there may be a better / more intuitve way | 05:19 |
| nihilifer | ok, thx | 05:21 |
| *** sdake_ has joined #openstack-kolla | 05:23 | |
| vhosakot | nihilifer: yes, adding an entry in /etc/fstab should make it persist thru reboot... I have not added it as I run the command after every reboot.. | 05:23 |
| vhosakot | nihilifer: you you see how /run is by doing cat /proc/self/mountinfo | grep run | 05:23 |
| *** sdake has quit IRC | 05:24 | |
| *** weiyu_ has joined #openstack-kolla | 05:26 | |
| *** daneyon_ has quit IRC | 05:30 | |
| vhosakot | sdake_: what is this midokura kolla webinar ? | 05:31 |
| sdake_ | something midokura setup | 05:34 |
| sdake_ | any chance you could register at zoom.com so i can see what its like to video conference with it? | 05:34 |
| sdake_ | would like to know the tool so I have some idea of what i've gotten myself into ;) | 05:35 |
| vhosakot | sdake_: I cannot access zoom.com | 05:37 |
| sdake_ | try off the vpn | 05:38 |
| sdake_ | its probably firewalled or something | 05:38 |
| sdake_ | sorry zoom.us | 05:38 |
| *** dimsum__ has quit IRC | 05:40 | |
| nihilifer | vhosakot: ok, another stupid question from me | 05:41 |
| nihilifer | i have the following entry in mountinfo | 05:41 |
| nihilifer | 21 22 0:16 / /run rw,nosuid,noexec,relatime shared:1 - tmpfs tmpfs rw,size=256604k,mode=755 | 05:41 |
| vhosakot | haha there is no sypid Q.. | 05:41 |
| nihilifer | how to pass the "shared:1" parameter to fstab? | 05:41 |
| nihilifer | adding it as an option after comma doesn't work for me | 05:42 |
| nihilifer | my fstab entry i've tried is | 05:42 |
| vhosakot | yep, I dont know that part.. I tried many things and did not work... that is why I have added it to /etc/fstab... | 05:43 |
| *** achanda has quit IRC | 05:43 | |
| nihilifer | tmpfs /run tmpfs rw,nosuid,noexec,relatime,shared:1 0 0 | 05:43 |
| vhosakot | nihilifer: what is /run filesystem type in your environmeny ? df /run | 05:43 |
| vhosakot | tmpfs is good.. cool | 05:43 |
| sdake_ | lets get that in the docs | 05:44 |
| sdake_ | plz ;0 | 05:44 |
| vhosakot | yes.. | 05:44 |
| sdake_ | we get this question all the time | 05:44 |
| sdake_ | nihilifer will you be at summit? | 05:45 |
| nihilifer | sdake_: i will not, unfortunately | 05:45 |
| * sdake_ whines | 05:45 | |
| *** sdake_ is now known as sdake | 05:45 | |
| sdake | same story with pbourke | 05:45 |
| *** weiyu_ has quit IRC | 05:46 | |
| vhosakot | oh, was hoping to meet everyone | 05:46 |
| sdake | nihilifer since your on our security team | 05:46 |
| sdake | i will need your help prepring architecture documenttion for a ta session | 05:46 |
| sdake | we will be using a tool called gliffy | 05:46 |
| sdake | plesae register :) | 05:46 |
| *** dims has joined #openstack-kolla | 05:47 | |
| sdake | yo dims | 05:47 |
| nihilifer | i'm registered, so no worries | 05:47 |
| nihilifer | vhosakot: here is the result of "df /run" | 05:47 |
| nihilifer | http://paste.openstack.org/show/493599/ - when running the mount command by hand | 05:48 |
| nihilifer | http://paste.openstack.org/show/493600/ - after my failed attempt with fstab | 05:48 |
| vhosakot | nihilifer: same output as mine... | 05:48 |
| vhosakot | nihilifer: the thing I have observed it the <options> column in the /etc/fstab file does not accept the "shared" value... http://man7.org/linux/man-pages/man8/mount.8.html#FILESYSTEM-INDEPENDENT_MOUNT%20OPTIONS | 05:49 |
| vhosakot | nihilifer: I have not been able to figure it out either, and hence run the mount command manually after eveyr reboot | 05:49 |
| nihilifer | :( | 05:50 |
| * nihilifer waits for 16.04 LTS and systemd | 05:50 | |
| vhosakot | or move to Centos | 05:51 |
| nihilifer | i cannot | 05:51 |
| nihilifer | my company forces me to use ubuntu, lol | 05:51 |
| vhosakot | so, after updating /etc/fstab ? did you reboot or re-mount the mountpoints using "mount -a" ? | 05:51 |
| nihilifer | i really wish i could use centos | 05:51 |
| *** haplo37 has quit IRC | 05:52 | |
| nihilifer | i did a reboot | 05:53 |
| vhosakot | cool... | 05:53 |
| *** jtriley has joined #openstack-kolla | 05:54 | |
| sdake | you aare forced to use ubuntu? | 05:56 |
| sdake | how can such a thing be forced? | 05:56 |
| nihilifer | i'm joking with "forceD" | 05:57 |
| nihilifer | ;) | 05:57 |
| sdake | you always hae a choice ;) | 05:57 |
| nihilifer | we just care about kolla working on ubuntu | 05:57 |
| nihilifer | without dirty hacks | 05:57 |
| sdake | roger and we intend to deliver on that | 05:57 |
| sdake | and thus far i think we have done a good job | 05:57 |
| sdake | as well as centos ole debian and lastly fedora ;) | 05:57 |
| sdake | i'd like suse support if any suse cats came to implement it | 05:58 |
| sdake | i reached out to them but snakeeyes | 05:58 |
| *** jtriley has quit IRC | 05:59 | |
| sdake | nihilifer jeffrey4l and I fixed the qemu bug on mesos | 06:02 |
| sdake | as well as everywhere else it was broken | 06:02 |
| *** karthiks has quit IRC | 06:03 | |
| sdake | if you stil lsee that kicking around let me know | 06:03 |
| sdake | this is our final week until release | 06:03 |
| sdake | if there is any more doa let me know plz :) | 06:03 |
| *** achanda has joined #openstack-kolla | 06:15 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add note about heat being busted for development https://review.openstack.org/290276 | 06:18 |
| *** weiyu has joined #openstack-kolla | 06:18 | |
| *** achanda has quit IRC | 06:20 | |
| *** daneyon_ has joined #openstack-kolla | 06:24 | |
| *** tfukushima has joined #openstack-kolla | 06:26 | |
| *** daneyon_ has quit IRC | 06:30 | |
| *** dims has quit IRC | 06:31 | |
| *** Serlex has joined #openstack-kolla | 06:33 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Drop root for HAProxy https://review.openstack.org/245013 | 06:33 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add quotes for RBMQ nodes definition https://review.openstack.org/303919 | 06:34 |
| vhosakot | sdake: I see you abadoned your rabbitmq crash review... I just need to add your comments in rabbitmq-env.conf.j2 in my PS right ? https://review.openstack.org/#/c/303837/4/ansible/roles/rabbitmq/templates/rabbitmq-env.conf.j2 | 06:42 |
| patchbot | vhosakot: patch 303837 - kolla - RabbitMQ -16 introduced ipv6 breakage (ABANDONED) | 06:42 |
| *** coolsvap has quit IRC | 06:43 | |
| *** achanda has joined #openstack-kolla | 06:44 | |
| sdake | ya just add my note | 06:44 |
| sdake | and comment out the entire section so we don't lose it | 06:44 |
| sdake | vhosakot ^^ | 06:44 |
| vhosakot | sdake: ok.. cool... should I add ::ffff: for ERL_EPMD_ADDRESS as well ? | 06:45 |
| vhosakot | sdake: ^^ | 06:47 |
| sdake | i wouldn't | 06:47 |
| sdake | or add both lines | 06:47 |
| sdake | say one is an option recommended in the launchpad bug above in comment #6 | 06:48 |
| sdake | but it results in a bind to all interfaces. | 06:48 |
| sdake | which is the same as not specifying anything here | 06:48 |
| *** dims has joined #openstack-kolla | 06:54 | |
| vhosakot | sdake: cool.. I wont add the ::ffff: for ERL_EPMD_ADDRESS then | 06:55 |
| *** coolsvap has joined #openstack-kolla | 06:57 | |
| *** dims has quit IRC | 06:58 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Drop root for HAProxy https://review.openstack.org/245013 | 07:00 |
| *** weiyu has quit IRC | 07:03 | |
| *** weiyu has joined #openstack-kolla | 07:04 | |
| *** mikelk has joined #openstack-kolla | 07:11 | |
| sdake | vhosakot can you sort out that gate change now | 07:12 |
| sdake | so I can ack it before bedtime | 07:12 |
| vhosakot | sure... doing it now | 07:12 |
| sdake | getting red hat to revert ipv6 support will be very difficult | 07:13 |
| sdake | and getting them to fix epmd to work with ipv4 and ipv6 also difficult | 07:13 |
| sdake | the epmd binding code is an absolute charlie foxtrot | 07:13 |
| sdake | i've looekd at it, I'd have to rewrite the entire binding to work correctly | 07:13 |
| sdake | and upstream may have already solved it | 07:13 |
| sdake | but I dont know | 07:14 |
| sdake | ipv4 and ipv6 binding code in corosync was over 2500 loc | 07:14 |
| sdake | its very difficult to get right | 07:14 |
| sdake | especially in a one-off port | 07:14 |
| *** jmccarthy has left #openstack-kolla | 07:14 | |
| sdake | as in something thatdoesn't go into master | 07:14 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add note about heat being busted for development https://review.openstack.org/290276 | 07:16 |
| *** weiyu has quit IRC | 07:17 | |
| coolsvap | sdake: s/Hea/Heat | 07:18 |
| sdake | coolsvap line # pleae | 07:18 |
| coolsvap | #22 | 07:19 |
| coolsvap | 21 | 07:19 |
| openstackgerrit | Merged openstack/kolla: Revert "Add general_log to mariadb container and make heka collect it" https://review.openstack.org/302401 | 07:19 |
| coolsvap | sdake: line 21 | 07:19 |
| *** weiyu has joined #openstack-kolla | 07:20 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Fix rabbitmq crash in centos gate https://review.openstack.org/303686 | 07:20 |
| vhosakot | sdake: ^^ | 07:20 |
| sdake | try that on | 07:20 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add note about heat being busted for development https://review.openstack.org/290276 | 07:20 |
| vhosakot | Jeffrey4l_: sdake https://review.openstack.org/#/c/303686/ | 07:20 |
| patchbot | vhosakot: patch 303686 - kolla - Fix rabbitmq crash in centos gate | 07:20 |
| openstackgerrit | Merged openstack/kolla: Add prechecks to validate kolla_internal_vip_address and kolla_external_vip_address https://review.openstack.org/302026 | 07:21 |
| coolsvap | sdake: should i correct it, i have a feeling you need to get to bed asap | 07:21 |
| sdake | beuno is spelled bueno | 07:21 |
| sdake | coolsvap i'm good | 07:21 |
| sdake | i'm up for a couple more hours | 07:21 |
| sdake | just tyring to get some adminstrative tasks done | 07:22 |
| coolsvap | s/beliee/believe | 07:22 |
| coolsvap | same line | 07:22 |
| coolsvap | line 21 | 07:22 |
| coolsvap | sdake: ^^ | 07:22 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add note about heat being busted for development https://review.openstack.org/290276 | 07:22 |
| vhosakot | Jeffrey4l_: sdake coolsvap: https://review.openstack.org/#/c/303686/ is updated | 07:24 |
| patchbot | vhosakot: patch 303686 - kolla - Fix rabbitmq crash in centos gate | 07:24 |
| sdake | beno/bueno | 07:25 |
| sdake | please change ;) | 07:25 |
| sdake | lets atleast spell it correctly | 07:25 |
| sdake | vhosakot ^^ | 07:25 |
| vhosakot | haha... sdake ok | 07:25 |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Fix rabbitmq crash in centos gate https://review.openstack.org/303686 | 07:28 |
| vhosakot | sdake: bueno pushed ^^ | 07:28 |
| sdake | coolsvap or nihilifer can you review https://review.openstack.org/#/c/303686/8 | 07:30 |
| patchbot | sdake: patch 303686 - kolla - Fix rabbitmq crash in centos gate | 07:30 |
| sdake | coolsvap will you be at summit? | 07:30 |
| coolsvap | sdake: yes | 07:30 |
| sdake | sweet | 07:30 |
| sdake | so only 3 people i think from the cor eteam wont make it | 07:30 |
| vhosakot | sdake: get us all beer ;) | 07:30 |
| sdake | 3/4 is better then 0/4ths :) | 07:31 |
| sdake | what do I look like - an atm? :) | 07:31 |
| vhosakot | Jeffrey4l_: ping | 07:31 |
| sdake | vhosakot agree with jeffrey4l | 07:31 |
| vhosakot | the rabbitmq crash is not seen on Ubuntu | 07:31 |
| sdake | that is because the cndition is never executed | 07:31 |
| vhosakot | ah, right... | 07:32 |
| vhosakot | let me remove it | 07:32 |
| sdake | so the variable is never set | 07:32 |
| coolsvap | atb (brewer) | 07:32 |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Fix rabbitmq crash in centos gate https://review.openstack.org/303686 | 07:32 |
| vhosakot | sdake: Jeffrey4l_ ^^^ | 07:32 |
| sdake | after this merges i am doing a massive recheck of all our outstanding patches | 07:33 |
| vhosakot | sdake: Jeffrey4l_ if condition is removed | 07:33 |
| vhosakot | hahah coolsvap atb (brewer) :) | 07:33 |
| *** shardy has joined #openstack-kolla | 07:34 | |
| vhosakot | if Jeffrey4l_ coolsvap acks it, our gates will be good.... and no rabbotmq crash | 07:35 |
| sdake | that wil lspeed up dev quite a bit ;) | 07:35 |
| vhosakot | sdake: we need a bug to track the nova vm issue not reaching active state in centos binary gate... new issue, not related to rabbitmq crash | 07:35 |
| sdake | 5 rechecks perpatch is annoing | 07:35 |
| sdake | vhosakot agree can youf ile one | 07:36 |
| sdake | and file a tech debt bug as requested for this particular EPMD issue | 07:36 |
| vhosakot | yep, I have debugeed it almost...a nova issue | 07:36 |
| vhosakot | ah ok | 07:36 |
| sdake | so its tracked as moer then just a TODO | 07:36 |
| vhosakot | 2 bugs | 07:36 |
| vhosakot | thanks for the +2 coolsvap | 07:37 |
| coolsvap | vhosakot: np! | 07:37 |
| * coolsvap away for lunch | 07:37 | |
| vhosakot | sdake: so, how will the EPDM IPv6 issue be solved ? do we need to upgrade rabbit-server ? | 07:37 |
| sdake | hell if I know | 07:37 |
| *** jmccarthy has joined #openstack-kolla | 07:37 | |
| sdake | I dont have a month to rewrite RHELs version of epmd binding | 07:38 |
| sdake | wich is far different from upstream | 07:38 |
| sdake | model is usptream first, then backport | 07:38 |
| sdake | I think we will jut have to live with it | 07:38 |
| sdake | maybe we can get red hat to fix it since its their port | 07:38 |
| sdake | but who knows | 07:38 |
| sdake | it will affect their customer base severely | 07:38 |
| vhosakot | sdake: cool agreed | 07:39 |
| sdake | and also our users will be negatively impacted by this problem | 07:39 |
| *** weiyu has quit IRC | 07:39 | |
| sdake | the issue is a wildcard bind on whatever port epmd runs on will trigger software that binds on that port from not being able to bind - theoretically | 07:39 |
| sdake | and also may have some other negative influence | 07:39 |
| *** dims has joined #openstack-kolla | 07:45 | |
| *** reddy has joined #openstack-kolla | 07:51 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Add prechecks to validate kolla_internal_vip_address and kolla_external_vip_address https://review.openstack.org/303938 | 07:52 |
| *** vhosakot has quit IRC | 07:53 | |
| *** jtriley has joined #openstack-kolla | 07:55 | |
| *** achanda has quit IRC | 07:56 | |
| *** mbound has joined #openstack-kolla | 07:57 | |
| *** jtriley has quit IRC | 08:00 | |
| sdake | Jeffrey4l_ or mandre can i get an ack on https://review.openstack.org/#/c/290276/ | 08:00 |
| patchbot | sdake: patch 290276 - kolla - Add note about heat being busted for development | 08:00 |
| *** gfidente has joined #openstack-kolla | 08:08 | |
| openstackgerrit | Merged openstack/kolla: Fix rabbitmq crash in centos gate https://review.openstack.org/303686 | 08:08 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix rabbitmq crash in centos gate https://review.openstack.org/303951 | 08:11 |
| sdake | need an ack on this cherrypick please ^^ | 08:12 |
| sdake | asalkeld nihilifer mandre Jeffrey4l_ ^^ | 08:12 |
| *** weiyu has joined #openstack-kolla | 08:15 | |
| mandre | sdake: I'm not sure I understand the resolution of https://bugs.launchpad.net/kolla/+bug/1562701 | 08:18 |
| openstack | Launchpad bug 1562701 in kolla mitaka "rabbitmq crashed on centos binary deploy" [Critical,Confirmed] - Assigned to Vikram Hosakote (vhosakot) | 08:18 |
| *** gfidente has quit IRC | 08:19 | |
| sdake | do you rreally want me to explain it to you | 08:20 |
| sdake | i am happy to do so | 08:20 |
| sdake | but it is time consuming and complicated :) | 08:21 |
| mandre | sdake, yes please :) | 08:21 |
| sdake | epmd which is some part of erlang is busted when compiled with ipv6 support | 08:21 |
| sdake | it wont bind properly to an ipv4 address | 08:21 |
| sdake | in some conditions | 08:21 |
| sdake | in others it will | 08:21 |
| sdake | I am not quite sure what the conditions are | 08:22 |
| mandre | can you first explain the comment you added? | 08:22 |
| sdake | i'm getting there | 08:22 |
| *** achanda has joined #openstack-kolla | 08:22 | |
| mandre | according to your comment, we should bind to a known IPv4 address and it will fix it for all cases | 08:22 |
| sdake | because erlang is broken and wont bind to ipv4, and kolla doesn't really work well with ipv6, the only option available to us is to remove the bind option | 08:22 |
| sdake | no, according to my comment we should be binding | 08:23 |
| sdake | but we cnanot ecause erlang is defective | 08:23 |
| mandre | that's not what your comment say | 08:23 |
| sdake | disagree | 08:23 |
| mandre | your comment says we should be binding, then the bind is commented out | 08:23 |
| sdake | i you read the bug log and the note it seems clear | 08:23 |
| mandre | that's what I don't understand | 08:23 |
| sdake | yes we should be binding | 08:24 |
| sdake | but we cant because as documented in the bug, doing so makes erlang bomb out | 08:24 |
| sdake | its a cherrypick, too late now to debate the language | 08:24 |
| sdake | if you want to clarify it submit a follow up patch and we can cherrypick that ;) | 08:24 |
| mandre | just trying to understand the resolution now... | 08:24 |
| *** weiyu has quit IRC | 08:25 | |
| *** gfidente has joined #openstack-kolla | 08:25 | |
| sdake | essentially its a problem in erlang | 08:25 |
| sdake | and erlang needs to fix it | 08:25 |
| sdake | and red hat needs to backport it into centos | 08:25 |
| sdake | and ubuntu the same | 08:26 |
| mandre | you're saying that binding with ERL_EPMD_ADDRESS is broken, correct? | 08:26 |
| sdake | yes precisely | 08:26 |
| sdake | the act o fbinding is correct | 08:26 |
| sdake | the implementation of erlang's binding is incorrect | 08:26 |
| mandre | ok, so I think we need to clarify your comment :) | 08:27 |
| sdake | ok i can do or you can - use same bug | 08:27 |
| sdake | whats it to be ;) | 08:27 |
| mandre | I'll clarify the comment, you'll tell me if I understood correctly in the patch review | 08:28 |
| sdake | cool but do it in master | 08:28 |
| sdake | in the meantime letes get the cherrypick merged | 08:28 |
| sdake | so i can start a massive recheck on all our patches | 08:29 |
| sdake | read the bug log too please | 08:29 |
| sdake | it has key information | 08:29 |
| sdake | what triggered this ws the introduction of ipv6 compile flags to erlang in -16 of erlang | 08:29 |
| sdake | t-2 weeks ago | 08:29 |
| *** weiyu has joined #openstack-kolla | 08:29 | |
| sdake | ubuntu never had the problem in the gate | 08:30 |
| sdake | even though ubuntu is equally broken | 08:30 |
| mandre | i see, nice find | 08:30 |
| sdake | because it had a conditional around it | 08:30 |
| sdake | it wsn't until we tried sbezverk's -17 patch that it all came together forme | 08:30 |
| sdake | and jeffrey's discovery of the syslog message epmd: failed to bind | 08:31 |
| sdake | when epmd fails to bind, erlang crashes | 08:31 |
| sdake | this causes anything running in erlang (rabbitmq for example) to damage the gate ;( | 08:31 |
| mandre | do you have a bug for broken binding with ERL_EPMD_ADDRESS? | 08:32 |
| sdake | i asked vhosakot to file one in kolla as a tracking bug | 08:32 |
| sdake | but i dont know if he did tonight or planned to tomorrow | 08:32 |
| sdake | but the bug info we know bout is in the bug in the cherrypick that comes from master | 08:33 |
| mandre | it should be an erlang bug not a kolla one | 08:33 |
| sdake | i dont know the erlang bug | 08:33 |
| sdake | but they are aware of it apparently | 08:33 |
| mandre | i'll look for it | 08:33 |
| sdake | making ipv4 and ipv6 bind properly is very difficult | 08:33 |
| sdake | and epmd has itwrong | 08:34 |
| sdake | the crux of the situation we are in now is that we are binding to all interfaces | 08:34 |
| sdake | which if controller code is collocated | 08:34 |
| sdake | could possibly cause disruptions of network services runnign on the same port as epmd | 08:34 |
| sdake | i suspect, but i am not sure | 08:34 |
| sdake | they have the wildcard bind working | 08:35 |
| sdake | they dont have specific interface binding woring | 08:35 |
| sdake | to make specific interface binding work, it is necessary to scan all interfaces, find the interface number, and pass it as an ioctl in a special bind oeration | 08:35 |
| sdake | this is done in corosync correctly but it took us literally 3-4 months to get right | 08:36 |
| sdake | epmd needs the same attention | 08:36 |
| sdake | its complicated by the fact that red hat's erlang is months/years behind upstream | 08:36 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix rabbitmq crash in centos gate https://review.openstack.org/303951 | 08:37 |
| *** fragatina has quit IRC | 08:43 | |
| *** salv-orl_ has joined #openstack-kolla | 08:44 | |
| *** salv-orlando has quit IRC | 08:47 | |
| *** dims has quit IRC | 08:48 | |
| openstackgerrit | Martin André proposed openstack/kolla: Clarify comment about binding erlang to IPv4 https://review.openstack.org/303969 | 08:50 |
| mandre | sdake: ^ | 08:53 |
| sdake | some suggestions in that doc mandre | 08:53 |
| sdake | ratherin the review | 08:53 |
| sdake | to clarify further | 08:53 |
| sdake | your totally right the comment was unclear | 08:54 |
| sdake | lets make it crystal clear :) | 08:54 |
| *** dims has joined #openstack-kolla | 08:55 | |
| sdake | mandre you might also include a link to my abaonded review | 08:58 |
| sdake | https://review.openstack.org/#/c/303837/ | 08:59 |
| patchbot | sdake: patch 303837 - kolla - RabbitMQ -16 introduced ipv6 breakage (ABANDONED) | 08:59 |
| mandre | sdake: I think this is information that should go in our issue, not in the code | 09:01 |
| mandre | I don't want to make the comment too verbose, it's already quite long | 09:01 |
| sdake | i hate to see someone uncomment for no good reason other then they think its fixed | 09:01 |
| sdake | i want them to be sure ;) | 09:02 |
| mandre | that's the reason why we're putting a link to the issue | 09:02 |
| sdake | change of topic | 09:02 |
| sdake | but the issue is closed and fixed | 09:03 |
| openstackgerrit | Martin André proposed openstack/kolla: Clarify comment about binding erlang to IPv4 https://review.openstack.org/303969 | 09:03 |
| sdake | people dont typically click through anyway :) | 09:03 |
| mandre | it doesn't mean we can't comment on it to provide more info | 09:03 |
| sdake | ok | 09:04 |
| sdake | there is also a red hat bug on this issue | 09:04 |
| sdake | https://bugzilla.redhat.com/show_bug.cgi?id=1324922 | 09:05 |
| openstack | bugzilla.redhat.com bug 1324922 in erlang "Log handler repeatedly crashes" [Urgent,On_qa] - Assigned to jeckersb | 09:05 |
| *** daneyon_ has joined #openstack-kolla | 09:07 | |
| sdake | mandre just for completeness can you add a link to that bug tracker | 09:11 |
| sdake | since that is our upstream for this bug | 09:11 |
| *** daneyon_ has quit IRC | 09:11 | |
| mandre | are you sure it's the same bug? I've seen you commented on it, but the trace looks somewhat different from what I remember | 09:12 |
| sdake | same bug | 09:14 |
| sdake | but your right different presentation | 09:14 |
| sdake | it revolves around epmd binding | 09:14 |
| sdake | whatever the case, -17 is totally broken | 09:14 |
| sdake | and I expect red hat will track it in that bug or provide links to other bugs where they sovlve the ipv6 problem with epmd :) | 09:15 |
| sdake | epmd is written in c | 09:16 |
| sdake | i've looked it over | 09:16 |
| sdake | the binding code is plain wrong | 09:16 |
| openstackgerrit | Martin André proposed openstack/kolla: Clarify comment about binding erlang to IPv4 https://review.openstack.org/303969 | 09:16 |
| *** jmccarthy has left #openstack-kolla | 09:16 | |
| mandre | sdake: should be good now | 09:17 |
| sdake | agree | 09:17 |
| sdake | coolsvap asalkeld nihilifer anyone else still awake can we get an ack on that | 09:17 |
| sdake | Jeffrey4l_ ? | 09:18 |
| coolsvap | done | 09:18 |
| openstackgerrit | Merged openstack/kolla: Fix rabbitmq crash in centos gate https://review.openstack.org/303951 | 09:20 |
| *** weiyu has quit IRC | 09:23 | |
| *** ccesario___ has quit IRC | 09:37 | |
| *** Kimmo_ has joined #openstack-kolla | 09:48 | |
| *** weiyu has joined #openstack-kolla | 09:49 | |
| *** allen_gao has quit IRC | 09:50 | |
| *** allen_gao has joined #openstack-kolla | 09:55 | |
| *** jtriley has joined #openstack-kolla | 09:55 | |
| *** jtriley has quit IRC | 10:00 | |
| *** weiyu has quit IRC | 10:10 | |
| *** coolsvap has quit IRC | 10:15 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Revert "Spec: Deploy Kolla images using Kubernetes" https://review.openstack.org/304010 | 10:17 |
| *** weiyu has joined #openstack-kolla | 10:19 | |
| *** reddy has quit IRC | 10:22 | |
| Jeffrey4l_ | sdake, back. | 10:23 |
| *** reddy has joined #openstack-kolla | 10:25 | |
| sdake | hey Jeffrey4l_ | 10:25 |
| sdake | Jeffrey4l_ can you ack the above patch | 10:25 |
| sdake | spec workflow process was not followed | 10:26 |
| Jeffrey4l_ | sdake, i am checking it. | 10:26 |
| sdake | we dont really have a specworkflow process | 10:26 |
| sdake | but historically we require majority vote on any specs | 10:26 |
| sdake | instead there was only 2 votes | 10:27 |
| *** mgoddard_ has joined #openstack-kolla | 10:27 | |
| sdake | and I would have voted -1 had I known it was hiding around in the review queue | 10:27 |
| sdake | since it is incomplete | 10:27 |
| *** andrewvwebber has joined #openstack-kolla | 10:28 | |
| *** mgoddard has quit IRC | 10:30 | |
| *** wmiller has quit IRC | 10:30 | |
| *** wmiller has joined #openstack-kolla | 10:31 | |
| *** weiyu has quit IRC | 10:32 | |
| *** weiyu has joined #openstack-kolla | 10:38 | |
| *** wmiller has quit IRC | 10:41 | |
| *** sdake_ has joined #openstack-kolla | 10:45 | |
| *** ccesario___ has joined #openstack-kolla | 10:47 | |
| *** sdake has quit IRC | 10:48 | |
| *** weiyu has quit IRC | 10:49 | |
| sdake_ | if ryan really wants to do kolla-kubernetes that wfm, but i'd prefer a well thought out specification on what the work is | 10:50 |
| sdake_ | kolla-kubernetes is a major undertaking, and just like kolla-mesos requires majority approval imo :) | 10:51 |
| *** weiyu has joined #openstack-kolla | 10:55 | |
| *** rhallisey has joined #openstack-kolla | 11:06 | |
| ccesario | morning guys!! | 11:12 |
| sdake_ | here is another gate failure that is semi-common | 11:13 |
| sdake_ | https://review.openstack.org/#/c/303969/3 | 11:13 |
| patchbot | sdake_: patch 303969 - kolla - Clarify comment about binding erlang to IPv4 (MERGED) | 11:13 |
| sdake_ | happens only on bluebox from what i can tell | 11:14 |
| sdake_ | hi ccesario | 11:14 |
| sdake_ | rhallisey can you ack that revert patch at the top of the queue | 11:14 |
| rhallisey | already did | 11:14 |
| sdake_ | and submit a new patch once the revert hits the repo | 11:14 |
| ccesario | sdake_, hello.... | 11:14 |
| sdake_ | thanks | 11:14 |
| rhallisey | shouldn't have been merged.. | 11:14 |
| sdake_ | sorry about that totaly my fault | 11:14 |
| sdake_ | i didn't tell vikram about our workflow for specs | 11:14 |
| rhallisey | sdake_, nah I should've -1 | 11:14 |
| rhallisey | d it | 11:14 |
| ccesario | sdake_, I saw your and Jeffrey4l_ comments.... | 11:15 |
| sdake_ | still my fault ;) | 11:15 |
| rhallisey | it was still WIP. Didn't make it clear | 11:15 |
| rhallisey | whatever it's fine | 11:15 |
| *** weiyu has quit IRC | 11:15 | |
| Jeffrey4l_ | ccesario, cool. what do you think? | 11:15 |
| *** tfukushima has quit IRC | 11:15 | |
| sdake_ | when yu get your spec resubmitted we can get action on the reviews going | 11:15 |
| rhallisey | ya agreed | 11:15 |
| sdake_ | so you can actually get a well written spec ready :) | 11:15 |
| rhallisey | ya that one was stale | 11:16 |
| ccesario | well, to clarify for me.... So, the ideia is make is dynamic using kolla_docker.py !? | 11:16 |
| sdake_ | i would hae done thta already but i didn't notie it | 11:16 |
| sdake_ | ccesario right | 11:16 |
| sdake_ | ccesario i dont know precisely how to implement it | 11:16 |
| ccesario | Jeffrey4l_, I agree with you | 11:16 |
| Jeffrey4l_ | ccesario, correct. just check the build_container function part. | 11:16 |
| ccesario | sdake_, I know | 11:16 |
| sdake_ | but i think Jeffrey4l_ has a good solution | 11:16 |
| ccesario | Jeffrey4l_, based on container name ? | 11:16 |
| Jeffrey4l_ | ccesario, yes. it will be enough, with less confused. | 11:17 |
| Jeffrey4l_ | when `docker exec -it nova_api bash`, show the container name is helpful. show other name will be confused. | 11:18 |
| ccesario | Jeffrey4l_, sdake_ well... I have this running in my env.... please look this and check if you agree.. http://paste.openstack.org/show/493631/ | 11:18 |
| sdake_ | Jeffrey4l_ even author agrees spec was prematurely merged | 11:18 |
| sdake_ | https://review.openstack.org/#/c/304010/ | 11:18 |
| patchbot | sdake_: patch 304010 - kolla - Revert "Spec: Deploy Kolla images using Kubernetes" | 11:18 |
| sdake_ | please lets get this fixed asap | 11:19 |
| sbezverk | sdake have you seen my reply to your comment? is it sufficient to change your mind from +1 to +2 ?? ;-) | 11:19 |
| sdake_ | sbezverk didn't see it | 11:19 |
| sdake_ | link? | 11:19 |
| sdake_ | sbezverk i've been tyring to cleanup the review queue and get the backports going all evening | 11:20 |
| sdake_ | hey look its 4:20 | 11:20 |
| sbezverk | https://review.openstack.org/#/c/300004/18 | 11:20 |
| rhallisey | sdake_, when does the sun come up in arizona these days? | 11:20 |
| patchbot | sbezverk: patch 300004 - kolla - Add iscsid/tgtd specific parameters to Kolla infr... | 11:20 |
| sdake_ | no idea | 11:20 |
| sdake_ | burning midnight oil today | 11:20 |
| Jeffrey4l_ | ccesario, your code has bug. the environment variable is ignored. | 11:21 |
| sdake_ | sbezverk where is it duplicated? | 11:21 |
| sdake_ | github link? | 11:21 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Set db connection retry to infinity https://review.openstack.org/302434 | 11:22 |
| ccesario | Jeffrey4l_, well... look this | 11:22 |
| ccesario | http://paste.openstack.org/show/493632/ | 11:22 |
| ccesario | line 138 | 11:22 |
| sbezverk | sdake iscsi uses port 3260 | 11:22 |
| sdake_ | looks good rhallisey :) | 11:23 |
| rhallisey | cool | 11:23 |
| ccesario | Jeffrey4l_, I have deployed and all container have the KOLLA_SERVICE_NAME env var | 11:23 |
| sbezverk | if you check the right side | 11:23 |
| sbezverk | we check already for the right group using config variable | 11:23 |
| sdake_ | line 138 of what sbezverk | 11:23 |
| sdake_ | line 138 on the right side is checking keystone | 11:24 |
| Jeffrey4l_ | sorry, i make a mistake. Your code works. bug this line is useless env = self.params.get('environment') | 11:24 |
| Jeffrey4l_ | sorry.. | 11:24 |
| Jeffrey4l_ | i am wrong. | 11:24 |
| sbezverk | sdake line 154 | 11:24 |
| ccesario | Jeffrey4l_, no replace _ to - ? | 11:24 |
| sdake_ | sbezverk thts to check if the port is free | 11:24 |
| ccesario | keep original name ? | 11:24 |
| sdake_ | got it | 11:25 |
| sdake_ | issed that | 11:25 |
| Jeffrey4l_ | it is a diff file, i just aware. | 11:25 |
| sdake_ | missed that | 11:25 |
| Jeffrey4l_ | OK. the only question: should we request replate _ to -? | 11:25 |
| Jeffrey4l_ | ccesario, ^ | 11:26 |
| sdake_ | sbezverk ok all set | 11:26 |
| sdake_ | sbezverk btw i added you to kolla group on fedorapeopole.org | 11:26 |
| ccesario | Jeffrey4l_, I don't know ... I jus replace to see better in PS1 var | 11:26 |
| sdake_ | or whereever the right url is | 11:26 |
| *** mlima has joined #openstack-kolla | 11:28 | |
| ccesario | Jeffrey4l_, may I send the patch set based in this code!? | 11:28 |
| Jeffrey4l_ | ccesario, cool. btw, your PS is very helpful for operators. thanks. | 11:29 |
| ccesario | Jeffrey4l_, thanks.... I 'll provide it in few minutes | 11:29 |
| Jeffrey4l_ | OK. | 11:29 |
| ccesario | thanks a lot guys by infos | 11:29 |
| sdake_ | Jeffrey4l_ now that we have fixed the epmd + erlang + rabbitmq crash a common gate fialure occurs only on bluebox | 11:30 |
| sdake_ | http://logs.openstack.org/40/299540/1/check/gate-kolla-dsvm-deploy-centos-source/6c69e15/console.html#_2016-04-11_09_06_37_454 | 11:30 |
| sdake_ | i have seen this a whoel lot today | 11:30 |
| sdake_ | every failed job is bluebox + link local waiting for keepalived | 11:30 |
| openstackgerrit | Merged openstack/kolla: Revert "Spec: Deploy Kolla images using Kubernetes" https://review.openstack.org/304010 | 11:30 |
| sdake_ | i'm thinking multicast is turned off possibly? | 11:31 |
| Jeffrey4l_ | sdake_, i am confused. let me checking the log more carefully. | 11:32 |
| sdake_ | keepalived uses multicast | 11:34 |
| sdake_ | if multicast is disabled i suppose its possible for keepalived not to assign a link local address | 11:34 |
| Jeffrey4l_ | sdake_, first of all, this should not related to the fix of rabbitmq. The rabbitmq even not be ran | 11:34 |
| Jeffrey4l_ | sdake_, maybe. may be need some debug/test for this issue in the gate. | 11:35 |
| sdake_ | i agree this is a different issue | 11:35 |
| sdake_ | our gates have been terribly flakey | 11:35 |
| *** andrewvwebber has quit IRC | 11:35 | |
| sdake_ | they revolve around the epmd crash | 11:35 |
| sdake_ | and the bluebox provider | 11:35 |
| sdake_ | the bluebox provider blows up on waiting for vip | 11:35 |
| *** Yogesh_Deshmukh has joined #openstack-kolla | 11:36 | |
| Jeffrey4l_ | yea. so many test nodes without the same configuration, which is annoying. | 11:37 |
| sdake_ | Jeffrey4l_ http://logs.openstack.org/40/299540/1/check/gate-kolla-dsvm-deploy-centos-source/6c69e15/logs/kolla/keepalived/keepalived.log | 11:38 |
| sdake_ | check out the bogus vrrp | 11:39 |
| Jeffrey4l_ | invalid password ? interesting. | 11:40 |
| sdake_ | yup | 11:41 |
| sdake_ | bluebox only | 11:41 |
| sdake_ | i have never seen it elsewhere | 11:41 |
| sdake_ | i'll file bug for that | 11:42 |
| sdake_ | moment | 11:42 |
| Jeffrey4l_ | maybe, there are two kolla tests running on bluebox. they can communicate. But we have different keepalived password for different test( gernerated randomly). So one keepalived received a vrrp package with wrong password? | 11:42 |
| Jeffrey4l_ | is this possible? | 11:42 |
| *** jtriley has joined #openstack-kolla | 11:43 | |
| Jeffrey4l_ | sdake_, when only one keepalived instance. it should work even the vrrp packet is blocked. | 11:45 |
| sdake_ | wackamole: https://bugs.launchpad.net/kolla/+bug/1568818 | 11:46 |
| openstack | Launchpad bug 1568818 in kolla mitaka "bluebox gate generates invalid password in keepalived logs" [High,Confirmed] | 11:46 |
| sdake_ | Jeffrey4l_ this is possible | 11:47 |
| Jeffrey4l_ | sdake_, notice this line (bogus VRRP packet received on eth0 !!!). I am a little sure for that. | 11:47 |
| Yogesh_Deshmukh | Hi, I'm deploying kolla on 3 node setup i.e. 1 control, compute and network node. | 11:47 |
| Yogesh_Deshmukh | without haproxy, I'm giving control node first interfaces IP address in kolla_internal_vip_address in globals.yml. | 11:47 |
| sdake_ | hey Yogesh_Deshmukh | 11:47 |
| Yogesh_Deshmukh | Is this correct? | 11:48 |
| sdake_ | why not run with haproxy? | 11:48 |
| *** jtriley has quit IRC | 11:48 | |
| sdake_ | running without haproxy really only works well on AIO deploys | 11:48 |
| sdake_ | i wouldnt recommend running without haproxy at all personally | 11:48 |
| Yogesh_Deshmukh | For single control node do we need haproxy? | 11:48 |
| sdake_ | I never run kolla that way and neither do a majority of the developers | 11:48 |
| sdake_ | haproxy sets up your internal and external VIPs | 11:49 |
| sdake_ | it could probablybe made to work without haproxy but why bother? | 11:49 |
| sdake_ | worried about performance? | 11:49 |
| *** daneyon_ has joined #openstack-kolla | 11:50 | |
| sdake_ | i am petty sure but not certian you cannot do mutinode of any type without haproxy | 11:50 |
| sdake_ | the person to talk to on this subject is inc0 | 11:50 |
| sdake_ | or dave-mccowan | 11:50 |
| Yogesh_Deshmukh | Yeah, but I'm running for testing purpose. | 11:50 |
| sdake_ | ya test with haproxy then ? :) | 11:50 |
| Yogesh_Deshmukh | Ok, will run with haproxy. | 11:50 |
| Yogesh_Deshmukh | what will be in kolla_internal_vip_address? | 11:51 |
| Yogesh_Deshmukh | any random non exist IP? | 11:51 |
| sdake_ | yes but it must be routable from your management network interface | 11:52 |
| *** vivek__ has joined #openstack-kolla | 11:53 | |
| *** daneyon_ has quit IRC | 11:54 | |
| *** wmiller has joined #openstack-kolla | 11:57 | |
| Jeffrey4l_ | sdake_, let me try to use a random `keepalived_virtual_router_id` in the gate test. | 11:59 |
| Jeffrey4l_ | I will this should fix this issue. | 11:59 |
| *** sdake_ is now known as sdake | 11:59 | |
| Jeffrey4l_ | s/will/think/ | 11:59 |
| *** mgoddard_ has quit IRC | 12:00 | |
| sdake | interesitng idea Jeffrey4l_ | 12:01 |
| *** allen_gao has quit IRC | 12:04 | |
| sdake | Jeffrey4l_ you mind assigning that bug to yourself then since your looking into it? | 12:04 |
| sdake | i'd say priortiize it after the 15th | 12:04 |
| *** allen_gao has joined #openstack-kolla | 12:04 | |
| sdake | but dontworry about it prior | 12:05 |
| Jeffrey4l_ | sdake, I have already done that. :D | 12:06 |
| sdake | cool | 12:06 |
| sdake | pretty sure this is our last gate failure | 12:07 |
| sdake | i am not sure i bluebox always fails | 12:07 |
| sdake | or if there are colocated jobs in bluebox | 12:07 |
| Jeffrey4l_ | how about the keystone issue in the gate. is it fixed? | 12:09 |
| Jeffrey4l_ | lets push the fix for keepalived and see what will happen. | 12:09 |
| *** sdake_ has joined #openstack-kolla | 12:10 | |
| sbezverk | sdake if you have time please check this: https://review.openstack.org/#/c/303858/ | 12:11 |
| patchbot | sbezverk: patch 303858 - kolla - Rabbitmq Upgrade restarts container only when imag... | 12:11 |
| *** sdake has quit IRC | 12:12 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Use a random virtual_router_id in the keepalived https://review.openstack.org/304080 | 12:13 |
| *** dave-mccowan has joined #openstack-kolla | 12:14 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Use a random virtual_router_id in the keepalived https://review.openstack.org/304080 | 12:14 |
| *** sdake_ has quit IRC | 12:18 | |
| *** achanda has quit IRC | 12:18 | |
| openstackgerrit | Martin André proposed openstack/kolla: Clarify comment about binding erlang to IPv4 https://review.openstack.org/304091 | 12:26 |
| Yogesh_Deshmukh | sdake, now I'm running with haproxy but it is giving error on 'Waiting for MariaDB service to be ready through VIP' | 12:32 |
| Yogesh_Deshmukh | Can't connect to MySQL server on '192.168.20.205' (113 "No route to host") | 12:32 |
| Yogesh_Deshmukh | the management IP is 192.168.20.202. i.e. 205 is on same network of management. I'm deplyoing Kolla on VMs. | 12:34 |
| *** achanda has joined #openstack-kolla | 12:35 | |
| *** achanda has quit IRC | 12:35 | |
| *** salv-orl_ has quit IRC | 12:36 | |
| sbezverk | Yogesh_Deshmukh check if you can ping 192.168.20.205, check if keepalived is running. | 12:36 |
| sbezverk | Yogesh_Deshmukh also check for haproxy container | 12:37 |
| Yogesh_Deshmukh | ok | 12:39 |
| *** mgoddard has joined #openstack-kolla | 12:40 | |
| *** achanda has joined #openstack-kolla | 12:44 | |
| *** jtriley has joined #openstack-kolla | 12:44 | |
| sbezverk | rhallisey if you have a minute could please you check this: https://review.openstack.org/#/c/300004/18 | 12:47 |
| patchbot | sbezverk: patch 300004 - kolla - Add iscsid/tgtd specific parameters to Kolla infr... | 12:47 |
| *** mwheckmann has joined #openstack-kolla | 12:47 | |
| rhallisey | sbezverk, looks great | 12:50 |
| sbezverk | rhallisey thank you.. There are two other parts which also need reviews, but I do not want to be pushy ;-) | 12:51 |
| rhallisey | ya I'm lookin | 12:51 |
| rhallisey | just a minute | 12:51 |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Identify container through PS1 shell variable https://review.openstack.org/297984 | 12:52 |
| ccesario | Jeffrey4l_, ^^ | 12:53 |
| *** ByPasS has joined #openstack-kolla | 12:54 | |
| rhallisey | sbezverk, do you use ubuntu or centos? | 12:55 |
| rhallisey | which have you been testing on | 12:55 |
| rhallisey | as a base OS not a base container os | 12:56 |
| sbezverk | rhallisey tested both | 12:56 |
| rhallisey | did you have any issue with /dev/pts/ptmx ? | 12:56 |
| rhallisey | centos by default sets that to 000 | 12:57 |
| sbezverk | rhallisey nope. never seen it, the only issue we hit on ubuntu, configfs has not been mounted by default | 12:57 |
| *** inc0 has joined #openstack-kolla | 12:57 | |
| *** klint has quit IRC | 12:57 | |
| rhallisey | https://www.kernel.org/doc/Documentation/filesystems/devpts.txt | 12:58 |
| rhallisey | sbezverk, I think it be worth mentioning in docs somewhere to set this | 12:59 |
| rhallisey | I'd imagine someone will run into this | 12:59 |
| sbezverk | rhallisey from the first read I do not see how it is applicable to our case, but I will read it again.. | 13:01 |
| rhallisey | sbezverk, I'm just saying that I have seen ptmx be 000 | 13:04 |
| sbezverk | rhallisey got it.. | 13:04 |
| rhallisey | and it will cause iscsi to fail to connect | 13:04 |
| sbezverk | rhallisey which was flavor/version and kernel version you have seen this condition? | 13:06 |
| rhallisey | I saw it in centos-atomic from ~month ago | 13:06 |
| rhallisey | had the same issue when I did cinder way back | 13:06 |
| Yogesh_Deshmukh | sbezverk, I checked, 192.168.20.205 is pinable. The haproxy and keepalived containers are present only on network node. Is that correct? | 13:07 |
| sbezverk | Yogesh_Deshmukh ah that is where I think your problem is, haproxy and keepalived should also be on controller node | 13:07 |
| sbezverk | basically their purpose to protect your control plane from a failure | 13:08 |
| sbezverk | rhallisey Interesting, never played with atomic flavor.. will keep that in mind. I have PS that adds documentation for iscsi, I will add a link to the page you gave me.. | 13:09 |
| *** ccesario___ has quit IRC | 13:09 | |
| rhallisey | sbezverk, yes please | 13:10 |
| *** stvnoyes has left #openstack-kolla | 13:11 | |
| *** ccesario___ has joined #openstack-kolla | 13:11 | |
| Yogesh_Deshmukh | sbezverk ahh, I also thought the same issue but in inventory file for multinode, [haproxy:children] network. | 13:12 |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Identify container through PS1 shell variable https://review.openstack.org/297984 | 13:13 |
| Yogesh_Deshmukh | So haproxy and keepalived are only deployed on network. I'm confused here. | 13:13 |
| Yogesh_Deshmukh | the haproxy deployment is conditional include: config.yml when: inventory_hostname in groups['haproxy'] in ansible/roles/haproxy/tasks/deploy.yml | 13:15 |
| Yogesh_Deshmukh | How haproxy is also deployed on control node? | 13:15 |
| sbezverk | Yogesh_Deshmukh yes | 13:16 |
| sbezverk | rhallisey I just checked my target server and it does have /dev/pts/ptmx : c---------. 1 root root 5, 2 Apr 11 09:59 /dev/pts/ptmx | 13:18 |
| rhallisey | sbezverk, so then it worked with that | 13:18 |
| rhallisey | hm | 13:18 |
| inc0 | good morning | 13:19 |
| sbezverk | rhallisey yep | 13:19 |
| sbezverk | it is not centos, it is rhel 7.2 but it should be the same thing.. | 13:20 |
| sbezverk | good morning inc0 | 13:21 |
| rhallisey | maybe scsi isn't create a session in this case.. idk | 13:21 |
| inc0 | https://www.youtube.com/watch?v=PivpCKEiQOQ&feature=youtu.be I hope all of you have seen it? | 13:22 |
| rhallisey | sbezverk, did you try multinode? | 13:22 |
| rhallisey | idk if that would make a difference | 13:23 |
| rhallisey | sbezverk, if you haven't I'll give it a try on centos and see | 13:23 |
| *** ccesario___ has quit IRC | 13:23 | |
| *** ccesario___ has joined #openstack-kolla | 13:24 | |
| sbezverk | rhallisey nope, do not have h/w yet | 13:25 |
| sbezverk | if you could test, that would be fantastic | 13:26 |
| rhallisey | sbezverk, sure | 13:26 |
| rhallisey | sbezverk, I actually don't have the h/w either. I cheat :) | 13:26 |
| rhallisey | when I finish the heat template I got I'll share them for dev purposes | 13:27 |
| sbezverk | rhallisey how ? please share? VMs? | 13:27 |
| rhallisey | could also use vagrant | 13:27 |
| rhallisey | sbezverk, I use virt on virt with tripleo | 13:28 |
| rhallisey | deploy nodes with ironic | 13:28 |
| rhallisey | but they are 'vms' | 13:28 |
| rhallisey | works great | 13:28 |
| rhallisey | I'm not familiar with vagrant so this works for me | 13:29 |
| *** banix has joined #openstack-kolla | 13:30 | |
| sbezverk | rhallisey Interesting, will look into virt with tripleo, thank you for info.. | 13:32 |
| rhallisey | sbezverk, if you do look into it. https://github.com/rthallisey/kolla-heat-templates | 13:33 |
| rhallisey | sbezverk, I'll try and come out with docs after I smooth eveything out in the templates | 13:34 |
| *** salv-orlando has joined #openstack-kolla | 13:37 | |
| *** alyson_ has joined #openstack-kolla | 13:38 | |
| *** daneyon_ has joined #openstack-kolla | 13:38 | |
| *** daneyon_ has quit IRC | 13:42 | |
| *** salv-orlando has quit IRC | 13:43 | |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Identify container through PS1 shell variable https://review.openstack.org/297984 | 13:45 |
| *** Hui has joined #openstack-kolla | 13:46 | |
| *** achanda has quit IRC | 13:50 | |
| *** achanda has joined #openstack-kolla | 13:58 | |
| ccesario | rhallisey, Kolla Triple-O ?! O_o | 13:58 |
| rhallisey | ccesario, I use it for multinode | 13:58 |
| rhallisey | since I don't have the hardware | 13:58 |
| rhallisey | I deploy stock undercloud and use my own heat-template to deploy nodes | 13:59 |
| ccesario | rhallisey, ohhh nice to know .. :) | 13:59 |
| rhallisey | feed those ips into kolla and done, multinode | 13:59 |
| ccesario | I don't have hardwarde too.... but I usage other method :P | 13:59 |
| rhallisey | you just leave the undercloud around and keep making new stacks | 13:59 |
| pabelanger | morning, we'd like to land https://review.openstack.org/#/c/303414/ if possible today. Its a requirement for openstack-infra to migrate kolla from devstack-centos7 to centos-7 | 13:59 |
| patchbot | pabelanger: patch 303414 - kolla - Make sure /etc/hostname is valid for SELinux | 13:59 |
| rhallisey | ccesario, ya this method is heavy I'd say | 14:00 |
| pabelanger | see https://review.openstack.org/#/c/301320/ for the openstack-infra change | 14:00 |
| patchbot | pabelanger: patch 301320 - openstack-infra/project-config - Migrate kolla jobs to centos-7 dib | 14:00 |
| rhallisey | ccesario, allows you to use ironic which is sweet | 14:00 |
| ccesario | rhallisey, VMware -> Virtual Machine - > 4 KVM machines to play Kolla :) | 14:01 |
| *** ccesario___ has quit IRC | 14:01 | |
| ccesario | rhallisey, nice... | 14:01 |
| rhallisey | ccesario, cool | 14:01 |
| ccesario | rhallisey, nested over nested :P | 14:01 |
| rhallisey | ccesario, I know :) | 14:02 |
| ccesario | :) | 14:03 |
| *** achanda has quit IRC | 14:07 | |
| *** mgoddard has quit IRC | 14:07 | |
| *** mgoddard has joined #openstack-kolla | 14:08 | |
| *** bmotz has left #openstack-kolla | 14:18 | |
| *** bmotz has joined #openstack-kolla | 14:18 | |
| *** dwalsh has joined #openstack-kolla | 14:24 | |
| *** reddy has quit IRC | 14:24 | |
| *** ccesario___ has joined #openstack-kolla | 14:30 | |
| *** haplo37 has joined #openstack-kolla | 14:35 | |
| *** vhosakot has joined #openstack-kolla | 14:37 | |
| *** tpeoplesvacation is now known as tpeoples | 14:40 | |
| *** Jeffrey4l_ has quit IRC | 14:51 | |
| *** inc0 has quit IRC | 14:59 | |
| *** mgoddard_ has joined #openstack-kolla | 15:01 | |
| *** bmotz_ has joined #openstack-kolla | 15:02 | |
| *** Serlex has quit IRC | 15:02 | |
| *** salv-orlando has joined #openstack-kolla | 15:03 | |
| *** salv-orlando has quit IRC | 15:03 | |
| *** salv-orlando has joined #openstack-kolla | 15:04 | |
| *** mgoddard has quit IRC | 15:04 | |
| *** wmiller has quit IRC | 15:04 | |
| *** wmiller has joined #openstack-kolla | 15:05 | |
| *** bmotz has quit IRC | 15:05 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Spec: Deploy Kolla images using Kubernetes https://review.openstack.org/304182 | 15:13 |
| *** Jeffrey4l_ has joined #openstack-kolla | 15:15 | |
| *** achanda has joined #openstack-kolla | 15:15 | |
| *** inc0 has joined #openstack-kolla | 15:16 | |
| *** openstackgerrit has quit IRC | 15:18 | |
| *** openstackgerrit has joined #openstack-kolla | 15:18 | |
| *** The_Ball has quit IRC | 15:20 | |
| *** daneyon_ has joined #openstack-kolla | 15:26 | |
| mlima | vhosakot, can you remove the -1 these patchsets https://review.openstack.org/#/c/303055/ and https://review.openstack.org/#/c/303575/ ? | 15:33 |
| patchbot | mlima: patch 303055 - kolla - Remove /etc/kolla configuration folders | 15:33 |
| patchbot | mlima: patch 303575 - kolla - Add a role to cleanup containers after deployment | 15:33 |
| vhosakot | mlima: yep, reviewing them now... few ming | 15:34 |
| vhosakot | mins | 15:34 |
| *** Yogesh_Deshmukh has quit IRC | 15:34 | |
| mlima | vhosakot, thank you :) | 15:34 |
| vhosakot | cool, np | 15:34 |
| openstackgerrit | Paul Belanger proposed openstack/kolla: Make sure /etc/hostname is valid for SELinux https://review.openstack.org/303414 | 15:38 |
| pabelanger | please review ^ per your requests | 15:39 |
| vhosakot | mlima: np! | 15:41 |
| *** mimizone has joined #openstack-kolla | 15:41 | |
| *** yogesh_deshmukh has joined #openstack-kolla | 15:45 | |
| gmmaha | vhosakot: mlima: quick Q about https://bugs.launchpad.net/kolla/+bug/1566538 | 15:45 |
| openstack | Launchpad bug 1566538 in kolla "generate_password.yml does not add new keys added to deployment" [Undecided,Incomplete] - Assigned to Mauricio Lima (mlima) (mauricio-lima) | 15:45 |
| gmmaha | You have mentioned that it works in the latest master, but i havent seen any changes in the generate file since the last time i tested. | 15:46 |
| mlima | yes gmmaha | 15:46 |
| gmmaha | maybe i am not explaining the scenario properly. | 15:46 |
| gmmaha | lets say i have an old install of kolla and there are new variables being added to the passwords.yml file. if i dont get the new values into /etc/kolla/passwords.yml, they will never get injected into the file | 15:47 |
| gmmaha | let me add the explanation to the bug.. I am sure you guys are busy with the release. maybe that might clarify | 15:47 |
| *** dwalsh has quit IRC | 15:47 | |
| mlima | I think we need to define a template to report bugs | 15:48 |
| mlima | :) | 15:48 |
| gmmaha | mlima: or i have to get a better hold on my english :( | 15:48 |
| mlima | no no gmmaha | 15:48 |
| mlima | I mean that many bugs does not have a step by step to test and confirm the bug. | 15:49 |
| mlima | Environment | 15:49 |
| gmmaha | mlima: true that. :) | 15:49 |
| mlima | Ubuntu/Source or Centos/binary | 15:49 |
| mlima | this is very useful | 15:50 |
| gmmaha | mlima: aah yes.. agreed.. | 15:51 |
| gmmaha | thanks.. will add them all | 15:51 |
| vhosakot | gmmaha: cool thanks! | 15:52 |
| *** dwalsh has joined #openstack-kolla | 15:53 | |
| gmmaha | vhosakot: mlima: added to the bug. thanks | 15:57 |
| mlima | ok | 15:57 |
| vhosakot | gmmaha: ok, thanks! | 15:57 |
| *** mikelk has quit IRC | 15:58 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the gate lying https://review.openstack.org/304205 | 15:59 |
| mlima | gmmaha, can you test using the master branch and instead of using "tools / generate_passwords.py" use "kolla-genpwd" | 16:01 |
| gmmaha | mlima: sure.. i can test master shortly | 16:01 |
| gmmaha | but i thought generate_passwords.yml was a symlink to kolla-genpwd | 16:01 |
| gmmaha | but yes, will definitely test it today and let you know if its different | 16:02 |
| *** inc0_ has joined #openstack-kolla | 16:08 | |
| pmisiak | hi guys | 16:09 |
| *** openstackstatus has joined #openstack-kolla | 16:09 | |
| *** ChanServ sets mode: +v openstackstatus | 16:09 | |
| pmisiak | there are still problems with nova/instances access rights | 16:09 |
| pmisiak | centos/source | 16:10 |
| pmisiak | qemu user is in nova group | 16:10 |
| pmisiak | but /var/lib/nova and /var/lib/nova/instances has wrong rights | 16:10 |
| pmisiak | folder /var/lib/nova hos no rights for group | 16:11 |
| *** inc0 has quit IRC | 16:11 | |
| pmisiak | folder /var/lib/nova/instances nas no write rights for group | 16:11 |
| pmisiak | and it is after applying this commit: https://github.com/openstack/kolla/commit/ad1461f46d730ed0393ffcb16b866ed2e0f34943 | 16:12 |
| *** dwalsh has quit IRC | 16:12 | |
| pmisiak | I had to do: chmod g+rx /var/lib/nova and chmod -R g+w /var/lib/nova/instances to be able to spawn a VM | 16:13 |
| *** mbound has quit IRC | 16:13 | |
| *** mbound has joined #openstack-kolla | 16:14 | |
| openstackgerrit | Merged openstack/kolla: Make sure /etc/hostname is valid for SELinux https://review.openstack.org/303414 | 16:14 |
| *** mbound has quit IRC | 16:18 | |
| *** mark-casey has joined #openstack-kolla | 16:19 | |
| *** ccesario___ has quit IRC | 16:21 | |
| yogesh_deshmukh | Hi, I stuck at 'Creating the Glance service and endpoint' when deploying Kolla on 3 Node setup (each control, compute and network). | 16:22 |
| yogesh_deshmukh | I looked for ansible.log from kolla_toolbox container it is showing 'Could not determine a suitable URL for the plugin' errror message. | 16:22 |
| yogesh_deshmukh | Any idea? | 16:23 |
| *** dwalsh has joined #openstack-kolla | 16:24 | |
| rhallisey | yogesh_deshmukh, that sounds like a configuration issue | 16:27 |
| rhallisey | something with you vips not be available on all nodes | 16:27 |
| rhallisey | or something | 16:27 |
| openstackgerrit | Merged openstack/kolla: Increase max pool size so conductor doesn't implode https://review.openstack.org/303677 | 16:27 |
| yogesh_deshmukh | rhallisey, how to check vip is working fine? | 16:29 |
| *** unicell has quit IRC | 16:30 | |
| rhallisey | yogesh_deshmukh, ping the addres you specified | 16:30 |
| rhallisey | for each of your hosts | 16:30 |
| *** ccesario___ has joined #openstack-kolla | 16:30 | |
| yogesh_deshmukh | rhallisey, I pinged vip from all i.e. 3 nodes that is working fine. | 16:32 |
| yogesh_deshmukh | haproxy and keepalived containers are deployed only on control node, is that correct or do we need them on all nodes? | 16:34 |
| rhallisey | that's fine | 16:34 |
| *** Hui has quit IRC | 16:40 | |
| *** Marga_ has joined #openstack-kolla | 16:42 | |
| *** inc0_ has quit IRC | 16:44 | |
| *** ccesario___ has quit IRC | 16:44 | |
| *** Marga_ has quit IRC | 16:51 | |
| *** Marga_ has joined #openstack-kolla | 16:52 | |
| *** Marga_ has quit IRC | 16:53 | |
| *** yogesh_deshmukh has quit IRC | 16:53 | |
| *** unicell has joined #openstack-kolla | 16:58 | |
| *** rajathagasthya has joined #openstack-kolla | 17:00 | |
| *** Marga_ has joined #openstack-kolla | 17:00 | |
| *** Marga_ has quit IRC | 17:00 | |
| *** Marga_ has joined #openstack-kolla | 17:01 | |
| *** daneyon_ has quit IRC | 17:01 | |
| *** inc0 has joined #openstack-kolla | 17:04 | |
| ccesario | Jeffrey4l_, just reply your questions | 17:06 |
| *** ccesario___ has joined #openstack-kolla | 17:11 | |
| *** unicell has quit IRC | 17:15 | |
| *** mgoddard has joined #openstack-kolla | 17:16 | |
| *** mgoddard_ has quit IRC | 17:17 | |
| *** unicell has joined #openstack-kolla | 17:17 | |
| *** ccesario___ has quit IRC | 17:20 | |
| *** Marga_ has quit IRC | 17:21 | |
| *** ccesario___ has joined #openstack-kolla | 17:21 | |
| *** Marga_ has joined #openstack-kolla | 17:23 | |
| *** Marga_ has quit IRC | 17:28 | |
| *** jasonsb has joined #openstack-kolla | 17:30 | |
| *** daneyon_ has joined #openstack-kolla | 17:31 | |
| *** neophy has joined #openstack-kolla | 17:40 | |
| *** ccesario___ has quit IRC | 17:41 | |
| *** shardy has quit IRC | 17:49 | |
| vhosakot | yogesh_deshmukh: did you run prechecks ? | 17:49 |
| *** JoseMello has joined #openstack-kolla | 17:50 | |
| vhosakot | yep haproxy and keepalived on control nodes is correct | 17:51 |
| *** shardy has joined #openstack-kolla | 17:53 | |
| *** achanda has quit IRC | 17:53 | |
| *** achanda has joined #openstack-kolla | 17:56 | |
| *** ccesario___ has joined #openstack-kolla | 17:56 | |
| *** daneyon_ has quit IRC | 17:58 | |
| *** Marga_ has joined #openstack-kolla | 18:00 | |
| *** achanda_ has joined #openstack-kolla | 18:04 | |
| *** achanda has quit IRC | 18:04 | |
| *** daneyon_ has joined #openstack-kolla | 18:04 | |
| *** daneyon_ has quit IRC | 18:08 | |
| *** inc0 has quit IRC | 18:11 | |
| *** sdake has joined #openstack-kolla | 18:18 | |
| *** shardy is now known as shardy_afk | 18:20 | |
| *** jasonsb has quit IRC | 18:21 | |
| *** rajathagasthya has quit IRC | 18:21 | |
| *** achanda_ has quit IRC | 18:21 | |
| *** achanda has joined #openstack-kolla | 18:21 | |
| sdake | hey folks | 18:27 |
| *** achanda has quit IRC | 18:27 | |
| *** achanda has joined #openstack-kolla | 18:30 | |
| openstackgerrit | Merged openstack/kolla: Clarify comment about binding erlang to IPv4 https://review.openstack.org/304091 | 18:33 |
| *** vhosakot has quit IRC | 18:36 | |
| *** vhosakot has joined #openstack-kolla | 18:36 | |
| *** achanda has quit IRC | 18:36 | |
| openstackgerrit | Merged openstack/kolla: Fix the neutron reconfigure issue when enable linuxbridge-agent https://review.openstack.org/303783 | 18:38 |
| *** vhosakot has quit IRC | 18:39 | |
| *** vhosakot has joined #openstack-kolla | 18:39 | |
| *** vhosakot has quit IRC | 18:59 | |
| *** vhosakot has joined #openstack-kolla | 18:59 | |
| mlima | heu sdake | 19:05 |
| mlima | hey* | 19:05 |
| sdake | hey mlima | 19:05 |
| mlima | :) | 19:05 |
| mlima | sdake, i think that this bug https://bugs.launchpad.net/kolla/+bug/1534547 was fixed. | 19:07 |
| openstack | Launchpad bug 1534547 in kolla mitaka "compare_environment compares incorrect" [High,Triaged] | 19:07 |
| sdake | mlima thanks do you have a review to point at? | 19:08 |
| daneyon | vhosakot sdake my builds are still failing. Here is the latest https://bugs.launchpad.net/kolla/+bug/1567657 | 19:08 |
| openstack | Launchpad bug 1567657 in kolla "Build Fails Using Tunk" [Undecided,New] | 19:08 |
| mlima | one sec | 19:08 |
| sdake | daneyon are you using https proxy properly ? | 19:09 |
| sdake | dave-mccowan can help you | 19:09 |
| mlima | sdake, https://review.openstack.org/#/c/283887/ | 19:09 |
| patchbot | mlima: patch 283887 - kolla - Switch ceph to use storage network (MERGED) | 19:09 |
| sdake | mlima studyign patch | 19:11 |
| daneyon | sdake thx, i'll sync with dave-mccowan | 19:11 |
| vhosakot | daneyon: i have seen that error too rarely on Ubuntu... I just rebuild and it works... did you rebuild ? may the mirror/repo was down | 19:12 |
| vhosakot | daneyon: HTTPConnectionPool(host='tarballs.openstack.org', port=80): Read timed out. | 19:12 |
| daneyon | vhosakot that error is about 2 hours old | 19:13 |
| vhosakot | daneyon: ah ok | 19:13 |
| vhosakot | daneyon: what container threw this error ? | 19:14 |
| daneyon | vhosakot do i need to wipe out my old docker images when i do a rebuild? | 19:14 |
| vhosakot | daneyon: just rebuidl what failed | 19:14 |
| daneyon | vhosakot error occured right after INFO:kolla.cmd.build:neutron-base:Pushed successfully | 19:15 |
| vhosakot | daneyon: this is that I do... kolla-build --base ubuntu --type source --registry <registry IP>:4000 <container name that failed> | 19:15 |
| vhosakot | daneyon: can you send the command you ran ? did you use --no-cache ? | 19:16 |
| daneyon | vhosakot this is what i do > # kolla-build --type source --registry 10.30.118.91:4000 --push | 19:16 |
| mlima | ok sdake | 19:17 |
| daneyon | vhosakot I just updated the bug, so you can see a bit more info | 19:19 |
| vhosakot | daneyon: --base is centos ? | 19:19 |
| daneyon | vhosakot i believe so, the docs state cent7 is def | 19:19 |
| daneyon | looks like the error is related to building the magnum image | 19:20 |
| vhosakot | yep.. cent is def | 19:20 |
| vhosakot | daneyon: can you try this ? | 19:20 |
| vhosakot | daneyon: kolla-build --type source --registry 10.30.118.91:4000 --push magnum | 19:20 |
| *** salv-orlando has quit IRC | 19:21 | |
| daneyon | vhosakot doing it now | 19:23 |
| vhosakot | daneyon: also, send entire output of kolla-build | 19:27 |
| daneyon | vhosakot: looks like magnum images built https://gist.github.com/danehans/c1a6d7178c820dc35af5dcabd5d3da81 | 19:27 |
| vhosakot | yes, all three were built - magnum-api, magnum-base and magnum-conductor.. cool | 19:28 |
| vhosakot | daneyon: now, can you rerun kolla-build --type source --registry 10.30.118.91:4000 --push | 19:28 |
| daneyon | vhosakot: in progress kolla-build --type source --registry 10.30.118.91:4000 --push | 19:29 |
| vhosakot | daneyon: passed the point where you saw failure last time ? | 19:30 |
| daneyon | vhosakot looks like it | 19:30 |
| daneyon | dave-mccowan fyi ^ looks like vhosakot helped me get past my latest build issue | 19:31 |
| vhosakot | daneyon: cool, BTW, the "Make sure Docker is running and that you have the correct privileges to run Docker (root)" in the bug may be due to permission issue in local env | 19:31 |
| *** Jeffrey4l__ has joined #openstack-kolla | 19:32 | |
| vhosakot | daneyon: let the build finish... some images fail due to intermittent mirror/repo issues from where they are pulled | 19:33 |
| *** neophy has quit IRC | 19:34 | |
| vhosakot | daneyon: usually, a retry works fine | 19:34 |
| *** Jeffrey4l_ has quit IRC | 19:34 | |
| daneyon | vhosakot: OK, I'll start doing that. Thanks again for all the help. | 19:34 |
| vhosakot | cool, np... | 19:34 |
| vhosakot | daneyon: by default, kolla-buila retried three times... you can control that by passing the --retries to kolla-build | 19:35 |
| vhosakot | retires* | 19:35 |
| sdake | mllima got toallly sidetracked | 19:36 |
| *** mwheckmann has quit IRC | 19:41 | |
| vhosakot | daneyon: when you see the error, should you pree CTRL+C to exit ? | 19:43 |
| sdake | mlima i am not confident the bug you linked is the same issue reported | 19:48 |
| sdake | rather the review you linked matche the bug reported | 19:49 |
| mlima | hmm, "But second tasks has no environment variables, and compare_environment make decision that environment is not different." | 19:49 |
| mlima | it is the bug summary ^ | 19:50 |
| sdake | didnt 'caatch that | 19:50 |
| sdake | just woke up | 19:50 |
| sdake | give me a minute to look again | 19:50 |
| sdake | mlima so your saying that review addresses the second task? | 19:51 |
| sdake | its really hard for me to tell | 19:51 |
| sdake | from the review | 19:51 |
| sdake | i see it now | 19:51 |
| sdake | mlima good spot!! | 19:51 |
| mlima | the bug was reported in 2016-01-15 and this patchset https://review.openstack.org/#/c/283887/ was merged in 2016-01-24 | 19:52 |
| patchbot | mlima: patch 283887 - kolla - Switch ceph to use storage network (MERGED) | 19:52 |
| mlima | 2016-02-24** | 19:52 |
| mlima | well, the environment variables are there sdake | 19:53 |
| sdake | ya that patch fixes multiple things | 19:53 |
| sdake | trivialfix abuse at it finest ;) | 19:53 |
| mlima | hahaha | 19:53 |
| *** salv-orlando has joined #openstack-kolla | 19:54 | |
| mlima | super patchset | 19:54 |
| mlima | :) | 19:54 |
| sdake | mlima i yuo find more already fixed please do let me know ;) | 19:54 |
| mlima | ok, I was looking for a bug to fix and i found it | 19:55 |
| mlima | -1 High bug to fix sdake :) | 19:57 |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Identify container through PS1 shell variable https://review.openstack.org/297984 | 19:58 |
| sdake | agreed | 19:58 |
| sdake | every thing counts at this point | 19:58 |
| sdake | 1 week to go | 19:58 |
| mlima | :) | 19:58 |
| sdake | also openstack-inra is blocked on us releasing | 19:58 |
| sdake | so we are prety hard committed to 4/15 at this point | 19:59 |
| sdake | so their work doesn't bleed over into summit | 19:59 |
| mlima | kolla team is working hard :) | 19:59 |
| sdake | i knwo everyone is | 19:59 |
| *** achanda has joined #openstack-kolla | 19:59 | |
| sdake | just pointing out there will be no more slips :) | 19:59 |
| sdake | pabelanger ^^ | 19:59 |
| sbezverk | sdake how about this one: it was marked as a critical? https://review.openstack.org/#/c/303858/ | 20:01 |
| patchbot | sbezverk: patch 303858 - kolla - Rabbitmq Upgrade restarts container only when imag... | 20:01 |
| sdake | i really want inc0 to review that one | 20:02 |
| sbezverk | sdake ok got it | 20:02 |
| sdake | i'd like inc0 to take lead on upgrade review changes | 20:02 |
| sdake | rather reviewing upgrade changes | 20:02 |
| sdake | to be more clear | 20:02 |
| sdake | it is also failing the gate sbezverk - i jsut triggered a recheck on it | 20:03 |
| sbezverk | sdake I am sorry but fixing haproxy and VIP failing at the gate is out of scope of this bug ;-) | 20:05 |
| sdake | sbezverk was it bluebox? | 20:05 |
| *** openstackgerrit has quit IRC | 20:05 | |
| sdake | sbezverk agree - that is why we recheck until gate is green | 20:05 |
| sdake | we are working on the keepalived bug separately | 20:05 |
| *** inc0 has joined #openstack-kolla | 20:05 | |
| sdake | to recheck a failed patch just type recheck in the review and click reply | 20:05 |
| sdake | inc0 sbezverk has a patch for you to review | 20:06 |
| sdake | and i'd really like if you take first to review on upgrade patches in this last week of our releaes | 20:06 |
| sbezverk | sdake how do I know if it is bluebox or red box or whatever color that box is ;-) ? | 20:06 |
| sdake | it says right at the top of the log | 20:06 |
| sdake | bluebox is a cloud provider | 20:06 |
| sdake | of which there are about 6 i think | 20:06 |
| sdake | they all have slightly different environments | 20:07 |
| sbezverk | sdake that is the problem :-) always go straight to the bottom where the error is .. | 20:07 |
| inc0 | ah, issue with gate? | 20:07 |
| inc0 | sbezverk, review about iscsi? | 20:08 |
| ccesario | gerrit is it down!>? | 20:08 |
| inc0 | sbezverk, I'll take a look later today | 20:08 |
| inc0 | I need to run few errands first | 20:08 |
| sbezverk | inc0 thank you.. | 20:08 |
| inc0 | (buying a new car...) | 20:08 |
| sdake | yes gerrit down for awhile | 20:08 |
| sdake | inc0 grats! | 20:09 |
| inc0 | or rather chosing one;) | 20:09 |
| sdake | what type | 20:09 |
| sbezverk | inc0 I hope it is mazerati | 20:09 |
| sdake | new car smell rocks ;) | 20:09 |
| inc0 | 2016 mazda miata is my fav now | 20:09 |
| sdake | sbezverk when we bought our panamera i reaally wanted a mazarati | 20:09 |
| sdake | but wife wouldn't have anything to do with it | 20:09 |
| sdake | mazda makes nice vehicles | 20:09 |
| inc0 | yeah | 20:09 |
| inc0 | and this one is so much fun | 20:10 |
| *** banix has quit IRC | 20:10 | |
| inc0 | I already have new subaru crosstrek, but that will be my wifes car | 20:11 |
| sdake | o folks | 20:11 |
| inc0 | so I'm buying something for myself;) and it doesn't have to be practical | 20:11 |
| sdake | gerrit is down until 21:00 | 20:11 |
| inc0 | ok | 20:11 |
| sdake | our gerrit bot updates were not coming because of the channel rename | 20:11 |
| inc0 | so I'll review stuff later | 20:11 |
| sdake | but that has been merged | 20:11 |
| sdake | so if there is any more gerrrit slip on schedule it should update our channel topic | 20:11 |
| inc0 | ok, afk | 20:12 |
| *** inc0 has quit IRC | 20:16 | |
| *** vhosakot has quit IRC | 20:17 | |
| *** shardy_afk has quit IRC | 20:18 | |
| *** neophy has joined #openstack-kolla | 20:21 | |
| *** achanda has quit IRC | 20:22 | |
| *** dwalsh has quit IRC | 20:24 | |
| *** banix has joined #openstack-kolla | 20:26 | |
| *** vhosakot has joined #openstack-kolla | 20:27 | |
| *** openstackgerrit has joined #openstack-kolla | 20:27 | |
| *** jtriley has quit IRC | 20:37 | |
| *** rstarmer has joined #openstack-kolla | 20:38 | |
| *** unicell has quit IRC | 20:38 | |
| *** openstackgerrit has quit IRC | 20:41 | |
| *** neophy has quit IRC | 20:53 | |
| *** openstackgerrit has joined #openstack-kolla | 20:55 | |
| *** rhallisey has quit IRC | 20:58 | |
| *** banix has quit IRC | 20:58 | |
| *** unicell1 has joined #openstack-kolla | 21:02 | |
| *** daneyon_ has joined #openstack-kolla | 21:02 | |
| *** ccesario___ has quit IRC | 21:03 | |
| *** ccesario___ has joined #openstack-kolla | 21:03 | |
| *** ccesario___ has quit IRC | 21:06 | |
| *** haplo37 has quit IRC | 21:07 | |
| manjeets | sdake ping | 21:07 |
| manjeets | i guess I reported a bug of linux bridge getting out of synce with ml2 both with kolla and neutron which i don't see any more | 21:08 |
| *** dwalsh has joined #openstack-kolla | 21:31 | |
| sdake | manjeets yesthat is because we merged aa fix for that i think ;) | 21:42 |
| sdake | dave-mccowan docker exec heka ls /var/log/kolla | 21:42 |
| manjeets | no sdake that was separate bug | 21:42 |
| manjeets | linux bridge was working when built from source | 21:42 |
| manjeets | but some times ml2 and linux bridge were getting out of sync | 21:43 |
| manjeets | like 300 vms getting error out of 1000 because | 21:43 |
| manjeets | one bug was linux bridge was not working totally when built from source ( thats fixed I guess ) | 21:43 |
| sdake | ya Jeffrey4l__ fixed that | 21:44 |
| manjeets | I reported this bug last friday | 21:44 |
| sdake | i can't tell yu the reviewbecause gerrit is offline | 21:44 |
| *** ccesario___ has joined #openstack-kolla | 21:44 | |
| manjeets | gerrit is up again and I know that change | 21:44 |
| vhosakot | manjeets: yes, Jeffrey4l__ fixed the neutron linuxbridge reconfigure bug... https://review.openstack.org/#/c/303783/ | 21:45 |
| patchbot | vhosakot: patch 303783 - kolla - Fix the neutron reconfigure issue when enable linu... (MERGED) | 21:45 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix the neutron reconfigure issue when enable linuxbridge-agent https://review.openstack.org/304320 | 21:47 |
| *** asalkeld has left #openstack-kolla | 21:48 | |
| manjeets | vhosakot it was something i reported friday | 21:48 |
| manjeets | I added neutron and kolla both | 21:48 |
| manjeets | to bug ticket | 21:48 |
| vhosakot | manjeets: https://review.openstack.org/#/c/302415/ is merged too | 21:48 |
| patchbot | vhosakot: patch 302415 - kolla - Increase max pool size so conductor doesn't implode (MERGED) | 21:48 |
| sdake | manjeets question | 21:48 |
| vhosakot | manjeets: can you send the bug ID pls | 21:48 |
| sdake | is your deployment workingflawlessly now? | 21:48 |
| manjeets | it was sdake | 21:49 |
| manjeets | but linuxbridge was getting ut of sync with ml2 | 21:49 |
| manjeets | while spining 1000 vms | 21:49 |
| manjeets | some times it worked 1000 vm successfully but after few runs I was having issues | 21:50 |
| sdake | manjeets is your deployment working flawlessly or are there still problems? | 21:50 |
| manjeets | sdake with ovs it did | 21:50 |
| manjeets | with lb there were issues | 21:50 |
| sdake | even with master? | 21:51 |
| *** asalkeld_ has quit IRC | 21:51 | |
| sdake | if so do you have logs | 21:51 |
| *** asalkeld has joined #openstack-kolla | 21:51 | |
| sdake | and a bug id to go on? | 21:51 |
| manjeets | sdake I am trying to find the bug ticket but not able to find it seems like somebody marked it invalid or duplicate | 21:51 |
| manjeets | but it was a different error | 21:52 |
| sdake | check your email for the bug id | 21:52 |
| manjeets | sdake it was you who marked that as duplicate | 21:54 |
| *** ccesario___ has quit IRC | 21:54 | |
| sdake | ok well i cna undo that | 21:54 |
| sdake | give me a bug link | 21:54 |
| manjeets | https://bugs.launchpad.net/kolla/+bug/1568202 | 21:55 |
| openstack | Launchpad bug 1567406 in kolla "duplicate for #1568202 neutron linux bridge agent is buggy and can not boot instance" [Critical,In progress] - Assigned to Jeffrey Zhang (jeffrey4l) | 21:55 |
| sdake | just to be clear, this bug exists in master, only for linuxbridge, but not ovs, when spinning 1000 vms | 21:55 |
| manjeets | spining 1000 vms was working with lb for initial runs | 21:55 |
| manjeets | but after two or three runs it lb was getting out of sync with ml2 | 21:55 |
| manjeets | and when i was spining 1000 vms for 4th time 300 out of 1000 were ending errir | 21:56 |
| manjeets | error | 21:56 |
| *** ccesario___ has joined #openstack-kolla | 21:56 | |
| sdake | manjeets you tried this review: https://review.openstack.org/#/c/302721/ | 21:56 |
| patchbot | sdake: patch 302721 - kolla - Fix launch vm failed in neutron-linuxbridge-agent env | 21:56 |
| sdake | in your logs brctl was missing | 21:56 |
| sdake | tht ptch adds brctl | 21:56 |
| *** ccesario___ has quit IRC | 21:57 | |
| manjeets | but 1000 vms worked for 2 or 3 runs | 21:57 |
| manjeets | not sure why they worked without brctl package | 21:58 |
| sdake | but your saying its fixed? | 21:59 |
| sdake | or still busted | 21:59 |
| sdake | with master | 21:59 |
| *** ccesario___ has joined #openstack-kolla | 21:59 | |
| manjeets | hav not tried current master | 22:01 |
| *** ccesario___ has quit IRC | 22:01 | |
| *** ccesario___ has joined #openstack-kolla | 22:01 | |
| sdake | could you try and report in that bug | 22:01 |
| sdake | we hve made alot of fixes | 22:01 |
| sdake | i'd like to make sure we aren't investigating solved problems ;) | 22:02 |
| sdake | note not everything has been backported in master to stable/mitaka hat needs to be yet | 22:02 |
| *** ccesario___ has quit IRC | 22:03 | |
| *** ccesario___ has joined #openstack-kolla | 22:04 | |
| *** manjeets is now known as manjeetsingh | 22:05 | |
| dave-mccowan | sdake if i have multi node, does a heka container run on every node? | 22:05 |
| sdake | yup | 22:05 |
| *** ccesario___ has quit IRC | 22:05 | |
| sdake | with relevant logs for that node | 22:05 |
| *** ccesario___ has joined #openstack-kolla | 22:05 | |
| *** manjeetsingh is now known as manjeets | 22:06 | |
| dave-mccowan | cool. which node(s) are the aggregate logs on? | 22:06 |
| sdake | control nodes | 22:06 |
| sdake | is where elasticsearch runs | 22:06 |
| sdake | akwasnie or elemoine_ is elastic search in its current implementtion in kolla ha? | 22:13 |
| *** vhosakot has quit IRC | 22:14 | |
| *** ccesario___ has quit IRC | 22:17 | |
| *** ccesario___ has joined #openstack-kolla | 22:21 | |
| *** dwalsh has quit IRC | 22:22 | |
| *** daneyon_ has quit IRC | 22:22 | |
| *** gfidente has quit IRC | 22:27 | |
| *** rstarmer has quit IRC | 22:29 | |
| *** ccesario___ has quit IRC | 22:29 | |
| *** rstarmer has joined #openstack-kolla | 22:38 | |
| *** rstarmer has quit IRC | 22:39 | |
| *** ccesario___ has joined #openstack-kolla | 22:39 | |
| *** vhosakot has joined #openstack-kolla | 22:43 | |
| *** daneyon_ has joined #openstack-kolla | 22:43 | |
| vhosakot | dave-mccowan: http://paste.openstack.org/show/493701/ | 22:47 |
| *** rstarmer has joined #openstack-kolla | 22:49 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Drop root for swift https://review.openstack.org/259374 | 22:50 |
| *** sdake_ has joined #openstack-kolla | 22:53 | |
| *** sdake has quit IRC | 22:55 | |
| *** JoseMello has quit IRC | 23:09 | |
| *** rstarmer has quit IRC | 23:09 | |
| *** rhallisey has joined #openstack-kolla | 23:18 | |
| *** salv-orlando has quit IRC | 23:21 | |
| *** banix has joined #openstack-kolla | 23:21 | |
| *** Jeffrey4l__ has quit IRC | 23:42 | |
| *** mwheckmann has joined #openstack-kolla | 23:47 | |
| *** sdake has joined #openstack-kolla | 23:52 | |
| *** sdake_ has quit IRC | 23:53 | |
| *** daneyon_ has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!