| *** achanda has quit IRC | 00:02 | |
| *** vhosakot has quit IRC | 00:05 | |
| sbezverk | sdake last night almost at the same time there was another guy who managed to get multinode going.. | 00:08 |
|---|---|---|
| sdake_ | multinode is definately busted | 00:10 |
| sdake_ | i can see in the logs | 00:10 |
| *** inc0 has joined #kolla | 00:11 | |
| *** jtriley has joined #kolla | 00:12 | |
| *** ssurana has quit IRC | 00:16 | |
| *** vishwanathj has quit IRC | 00:37 | |
| *** achanda has joined #kolla | 00:39 | |
| *** jtriley has quit IRC | 00:43 | |
| inc0 | hey sdake_ any progress with manjeets setup? | 00:44 |
| inc0 | I tried to deploy latest mariadb and it worked just fine | 00:44 |
| manjeets | inc0: we are still debugging | 00:44 |
| manjeets | what version is it ? | 00:45 |
| manjeets | inc0: | 00:45 |
| manjeets | mariadb | 00:45 |
| inc0 | I deployed todays kolla | 00:45 |
| inc0 | with full rebuild | 00:45 |
| manjeets | i did that to after ip table clean up | 00:45 |
| *** jtriley has joined #kolla | 00:47 | |
| *** vhosakot has joined #kolla | 00:53 | |
| sdake_ | inc0 9 day old images work 2 hour old images dont work | 01:05 |
| inc0 | hmm | 01:06 |
| inc0 | on same setup? | 01:06 |
| inc0 | so it's a bug in mariadb? | 01:06 |
| inc0 | mariadb 10.1.13 got released yesterday | 01:07 |
| inc0 | can you pin mariadb to 10.1.12 and try? | 01:07 |
| *** sdake has joined #kolla | 01:11 | |
| *** marmat has quit IRC | 01:12 | |
| *** sdake_ has quit IRC | 01:14 | |
| *** iceyao has joined #kolla | 01:16 | |
| vhosakot | sdake: I pulled yesterday, and have mariadb 10.0.24.... it is stable.. | 01:16 |
| vhosakot | sdake: http://paste.openstack.org/show/491959/ | 01:16 |
| *** unicell has quit IRC | 01:19 | |
| sdake | vhosakot how do i fidn out which vesion of mariadb is installed in ubuntu | 01:21 |
| vhosakot | docker exec mariadb dpkg -l | grep mariadb | 01:21 |
| vhosakot | sdake: http://paste.openstack.org/show/491959/ | 01:22 |
| sdake | vhosakot areyou duoing multinode deploys? | 01:22 |
| vhosakot | no.. AIO... | 01:23 |
| sdake | manjeets has 10.0.24 | 01:23 |
| sdake | aio works for manjeets | 01:23 |
| sdake | multinode does not work for manjeets | 01:23 |
| vhosakot | yep, as of yesterday, we pull 10.0.24 on ubuntu..... | 01:23 |
| vhosakot | oh ok, I tried helping manjeet yesterday... wetried couple of things... nothing worked... we restarted mariadb and removed network nodes (rather combined network group and control group in multinode inventory)... is marisdb still down.. | 01:24 |
| *** jtriley has quit IRC | 01:26 | |
| sbezverk | vhosakot I found a couple of similar mariadb issues reported on the internet, but they are all resolved by cleaning mariadb binaries and reinstalling it.. not very helpfull for this case | 01:28 |
| vhosakot | sbezverk: ah ok... cleaning meaning... uninstall the pakcges and re-install ? | 01:28 |
| vhosakot | bbiab | 01:30 |
| sbezverk | vhosakot yep | 01:30 |
| vhosakot | cool | 01:31 |
| manjeets | mariadb is giving me hard time on mutinode | 01:31 |
| sbezverk | manjeets have you tried limit the number of server to let's say 3 instead of 60.. to see if it makes any difference | 01:32 |
| manjeets | i tried every single thing with different setup | 01:33 |
| manjeets | 2,3 node control 1 compute and storage | 01:33 |
| sbezverk | manjeets and saw same failure everytime? | 01:33 |
| manjeets | every single time same failure | 01:34 |
| inc0 | manjeets, try checking out this rev cfbd96bbb9ec2ee9d0983c4997e1f6300e61c7c1 | 01:36 |
| inc0 | and rebuild -> redeploy | 01:36 |
| inc0 | remember to use --no-cache | 01:36 |
| vhosakot | manjeets: can you stop mariadb container, remove it, remove its shared volume (docker stop mariadb && docker rm mariadb && docker volume rm mariadb)... and then, check if you see the file /var/lib/docker/volumes/mariadb/_data/mariadb.pid on the control node..... I start to wonder if the stale /var/lib/docker/volumes/mariadb/_data/mariadb.pid hangs around on the host after the maruadb container is deleted.. | 01:37 |
| inc0 | vhosakot, he did cleanup-containers multiple time | 01:38 |
| inc0 | s | 01:38 |
| inc0 | and it deletes volume as well | 01:38 |
| vhosakot | inc0: cool, I'm just curious if /var/lib/docker/volumes/mariadb/_data/mariadb.pid still stays on the host aftrer cleanup | 01:38 |
| vhosakot | inc0: rather, is the volume really deleted _properly_ ? | 01:39 |
| inc0 | worth to check vhosakot | 01:39 |
| sbezverk | vhosakot last night we cleaned everything under docker/* that should have wiped out everything | 01:40 |
| vhosakot | sbezverk: ah ok.. so, then that is not the issue.. cool | 01:40 |
| inc0 | I have 2 suspicions | 01:41 |
| vhosakot | as inc0 suggested, trying --no-cache is good to eliminate local registry's cache issues | 01:41 |
| inc0 | 1 is mariadb release | 01:41 |
| inc0 | 2 is change in startup script that merged 5 days ago | 01:41 |
| inc0 | that being said, today I deployed mariadb no problems | 01:41 |
| sbezverk | vhosakot have disagree about registry theory as AIO works just fine.. | 01:41 |
| vhosakot | sbezverk: ah ok... right | 01:42 |
| inc0 | also something funky may happen on networking layer | 01:42 |
| sdake | the issue is mariadb starts up only on one node | 01:42 |
| sdake | the other nodes are skilpped | 01:42 |
| sdake | yet mariadb config tries to connect to all nodes | 01:42 |
| sdake | and mariadb fails and exits | 01:42 |
| sdake | inc0 tips on that? | 01:42 |
| inc0 | hmm | 01:43 |
| inc0 | so I think it's bootstrap | 01:43 |
| inc0 | so it should run only on one node | 01:43 |
| vhosakot | manjeets: inc0 has a good point... can you check https://github.com/openstack/kolla/commit/afce10dea418dcf691e8f4539f5fb7dc766235a3... this has the update to maridb's startup script for lights out recovery.... wonder it affects multinode | 01:43 |
| vhosakot | it was done 5 days ago.. yes | 01:44 |
| inc0 | vhosakot, I deployed multinode | 01:44 |
| inc0 | today | 01:44 |
| inc0 | so it's non trivial | 01:44 |
| inc0 | however worth to try | 01:44 |
| vhosakot | inc0: multinode works for you ? | 01:44 |
| inc0 | yes | 01:44 |
| inc0 | on 3 vms tho | 01:44 |
| inc0 | so scale is different | 01:44 |
| vhosakot | ah ok | 01:45 |
| vhosakot | would be a good test to scale down then... dont' know how feasable it is for manjeets | 01:46 |
| inc0 | we tried that vhosakot | 01:46 |
| sdake | we have already scaled down to3 nodes | 01:46 |
| sdake | we are teting reverts now | 01:46 |
| vhosakot | ah ok... clearly i am catching up on irc :) | 01:46 |
| * vhosakot digs thru irc logs of the past 2 days and manjeets' pastebin links in logs | 01:47 | |
| sdake | so we reverted the last two mariadb commits | 01:47 |
| manjeets | it actually worked | 01:48 |
| sdake | and manjeets deploy works | 01:48 |
| manjeets | after reverting two mariadb commits | 01:48 |
| vhosakot | in master or in manjeets local env ? | 01:48 |
| sdake | master | 01:48 |
| vhosakot | wow, missed those reviwes | 01:49 |
| vhosakot | cool.. | 01:49 |
| manjeets | http://paste.openstack.org/show/491960/ | 01:49 |
| * manjeets is having issues in progressive manner | 01:50 | |
| vhosakot | ah, so next issue... ok.. | 01:50 |
| sdake | get me some anestaphine from the medlock | 01:50 |
| vhosakot | haha | 01:51 |
| inc0 | yo SamYaple | 01:51 |
| inc0 | mariadb is broken;) | 01:51 |
| * manjeets needs a giant dinner as he had shorter lunch | 01:51 | |
| vhosakot | sdake: so, are those commits reverted in master... aren't they needed ? | 01:51 |
| manjeets | not reverted in master | 01:51 |
| manjeets | though | 01:51 |
| inc0 | manjeets, this is an easy one...cp etc/kolla /etc/kolla;) | 01:51 |
| inc0 | we added new conf that needs to be in /etc/kolla/passwords.yml | 01:52 |
| SamYaple | whats wrong with mariadb? | 01:53 |
| inc0 | manjeets, care to bring Sam up to speed? | 01:54 |
| manjeets | afce10dea418dcf691e8f4539f5fb7dc766235a3 this commits | 01:54 |
| manjeets | 2aaaed770e158c3dc0d6e1895021c0e0475d200d | 01:54 |
| manjeets | after reverting these two its cracked | 01:54 |
| manjeets | https://bugs.launchpad.net/kolla/+bug/1562231 | 01:56 |
| openstack | Launchpad bug 1562231 in kolla "mariadb will end up in issues with multinode" [Undecided,New] | 01:56 |
| sdake | fwiw the bootstrapping and starting didn't look correct during debug | 01:56 |
| sdake | only thing i can think of is manjeets hostnames are manjeets85094 | 01:57 |
| sdake | or whatenot | 01:57 |
| sdake | maybe numbers at the end damage the parsing | 01:57 |
| SamYaple | i still havent heard the issue... | 01:57 |
| SamYaple | but my guess, like always, is most likely the evironment wasnt clean | 01:58 |
| sdake | mariadb stops at first mariadb continer startup | 01:58 |
| SamYaple | in the case of mariadb that means leftover volumes | 01:58 |
| sdake | environment was clean and fresh | 01:58 |
| inc0 | SamYaple, trust me it was | 01:58 |
| *** vhosakot has quit IRC | 01:58 | |
| sdake | it ws cleaned atleast 20 times and rebuilt several during our debug session | 01:58 |
| inc0 | issue was that clusted refused to stand up and turned mariadb into restarting loop | 01:58 |
| SamYaple | i mean in the corse of this conversation i just ran a multinode deploy no issue..... | 01:58 |
| inc0 | SamYaple, me too, but it was consistent on manjeets infra | 01:59 |
| sdake | yup it does seem to wrk for mot people SamYaple | 01:59 |
| SamYaple | is there already a maradin running on the host? (did the prechecks pass) | 01:59 |
| inc0 | in fact I am prepping exactly same hardware now | 01:59 |
| SamYaple | in the case of that hp gear there IS a mariadb running on the host inc0 | 01:59 |
| SamYaple | just fyi | 01:59 |
| SamYaple | well its mysql | 01:59 |
| manjeets | inc0: i have a password.yml which has a todo activity for Samyaple | 01:59 |
| SamYaple | but same port | 01:59 |
| inc0 | not for, by SamYaple;) | 02:00 |
| manjeets | i should update ? | 02:00 |
| inc0 | wait...what? | 02:00 |
| inc0 | what do you mean? | 02:00 |
| sdake | the fact that reverting those two patches is a good sign there is a problem | 02:00 |
| *** vhosakot has joined #kolla | 02:01 | |
| sdake | manjeets lets set up another google groups | 02:02 |
| sdake | so i can hep you capute appropriate information | 02:02 |
| sdake | now tht my shit ass pizza dinner is done | 02:02 |
| vhosakot | sdake: what is this google groups ? online debug session thru google hangout ? | 02:03 |
| sdake | yup | 02:03 |
| vhosakot | sdake: wow, I'd like to participate too | 02:03 |
| manjeets | sdake is it possible after some time ? | 02:03 |
| manjeets | i need to go home and get dinner | 02:03 |
| vhosakot | sdake: how will i know about online debug sessions ? are they decided in IRC or email is sent to ML ? | 02:03 |
| sdake | well it wont be fixed until we hve approprite logs | 02:03 |
| manjeets | every body left office i am only one and lights are off now | 02:04 |
| vhosakot | manjeets: mariabdb has lights out recovery;) | 02:04 |
| sdake | because nobody can reproduce it but you | 02:04 |
| sdake | *had* | 02:04 |
| vhosakot | yeah | 02:04 |
| vhosakot | sdake: how will i know about online debug sessions ? are they decided in IRC or email is sent to ML ? | 02:04 |
| sdake | this was a 1 time thing | 02:05 |
| sdake | because manjeets had 64 nodes | 02:05 |
| vhosakot | ah ok | 02:05 |
| sdake | i wanted to see how long it took to deploy kolla | 02:05 |
| sdake | the answer was 21 minutes for 64 nodes | 02:05 |
| sdake | with cinder | 02:05 |
| vhosakot | cool | 02:05 |
| manjeets | is it okay to have password less password.yml ? | 02:06 |
| manjeets | or it is must to generate passwords | 02:06 |
| vhosakot | they cant be empty.. run kolla-genpwd | 02:07 |
| SamYaple | thud far ive seen no logs or posts about any issue | 02:07 |
| SamYaple | just "it doesnt work" | 02:07 |
| SamYaple | i cant do anything wit hthat | 02:08 |
| inc0 | hold on SamYaple there are pastes | 02:08 |
| manjeets | kolla-genpwd does not work | 02:08 |
| vhosakot | manjeets: what happened to kolla-genpwd ? | 02:09 |
| manjeets | no command | 02:09 |
| manjeets | i did tools/generate_password.py | 02:09 |
| SamYaple | manjeets: you are clearly not on the latest version | 02:09 |
| SamYaple | if you arent on the latest version, you are running old code | 02:09 |
| vhosakot | manjeets: yep, would be good to move to latest master | 02:09 |
| SamYaple | pip uninstall kolla | 02:09 |
| manjeets | did that and pip install -U kolla | 02:10 |
| SamYaple | dont install kolla | 02:10 |
| SamYaple | just run it from folder | 02:10 |
| manjeets | pip shows kolla says 2.0.0.rc1 | 02:10 |
| SamYaple | pip uninstall kolla | 02:11 |
| vhosakot | git pull origin/master before next install | 02:11 |
| SamYaple | yea thats not going to be quite accurate since the reno stuff requires pbr changes | 02:11 |
| manjeets | i ran tools/generate_password | 02:11 |
| SamYaple | i need many thing to tell you what the issue is. first and formost, logs | 02:11 |
| SamYaple | ansible logs | 02:12 |
| SamYaple | mariabd logs | 02:12 |
| SamYaple | all of them | 02:12 |
| SamYaple | at this point though im ~90% sure it just outdated kolla and/or images | 02:12 |
| inc0 | SamYaple, http://paste.openstack.org/show/491940/ | 02:12 |
| manjeets | let me grab dinner ill crack on it after dinner | 02:13 |
| manjeets | 8:30 PST probably | 02:13 |
| manjeets | starving to death now | 02:13 |
| manjeets | lol | 02:13 |
| SamYaple | im going to sleep now. ill see you guys later | 02:13 |
| sdake | SamYaple i spent 3 hours debugging with manjeets | 02:14 |
| sdake | the reverts fix the problem definately | 02:14 |
| manjeets | I am not sure of kolla yet but kolla community is great | 02:14 |
| sdake | but agree logs are needed | 02:14 |
| manjeets | thanks for help everybody | 02:14 |
| SamYaple | sdake: i dont trust your debugging. | 02:14 |
| inc0 | SamYaple, I did the same;) | 02:14 |
| manjeets | i guess i sent all pastebin to this if you can get over history all logs can be found | 02:15 |
| manjeets | I have saved mariadb logs and ansible console output for failures | 02:15 |
| manjeets | will post link to them after i grab dinner | 02:15 |
| manjeets | thank you so much everyone i really appreciate your help guys | 02:16 |
| manjeets | pardon me if I was dumb on some things | 02:16 |
| *** sdake has quit IRC | 02:16 | |
| inc0 | manjeets, we need to get to the bottom of this | 02:16 |
| inc0 | but thanks for patience | 02:16 |
| *** vhosakot has quit IRC | 02:17 | |
| *** sdake has joined #kolla | 02:19 | |
| inc0 | manjeets, soo...it worked for me | 02:20 |
| inc0 | no issues | 02:20 |
| inc0 | latest master | 02:20 |
| inc0 | same hardware | 02:20 |
| *** vhosakot has joined #kolla | 02:21 | |
| sdake | manjeets follow this bug requirement: | 02:23 |
| sdake | https://bugs.launchpad.net/kolla/+bug/1562231 | 02:23 |
| openstack | Launchpad bug 1562231 in kolla "mariadb will end up in issues with multinode" [Critical,Confirmed] | 02:23 |
| *** sdake_ has joined #kolla | 02:25 | |
| *** sdake_ has quit IRC | 02:27 | |
| *** sdake has quit IRC | 02:27 | |
| *** sdake has joined #kolla | 02:27 | |
| inc0 | sdake, soooo...I just deployed kolla | 02:28 |
| inc0 | on the very same hardware manjeets was using | 02:28 |
| sdake | 19:22:51] <sdake>manjeets follow this bug requirement: | 02:28 |
| sdake | [19:22:52] <sdake> https://bugs.launchpad.net/kolla/+bug/1562231 | 02:28 |
| sdake | [19:23:15] <sdake>my speculation is it has to do with his hostnames | 02:28 |
| sdake | [19:23:21] <sdake>and their relative sorting | 02:28 |
| sdake | [19:23:33] <sdake>but i'm not reallly certain | 02:28 |
| sdake | [ | 02:28 |
| openstack | Launchpad bug 1562231 in kolla "mariadb will end up in issues with multinode" [Critical,Confirmed] | 02:28 |
| inc0 | same servers, not same environment | 02:28 |
| inc0 | I'll try one thing when he gets back | 02:28 |
| SamYaple | sdake: crazy thing to say considering mariadb has nothing to do with hostnames | 02:29 |
| *** iceyao has quit IRC | 02:29 | |
| *** sdake has quit IRC | 02:30 | |
| *** sdake has joined #kolla | 02:32 | |
| *** sdake has quit IRC | 02:33 | |
| *** klint has joined #kolla | 02:42 | |
| *** iceyao has joined #kolla | 02:47 | |
| vhosakot | sdake: is there a plan for multinode gate ? | 02:49 |
| inc0 | vhosakot, there is | 02:50 |
| inc0 | we were constrained by ram on gates tho | 02:50 |
| inc0 | but we need to make it work | 02:50 |
| inc0 | SamYaple wrote nice thing that uses docker to simulate multinode | 02:51 |
| inc0 | dind - docker in a docker;) | 02:51 |
| inc0 | we have code for that in tree | 02:51 |
| vhosakot | inc0: ah ok... I see this bp https://blueprints.launchpad.net/kolla/+spec/multinode-gate.. cool | 02:52 |
| inc0 | ok I'm off | 02:52 |
| inc0 | more debugging tomorrow | 02:52 |
| inc0 | cya guys | 02:52 |
| *** inc0 has quit IRC | 02:53 | |
| openstackgerrit | Nguyen Hung Phuong proposed openstack/kolla: Fix typos in Kolla https://review.openstack.org/297909 | 02:56 |
| manjeets | inc0: thanks just before i start anotther round having a dinner and redoing stuff from scratch | 03:05 |
| *** vhosakot has quit IRC | 03:07 | |
| *** Marga_ has quit IRC | 03:42 | |
| *** vhosakot has joined #kolla | 03:43 | |
| *** Marga_ has joined #kolla | 03:48 | |
| *** achanda has quit IRC | 03:50 | |
| *** Marga_ has quit IRC | 03:52 | |
| *** Marga_ has joined #kolla | 03:58 | |
| *** Marga_ has quit IRC | 04:02 | |
| *** vishwanathj has joined #kolla | 04:16 | |
| manjeets | vhosakot just did everything from scratch | 04:35 |
| manjeets | sdake | 04:35 |
| manjeets | neutron and compute succeded | 04:35 |
| manjeets | but unfortunately cinder failed | 04:36 |
| manjeets | thise time | 04:36 |
| vhosakot | Could you append -D for “DOCKER_OPTS=“ in /etc/default/docker and restart the docker daemon (sudo service docker restart) in debug mode on all the nodes on which the mariadb container is run. | 04:36 |
| vhosakot | Could you add wsrep_debug=1 under the [mysqld] section in galera.cnf and re-deploy mariadb in debug modes on all the nodes. | 04:36 |
| manjeets | its already in every single node | 04:36 |
| vhosakot | cool | 04:36 |
| manjeets | wsrep_debug = 1 is not there | 04:37 |
| manjeets | but docker_opts i have it in all | 04:37 |
| manjeets | http://paste.openstack.org/show/491963/ | 04:37 |
| manjeets | this is failure I am getting | 04:37 |
| manjeets | cinder ends up in issue | 04:37 |
| *** jasonsb has joined #kolla | 04:40 | |
| vhosakot | add -D to DOCKER_OPTS and restart docker daemon in debug mode | 04:40 |
| manjeets | -D ? | 04:40 |
| vhosakot | yep | 04:41 |
| manjeets | DOCKER_OPTS="--insecure-registry 172.87.0.201:4000" | 04:41 |
| vhosakot | make it DOCKER_OPTS="--insecure-registry 172.87.0.201:4000 -D" | 04:41 |
| manjeets | on every single node ? | 04:41 |
| manjeets | how do restart docker in debug mode ? | 04:42 |
| manjeets | i just do service docker restart | 04:42 |
| vhosakot | sudo service docker restart.. yes, on all nodes | 04:42 |
| vhosakot | https://bugs.launchpad.net/kolla/+bug/1562231 | 04:42 |
| openstack | Launchpad bug 1562231 in kolla "mariadb will end up in issues with multinode" [Critical,Confirmed] | 04:42 |
| *** iceyao has quit IRC | 04:43 | |
| manjeets | this time i don't need to rebuild images | 04:46 |
| manjeets | ? | 04:46 |
| *** salv-orl_ has joined #kolla | 05:02 | |
| *** salv-orl_ has quit IRC | 05:03 | |
| *** salv-orl_ has joined #kolla | 05:03 | |
| *** salv-orlando has quit IRC | 05:05 | |
| manjeets | i think i got the the issue i did not enable cinder in all.yml | 05:07 |
| *** salv-orl_ has quit IRC | 05:08 | |
| vhosakot | cool | 05:09 |
| *** sdake has joined #kolla | 05:14 | |
| *** iceyao has joined #kolla | 05:15 | |
| *** sdake has quit IRC | 05:19 | |
| *** Marga_ has joined #kolla | 05:25 | |
| *** Marga__ has joined #kolla | 05:26 | |
| *** iceyao has quit IRC | 05:30 | |
| *** Marga_ has quit IRC | 05:30 | |
| openstackgerrit | Nguyen Hung Phuong proposed openstack/kolla: Fix typos in Kolla https://review.openstack.org/297909 | 05:32 |
| *** vhosakot has quit IRC | 05:54 | |
| *** allen_gao has joined #kolla | 05:55 | |
| *** ccesario has quit IRC | 06:00 | |
| *** unicell has joined #kolla | 06:16 | |
| *** manjeets has left #kolla | 06:19 | |
| *** manjeets has joined #kolla | 06:20 | |
| *** Jeffrey4l_ has joined #kolla | 06:25 | |
| *** unicell has quit IRC | 06:30 | |
| *** unicell has joined #kolla | 06:38 | |
| *** iceyao has joined #kolla | 06:46 | |
| *** daneyon has quit IRC | 07:10 | |
| *** achanda has joined #kolla | 07:19 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Restrict the rabbitmq only start on rabbitmq group https://review.openstack.org/297929 | 07:54 |
| *** salv-orlando has joined #kolla | 07:56 | |
| *** salv-orlando has quit IRC | 07:59 | |
| openstackgerrit | Nguyen Hung Phuong proposed openstack/kolla: Fix typos in Kolla https://review.openstack.org/297909 | 08:06 |
| *** stvnoyes has quit IRC | 08:08 | |
| *** stvnoyes has joined #kolla | 08:09 | |
| *** dwalsh has joined #kolla | 08:17 | |
| *** Jeffrey4l_ has quit IRC | 08:32 | |
| *** Jeffrey4l_ has joined #kolla | 08:32 | |
| *** jmccarthy has quit IRC | 08:36 | |
| *** jmccarthy has joined #kolla | 08:36 | |
| *** dwalsh has quit IRC | 08:49 | |
| openstackgerrit | Kuo-tung Kao proposed openstack/kolla: add unit tests for build script https://review.openstack.org/290423 | 08:53 |
| *** achanda has quit IRC | 09:13 | |
| *** Marga__ has quit IRC | 09:34 | |
| *** allen_gao has quit IRC | 10:12 | |
| *** achanda has joined #kolla | 10:13 | |
| *** pbourke has quit IRC | 10:15 | |
| *** pbourke has joined #kolla | 10:16 | |
| *** achanda has quit IRC | 10:19 | |
| *** iceyao has quit IRC | 10:21 | |
| *** iceyao has joined #kolla | 10:21 | |
| *** allen_gao has joined #kolla | 10:24 | |
| *** salv-orlando has joined #kolla | 10:37 | |
| *** allen_gao has quit IRC | 10:43 | |
| *** salv-orlando has quit IRC | 10:46 | |
| *** allen_gao has joined #kolla | 10:49 | |
| *** pbourke-home has joined #kolla | 10:50 | |
| *** Serlex has joined #kolla | 10:59 | |
| *** dwalsh has joined #kolla | 11:07 | |
| *** achanda has joined #kolla | 11:18 | |
| *** achanda has quit IRC | 11:23 | |
| *** allen_gao has quit IRC | 12:03 | |
| *** dwalsh has quit IRC | 12:05 | |
| *** ccesario has joined #kolla | 12:07 | |
| *** sdake has joined #kolla | 12:10 | |
| *** jtriley has joined #kolla | 12:35 | |
| *** iceyao has quit IRC | 12:46 | |
| *** jtriley has quit IRC | 12:51 | |
| *** klint has quit IRC | 12:59 | |
| *** SiRiuS_ has joined #kolla | 13:04 | |
| *** achanda has joined #kolla | 13:20 | |
| *** achanda has quit IRC | 13:25 | |
| *** salv-orlando has joined #kolla | 13:33 | |
| *** Jeffrey4l has joined #kolla | 13:37 | |
| *** salv-orlando has quit IRC | 13:38 | |
| *** allen_gao has joined #kolla | 13:44 | |
| *** Jeffrey4l has quit IRC | 13:47 | |
| *** Jeffrey4l has joined #kolla | 13:48 | |
| *** ccesario has quit IRC | 13:51 | |
| sdake | hey folks, testr seems to be missing from the gate | 13:52 |
| *** iceyao has joined #kolla | 13:57 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate https://review.openstack.org/297945 | 14:09 |
| *** Jeffrey4l has quit IRC | 14:15 | |
| *** Jeffrey4l_ has quit IRC | 14:16 | |
| *** Jeffrey4l_ has joined #kolla | 14:27 | |
| *** Jeffrey4l has joined #kolla | 14:27 | |
| *** Jeffrey4l has quit IRC | 14:54 | |
| *** salv-orlando has joined #kolla | 14:57 | |
| sdake | Jeffrey4l_ around? | 14:57 |
| sdake | if so please ackc https://review.openstack.org/#/c/296619/ | 14:57 |
| patchbot | sdake: patch 296619 - kolla (stable/mitaka) - Enable horizon domain logins | 14:57 |
| Jeffrey4l_ | sdake, yea. | 14:58 |
| openstackgerrit | Merged openstack/kolla: Re-set external_network_bridge for l3 agent https://review.openstack.org/297209 | 14:58 |
| Jeffrey4l_ | done | 14:58 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Changes horizon log location to kolla_logs volume https://review.openstack.org/297952 | 15:01 |
| sdake | Jeffrey4l_ can you +2 this | 15:02 |
| sdake | https://review.openstack.org/#/c/297952/ | 15:02 |
| patchbot | sdake: patch 297952 - kolla (stable/mitaka) - Changes horizon log location to kolla_logs volume | 15:02 |
| openstackgerrit | Merged openstack/kolla: Enable horizon domain logins https://review.openstack.org/296619 | 15:03 |
| Jeffrey4l_ | sdake, done. you are the owner. So i just +2 for it. no +workflow. | 15:05 |
| sdake | cherryppicks we permit cherrypicker to approve if there are not conflicts | 15:06 |
| *** salv-orlando has quit IRC | 15:06 | |
| sdake | if there are conflicts it needs to go through a thorough code review | 15:07 |
| *** salv-orlando has joined #kolla | 15:12 | |
| openstackgerrit | Merged openstack/kolla: Add conntrack-tools https://review.openstack.org/296587 | 15:15 |
| Jeffrey4l_ | sdake, so I can +1 workflow directly? | 15:16 |
| sdake | still requires two reviews | 15:16 |
| *** salv-orlando has quit IRC | 15:16 | |
| sdake | but the second reviewer can workflow in cherrypick backports | 15:17 |
| sdake | even if one of the cmmitters (but not the author) is the backporter | 15:17 |
| *** salv-orlando has joined #kolla | 15:22 | |
| Jeffrey4l_ | Got it. Reasonal. | 15:22 |
| Jeffrey4l_ | reasonable. | 15:23 |
| *** achanda has joined #kolla | 15:23 | |
| *** sdake_ has joined #kolla | 15:26 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate https://review.openstack.org/297945 | 15:26 |
| *** sdake has quit IRC | 15:27 | |
| *** achanda has quit IRC | 15:28 | |
| *** salv-orlando has quit IRC | 15:37 | |
| sdake_ | rc2 out the door ^^ | 15:40 |
| sdake_ | https://github.com/openstack/kolla/releases/tag/2.0.0.0rc2 | 15:40 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add note about heat being busted for development https://review.openstack.org/290276 | 15:49 |
| *** iceyao has quit IRC | 15:53 | |
| *** daneyon has joined #kolla | 16:05 | |
| *** Jeffrey4l__ has joined #kolla | 16:15 | |
| *** Jeffrey4l_ has quit IRC | 16:17 | |
| *** sdake has joined #kolla | 16:28 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add reno support https://review.openstack.org/297959 | 16:29 |
| sdake | Jeffrey4l__ can you ack that | 16:30 |
| sdake | i can't because i was an original author | 16:30 |
| sdake | so it will require one more person | 16:30 |
| Jeffrey4l__ | sdake, np | 16:31 |
| *** sdake_ has quit IRC | 16:31 | |
| sdake | Jeffrey4l__ you had a blueprint in rc2 didnt yu | 16:33 |
| sdake | i so can you click the cherrypick button in gerrit and type in stable/mitaka | 16:33 |
| sdake | i/if | 16:33 |
| Jeffrey4l__ | sdake, seems yes. I will check it and backport to mitaka if needed. | 16:35 |
| sdake | Jeffrey4l__ please email me your address as i would provide to the us postal service for delivery of an item of nominal value to your home | 16:35 |
| sdake | a gift from the foundation | 16:35 |
| Jeffrey4l__ | sdake, cool. what is it? I will email you the address. | 16:36 |
| *** salv-orlando has joined #kolla | 16:36 | |
| *** Jeffrey4l__ is now known as Jeffrey4l | 16:38 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the wrong refer self in kolla_docker.py https://review.openstack.org/297962 | 16:41 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: [WIP] Implement nova-ssh container https://review.openstack.org/297963 | 16:41 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: [WIP] Implement nova-ssh container https://review.openstack.org/297963 | 16:42 |
| *** ccesario has joined #kolla | 16:44 | |
| *** salv-orlando has quit IRC | 16:46 | |
| sdake | Jeffrey4l its a surprise - see ml :) | 16:59 |
| Jeffrey4l | :D | 16:59 |
| *** sdake_ has joined #kolla | 17:01 | |
| *** sdake has quit IRC | 17:04 | |
| Jeffrey4l | sdake_, could u explain whats mean by 'add reno support'? I can not get the point of the `reno` mean. I just get it is a city from dictionary. | 17:06 |
| sdake_ | reno is a release notes tool | 17:06 |
| sdake_ | sudo pip install reno | 17:06 |
| sdake_ | reno new-feature | 17:06 |
| sdake_ | e.g. reno add-reno | 17:06 |
| sdake_ | this creates a yml file | 17:06 |
| sdake_ | which is published on the internets here: | 17:06 |
| sdake_ | http://docs.openstack.org/releasenotes/kolla/ | 17:07 |
| sdake_ | note we have a reno gate job now to generate this documentation | 17:07 |
| Jeffrey4l | got it. | 17:09 |
| openstackgerrit | Steven Dake proposed openstack/kolla: DONOTMERGE: Run upgrade and reconfigure playbooks https://review.openstack.org/289702 | 17:09 |
| sdake_ | groan | 17:09 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate https://review.openstack.org/297945 | 17:11 |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Change keystone log dir https://review.openstack.org/297967 | 17:18 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Provide a release notes link in our main index.rst https://review.openstack.org/297968 | 17:18 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Upgrade elasticsearch https://review.openstack.org/297969 | 17:20 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: [WIP] Implement nova-ssh container https://review.openstack.org/297963 | 17:26 |
| *** achanda has joined #kolla | 17:26 | |
| *** achanda has quit IRC | 17:31 | |
| *** salv-orlando has joined #kolla | 17:36 | |
| *** achanda has joined #kolla | 17:39 | |
| *** Serlex has quit IRC | 17:42 | |
| *** Jeffrey4l has quit IRC | 17:45 | |
| *** salv-orlando has quit IRC | 18:06 | |
| *** SiRiuS_ has quit IRC | 18:14 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate https://review.openstack.org/297945 | 18:29 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate https://review.openstack.org/297945 | 18:33 |
| *** sdake has joined #kolla | 18:39 | |
| *** sdake_ has quit IRC | 18:40 | |
| *** ayoung has quit IRC | 18:52 | |
| *** iceyao has joined #kolla | 18:54 | |
| *** sdake has quit IRC | 18:57 | |
| *** iceyao has quit IRC | 18:58 | |
| *** salv-orlando has joined #kolla | 19:07 | |
| *** sdake has joined #kolla | 19:15 | |
| *** daneyon has quit IRC | 19:16 | |
| *** achanda has quit IRC | 19:19 | |
| *** salv-orlando has quit IRC | 19:21 | |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Change keystone log dir https://review.openstack.org/297967 | 19:28 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate https://review.openstack.org/297945 | 19:29 |
| *** daneyon has joined #kolla | 19:37 | |
| *** vhosakot has joined #kolla | 19:40 | |
| *** vhosakot has quit IRC | 19:42 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix gate to use newgrp instead of sudo https://review.openstack.org/297945 | 19:43 |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Added hostname param into kolla_docker https://review.openstack.org/297984 | 19:51 |
| *** achanda has joined #kolla | 19:57 | |
| *** vhosakot has joined #kolla | 20:04 | |
| ccesario | sdake: ping ? | 20:04 |
| sdake | shoot | 20:05 |
| ccesario | https://bugs.launchpad.net/kolla/+bug/1531407 is it needed ? | 20:05 |
| openstack | Launchpad bug 1531407 in kolla "add new script to clean named volume" [Low,Confirmed] - Assigned to Kuo-tung Kao (jelly) (coding1314) | 20:05 |
| vhosakot | ccesario: you took my bug :) I was working on it.... :) | 20:05 |
| vhosakot | ccesario: let me catch up the IRC logs :) | 20:06 |
| sdake | ya ccesario its not ideal to take peoples ugs because htey may already be done with the work and not git reviewed it yet | 20:06 |
| sdake | ccesario i dont really know for sure | 20:07 |
| sdake | i dont see named volumes in my host after i run cleanup-containers | 20:07 |
| vhosakot | ccesario: i had done 90% of the fix and goiing to submit patch set for review today :) | 20:07 |
| vhosakot | ccesario: please ask next time... :) | 20:08 |
| vhosakot | sdake: I see you moved https://bugs.launchpad.net/kolla/+bug/1560620 to ccesario | 20:08 |
| openstack | Launchpad bug 1560620 in kolla "Move keystone-apache-*.log files to /var/log/kolla/keystone directory in keystone container" [Critical,In progress] - Assigned to Carlos Cesario (ccesario) | 20:08 |
| sdake | vhosakot yes because he submitted a patch for it | 20:08 |
| vhosakot | ah cool | 20:08 |
| sdake | and the patch loosk correct to me | 20:08 |
| vhosakot | sdake: cool, np.. I will stop my work and review ccesario's patch | 20:09 |
| vhosakot | sdake: yep, looks correct | 20:09 |
| sdake | sorry for the wated effort vhosakot - wasn't my doing ;) | 20:09 |
| vhosakot | sdake: cool, np... :) | 20:10 |
| vhosakot | sdake: so, did you have any bug in mind I need to pick up ? if not, I'll just browse launchpad and take one | 20:10 |
| sdake | coolsnap said he was going to finish dorp root on swift | 20:11 |
| sdake | but i havne't seen it yet | 20:11 |
| sdake | we need drop root for haproxy | 20:11 |
| ccesario | I 'm sorry by post first vhosakot ... I 'm newbie in this task ... sorry by that | 20:11 |
| ccesario | and I 'm looking for easy fix for ME ... :P | 20:11 |
| sdake | vhosakot if you want something hard sort out drop root for haproxy | 20:11 |
| vhosakot | sdake: ha, droproot for hap... cool,... | 20:12 |
| sdake | read the bug log first | 20:12 |
| sdake | it hsa recommendations to not actually dorp root | 20:12 |
| vhosakot | ccesario: cool, np... please ask in IRC next time becasue people usually start fixing it once thay assign it to themselves :) | 20:12 |
| vhosakot | sdake: ah ok, will read the bug | 20:12 |
| ccesario | sdake: ok | 20:12 |
| vhosakot | ccesario: cheers :) | 20:13 |
| vhosakot | ccesario: will review your fix and let me know if there are any comments... it looks as of now.. :) | 20:13 |
| vhosakot | sdake: Michal Rostecki has a patch set in queue for haproxy droproot... https://review.openstack.org/#/c/245013/... I will pick a different bug... cool | 20:15 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix gate to use newgrp instead of sudo https://review.openstack.org/297945 | 20:15 |
| patchbot | vhosakot: patch 245013 - kolla - Drop root for HAProxy | 20:15 |
| sdake | he hasn't worked on it for 3-4 weeks | 20:16 |
| vhosakot | ah ok.. cool | 20:16 |
| vhosakot | sdake: will work on it | 20:16 |
| ccesario | vhosakot: as told you https://review.openstack.org/#/c/297984/ | 20:16 |
| patchbot | ccesario: patch 297984 - kolla - Added hostname param into kolla_docker | 20:16 |
| vhosakot | sdake: I need this to be merged https://review.openstack.org/#/c/296649/ | 20:17 |
| patchbot | vhosakot: patch 296649 - kolla - Revert libvirt change in heka-openstack.toml.j2 in... | 20:17 |
| sdake | i am waiting on merges for the gate to be fixed | 20:17 |
| vhosakot | sdake: I am working working on implementing lus parser in heka for libvirtd.loh | 20:17 |
| sdake | so hang tight | 20:17 |
| ccesario | if you could test it | 20:17 |
| vhosakot | sdake: vool | 20:17 |
| vhosakot | cool* | 20:17 |
| vhosakot | sdake: https://review.openstack.org/#/c/296012/ needs to merge as well... Eric have +1... | 20:18 |
| patchbot | vhosakot: patch 296012 - kolla (stable/mitaka) - Make heka collect libvirtd.log in nova_libvirt con... | 20:18 |
| vhosakot | gave* | 20:18 |
| sdake | i am not acking patches until the gate is fixed | 20:18 |
| sdake | i do reviews several times a day | 20:18 |
| sdake | i go throug hthe whole lsit - and bitch on reviews that need rebases for weeks or have no activity for weeks :) | 20:19 |
| vhosakot | sdake: cool, perfect :) | 20:19 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Implement kuryr neutron plugin https://review.openstack.org/297987 | 20:19 |
| vhosakot | ccesario: cool, will review the other patch set as well.. | 20:19 |
| openstackgerrit | Merged openstack/kolla: Fix typos in Kolla https://review.openstack.org/297909 | 20:20 |
| sdake | vhosakot i will get the gate fixed today | 20:21 |
| sdake | i nkow how to fix it | 20:21 |
| sdake | i am just learning how the env is setup now since infra changed the base images | 20:21 |
| vhosakot | sdake: what happened to gate ? testr: command not found ?? | 20:22 |
| sdake | because it is being ran via sudo | 20:23 |
| sdake | sudo is used to change the group | 20:23 |
| sdake | the proper way to change groups is newgrp command | 20:24 |
| *** SiRiuS_ has joined #kolla | 20:24 | |
| vhosakot | ah ok.. saw your steak debug patch set | 20:24 |
| *** vhosakot has quit IRC | 20:28 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix gate to use newgrp instead of sudo https://review.openstack.org/297945 | 20:34 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Implement kuryr neutron plugin https://review.openstack.org/297987 | 20:38 |
| *** fbarilla has joined #kolla | 20:42 | |
| sdake | hui ping | 20:46 |
| sdake | hui kang if your around want to chat | 20:48 |
| *** achanda has quit IRC | 20:52 | |
| *** fbarilla has quit IRC | 21:06 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix gate to use newgrp instead of sudo https://review.openstack.org/297945 | 21:11 |
| *** salv-orlando has joined #kolla | 21:11 | |
| *** salv-orlando has quit IRC | 21:26 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Use debian repos for debian base docker image https://review.openstack.org/288936 | 21:38 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix gate to use newgrp instead of sudo https://review.openstack.org/297945 | 21:49 |
| *** openstackgerrit has quit IRC | 22:02 | |
| *** openstackgerrit has joined #kolla | 22:03 | |
| *** ccesario has quit IRC | 22:28 | |
| *** sdake_ has joined #kolla | 22:28 | |
| *** sdake has quit IRC | 22:29 | |
| *** salv-orlando has joined #kolla | 22:38 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix gate to use newgrp instead of sudo https://review.openstack.org/297945 | 22:39 |
| *** salv-orlando has quit IRC | 22:45 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate to use newgrp instead of sudo https://review.openstack.org/297945 | 23:00 |
| sdake_ | groan | 23:00 |
| sdake_ | you would think bash woudl read newgrp and say "hey let me execute a setgid" | 23:00 |
| *** achanda has joined #kolla | 23:03 | |
| *** ccesario has joined #kolla | 23:04 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate to use sgid tox https://review.openstack.org/297945 | 23:24 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate to use sgid tox https://review.openstack.org/297945 | 23:39 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate to use world writeable docker socket https://review.openstack.org/297945 | 23:40 |
| sdake_ | running out of ideas | 23:40 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate to use world writeable docker socket https://review.openstack.org/297945 | 23:44 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate to use world writeable docker socket https://review.openstack.org/297945 | 23:49 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix gate to use world writeable docker socket https://review.openstack.org/297945 | 23:53 |
| *** sdake has joined #kolla | 23:55 | |
| *** allen_gao has quit IRC | 23:56 | |
| *** sdake_ has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!