| vhosakot | Jeffrey4l_: sdake corrected it.. check like 62 in PS 8 https://review.openstack.org/#/c/298479/8/docker/kolla-toolbox/Dockerfile.j2 | 00:03 |
|---|---|---|
| Jeffrey4l_ | vhosakot, saw that, thanks. | 00:03 |
| vhosakot | cool.. np... | 00:03 |
| sdake | atleat we have a low count of triaged bugs | 00:03 |
| sdake | everything is either confirmed or in progress | 00:04 |
| vhosakot | as far as I know, haproxy is the only container running as root, and I am working on dropping it | 00:05 |
| sdake | swift | 00:06 |
| sdake | haproxy cannot drop root | 00:06 |
| manjeets | sdake is horizon logging centralized with kolla ? | 00:08 |
| manjeets | I mean can i find all in just one horizon.log ? | 00:08 |
| sdake | yup if you enable_central_looggin:yes | 00:08 |
| manjeets | oke | 00:08 |
| manjeets | where do i enable that ? | 00:08 |
| manjeets | which file ? | 00:08 |
| sdake | ut it will be stored on the controller nodes in the heka container as well | 00:08 |
| *** salv-orl_ has quit IRC | 00:09 | |
| sdake | enable_central_logging is hard to use because we lack a kibana dashboard for it | 00:09 |
| sdake | just use the heka container on the controller node | 00:09 |
| sdake | docker exec heka cat /var/log/kolla/horizon/* | 00:09 |
| sdake | note horizon changes are either in flight or just recently landed | 00:09 |
| sdake | so your code may not have them in it atm | 00:09 |
| *** jtriley has joined #openstack-kolla | 00:10 | |
| gmmaha | sdake: vhosakot: once the containers are created, should libvirtd start in the host? | 00:12 |
| *** rajathagasthya has quit IRC | 00:14 | |
| Jeffrey4l_ | gmmaha, libvirtd is ran in the docker too and the ansible will start it automatically. | 00:18 |
| gmmaha | Jeffrey4l_: on the host as well? | 00:19 |
| gmmaha | Just checking.. cause all this while i was beating to get it removed from the system | 00:19 |
| Jeffrey4l_ | gmmaha, in a container. | 00:19 |
| Jeffrey4l_ | yea. you need remove the libvirt in the host. | 00:19 |
| gmmaha | Jeffrey4l_: aah ok.. Yeah then something is wrong.. | 00:20 |
| gmmaha | i removed it from the host and post deploy i see libvirtd running on the host | 00:20 |
| gmmaha | :| | 00:20 |
| Jeffrey4l_ | gmmaha, that is not happend. can u make sure is there a container named nova_libvirt? | 00:21 |
| gmmaha | Jeffrey4l_: yeah it is there.. | 00:22 |
| gmmaha | and ight after i remove the containers the livirtd --listen process is gone :O | 00:22 |
| gmmaha | :( | 00:22 |
| Jeffrey4l_ | gmmaha, that's right. why u remove the container | 00:22 |
| *** diogogmt has joined #openstack-kolla | 00:22 | |
| gmmaha | Jeffrey4l_: the deployment failed.. | 00:22 |
| gmmaha | when it gets to rabbitmq | 00:23 |
| openstackgerrit | Merged openstack/kolla: Update swift name https://review.openstack.org/299351 | 00:23 |
| Jeffrey4l_ | please post the config file and deploy log. | 00:23 |
| gmmaha | http://paste.openstack.org/show/492520/ | 00:23 |
| vhosakot | gmmaha: correct.. libvirt must not be running on the compute host... it will run inside the nova_libvirt container... | 00:23 |
| vhosakot | gmmaha: can you do docker exec nova_libvirt ps aux | grep libvirt | 00:23 |
| gmmaha | vhosakot: just cleaned up the container.. | 00:24 |
| gmmaha | vhosakot: but the libvirtd --listen process is runnin gon the host | 00:24 |
| gmmaha | and thats right after all the nova containers are deployed | 00:24 |
| vhosakot | the is te same process inside the nova_libvirt container.. the PID matches both inside the cotnainer and on host | 00:25 |
| gmmaha | vhosakot: aaah | 00:25 |
| *** jtriley has quit IRC | 00:25 | |
| gmmaha | hmm | 00:25 |
| vhosakot | that is fine and expected | 00:25 |
| gmmaha | Ok.. | 00:25 |
| gmmaha | but then the deployment is failing for sure | 00:25 |
| * gmmaha goes to disable heat and see if that works | 00:25 | |
| *** jtriley has joined #openstack-kolla | 00:26 | |
| gmmaha | vhosakot: youa re right.. | 00:32 |
| gmmaha | its the same PID | 00:32 |
| vhosakot | yep.. that is fine.. not an issue | 00:32 |
| vhosakot | ann processes inside container have same PID on host | 00:33 |
| vhosakot | the way docker setups process tables I think | 00:33 |
| gmmaha | vhosakot: aah ok.. | 00:33 |
| gmmaha | i ned to read up on docker more | 00:33 |
| *** sdake has quit IRC | 00:34 | |
| *** weiyu_ has joined #openstack-kolla | 00:34 | |
| ccesario_ | hey, someone did run deploy from master branch today ? | 00:35 |
| gmmaha | ccesario_: are you hitting with it? | 00:36 |
| *** sdake has joined #openstack-kolla | 00:36 | |
| gmmaha | i am seeing these errors.. http://paste.openstack.org/show/492520/ | 00:36 |
| *** Jeffrey4l_ has quit IRC | 00:38 | |
| ccesario_ | I'm just update my local rep cleanup all nodes and try deploy .... I 'm getting problem on maridb task | 00:38 |
| *** achanda has quit IRC | 00:41 | |
| ccesario_ | gmmaha: checking the error | 00:42 |
| gmmaha | ccesario_: thanks | 00:42 |
| *** asalkeld_ has quit IRC | 00:45 | |
| *** rhallisey has quit IRC | 00:45 | |
| ccesario_ | http://sprunge.us/RTHO?debug my error it s different..... generic error..... | 00:45 |
| *** Marga_ has quit IRC | 00:48 | |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add Kuryr ansible role https://review.openstack.org/298894 | 00:50 |
| *** dave-mccowan has quit IRC | 00:53 | |
| *** banix has quit IRC | 00:55 | |
| *** dave-mccowan has joined #openstack-kolla | 00:58 | |
| *** banix has joined #openstack-kolla | 00:59 | |
| gmmaha | vhosakot: its not the latest master.. My deployments even with previous code where they worked is halting now with the same error.. | 01:00 |
| vhosakot | libvirt permission error ? | 01:01 |
| gmmaha | vhosakot: no.. libvirt is fine now | 01:01 |
| gmmaha | but this. http://paste.openstack.org/show/492520/ | 01:01 |
| gmmaha | vhosakot: https://github.com/openstack/kolla/blob/master/ansible/site.yml#L133 | 01:02 |
| gmmaha | at that spot | 01:02 |
| * manjeets just built 2000 vms | 01:04 | |
| * manjeets crosses fingers | 01:05 | |
| ccesario_ | manjeets: O_o | 01:07 |
| * manjeets sees 600 active so and 0 with error status | 01:08 | |
| manjeets | so far88 | 01:08 |
| sdake | gmmaha i am avaialble again for 1:1 debug | 01:11 |
| sdake | manjeets did the2000 vms produce useful data | 01:12 |
| sdake | such as highwter mark on database connectivity | 01:12 |
| manjeets | on did not see more than 1600 connections | 01:13 |
| manjeets | running a watch on connection list | 01:13 |
| gmmaha | sdake: sweet.. let me start it again and send you an invite | 01:13 |
| gmmaha | give me 2 | 01:13 |
| sdake | how long id it take to deploy 2000 vms | 01:14 |
| sdake | gmmaha give me a few too | 01:15 |
| ccesario_ | vhosakot: https://bugs.launchpad.net/kolla/+bug/1563506 it is extremly important .... without it the generic error is reported ... http://paste.openstack.org/show/492524/ | 01:15 |
| openstack | Launchpad bug 1563506 in kolla "prechecks must check empty passwords in /etc/kolla/passwords.yml" [Undecided,In progress] - Assigned to Vikram Hosakote (vhosakot) | 01:15 |
| manjeets | 23 minutes still counting | 01:15 |
| sdake | i am setting up my daugther's computer | 01:15 |
| manjeets | nova list is not showing me more than 1100 | 01:16 |
| manjeets | good news is 1100 are active but 900 missing | 01:16 |
| manjeets | horizon is back to business viewing 100 per page | 01:17 |
| manjeets | sdake i delete 100 vms 3 4 times | 01:25 |
| manjeets | database still shows 1000 | 01:25 |
| manjeets | feels like 2000 were active somehow database is enable to show all at once | 01:25 |
| sdake | sounds like a hard limit somewhere | 01:25 |
| sdake | ook at the hpervisor tab in horizon | 01:25 |
| manjeets | horizon is on and off nova list is nicer than horizon | 01:26 |
| manjeets | sdake may be tomorrow need to figure out multiple ip issue that still exists for some vms | 01:27 |
| manjeets | barely 5 or 10 out of 1000 have that problem | 01:27 |
| kklimonda | manjeets: what storage do you use for spawning them? | 01:28 |
| manjeets | storage ? | 01:28 |
| manjeets | u mean storage driver ? | 01:28 |
| kklimonda | yeah, underlying storage for VM images | 01:28 |
| manjeets | i choose one of default flavors | 01:28 |
| manjeets | m1.tiny | 01:29 |
| kklimonda | I remember spawning ~1k VMs while testing, and quite a few of them unable to boot, due to I/O starvation | 01:29 |
| manjeets | i saw active status of all of them | 01:30 |
| manjeets | actually i deleted 400 vms it still shows 1000 active no error 600 are lost | 01:31 |
| manjeets | nova list | grep ACTIVE | wc -l | 01:31 |
| manjeets | 1000 | 01:31 |
| *** weiyu_ has quit IRC | 01:33 | |
| openstackgerrit | Dave McCowan proposed openstack/kolla: When two interfaces are used for two VIPs, tie them together https://review.openstack.org/299704 | 01:33 |
| manjeets | sdake i see on average 32 instances per compute node | 01:34 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix gate by adding MariaDB-shared dependnecy https://review.openstack.org/299681 | 01:34 |
| manjeets | for 53 computes | 01:34 |
| manjeets | that means more than 1700 vms and nova list grep error gives me 0 | 01:34 |
| manjeets | need to figure where 600 or 700 are hiding | 01:35 |
| vhosakot | ccesario_: yes, I am working on flagging an error if kolla is run with all empty passwords.. will keep upi psted | 01:37 |
| *** weiyu has joined #openstack-kolla | 01:38 | |
| sdake | manjeets i think you could be running into another quota | 01:38 |
| ccesario_ | vhosakot: and if this check was added into kolla-genpw ? | 01:39 |
| sdake | manjeets nm i see what you mean | 01:39 |
| sdake | you see 32 qemu processes | 01:39 |
| vhosakot | ccesario_: no,, I am adding a precehck | 01:40 |
| vhosakot | ccesario_: run kolla-genpwd to resolve errors due to empty passwords | 01:41 |
| *** weiyu has quit IRC | 01:41 | |
| *** achanda has joined #openstack-kolla | 01:41 | |
| ccesario_ | vhosakot: yes yes .. I meant this.... but kolla-genpwd check if is empty and then precheck call kolla-genpwd --check :P | 01:42 |
| vhosakot | ccesario_: hmmm... dont know if kolla-genpwd should be automated... I think operators must know what passwrods they set for which service and also if they use a random-generator like kolla-genpwd... | 01:45 |
| vhosakot | let us now force/automate passwords that operators want :) right ? :) | 01:46 |
| *** achanda has quit IRC | 01:46 | |
| ccesario_ | vhosakot: yes yes I agree..... but think about.. If the operator run kolla-genpwd, and deploy ... and after that he run again (unintentionally) it overwrite the passwords ... it can cause a big problem for operator :P | 01:48 |
| *** sdake_ has joined #openstack-kolla | 01:49 | |
| ccesario_ | vhosakot: ohhh ignore my comment.... | 01:49 |
| *** Marga_ has joined #openstack-kolla | 01:49 | |
| ccesario_ | double kolla-genpw does not overwrite the pass :P | 01:50 |
| vhosakot | yep :) | 01:50 |
| vhosakot | that is what I meant | 01:50 |
| vhosakot | :) | 01:50 |
| *** sdake has quit IRC | 01:50 | |
| ccesario_ | vhosakot: sorry :) | 01:50 |
| vhosakot | cool, np | 01:50 |
| vhosakot | bbiab 30 mins | 01:50 |
| *** vhosakot has quit IRC | 01:51 | |
| *** Marga_ has quit IRC | 01:54 | |
| *** achanda has joined #openstack-kolla | 01:56 | |
| *** weiyu has joined #openstack-kolla | 01:59 | |
| *** unicell1 has quit IRC | 02:02 | |
| *** Marga_ has joined #openstack-kolla | 02:10 | |
| *** Marga_ has quit IRC | 02:14 | |
| *** banix has quit IRC | 02:16 | |
| *** Jeffrey4l has joined #openstack-kolla | 02:17 | |
| Jeffrey4l | vhosakot, pls stop using `recheck` on the mitaka branch. the gate is broken. recheck is not helpful. | 02:20 |
| Jeffrey4l | vhosakot, when this https://review.openstack.org/299681 is merged, the gate will be fixed. | 02:20 |
| *** tfukushima has joined #openstack-kolla | 02:21 | |
| *** salv-orlando has joined #openstack-kolla | 02:21 | |
| *** alisonh has quit IRC | 02:21 | |
| ccesario_ | gmmaha: only to keep you notfied .... y deployment it is working ... the problem it was empty passwords :P | 02:26 |
| gmmaha | ccesario_: thanks.. :) | 02:26 |
| gmmaha | guessing i am a special breed with toooo many ways of destroying kolla.. :P | 02:26 |
| ccesario_ | gmmaha: any news in your deployment ? | 02:27 |
| *** salv-orlando has quit IRC | 02:32 | |
| *** salv-orlando has joined #openstack-kolla | 02:36 | |
| *** alisonh has joined #openstack-kolla | 02:39 | |
| *** alisonh has quit IRC | 02:44 | |
| *** Marga_ has joined #openstack-kolla | 02:48 | |
| *** alisonh has joined #openstack-kolla | 02:50 | |
| *** salv-orlando has quit IRC | 02:52 | |
| *** Marga_ has quit IRC | 02:53 | |
| *** tfukushima has quit IRC | 02:59 | |
| *** ccesario_ has quit IRC | 03:03 | |
| *** achanda has quit IRC | 03:03 | |
| *** sdake has joined #openstack-kolla | 03:07 | |
| *** Marga_ has joined #openstack-kolla | 03:07 | |
| *** Marga_ has quit IRC | 03:07 | |
| *** Marga_ has joined #openstack-kolla | 03:08 | |
| *** Marga_ has quit IRC | 03:08 | |
| *** weiyu has quit IRC | 03:08 | |
| *** Marga_ has joined #openstack-kolla | 03:09 | |
| *** weiyu has joined #openstack-kolla | 03:09 | |
| *** sdake_ has quit IRC | 03:10 | |
| *** yuanying has quit IRC | 03:21 | |
| *** jeblair_ has joined #openstack-kolla | 03:27 | |
| sdake | Jeffrey4l ping | 03:28 |
| Jeffrey4l | sdake, pong | 03:28 |
| sdake | neutron problem | 03:28 |
| *** vhosakot has joined #openstack-kolla | 03:28 | |
| sdake | complains aout incorrect json format | 03:28 |
| Jeffrey4l | sdake, neutron reconfigure? | 03:28 |
| sdake | not reconfigure deploy | 03:28 |
| vhosakot | evening | 03:29 |
| sdake | there was a workaround in the bug log | 03:29 |
| sdake | hey vhosakot | 03:29 |
| sdake | i dont knwo where the bug log is | 03:29 |
| Jeffrey4l | sdake, could you paste the log out? | 03:29 |
| sdake | it involved moving something in site.yml | 03:29 |
| Jeffrey4l | sdake, I haven't meet that when deploy multi node. | 03:29 |
| sdake | does tht ring a bell? | 03:29 |
| Jeffrey4l | so logs will be helpful. | 03:29 |
| sdake | Jeffrey4l can you join a google hangout | 03:29 |
| Jeffrey4l | np | 03:30 |
| Jeffrey4l | link? | 03:30 |
| vhosakot | I remmeber reordering hosts in multinode inventory to work-around Ansible 1.9.4 bug.. is that what you're talking about sdake | 03:30 |
| Jeffrey4l | btw, fyi the neutron reconfigure is buggy, which i am working on it. | 03:30 |
| sdake | vhosakot yes link on the bug needed plz | 03:30 |
| Jeffrey4l | vhosakot, maybe. | 03:30 |
| vhosakot | I'd like to join google hangout too | 03:30 |
| vhosakot | well, ccesario or mlima submitted a PS for that, and Sam ask to abandon it.. let me search.. 1 sec | 03:31 |
| sdake | its broken it needs fixing i dont care how | 03:31 |
| sdake | i thin kthe problem is its a global variable | 03:31 |
| sdake | Jeffrey4l doe register prodce globals or locals? | 03:31 |
| Jeffrey4l | gmmaha, sent the link to me | 03:31 |
| gmmaha | Jeffrey4l: hey yeah | 03:32 |
| *** jeblair has quit IRC | 03:32 | |
| *** serverascode has quit IRC | 03:33 | |
| *** tfukushima has joined #openstack-kolla | 03:34 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 03:34 | |
| Jeffrey4l_ | sdake, whats mean by 'doe register prodce globals or locals' | 03:35 |
| sdake | register: xyz | 03:35 |
| sdake | does that produce a global | 03:35 |
| sdake | or a local to the role | 03:35 |
| sdake | because we have alot of register; database operations | 03:35 |
| *** jtriley has quit IRC | 03:37 | |
| vhosakot | https://review.openstack.org/#/c/285408/1 | 03:37 |
| *** Jeffrey4l has quit IRC | 03:37 | |
| *** coolsvap has joined #openstack-kolla | 03:38 | |
| *** serverascode has joined #openstack-kolla | 03:38 | |
| vhosakot | https://bugs.launchpad.net/kolla/+bug/1546789 | 03:38 |
| openstack | Launchpad bug 1546789 in kolla mitaka "neutron failed to deploy in a multi-node deployment" [High,Confirmed] - Assigned to Michał Jastrzębski (inc007) | 03:38 |
| sdake | http://eavesdrop.openstack.org/irclogs/%23kolla/%23kolla.2016-03-11.log.html#t2016-03-11T19:56:39 | 03:38 |
| Jeffrey4l_ | sdake, sorry still do not get your point. I know the register stuff now. whats means by `global or a local to the role` | 03:43 |
| vhosakot | https://github.com/openstack/kolla/blob/master/ansible/roles/neutron/tasks/bootstrap.yml#L14 | 03:51 |
| *** salv-orlando has joined #openstack-kolla | 03:52 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Update swift name https://review.openstack.org/299736 | 03:52 |
| *** salv-orlando has quit IRC | 03:55 | |
| dave-mccowan | my deploy is failing on image does not exist for centos-binary-cron. has anyone else seen this? | 04:01 |
| *** achanda has joined #openstack-kolla | 04:04 | |
| *** yuanying has joined #openstack-kolla | 04:07 | |
| *** achanda has quit IRC | 04:09 | |
| *** achanda has joined #openstack-kolla | 04:09 | |
| Jeffrey4l_ | dave-mccowan, pull the latest code and rebuild the code. then try deploy again. | 04:09 |
| dave-mccowan | Jeffrey4l_ thanks. i figured it out. i forgot to specify my local registry when i did the build. | 04:10 |
| vhosakot | groups['neutron-server'] | 04:18 |
| *** salv-orlando has joined #openstack-kolla | 04:19 | |
| Jeffrey4l_ | vhosakot, sorry , i can not speak in the hangout. but could you post the multinode inventory and ansible run result out? | 04:19 |
| vhosakot | Jeffrey4l_: sure, I will.. 1 sec | 04:21 |
| Jeffrey4l_ | vhosakot, thanks. | 04:22 |
| *** salv-orlando has quit IRC | 04:23 | |
| *** sdake_ has joined #openstack-kolla | 04:28 | |
| gmmaha | when: inventory_hostname == neutron-server[0] | 04:30 |
| sdake_ | my a dio no beuno | 04:30 |
| sdake_ | why not and all three variables | 04:30 |
| sdake_ | is there a way to do such a thing | 04:30 |
| *** sdake has quit IRC | 04:30 | |
| sdake_ | AND | 04:30 |
| sdake_ | andthe sets | 04:30 |
| sdake_ | we tried in | 04:32 |
| sdake_ | bu we didn't try == | 04:32 |
| sdake_ | dont use in | 04:32 |
| sdake_ | use == | 04:32 |
| sdake_ | wtb buy working audio | 04:33 |
| sdake_ | because that could break something else | 04:33 |
| sdake_ | if delegate_to is broken we need to nuke it | 04:34 |
| *** Jeffrey4l_ has quit IRC | 04:34 | |
| gmmaha | sdake_: you are showing as on mute :) | 04:35 |
| sdake_ | lets try == now tho beuseits multinode | 04:35 |
| *** weiyu has quit IRC | 04:37 | |
| gmmaha | https://github.com/ansible/ansible/issues/14684 | 04:41 |
| gmmaha | https://github.com/ansible/ansible/pull/15024 | 04:41 |
| sdake_ | run once means i runs only one time ever | 04:47 |
| sdake_ | bootstrap only runs once ever | 04:47 |
| sdake_ | otherwise its skipped | 04:47 |
| sdake_ | sorry my adio is bad | 04:48 |
| gmmaha | https://github.com/ansible/ansible/issues/13226 | 04:48 |
| sdake_ | i would highly recommend anding the 3 variables | 04:48 |
| sdake_ | i think that would work | 04:48 |
| sdake_ | becasue its not == and not in | 04:48 |
| vhosakot | https://www.elmund.io/configuration%20management/2015/07/23/ansible-delegate_to-and-variables/ | 04:49 |
| sdake_ | == = in | 04:49 |
| sdake_ | anding the 3 lists will produce 1 item | 04:50 |
| sdake_ | [a,b,c] & [a] & [a] = a | 04:50 |
| sdake_ | i can hear but cant speak | 04:50 |
| sdake_ | if that can be done in ansible its a different syntax | 04:51 |
| sdake_ | and might avoid the problem | 04:51 |
| sdake_ | because debug works with the printing the 3 variables | 04:51 |
| sdake_ | i dont now the syntax | 04:52 |
| sdake_ | yes right | 04:52 |
| sdake_ | but it will produce th correct variable without indexing ;) | 04:52 |
| sdake_ | and without = nor in | 04:52 |
| sdake_ | git diff | 04:53 |
| sdake_ | real quick | 04:53 |
| sdake_ | whatever was in th edebug ;) | 04:53 |
| sdake_ | i dont no wif that is correct syntax | 04:54 |
| sdake_ | and the sets | 04:55 |
| sdake_ | and the sets | 04:55 |
| sdake_ | not and in boolean | 04:55 |
| sdake_ | dont treat it like a bollean treat it like a set | 04:55 |
| openstackgerrit | Dave McCowan proposed openstack/kolla: When two interfaces are used for two VIPs, tie them together https://review.openstack.org/299704 | 04:56 |
| sdake_ | ok thorw that one out | 04:56 |
| sdake_ | no in | 04:56 |
| sdake_ | dont use in | 04:56 |
| sdake_ | it needs to get a set and | 04:56 |
| sdake_ | no dont try that | 04:56 |
| sdake_ | wait just a sec | 04:56 |
| sdake_ | ['a,b,c']&['a','b'] = [a,b] | 04:57 |
| sdake_ | this is a set and | 04:57 |
| sdake_ | we need a set and not a boolean and | 04:57 |
| *** bigjools has joined #openstack-kolla | 05:04 | |
| *** bigjools has left #openstack-kolla | 05:05 | |
| *** asalkeld_ has joined #openstack-kolla | 05:07 | |
| *** dave-mccowan has quit IRC | 05:13 | |
| *** sdake_ is now known as sdake | 05:14 | |
| *** coolsvap has quit IRC | 05:17 | |
| *** coolsvap has joined #openstack-kolla | 05:17 | |
| *** unicell has joined #openstack-kolla | 05:23 | |
| *** salv-orlando has joined #openstack-kolla | 05:23 | |
| *** coolsvap has quit IRC | 05:26 | |
| *** salv-orlando has quit IRC | 05:28 | |
| gmmaha | https://bugs.launchpad.net/kolla/+bug/1546789 | 05:28 |
| openstack | Launchpad bug 1546789 in kolla mitaka "neutron failed to deploy in a multi-node deployment" [High,Confirmed] - Assigned to Michał Jastrzębski (inc007) | 05:28 |
| vhosakot | https://bugs.launchpad.net/kolla/+bug/1546789 | 05:29 |
| *** coolsvap has joined #openstack-kolla | 05:32 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 05:39 | |
| sdake | http://docs.ansible.com/ansible/playbooks_delegation.html#run-once | 05:43 |
| *** daneyon_ has joined #openstack-kolla | 05:53 | |
| *** weiyu has joined #openstack-kolla | 05:53 | |
| *** Jeffrey4l_ has quit IRC | 05:54 | |
| *** daneyon_ has quit IRC | 05:58 | |
| *** unicell has quit IRC | 06:16 | |
| *** weiyu has quit IRC | 06:17 | |
| *** unicell has joined #openstack-kolla | 06:17 | |
| *** vhosakot has quit IRC | 06:18 | |
| *** mikelk has joined #openstack-kolla | 06:21 | |
| *** allen_gao has joined #openstack-kolla | 06:24 | |
| *** jiriprox has quit IRC | 06:24 | |
| *** SiRiuS_ has joined #openstack-kolla | 06:28 | |
| *** mikelk has quit IRC | 06:29 | |
| *** mikelk has joined #openstack-kolla | 06:34 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Fix gate by adding MariaDB-shared dependnecy https://review.openstack.org/299681 | 06:35 |
| *** sdake_ has joined #openstack-kolla | 06:37 | |
| *** sdake has quit IRC | 06:40 | |
| *** weiyu has joined #openstack-kolla | 06:41 | |
| openstackgerrit | Merged openstack/kolla: Revert libvirt change in heka-openstack.toml.j2 in master https://review.openstack.org/296649 | 06:43 |
| *** sdake has joined #openstack-kolla | 06:45 | |
| coolsvap | sdake: you wanted to do Workflow +1 or -1 for the keystone change set? | 06:46 |
| *** salv-orlando has joined #openstack-kolla | 06:48 | |
| *** sdake_ has quit IRC | 06:48 | |
| sdake | workflow -1 | 06:49 |
| sdake | did I do =1? | 06:49 |
| sdake | =etired | 06:50 |
| *** salv-orlando has quit IRC | 06:51 | |
| *** Serlex has joined #openstack-kolla | 07:15 | |
| *** coolsvap has quit IRC | 07:28 | |
| *** coolsvap has joined #openstack-kolla | 07:29 | |
| *** wuhg has joined #openstack-kolla | 07:38 | |
| *** coolsvap has quit IRC | 07:39 | |
| *** coolsvap has joined #openstack-kolla | 07:42 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Increase database connections to 10000 https://review.openstack.org/299791 | 07:42 |
| *** shardy has joined #openstack-kolla | 07:48 | |
| sdake | hey shardy | 07:49 |
| *** mgoddard has joined #openstack-kolla | 07:50 | |
| coolsvap | we need the sudo -E and mariadb-shared changes merged for the stable/mitaka gate | 07:58 |
| *** mbound has joined #openstack-kolla | 07:59 | |
| *** cristicalin has joined #openstack-kolla | 07:59 | |
| *** salv-orlando has joined #openstack-kolla | 08:07 | |
| sdake | coolsvap yup | 08:09 |
| sdake | coolsvap i'm pretty sure they have been cherrypicked | 08:09 |
| *** salv-orlando has quit IRC | 08:09 | |
| coolsvap | sdake: yes | 08:10 |
| sdake | asalkeld if your around to approv | 08:10 |
| sdake | or nihilifer | 08:10 |
| sdake | or pbourke | 08:11 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Workaround ansible bug related to delegate_to https://review.openstack.org/299803 | 08:11 |
| *** kproskurin has joined #openstack-kolla | 08:12 | |
| coolsvap | https://review.openstack.org/#/c/299681 and https://review.openstack.org/#/c/299681 | 08:12 |
| coolsvap | pbourke: nihilifer asalkeld ^^ | 08:12 |
| *** stvnoyes has joined #openstack-kolla | 08:13 | |
| *** sdake_ has joined #openstack-kolla | 08:17 | |
| SiRiuS_ | kproskurin, ping | 08:18 |
| *** sdake has quit IRC | 08:19 | |
| kproskurin | SiRiuS_: Hi! | 08:20 |
| SiRiuS_ | kproskurin, Hi :), so I tried deploying, and something is just weird | 08:20 |
| kproskurin | :-D | 08:20 |
| kproskurin | SiRiuS_: Make sure you use latest master, we fixed like 2-3 bugs yesterday btw and still fixing | 08:21 |
| SiRiuS_ | kproskurin, http://paste.openstack.org/show/492541/ | 08:21 |
| SiRiuS_ | kproskurin, I cherrypicked everything :) | 08:22 |
| *** sdake has joined #openstack-kolla | 08:22 | |
| SiRiuS_ | it says all the apps are started | 08:22 |
| SiRiuS_ | but sometime I only get kolla_toolbox | 08:22 |
| SiRiuS_ | sometimes I get nothing | 08:23 |
| kproskurin | SiRiuS_: Open marathon UI and go into mariadb container for example | 08:23 |
| kproskurin | Open configuration and paste it to me | 08:23 |
| *** sdake_ has quit IRC | 08:24 | |
| SiRiuS_ | kproskurin, checking | 08:25 |
| SiRiuS_ | kproskurin, all the apps appear to be suspended | 08:25 |
| SiRiuS_ | 21 suspended | 08:25 |
| kproskurin | I belive you still could get configuration from an app | 08:26 |
| kproskurin | Could you? | 08:26 |
| *** achanda has quit IRC | 08:30 | |
| SiRiuS_ | kproskurin, http://paste.openstack.org/show/492543/ | 08:31 |
| SiRiuS_ | is there a better way to paste the config from the web page? | 08:32 |
| kproskurin | SiRiuS_, Alright, so we need to check few things. 1) "image": "operator.local:5000/kollaglue/centos-source-mariadb:2.0.0" - is this a right path to your docker registry and image? | 08:32 |
| SiRiuS_ | yes | 08:33 |
| *** achanda has joined #openstack-kolla | 08:34 | |
| *** daneyon_ has joined #openstack-kolla | 08:35 | |
| kproskurin | SiRiuS_: 2) Could you exec into your mesos-slave container on this host and check something like “ps axuf” and look for “--attributes=openstack_role:controller” | 08:36 |
| *** mbound has quit IRC | 08:38 | |
| *** daneyon_ has quit IRC | 08:39 | |
| SiRiuS_ | kproskurin, if I run ps aux inside mesos_slave container on controller01, I don't see “--attributes=openstack_role:controller” | 08:41 |
| SiRiuS_ | root 25 0.0 0.1 13372 2020 ? S 08:39 0:00 -bash | 08:41 |
| SiRiuS_ | root 41 0.0 0.0 49016 1704 ? R+ 08:40 0:00 \_ ps axuf | 08:41 |
| SiRiuS_ | root 1 0.1 1.0 949136 19160 ? Ssl+ 08:12 0:03 mesos-slave | 08:41 |
| kproskurin | oh, we pass it via envs, 1 sec | 08:42 |
| kproskurin | SiRiuS_: exec: “env | grep MESOS_ATTRIBUTES” | 08:43 |
| kproskurin | inside the container | 08:43 |
| SiRiuS_ | kproskurin, quick question: does the IP address for the mesos host inside the kolla-mesos.conf file need to be the leader? because I have it set to the master03 node, and now the leader is master01 | 08:43 |
| *** pbourke has quit IRC | 08:43 | |
| *** pbourke has joined #openstack-kolla | 08:44 | |
| kproskurin | SiRiuS_: mesos slaver do an election of their own and chose leader | 08:44 |
| *** sbezverk has quit IRC | 08:44 | |
| *** sbezverk has joined #openstack-kolla | 08:44 | |
| kproskurin | But the redirect all calls to current leader, no matter to which mesos you are connected, so dont worry | 08:44 |
| *** jiriprox has joined #openstack-kolla | 08:46 | |
| SiRiuS_ | kproskurin, I did not find the attributes in the environment | 08:46 |
| SiRiuS_ | kproskurin, this is all I have | 08:46 |
| SiRiuS_ | kproskurin, http://paste.openstack.org/show/492545/ | 08:46 |
| *** cristicalin has quit IRC | 08:47 | |
| *** achanda has quit IRC | 08:48 | |
| kproskurin | SiRiuS_, Im confused. :-( Sadly I dont use this ansible mesos stack myself right now, but it clearly set a lot of stuff via envs: https://github.com/openstack/kolla-mesos/blob/master/ansible/roles/mesos-slave/tasks/start.yml | 08:48 |
| SiRiuS_ | kproskurin, the thing is, sometimes it starts some containers, sometimes it does not start any at all | 08:49 |
| SiRiuS_ | without doing anything to the config | 08:49 |
| *** achanda has joined #openstack-kolla | 08:49 | |
| kproskurin | SiRiuS_: last one thing to check is resourses avalible | 08:50 |
| kproskurin | https://www.dropbox.com/s/g9jlaiay3nflvwa/HW.png?dl=0 | 08:50 |
| kproskurin | It will loke something like this ^^ | 08:50 |
| kproskurin | RAM and CPU | 08:50 |
| kproskurin | To check how much does your mesos-slave have you need to go to “http://mesos-master-ip:5050/#/slaves” | 08:51 |
| kproskurin | and if it has less that marathon expect - it will not run any apps | 08:52 |
| *** achanda has quit IRC | 08:52 | |
| SiRiuS_ | kproskurin, it says I have 4 slaves (which I do) | 08:53 |
| SiRiuS_ | 7 CPUs | 08:53 |
| SiRiuS_ | 3.6GB ram | 08:53 |
| SiRiuS_ | used 0 CPUs | 08:54 |
| kproskurin | SiRiuS_: compare it with marathon expectations from marathon UI | 08:54 |
| SiRiuS_ | used 0 ram | 08:54 |
| SiRiuS_ | kproskurin, in marathon it appear to be 0.0 CPU and O B ram | 08:54 |
| SiRiuS_ | hmm | 08:54 |
| kproskurin | Oh, I see | 08:55 |
| kproskurin | So, my bet it’s a attributes problem | 08:55 |
| kproskurin | Attributes is not set and marathon dont know to which mesos it should pass the app, since it expects this constrains. And kolla-toolbox is a chronos job, which dont use constrains yet, so it works | 08:56 |
| SiRiuS_ | kproskurin, funny thing is, I cleaned everything up, I deployed the mesos cluster again, and deployed the apps again, and this time I don't even have kolla_toolbox running :D | 08:57 |
| kproskurin | Could be a chronos problem, but it’s not the main case right now.Im thinking how to check attributes… | 08:58 |
| *** achanda has joined #openstack-kolla | 08:59 | |
| nihilifer | are they obstackles to get this one merged, except the failing gate? https://review.openstack.org/#/c/297517/ | 09:01 |
| nihilifer | i see that two +2 were given some time ago | 09:01 |
| nihilifer | just wondering whether it needs more cores or discussion | 09:01 |
| SiRiuS_ | kproskurin, | 09:01 |
| SiRiuS_ | kproskurin, http://paste.fedoraproject.org/347780/41489114 | 09:01 |
| SiRiuS_ | kproskurin, http://paste.fedoraproject.org/347782/59414928 | 09:02 |
| SiRiuS_ | kproskurin, that's how my config files look like | 09:02 |
| kproskurin | SiRiuS_: looks ok | 09:02 |
| kproskurin | nihilifer: Do you remember mesos url to check attributes? :-) | 09:03 |
| kproskurin | Fiund it | 09:04 |
| kproskurin | SiRiuS_: http://mesos-master-ip:5050/state | 09:04 |
| nihilifer | :5050/slaves | 09:04 |
| nihilifer | or if it doesn't work, then :5050/state.json | 09:04 |
| kproskurin | Yeah, slaves are better | 09:04 |
| kproskurin | SiRiuS_: ^^ | 09:04 |
| SiRiuS_ | kproskurin, I've just cleaned up and redeployed the mesos cluster, but did not deploy the apps yet | 09:05 |
| kproskurin | It doest matter | 09:05 |
| kproskurin | Attributes should be set at mesos-slave start | 09:06 |
| SiRiuS_ | kproskurin, should I deploy the apps and debug, or is there something interesting to look at before deploying the apps | 09:06 |
| SiRiuS_ | ? | 09:06 |
| SiRiuS_ | ok | 09:06 |
| kproskurin | SiRiuS_: http://mesos-master-ip:5050/slaves | 09:06 |
| kproskurin | And paste it | 09:06 |
| *** coolsvap has quit IRC | 09:07 | |
| SiRiuS_ | http://paste.openstack.org/show/492548/ | 09:07 |
| SiRiuS_ | also this is only for master03, because for master01:5050/slaves and master02:5050/slaves I get only {"slaves":[]} | 09:09 |
| openstackgerrit | Steven Dake proposed openstack/kolla: A Fix gate to use world writeable docker socket https://review.openstack.org/298098 | 09:09 |
| openstackgerrit | Steven Dake proposed openstack/kolla: A Fix gate by adding MariaDB-shared dependnecy https://review.openstack.org/299681 | 09:09 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix gate to use world writeable docker socket https://review.openstack.org/298098 | 09:10 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix gate by adding MariaDB-shared dependnecy https://review.openstack.org/299681 | 09:10 |
| sdake | core reviewers - need some love on those two patches ^^ | 09:10 |
| kproskurin | SiRiuS_: it does look ok. ;-\ Alright, pls run this: “kolla-mesos --profiles-default mariadb deployment run” | 09:11 |
| sdake | nihilifer asalkeld pbourke ^^ | 09:11 |
| kproskurin | SiRiuS_: it will deploy only mariadb, go to it in marathon UI and chose “debug” section | 09:11 |
| pbourke | looking now | 09:11 |
| sdake | thanks | 09:12 |
| sdake | pbourke do y ouhae a ultinode setup? | 09:12 |
| pbourke | yes | 09:12 |
| SiRiuS_ | kproskurin, output of command: | 09:13 |
| SiRiuS_ | INFO - Marathon framework: marathon | 09:13 |
| SiRiuS_ | INFO - Marathon app "/default/infra/mariadb/mariadb" is started | 09:13 |
| SiRiuS_ | INFO - Written OpenStack env to "default-openrc" | 09:13 |
| kproskurin | SiRiuS_: its ok, just go to marathon UI | 09:13 |
| sdake | pbourke cn yu test https://review.openstack.org/#/c/299803/ | 09:13 |
| sdake | pbourke and see if it implodes or works | 09:13 |
| sdake | currently ansible has a bug - this is a workaround | 09:14 |
| sdake | if not i'll see if manjeets or nttptr can test in the morning | 09:14 |
| SiRiuS_ | kproskurin, I have mariadb in "Waiting" state | 09:14 |
| sdake | i am ptfoing | 09:14 |
| sdake | night | 09:14 |
| kproskurin | SiRiuS_: open it and push “Debug” tab, near the “configuration” | 09:15 |
| SiRiuS_ | kproskurin, "This app does not have failed tasks" | 09:16 |
| SiRiuS_ | This app does not have task statistics | 09:16 |
| SiRiuS_ | No operation since last config change | 09:16 |
| kproskurin | SiRiuS_, Alright, so it cant run it | 09:16 |
| kproskurin | SiRiuS_: so it attributes error OR resourses error | 09:17 |
| SiRiuS_ | CPU 0.9 Memory 384 | 09:17 |
| kproskurin | SiRiuS_: let me think a bit | 09:18 |
| SiRiuS_ | kproskurin, ok :) | 09:18 |
| jiriprox | Hi, my kolla deployment with HAproxy is failing in "haproxy | Waiting for virtual IP to appear" msg: Timeout when waiting for x.x.x.x:3306. It was working two weeks back. Do you know what was changed? What needs to be changed to resolve it? | 09:20 |
| sdake | /query nihilifer | 09:21 |
| kproskurin | SiRiuS_: Add to globals.yml: | 09:21 |
| kproskurin | mariadb_mem: “32” | 09:21 |
| kproskurin | mariadb_cpus: “0.1” | 09:21 |
| kproskurin | and re-run deploy | 09:21 |
| *** salv-orlando has joined #openstack-kolla | 09:23 | |
| kproskurin | SiRiuS_: Than, open marathon UI for mariadb in the tab “Instances”(left of configuration) | 09:23 |
| kproskurin | You should see 3 instances below(on each “controller” node) | 09:24 |
| kproskurin | If you could make a screenshoot, it will be helpful | 09:24 |
| *** achanda has quit IRC | 09:26 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 09:26 | |
| SiRiuS_ | kproskurin, http://picpaste.com/SD1GGpZ1.png | 09:28 |
| SiRiuS_ | kproskurin, now it is suspended | 09:29 |
| kproskurin | SiRiuS_: :-\ | 09:29 |
| *** coolsvap has joined #openstack-kolla | 09:30 | |
| *** sdake_ has joined #openstack-kolla | 09:32 | |
| *** sdake has quit IRC | 09:32 | |
| *** sdake has joined #openstack-kolla | 09:33 | |
| *** sdake_ has quit IRC | 09:36 | |
| kproskurin | SiRiuS_: we have a plan to patch mesos\marathon stuff to be more verbose on wtf is wrong with deploy, but its not done yet, so right now only option is to go through mesos huge logs and look at the reason | 09:37 |
| SiRiuS_ | kproskurin, yeah, right now it does not even enter waiting state | 09:39 |
| SiRiuS_ | kproskurin, I cleaned everything up, redeployed the mesos stack | 09:39 |
| *** salv-orlando has quit IRC | 09:39 | |
| SiRiuS_ | kproskurin, and still it won't reply mariadb, | 09:39 |
| SiRiuS_ | kproskurin, it's in suspended state | 09:39 |
| SiRiuS_ | kproskurin, it's random, that's the problem :) | 09:40 |
| kproskurin | SiRiuS_: suspended means it doesnt have some resourses OR constrains to launch. | 09:40 |
| kproskurin | But I dunno why in your case | 09:40 |
| SiRiuS_ | kproskurin, where are those huge mesos logs located? | 09:42 |
| kproskurin | SiRiuS_: Try to deploy AIO. Uncomment “mesos_aio_hostname” in globals.yml and put some any mesos-slave hostname in it. And set multinode: no | 09:42 |
| kproskurin | SiRiuS_: I just wonder if AIO will deploy ok | 09:43 |
| SiRiuS_ | if I connect to the mesos UI, there is a LOG uri | 09:43 |
| kproskurin | SiRiuS_: its not in the UI, it should be in “docker logs CONT_ID” | 09:43 |
| SiRiuS_ | but if I click it I receive :"Failed to initialize ... retrying" | 09:43 |
| kproskurin | SiRiuS_: there is logs for deployment, but you dont have any, so it is useless foe you right now, you need logs of the meos itself and you could access it via “docker logs” | 09:44 |
| kproskurin | SiRiuS_: Try AIO in a spare time, if it will work, then problem is with constrains if not - I give up :-( | 09:45 |
| SiRiuS_ | kproskurin, http://paste.openstack.org/show/492552/ | 09:47 |
| SiRiuS_ | docker logs for mesos_slave on controller01 | 09:48 |
| kproskurin | SiRiuS_, Looks totaly fine but I dont see any deployment atempt, so maybe its marathon logs what we needed | 09:50 |
| *** coolsvap has quit IRC | 09:54 | |
| *** coolsvap has joined #openstack-kolla | 09:55 | |
| sdake | cool we got a good haul for summit | 09:55 |
| sdake | 14 sessions | 09:55 |
| *** rmart04 has joined #openstack-kolla | 09:59 | |
| *** kproskurin has quit IRC | 10:02 | |
| *** sdake has quit IRC | 10:03 | |
| *** ccesario_ has joined #openstack-kolla | 10:04 | |
| *** kproskurin has joined #openstack-kolla | 10:13 | |
| *** ccesario_ has quit IRC | 10:15 | |
| openstackgerrit | Merged openstack/kolla-mesos: Move memcached deps to bootstrap section for horizon https://review.openstack.org/298741 | 10:20 |
| *** coolsvap has quit IRC | 10:24 | |
| *** banix has joined #openstack-kolla | 10:30 | |
| *** rhallisey has joined #openstack-kolla | 10:34 | |
| *** coolsvap has joined #openstack-kolla | 10:38 | |
| dims_ | howdy folks : trying quickstart steps, can't seem to get past this problem with mariadb not coming up - http://paste.openstack.org/show/492559/ | 10:45 |
| dims_ | mkdir: cannot create directory '/var/log/kolla/mariadb': Permission denied | 10:45 |
| rhallisey | add a sudo | 10:46 |
| rhallisey | in front of your start command | 10:46 |
| rhallisey | only need to do that once because we need to create those directories on the host | 10:47 |
| *** tfukushima has quit IRC | 10:49 | |
| *** mbound has joined #openstack-kolla | 10:50 | |
| *** salv-orlando has joined #openstack-kolla | 10:51 | |
| dims_ | rhallisey : y, does not seem to help will try again | 10:58 |
| rhallisey | do you get past the perm error though? | 10:59 |
| dims_ | nope | 10:59 |
| dims_ | rhallisey : i removed the mariadb container, tried deploy again as root. | 11:00 |
| dims_ | tried creating that directory by hand with 777 no luck | 11:00 |
| dims_ | rhallisey : heka memcached etc started ok | 11:05 |
| rhallisey | weird | 11:05 |
| *** weiyu has quit IRC | 11:08 | |
| ccesario | good morning | 11:08 |
| dims_ | rhallisey : ack will dig more | 11:09 |
| ccesario | rhallisey, testing the second deploy based in directory structure | 11:09 |
| rhallisey | dims_, ya not sure. Usually only perms is required to make the dir | 11:10 |
| rhallisey | ccesario, how's it going? | 11:10 |
| ccesario | rhallisey, just fixing some prechecks on test node.... give me a bit time | 11:12 |
| rhallisey | sure | 11:12 |
| *** mbound has quit IRC | 11:13 | |
| *** mbound has joined #openstack-kolla | 11:13 | |
| *** coolsvap has quit IRC | 11:14 | |
| *** mbound has quit IRC | 11:15 | |
| *** SiRiuS_ has quit IRC | 11:22 | |
| *** unicell has quit IRC | 11:22 | |
| *** allen_gao has quit IRC | 11:22 | |
| *** allen_ga- has joined #openstack-kolla | 11:22 | |
| *** unicell has joined #openstack-kolla | 11:23 | |
| *** banix has quit IRC | 11:34 | |
| *** kjelly has joined #openstack-kolla | 11:35 | |
| *** rbrady has joined #openstack-kolla | 11:41 | |
| *** alyson_ has joined #openstack-kolla | 11:51 | |
| *** gfidente has joined #openstack-kolla | 12:06 | |
| *** dwalsh has joined #openstack-kolla | 12:08 | |
| *** ayoung has quit IRC | 12:10 | |
| *** banix has joined #openstack-kolla | 12:15 | |
| *** dave-mccowan has joined #openstack-kolla | 12:20 | |
| *** ccesario_ has joined #openstack-kolla | 12:25 | |
| sbezverk | good morning | 12:27 |
| sbezverk | rhallisey qq if I want to split one PS into pieces, do I abandon the original PS? what is the right aproach here? | 12:28 |
| rhallisey | sbezverk, probably easiest to abandon and split into pieces | 12:30 |
| *** banix has quit IRC | 12:31 | |
| sbezverk | rhallisey got it, thank you | 12:32 |
| rhallisey | no problem | 12:33 |
| *** dwalsh has quit IRC | 12:41 | |
| *** dwalsh has joined #openstack-kolla | 12:41 | |
| *** patchbot has joined #openstack-kolla | 12:48 | |
| *** mbound has joined #openstack-kolla | 12:48 | |
| *** mbound has quit IRC | 12:49 | |
| *** mbound has joined #openstack-kolla | 12:49 | |
| dims_ | rhallisey : am getting the hang of this hopefully :) did a docker inspect on mariadb container, (http://paste.openstack.org/show/492568/) looked at "Mounts"/"Source" and ensured that those directories are present with appropriate permissions | 12:50 |
| *** mbound has quit IRC | 12:51 | |
| *** achanda has joined #openstack-kolla | 13:00 | |
| ccesario | rhallisey, the second deploy pause on this task: TASK: [haproxy | Waiting for virtual IP to appear] **************************** | 13:01 |
| ccesario | it seems the HA address does not goes "UP" | 13:01 |
| ccesario | http://paste.openstack.org/show/492571/ | 13:02 |
| rhallisey | dims_, do you have selinux on or something? | 13:02 |
| rhallisey | dims_, I don't know why that originally happened | 13:03 |
| dims_ | rhallisey : "15.10 (Wily Werewolf)" | 13:03 |
| dims_ | ccesario : i did (enable_haproxy: "No") | 13:04 |
| rhallisey | dims_, gotcha.. so did making those dirs by hand work? | 13:04 |
| rhallisey | seems like you got past the issue | 13:05 |
| ccesario | do you know if is there any "exception" when running two hapry in same subnet ? | 13:05 |
| ccesario | dims_, in the first deploy it is running with haproxy Yes | 13:05 |
| dims_ | rhallisey : yes thanks, got past heka, mariadb... now keystone centos is not building, going to try keystone ubuntu | 13:06 |
| *** achanda has quit IRC | 13:06 | |
| *** daneyon_ has joined #openstack-kolla | 13:06 | |
| ccesario | the config of the both "deploys" http://paste.openstack.org/show/492572/ | 13:06 |
| rhallisey | dims_, maybe that's the issue. Ya use ubuntu since that is your base os | 13:06 |
| dims_ | rhallisey : interesting | 13:07 |
| *** stvnoyes has quit IRC | 13:07 | |
| *** kjelly has quit IRC | 13:09 | |
| *** daneyon_ has quit IRC | 13:10 | |
| ccesario | rhallisey, it seems a haproxy behaivor .... because if I deploy firtly the "cloud2" it works.... and "cloud1" presents the same problem | 13:15 |
| rhallisey | ccesario, can you explain this in different way. You have one cloud that's working (cloud1) and cloud2 is having the haprxy issue? | 13:17 |
| rhallisey | is that what your seeing? | 13:17 |
| ccesario | yes..... | 13:17 |
| ccesario | and now I cleanup cloud1 containers | 13:18 |
| *** salv-orlando has quit IRC | 13:18 | |
| rhallisey | ok let me just do some reading | 13:18 |
| ccesario | and run cloud2 deploy (without touch in config files) | 13:18 |
| ccesario | and the deploy it is running :) | 13:19 |
| *** salv-orlando has joined #openstack-kolla | 13:19 | |
| ccesario | when the cloud2 deploy finish, I will try deploy cloud1 (without cleanup cloud2 containers) | 13:19 |
| rhallisey | ccesario, ok | 13:20 |
| *** stvnoyes has joined #openstack-kolla | 13:20 | |
| ccesario | rhallisey, cloud2 finish!! | 13:20 |
| ccesario | PLAY RECAP ******************************************************************** | 13:20 |
| ccesario | 192.168.201.3 : ok=293 changed=98 unreachable=0 failed=0 | 13:20 |
| rhallisey | ccesario, you are using the multinode inventory correct? | 13:21 |
| ccesario | yes (with 1 node only) | 13:22 |
| ccesario | the prechecks in cloud1 it is OK .... | 13:23 |
| ccesario | PLAY RECAP ******************************************************************** | 13:23 |
| ccesario | 192.168.201.2 : ok=58 changed=0 unreachable=0 failed=0 | 13:23 |
| ccesario | let me try deploy cloud1 | 13:23 |
| rhallisey | k | 13:23 |
| *** salv-orl_ has joined #openstack-kolla | 13:24 | |
| *** salv-orlando has quit IRC | 13:26 | |
| openstackgerrit | Merged openstack/kolla-mesos: Fix rabbitmq version for centos https://review.openstack.org/298720 | 13:30 |
| *** diogogmt has quit IRC | 13:30 | |
| ccesario | rhallisey, http://sprunge.us/XfVh?log | 13:32 |
| ccesario | the process stops in the same point that previous deploy | 13:32 |
| *** diogogmt has joined #openstack-kolla | 13:35 | |
| rhallisey | ccesario, ya you need to use a different vip address since that one is in use | 13:36 |
| ccesario | rhallisey, but I 'm using | 13:37 |
| ccesario | look the http://paste.openstack.org/show/492571 | 13:37 |
| ccesario | :D | 13:38 |
| *** salv-orl_ has quit IRC | 13:38 | |
| ccesario | so it is that running IP is 192.168.201.100 :) | 13:38 |
| rhallisey | what do you have for kolla_internal_vip_address | 13:39 |
| rhallisey | it's that ^ right? | 13:39 |
| ccesario | if I cleanup all and run cloud1 deploy the running ip 192.168.201.200 it works | 13:39 |
| rhallisey | ccesario, I thought you said you are using the same globals.yml | 13:42 |
| rhallisey | so you're using 192.168.201.100 & 192.168.201.200 as your vips for cloud 1 and 2 | 13:42 |
| rhallisey | ? | 13:42 |
| ccesario | rhallisey, no no.... specific config for each "cloud" | 13:42 |
| ccesario | yes yes | 13:42 |
| ccesario | 192.168.201.100 & 192.168.201.200 | 13:42 |
| *** salv-orlando has joined #openstack-kolla | 13:43 | |
| ccesario | rhallisey, about kolla_internal_vip_address ... in this case are the address defined in kolla_internal_address in each config.... right!? | 13:44 |
| ccesario | --->>> ansible/group_vars/all.yml:kolla_internal_vip_address: "{{ kolla_internal_address }}" | 13:45 |
| *** vhosakot has joined #openstack-kolla | 13:45 | |
| *** daneyon has quit IRC | 13:45 | |
| *** daneyon has joined #openstack-kolla | 13:46 | |
| ccesario | rhallisey, right!!? | 13:47 |
| rhallisey | ya | 13:47 |
| vhosakot | morning! | 13:48 |
| *** huikang has joined #openstack-kolla | 13:49 | |
| ccesario | morning vhosakot ! | 13:49 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/kolla: Update Quickstart with example for distro and install type https://review.openstack.org/299945 | 13:50 |
| *** ayoung has joined #openstack-kolla | 13:50 | |
| ccesario | rhallisey, any clue!? | 13:51 |
| rhallisey | ccesario, are you setting kolla_internal_address ? | 13:51 |
| rhallisey | let me get my setup going | 13:51 |
| ccesario | rhallisey, ya | 13:52 |
| ccesario | http://paste.openstack.org/show/492572 | 13:52 |
| rhallisey | ah ok | 13:52 |
| ccesario | :D | 13:52 |
| *** salv-orlando has quit IRC | 13:52 | |
| *** diogogmt has quit IRC | 13:54 | |
| rhallisey | ccesario, just need to do some reading on this | 13:54 |
| ccesario | rhallisey, do not worry with a lot enable_{Service} .... only tests :P | 13:54 |
| rhallisey | gotcha | 13:54 |
| ccesario | rhallisey, tell me | 13:54 |
| *** banix has joined #openstack-kolla | 13:56 | |
| *** mgoddard_ has joined #openstack-kolla | 14:00 | |
| *** salv-orlando has joined #openstack-kolla | 14:02 | |
| *** achanda has joined #openstack-kolla | 14:02 | |
| *** mgoddard has quit IRC | 14:03 | |
| *** ccesario_ has quit IRC | 14:03 | |
| *** dougs1 has quit IRC | 14:04 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: lvm2/iscsi backend support for cinder https://review.openstack.org/299956 | 14:07 |
| *** achanda has quit IRC | 14:08 | |
| *** pmisiak has joined #openstack-kolla | 14:18 | |
| pmisiak | hi, hello | 14:18 |
| ccesario | rhallisey, tried again.... the same problem returned | 14:20 |
| rhallisey | ccesario, still reading about how to properly config this | 14:21 |
| rhallisey | you get a 301 error from haproxy | 14:21 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: lvm2/iscsi backend support for cinder https://review.openstack.org/299956 | 14:22 |
| dave-mccowan | are there still some issues with rabbit and hostnames? i'm getting the error "Hostname has to resolve to IP address of api_interface" with an all-in-one deploy. | 14:23 |
| ccesario | rhallisey, let me try check heka logs | 14:24 |
| vhosakot | dave-mccowan: yes, seen just with rax gate.... https://bugs.launchpad.net/kolla/+bug/1561563 | 14:25 |
| openstack | Launchpad bug 1561563 in kolla "rax gates fail with ip != hostname" [High,In progress] - Assigned to Sam Yaple (s8m) | 14:25 |
| ccesario | rhallisey, http://paste.openstack.org/show/492589/ | 14:25 |
| vhosakot | dave-mccowan: not able to reproduce it in my local env on Ubuntu... | 14:25 |
| vhosakot | dave-mccowan: looks like /etc/hosts gets bad often on rax gate.. | 14:25 |
| rhallisey | dave-mccowan, ping `hostname` | 14:26 |
| *** SiRiuS_ has joined #openstack-kolla | 14:26 | |
| huikang | ping: vhosakot | 14:26 |
| vhosakot | huikang: saw all your comments :) | 14:26 |
| huikang | great | 14:26 |
| vhosakot | huikang: I need to some to reply :) | 14:26 |
| rhallisey | dave-mccowan, add you hostname into /etc/hosts and it should work | 14:26 |
| huikang | vhosakot, no problem : ) | 14:27 |
| dave-mccowan | what's weird is the error message doesn't have my hostname address, it has my kolla_external_vip_address | 14:30 |
| vhosakot | rhallisey: don't we already add hostname into /etc/hosts in setup_gate.sh ? https://github.com/openstack/kolla/blob/master/tools/setup_gate.sh#L80-L89 for some reason, that code does not work on rax gate | 14:30 |
| dave-mccowan | i can reproduce it on my local setup. i'd hate to "fix" it, if there is a hard to reproduce bug that needs to be debugged. | 14:31 |
| rhallisey | vhosakot, I thought he was just doing a regular deployment | 14:32 |
| vhosakot | ah ok.. I thought dave-mccowan saw the error in gate... ok | 14:32 |
| dave-mccowan | all in one on centos7 | 14:33 |
| vhosakot | dave-mccowan: if you know the fix and busy to fix, you could send the steps please, and one of us can fix it and add you as co-author sure | 14:33 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Enable the kibana access from the external network https://review.openstack.org/299994 | 14:34 |
| *** Jeffrey4l_ has quit IRC | 14:35 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 14:35 | |
| dave-mccowan | i'm happy to debug and fix. what i'm not sure is what this check is supposed to do. it says my hostname doesn't match the address on api_interface, but the values it's giving are not my hostname and not the address on api_interface. | 14:35 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: lvm2/iscsi backend support for cinder (part 2) https://review.openstack.org/299995 | 14:35 |
| dave-mccowan | where does the value for ansible_hostname come from, that might be the root of my error. | 14:37 |
| rhallisey | dave-mccowan, what address does your hostname resolve to? | 14:38 |
| dave-mccowan | ah... the wrong fqdn name. looks like i messed up my linux config. | 14:40 |
| openstackgerrit | Merged openstack/kolla-mesos: Fix marathon framework autodetection https://review.openstack.org/297054 | 14:40 |
| ccesario | rhallisey, fresh haproxy logs http://paste.openstack.org/show/492595/ | 14:41 |
| openstackgerrit | Merged openstack/kolla: Fix gate to use world writeable docker socket https://review.openstack.org/298098 | 14:41 |
| openstackgerrit | Merged openstack/kolla: Fix gate by adding MariaDB-shared dependnecy https://review.openstack.org/299681 | 14:41 |
| dims_ | rhallisey : thanks! i got a deploy to succeed (http://paste.openstack.org/show/492594/) with one doc suggestion :) https://review.openstack.org/#/c/299945/ | 14:42 |
| patchbot | dims_: patch 299945 - kolla - Update Quickstart with example for distro and inst... | 14:42 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/kolla: Update Quickstart with example for distro and install type https://review.openstack.org/299945 | 14:42 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: lvm2/iscsi backend support for cinder (part 3) https://review.openstack.org/300004 | 14:42 |
| rhallisey | dims_, excellent! | 14:43 |
| rhallisey | ccesario, ok so connection refused.. | 14:47 |
| rhallisey | ccesario, what containers do you have up at this point? | 14:47 |
| rhallisey | in cloud 1 I think it is | 14:47 |
| rhallisey | whichever one is getting the error | 14:47 |
| *** wuhg has quit IRC | 14:49 | |
| ccesario | rhallisey, http://paste.openstack.org/show/492597/ | 14:51 |
| ccesario | on "cloud1" | 14:51 |
| *** sdake has joined #openstack-kolla | 14:57 | |
| huikang | morning, sdake | 14:57 |
| *** mgoddard_ has quit IRC | 14:57 | |
| sdake | morning huikang | 14:58 |
| sdake | hey is gmmaha around | 14:58 |
| *** mgoddard has joined #openstack-kolla | 14:58 | |
| huikang | sdake, is there a kolla design session in austin summit? | 14:58 |
| sdake | yes we have 14 sessions | 14:58 |
| huikang | sdake, sounds great | 14:59 |
| sdake | last time we had 6 | 14:59 |
| huikang | I can join this time | 14:59 |
| sdake | sweet | 15:02 |
| sdake | on tuesday is a complete cross-sesssion day | 15:02 |
| sdake | monday is the ops susmmit -i'd recommend attending that | 15:02 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: lvm2/iscsi backend support for cinder (part 3) https://review.openstack.org/300004 | 15:03 |
| huikang | is the schedule available online? | 15:03 |
| kproskurin | rhallisey: Hey, looks like you didnt create a patch\bug about nova chown. Something wrong? | 15:04 |
| rhallisey | kproskurin, not sure if that solved the issue | 15:05 |
| rhallisey | kproskurin, I also don't think we need chown -R | 15:06 |
| rhallisey | the perms are being set properly | 15:06 |
| rhallisey | user rather | 15:06 |
| kproskurin | rhallisey: What do you mean? I rebuilt nova-compute with chown -R /var/lib/nova and everything start to work for me. Right now ew have a situation what /var/lib/nova/instanses call is not working at all. But fixing it not solve all problemes, since there is a subdirts inside instances | 15:07 |
| rhallisey | I started the container by hand and saw everything was given the nova user | 15:08 |
| rhallisey | kproskurin, I guess -R won't really cause any harm here, but I'm not seeing the same issue | 15:10 |
| kproskurin | rhallisey: are you sure you checked subdirs? | 15:10 |
| rhallisey | kproskurin, ya I'm seeing everything as 'nova nova' | 15:11 |
| kproskurin | rhallisey: I cant imagene how it could be different in Kolla… its the same call. And sudo 100% wrong for instances | 15:11 |
| *** salv-orlando has quit IRC | 15:11 | |
| rhallisey | kproskurin, so this was just in kolla-mesos you saw this right? | 15:12 |
| kproskurin | rhallisey: yes… BUT it’s same container, built from kolla master. And it’s the same code, with the same calls, same sudo and everything | 15:12 |
| ccesario | rhallisey, I tried increase the timeout of task ... but no sucess too :( | 15:12 |
| rhallisey | ya it's odd | 15:13 |
| sdake | gmmaha ping | 15:13 |
| rhallisey | ccesario, still investigating the cause of 'connection refused' | 15:13 |
| rhallisey | ccesario, I'll get back to it in a minute | 15:14 |
| ccesario | rhallisey, http://paste.openstack.org/show/492601/ stopped in this point | 15:14 |
| ccesario | only to keep you notified :) | 15:14 |
| rhallisey | thanks :_ | 15:14 |
| rhallisey | :) | 15:14 |
| rhallisey | kproskurin, does kolla-mesos use all of kolla's containers? | 15:15 |
| rhallisey | wondering if there is some disconnect here | 15:15 |
| rhallisey | either way I'll add -R | 15:15 |
| rhallisey | but just curious | 15:15 |
| kproskurin | rhallisey: yes it does, we use most of the things from Kolla, Ill do some basic re-check right now about this issue... | 15:16 |
| rhallisey | kproskurin, is /var/lib/nova 'nova nova' user & group | 15:17 |
| *** pmisiak has quit IRC | 15:18 | |
| *** blahRus has joined #openstack-kolla | 15:22 | |
| *** ayoung has quit IRC | 15:24 | |
| *** ayoung has joined #openstack-kolla | 15:25 | |
| *** kproskurin has quit IRC | 15:35 | |
| *** kproskurin has joined #openstack-kolla | 15:39 | |
| *** tobe has joined #openstack-kolla | 15:43 | |
| dims_ | ccesario : rhallisey : like what i see here? http://paste.openstack.org/show/492605/ | 15:45 |
| dims_ | libvirtError: unable to connect to server at '10.0.0.3:16509': Connection refused | 15:45 |
| rhallisey | dims_, ccesario is doing something different. He is created 2 clouds and haproxy is giving connections refused on the second | 15:47 |
| sdake | dims is libvirtd running on the host? | 15:47 |
| rhallisey | dims_, what's the libvirt container status? | 15:48 |
| *** tobe has quit IRC | 15:48 | |
| dims_ | rhallisey : sdake : no. i did the "service libvirt-bin stop;update-rc.d libvirt-bin disable" per quickstart | 15:49 |
| dims_ | rhallisey : ah ok | 15:49 |
| sdake | dims_ docker ps -a | grep libvirt | 15:49 |
| dims_ | ah. its running but ran into trouble | 15:50 |
| dims_ | Running command: '/usr/sbin/libvirtd --listen' | 15:50 |
| dims_ | /usr/sbin/libvirtd: error while loading shared libraries: libvirt-admin.so.0: cannot open shared object file: Permission denied | 15:50 |
| sdake | dims_ if you type dmesg I'm sure you will see the problem - apparmor | 15:51 |
| rhallisey | this needs to be in the quickstart.. | 15:52 |
| rhallisey | how do you turn off apparmor | 15:52 |
| rhallisey | I'll add it in there | 15:52 |
| dims_ | sdake rhallisey : ah thanks | 15:53 |
| *** rmart04 has quit IRC | 15:56 | |
| *** mgoddard_ has joined #openstack-kolla | 15:59 | |
| *** sdake has quit IRC | 16:01 | |
| ccesario | rhallisey, http://paste.openstack.org/show/492615/ | 16:02 |
| ccesario | :D | 16:02 |
| *** mgoddard has quit IRC | 16:02 | |
| *** Jeffrey4l_ has quit IRC | 16:04 | |
| dims_ | rhallisey : got a tip for me to try? | 16:05 |
| *** Jeffrey4l_ has joined #openstack-kolla | 16:05 | |
| *** achanda has joined #openstack-kolla | 16:05 | |
| ccesario | rhallisey, bingo! discovered the problem!!! | 16:08 |
| *** salv-orlando has joined #openstack-kolla | 16:08 | |
| rhallisey | ccesario, what is it!? | 16:08 |
| rhallisey | dims_, did you disable it? | 16:09 |
| *** daneyon_ has joined #openstack-kolla | 16:09 | |
| rhallisey | sudo /etc/init.d/apparmor stop | 16:09 |
| rhallisey | sudo update-rc.d -f apparmor remove | 16:09 |
| *** achanda has quit IRC | 16:11 | |
| ccesario | rhallisey, http://paste.openstack.org/show/492615/ from the "kolla deployer" | 16:11 |
| dims_ | rhallisey : ack. after that should i just nuke nova-compute and libvirt containers and try deploy again? | 16:11 |
| rhallisey | dims_, ya | 16:11 |
| *** Jeffrey4l_ has quit IRC | 16:12 | |
| ccesario | the problem is in keepalive.... due the default template usage virtual_router_id with static value 51 | 16:12 |
| *** daneyon has quit IRC | 16:12 | |
| ccesario | I have changed the config of cloud1 to virtual_router_id 52 ... and redeploy | 16:13 |
| ccesario | the Ip address goes UP and the tasks run as expected | 16:13 |
| rhallisey | nice | 16:14 |
| *** unicell1 has joined #openstack-kolla | 16:15 | |
| ccesario | rhallisey, now, just know how to solve this in the code :P | 16:15 |
| *** unicell has quit IRC | 16:15 | |
| rhallisey | ccesario, I think it just needs to be configurable | 16:16 |
| dims_ | rhallisey : did not help | 16:16 |
| rhallisey | dims_, sudo apparmour status | 16:16 |
| rhallisey | just curious if that disable it | 16:16 |
| ccesario | rhallisey, kind of param on globals ? | 16:16 |
| rhallisey | ccesario, ya | 16:16 |
| rhallisey | ccesario, we also need to document what you did | 16:17 |
| rhallisey | ccesario, because people will be asking about it | 16:17 |
| dims_ | rhallisey : http://paste.openstack.org/show/492618/ | 16:17 |
| ccesario | rhallisey, until this point I think important tw things/(maybe features) | 16:18 |
| dims_ | am running it on my box (ubuntu-brix) not in the libvirt container to be specific | 16:18 |
| *** shardy has quit IRC | 16:19 | |
| rhallisey | dims_, let me check the irc logs.. someone got this working yesterday | 16:19 |
| ccesario | rhallisey, these points | 16:20 |
| ccesario | 1 - make kolla-genpwd optionaly write on specifc folder | 16:20 |
| ccesario | 2 - make virtual_router_id keepalive param configurable | 16:20 |
| rhallisey | ccesario, ok. I'll make a bp for N | 16:20 |
| *** jiriprox has quit IRC | 16:21 | |
| rhallisey | shouldn't be too hard | 16:21 |
| *** shardy has joined #openstack-kolla | 16:21 | |
| ccesario | until now I have mapped these two points | 16:22 |
| dims_ | rhallisey : thanks a ton you have been very helpful | 16:22 |
| ccesario | rhallisey, good...! if you could make a bp, would be nice! | 16:22 |
| rhallisey | dims_, no problem | 16:23 |
| rhallisey | ccesario, ya will do | 16:23 |
| *** ChanServ changes topic to "Kolla IRC meetngs on Wednesdays @ 16:30 UTC even weeks, 23:00 UTC odd weeks - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA)" | 16:24 | |
| *** mikelk has quit IRC | 16:24 | |
| *** sdake has joined #openstack-kolla | 16:24 | |
| *** inc0 has joined #openstack-kolla | 16:26 | |
| inc0 | howdy | 16:26 |
| ccesario | rhallisey, other point | 16:27 |
| rhallisey | dims_, goo.gl/2F4suI | 16:27 |
| rhallisey | http://goo.gl/2F4suI | 16:27 |
| ccesario | the param vrrp_instance needs to be diferent | 16:27 |
| ccesario | by default it is vrrp_instance kolla_internal_vip | 16:28 |
| *** sdake has quit IRC | 16:28 | |
| *** sdake has joined #openstack-kolla | 16:29 | |
| rhallisey | ccesario, I think the issue is having two vrrp_instances with the same router_id | 16:29 |
| *** sdake has quit IRC | 16:30 | |
| rhallisey | gmmaha, where did you disable apparmor to get libvirt working? On your host? | 16:31 |
| ccesario | an automation like this can be easly too.... vrrp_instance kolla_internal_vip_{{ router_id_var }} | 16:31 |
| *** mbound has joined #openstack-kolla | 16:31 | |
| *** daneyon has joined #openstack-kolla | 16:32 | |
| rhallisey | ccesario, ya I think it's an easy patch | 16:32 |
| rhallisey | brb a min lunch | 16:32 |
| *** daneyon_ has quit IRC | 16:33 | |
| ccesario | rhallisey, have a good lunch! | 16:33 |
| *** mbound has quit IRC | 16:33 | |
| rhallisey | thanks :) | 16:33 |
| *** daneyon_ has joined #openstack-kolla | 16:36 | |
| *** daneyon__ has joined #openstack-kolla | 16:38 | |
| *** daneyon has quit IRC | 16:38 | |
| *** daneyon_ has quit IRC | 16:41 | |
| *** daneyon has joined #openstack-kolla | 16:42 | |
| gmmaha | rhallisey: i just pulled it out.. apt-get remove -y apparmor | 16:42 |
| gmmaha | crude, yes i know! :( | 16:42 |
| rhallisey | dims_, ^^ see if that helps | 16:43 |
| *** daneyon has quit IRC | 16:43 | |
| *** daneyon__ has quit IRC | 16:43 | |
| *** daneyon has joined #openstack-kolla | 16:43 | |
| gmmaha | dims_: that and make sure you disable libvirtd on the host.. (update-rc.d libvirt-bin disable) | 16:43 |
| *** daneyon has quit IRC | 16:44 | |
| dims_ | gmmaha : y already did the libvirtd. will try removing apparmor completely | 16:45 |
| *** kproskurin has quit IRC | 16:47 | |
| *** sdake has joined #openstack-kolla | 16:48 | |
| *** vhosakot has quit IRC | 16:48 | |
| gmmaha | dims_: cool.. | 16:49 |
| gmmaha | sdake: hey | 16:49 |
| sdake | gmmaha ping re tet | 16:49 |
| sdake | hey bro | 16:49 |
| sdake | did you try my patch | 16:49 |
| gmmaha | sorry couldnt get up early | 16:49 |
| gmmaha | sdake: did you push a patch? I didnt see any yet.. | 16:49 |
| sdake | gmma moment i'll give you a link | 16:49 |
| gmmaha | sdake: thanks | 16:50 |
| sdake | https://review.openstack.org/#/c/299803/ | 16:50 |
| patchbot | sdake: patch 299803 - kolla - [WIP] Workaround ansible bug related to delegate_to | 16:50 |
| gmmaha | sdake: testing it now | 16:51 |
| sdake | sort of bsed aroudn vhosakot's idea at the last minute before i ptfoed | 16:51 |
| sdake | make sure to revert any other changes you have in your repo | 16:51 |
| sdake | but you know that I htink :) | 16:51 |
| gmmaha | sdake: aaaahhh | 16:51 |
| gmmaha | this might work | 16:51 |
| sdake | i am hoepful | 16:51 |
| gmmaha | sdake: for sure.. | 16:51 |
| sdake | i dont know enough about ansible variable calculation to know for certain | 16:51 |
| *** inc0 has quit IRC | 16:53 | |
| gmmaha | sdake: might work, given that we arent dynamically reading that value when that play is being run.. | 16:53 |
| *** mgoddard has joined #openstack-kolla | 17:01 | |
| *** diogogmt has joined #openstack-kolla | 17:01 | |
| *** mgoddard_ has quit IRC | 17:02 | |
| *** gfidente has quit IRC | 17:02 | |
| gmmaha | sdake: that didnt work :( | 17:06 |
| gmmaha | but run once failed completely | 17:06 |
| gmmaha | http://paste.openstack.org/show/492628/ | 17:06 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: lvm2/iscsi backend support for cinder (part 3) https://review.openstack.org/300004 | 17:07 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: lvm2/iscsi backend support for cinder https://review.openstack.org/291285 | 17:07 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: lvm2/iscsi backend support for cinder (part 2) https://review.openstack.org/299995 | 17:07 |
| *** mlima has joined #openstack-kolla | 17:08 | |
| gmmaha | sdake: sorry run once works.. :) | 17:08 |
| *** jmccarthy has quit IRC | 17:08 | |
| *** daneyon has joined #openstack-kolla | 17:09 | |
| sdake | gmmaha so did the patch work or not work? | 17:09 |
| gmmaha | sdake: no.. didnt work | 17:09 |
| gmmaha | it still tried runnign it on the compute node | 17:09 |
| sdake | sam error? | 17:09 |
| *** jmccarthy has joined #openstack-kolla | 17:09 | |
| sdake | sam/same | 17:09 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Add integration with Kolla infrastructure https://review.openstack.org/300004 | 17:10 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Adds iscsid and tgtd docker containers https://review.openstack.org/291285 | 17:10 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Adds ansible code for iscsid and tgtd containers https://review.openstack.org/299995 | 17:10 |
| *** mlima_ has quit IRC | 17:10 | |
| gmmaha | sdake: yeah | 17:11 |
| sdake | gmmaha damn i really thought that might work | 17:14 |
| sdake | gmmaha can you leave a note in the review its no beuno | 17:14 |
| gmmaha | sdake: for sure | 17:14 |
| *** mlima_ has joined #openstack-kolla | 17:14 | |
| gmmaha | i thought the same.. let me test something out | 17:14 |
| *** achanda has joined #openstack-kolla | 17:15 | |
| *** mlima has quit IRC | 17:17 | |
| *** vhosakot has joined #openstack-kolla | 17:18 | |
| *** mbound has joined #openstack-kolla | 17:19 | |
| manjeets | where all configs are saved ? | 17:21 |
| manjeets | i want to edit nova config for api_return result limit ? | 17:22 |
| vhosakot | manjeets: /etc/kolla/nova-compute/nova.conf | 17:23 |
| manjeets | i see that on control node | 17:23 |
| vhosakot | /etc/kolla/nova has nova confis | 17:23 |
| manjeets | I don't see this on host from where i deployed | 17:24 |
| sdake | from host you deploy ou need to override in /etc/kolla/config | 17:24 |
| sdake | manjeets this technique is currently undocumented | 17:24 |
| sdake | but basically create a file called nova-compute.ini | 17:25 |
| sdake | in /etc/kolla/config | 17:25 |
| sdake | add [iniheader] | 17:25 |
| sdake | key=value | 17:25 |
| *** stvnoyes has left #openstack-kolla | 17:25 | |
| sdake | and run reconfgiure | 17:25 |
| sdake | if reconfigure is still broken run redeploy and it will merge the configs in with the defaults | 17:25 |
| sdake | sorry file should be called nova-compute.conf | 17:26 |
| sdake | if you want it to apply to all of nova, call it nova.conf | 17:26 |
| sdake | I think | 17:26 |
| manjeets | there is no config dir in /etc/kolla | 17:27 |
| manjeets | i created one and copied nova.conf | 17:28 |
| manjeets | sdake should libvirt-bin be stopped before i run re configure | 17:30 |
| manjeets | ? | 17:30 |
| manjeets | or anything that need to be stopped | 17:30 |
| *** alyson_ has quit IRC | 17:32 | |
| *** inc0 has joined #openstack-kolla | 17:33 | |
| ccesario | sdake, do you have feedback about this ticket ? #299803 | 17:36 |
| ccesario | rhallisey, ping ? | 17:37 |
| rhallisey | ccesario, hey | 17:37 |
| manjeets | vhosakot if i changed in /etc/nova-compute/nova.conf and restarted nova_api container is that enough to reflect the changes | 17:38 |
| ccesario | rhallisey, http://sprunge.us/JFTR?diff | 17:38 |
| vhosakot | manjeets: can you follow this for reconfig - https://etherpad.openstack.org/p/kolla-mitaka-testing-reconfigure | 17:39 |
| ccesario | rhallisey, I just test it now..... working as expected ... only changing the keepalived_virtual_router_id: "100" in second "cloud" | 17:39 |
| *** dims_ has quit IRC | 17:39 | |
| rhallisey | ccesario, looks good | 17:40 |
| *** dims has joined #openstack-kolla | 17:40 | |
| ccesario | rhallisey, ;) | 17:42 |
| manjeets | vhosakot i need to add only the option i want to chane ? | 17:43 |
| vhosakot | yes.. along with the section name | 17:43 |
| manjeets | okay cool thansk | 17:44 |
| ccesario | rhallisey, when you create a bp or bug ... ping me that I submit it | 17:44 |
| rhallisey | k will do | 17:44 |
| ccesario | rhallisey, do you think that need more test abou it ? | 17:44 |
| *** shardy has quit IRC | 17:45 | |
| rhallisey | that patch shouldn't disrupt anything | 17:45 |
| rhallisey | since it's beingdefaulted | 17:45 |
| ccesario | rhallisey, yes.... I think too... in my tests nothing it was broken :P | 17:47 |
| ccesario | expect if I did not change the param, the second deploy stops (as previously I told you) | 17:47 |
| ccesario | except* | 17:47 |
| ccesario | 1 - make kolla-genpwd optionaly write on specifc folder | 17:49 |
| ccesario | 2 - make virtual_router_id keepalive param configurable | 17:49 |
| ccesario | 3 - vrrp_instance kolla_internal_vip_{{ router_id_var }} | 17:49 |
| ccesario | 2 and 3 ... done | 17:49 |
| *** unicell has joined #openstack-kolla | 17:54 | |
| mlima_ | guys, have i can two identical containers ( two cinder_volume container ), but each container with your backend? | 17:54 |
| sdake | not wit hthe same name | 17:54 |
| mlima_ | hmm | 17:55 |
| *** dwalsh has quit IRC | 17:55 | |
| sdake | name it cinder_voolume_bakcendname | 17:55 |
| inc0 | not sure how it will work tho | 17:55 |
| inc0 | not a normal use case | 17:55 |
| sdake | yup a new one | 17:55 |
| mlima_ | it need some configurations | 17:55 |
| inc0 | on that note, how do we look like on iscsi+lvm front? | 17:55 |
| *** unicell1 has quit IRC | 17:56 | |
| sdake | ccesario can you link the bug plz you are talking about | 17:56 |
| sdake | manjeets libvirt-bin should not be running at aall! | 17:56 |
| sdake | manjeets you have to mkdir /etc/kolla/config | 17:56 |
| manjeets | sdake http://paste.openstack.org/show/492634/ vhosakot got this issue while reconfigure | 17:56 |
| sdake | apologie sfor lag was in security meeting | 17:56 |
| manjeets | i guess my libvirt-bin is running | 17:56 |
| sdake | manjeets thats a problem - with gmmaha we removed libvirt-bin | 17:57 |
| manjeets | I had that problem yesterday kolla cleanups do not force stop libvirt bin | 17:57 |
| manjeets | i froze yesterday i did reboot and still its was giving me issues | 17:58 |
| manjeets | it** | 17:58 |
| manjeets | then i manually stopped the container and killed the libvirt processes in every single node and redeploy worked without reboot | 17:58 |
| inc0 | yeah libvirt needs to die | 17:59 |
| inc0 | on host | 17:59 |
| gmmaha | sdake: i have a theory | 17:59 |
| gmmaha | about the bug.. | 17:59 |
| gmmaha | vhosakot: ^^ | 17:59 |
| gmmaha | and i think re-ordering wont be a bad thing.. | 17:59 |
| gmmaha | want me to put it down in an email and send it your way? | 17:59 |
| manjeets | libvirt did not die even after reboot | 17:59 |
| inc0 | gmmaha, what's your idea? | 17:59 |
| sdake | gmmaha use the ebug tracker | 18:00 |
| manjeets | not sure if it should or not | 18:00 |
| inc0 | also, do you have an env on which this is constantly failing? | 18:00 |
| inc0 | mind if I do some debugging on my own | 18:00 |
| inc0 | ? | 18:00 |
| ccesario | sdake, https://review.openstack.org/#/c/299803/ | 18:00 |
| patchbot | ccesario: patch 299803 - kolla - [WIP] Workaround ansible bug related to delegate_to | 18:00 |
| gmmaha | inc0: are you talking about the neutron play failing? | 18:00 |
| gmmaha | sdake: sure.. | 18:00 |
| inc0 | that's the one gmmaha | 18:00 |
| sdake | inc0 we had a 6 hour debug session on google groups last night with 6 people | 18:00 |
| sdake | it is definately an ansible bug | 18:00 |
| inc0 | sure, which one? | 18:00 |
| inc0 | anyone know what is exactly happening? | 18:00 |
| sdake | yup | 18:00 |
| gmmaha | inc0: sure.. | 18:00 |
| sdake | its running bootstrap on compute | 18:00 |
| inc0 | do share | 18:01 |
| inc0 | but why? | 18:01 |
| gmmaha | inc0: let me put it down | 18:01 |
| gmmaha | on the bug | 18:01 |
| sdake | and running the check of the bootstrap results on some other node | 18:01 |
| inc0 | yeah | 18:01 |
| * gmmaha goes to find the bug | 18:01 | |
| inc0 | it's about delegate_to right? | 18:01 |
| gmmaha | inc0: a little.. i think its clear delegate_to is broken in this version | 18:02 |
| gmmaha | and there are bugs around that | 18:02 |
| gmmaha | its about how we can fix it | 18:02 |
| *** rajathagasthya has joined #openstack-kolla | 18:02 | |
| manjeets | sdake vhosakot i stoped nova_libvirt and then did refconfigure it failed | 18:03 |
| sdake | ya reconfigure may be busted atm | 18:04 |
| sdake | inc0 look t my review where i made an attempt at a fix | 18:05 |
| inc0 | I -1 it, it's not how we should do it | 18:05 |
| inc0 | so my guess is | 18:05 |
| sdake | oh i see you already did | 18:05 |
| rhallisey | ccesario, https://blueprints.launchpad.net/kolla/+spec/multiple-clouds | 18:06 |
| inc0 | https://github.com/openstack/kolla/blob/master/ansible/site.yml#L107 | 18:06 |
| rhallisey | ccesario, going to also add a bug | 18:06 |
| inc0 | ansible probably builds a list of servers | 18:06 |
| inc0 | based on order we used in site.yml | 18:06 |
| sdake | inc0 Ihadn't thought of that! | 18:06 |
| inc0 | and takes first one from this | 18:07 |
| inc0 | we can try to change order in our very own site.yml and it should help in most cases | 18:07 |
| sdake | gmmaha revert my chnges and enable manila and repdeloy please | 18:07 |
| inc0 | that too | 18:07 |
| sdake | i am not keen on order change | 18:07 |
| inc0 | why? what does it change really? | 18:07 |
| sdake | lets see if i ts its the enable manila thing | 18:07 |
| sdake | inc0 my concern is delegate-to is broken in genernal | 18:08 |
| sdake | and we use it in all our bootstrap scriips | 18:08 |
| sdake | so I'd rather fix it everywhere then where it happens to pop up randomly | 18:08 |
| inc0 | sdake, move to ansible 2.0 is on top of my priorities for N | 18:08 |
| sdake | inc0 that isn't an ansswer for mitaka | 18:08 |
| inc0 | so see if you can enable manila and it helps | 18:08 |
| rhallisey | ccesario, https://bugs.launchpad.net/kolla/+bug/1564547 | 18:09 |
| openstack | Launchpad bug 1564547 in kolla "The virtual_router_id needs to be configurable" [Medium,New] | 18:09 |
| sdake | gmmaha inc0's theory is that enable_manila bool is breaking the playbook | 18:09 |
| sdake | because it always evaluates to false | 18:09 |
| gmmaha | sdake: inc0: vhosakot: https://bugs.launchpad.net/kolla/+bug/1546789/comments/11 | 18:09 |
| openstack | Launchpad bug 1546789 in kolla "neutron failed to deploy in a multi-node deployment" [Critical,In progress] - Assigned to Steven Dake (sdake) | 18:09 |
| gmmaha | sdake: aaah i can try it out | 18:09 |
| gmmaha | let me see | 18:09 |
| sdake | "where did you get that theory from? A unicorn's ass? He isn't some zulu warrior." :) | 18:10 |
| vhosakot | hahahahahahahahahahahahahahahahahahahaha | 18:10 |
| ntpttr | gmmaha: I don't think that it's delegate_to that's broken, but just the group lookup itself | 18:10 |
| *** jasonsb has joined #openstack-kolla | 18:10 | |
| gmmaha | sdake: are you talking about me? | 18:11 |
| gmmaha | :) | 18:11 |
| ntpttr | gmmaha: since a bunch of different options that didn't use delegate_to had the same result | 18:11 |
| vhosakot | yes, let us move to Ansible 2.0.... it has better error handling I read | 18:11 |
| gmmaha | ntpttr: the other ones the first that runs is the right host.. check nova | 18:11 |
| inc0 | ntpttr, that's my take as well | 18:11 |
| dmsimard | vhosakot: you have the blocks feature for error handling | 18:11 |
| dmsimard | but it's still fairly limited | 18:12 |
| inc0 | also try adding this to play | 18:12 |
| ntpttr | gmmaha: I know, it's a bit of a random bug. Don't know why it only happens some of the time | 18:12 |
| vhosakot | dmsimard: what is blocks feature ? | 18:12 |
| dmsimard | vhosakot: http://docs.ansible.com/ansible/playbooks_blocks.html | 18:12 |
| inc0 | http://paste.openstack.org/show/492640/ ntpttr gmmaha | 18:13 |
| ntpttr | inc0: I believe we did that, it's getting the compute host | 18:13 |
| gmmaha | inc0: that was there.. | 18:13 |
| dmsimard | vhosakot: I'm personally keeping an eye on two feature requests that'll greatly improve things: https://github.com/ansible/ansible/issues/13587 and https://github.com/ansible/ansible/issues/12341 | 18:13 |
| *** Serlex has left #openstack-kolla | 18:13 | |
| ntpttr | inc0: the group lookup in just that one instance in the whole ansible run is incurring this bug, for who knows what reason >.< | 18:13 |
| *** jasonsb has quit IRC | 18:13 | |
| vhosakot | so, it blocks similar to tags ? group tasks together ? | 18:14 |
| dmsimard | vhosakot: no, blocks are like "try except finally" | 18:14 |
| dmsimard | vhosakot: kind of | 18:14 |
| vhosakot | ah ok.. wow | 18:14 |
| inc0 | ntpttr, remove [0] and let's see what it thinks "neutron-server" is | 18:14 |
| inc0 | what hosts are there | 18:14 |
| dmsimard | except instead of "try except finally", it's "block rescue always" :) | 18:14 |
| vhosakot | ah | 18:15 |
| ntpttr | inc0: we checked that too, and it's the compute group :) | 18:15 |
| gmmaha | inc0: we ran through all these last night.. :) | 18:15 |
| gmmaha | hence the completely bonkers theory of mine | 18:15 |
| inc0 | ok...and ordering change changes?:) | 18:15 |
| dmsimard | vhosakot: it's pretty nice, but you can't wrap anything you want in it right now | 18:15 |
| gmmaha | but atleast the logs seem to be consistent to what i am seeing | 18:15 |
| ntpttr | gmmaha: what's your theory then? | 18:15 |
| ntpttr | inc0: yeah putting the neutron groups at the top of the hosts list makes it resolve correctly for that play | 18:16 |
| vhosakot | yep, updating kolla playbooks with new Ansible syntax and testing them is a huge task also | 18:16 |
| *** diogogmt_ has joined #openstack-kolla | 18:16 | |
| *** diogogmt has quit IRC | 18:17 | |
| *** diogogmt_ is now known as diogogmt | 18:17 | |
| inc0 | gmmaha your theory makes sense | 18:18 |
| inc0 | gmmaha, try to remove run_once and see if it helps | 18:18 |
| gmmaha | inc0: the problem with that is that it will try to create the db 3 times.. :( | 18:18 |
| inc0 | no matter | 18:19 |
| inc0 | will validate theory | 18:19 |
| inc0 | and then we can think how to resolve it | 18:19 |
| gmmaha | inc0: yeah that worked yesterday.. | 18:19 |
| gmmaha | when we remove the run_once | 18:19 |
| *** dwalsh has joined #openstack-kolla | 18:20 | |
| * manjeets how nova_libvirt automatically restarts | 18:21 | |
| manjeets | i killed 2 3 times and did reboor | 18:21 |
| vhosakot | all DB creation plays have run_once:True... dont think that should be removed and causing any issue... | 18:21 |
| gmmaha | sdake: inc0: with enable_manila it still doesnt work | 18:22 |
| inc0 | that would confirm your theory | 18:22 |
| inc0 | sooo | 18:22 |
| inc0 | hmm | 18:22 |
| gmmaha | inc0: and if i remvoe run_once it will worjk | 18:22 |
| inc0 | getting neutron-servers to top of the list will work | 18:22 |
| gmmaha | inc0: right | 18:23 |
| inc0 | and find all other places like that | 18:23 |
| inc0 | we need to make it work for Mitaka | 18:23 |
| gmmaha | and I & ntpttr think its a less risky change | 18:23 |
| vhosakot | gmmaha: you'll need to first do https://github.com/openstack/kolla/blob/master/doc/manila-guide.rst before enabling manila | 18:23 |
| inc0 | and we have only so many places that match this criterion | 18:23 |
| sdake | reading backlog | 18:23 |
| inc0 | and focus on move to Ansible 2.0 in N | 18:23 |
| inc0 | vhosakot, that was just to check if neutron fails | 18:23 |
| sdake | match which criteria inc0 | 18:24 |
| vhosakot | inc0: ah ok | 18:24 |
| sdake | inc0 that doesn't confirm the theory | 18:24 |
| inc0 | has run_once: True and delegate_to | 18:24 |
| inc0 | in same task | 18:24 |
| *** dims_ has joined #openstack-kolla | 18:24 | |
| *** dims has quit IRC | 18:24 | |
| sdake | oh the runonce delgate_to theory | 18:24 |
| sdake | not the boolean theory | 18:24 |
| sdake | every bootstrap has runonce and delegate to | 18:25 |
| sdake | how do you propose to move them all to the top ;) | 18:25 |
| vhosakot | sdake: when does kolla move to Ansible 2.0 ? | 18:25 |
| sdake | gmmaha the unicorns ass thing is a quote from a riddick movie :) | 18:26 |
| *** salv-orlando has quit IRC | 18:26 | |
| sdake | it wasnt dircted at anyone i just thought it was funny :) | 18:26 |
| vhosakot | sdake: when does kolla move to Ansible 2.0 ? | 18:26 |
| gmmaha | sdake: the move to top is only needed for neutron | 18:26 |
| gmmaha | all the other services have the right server names at the top | 18:26 |
| gmmaha | nova, keystone, heat, murano etc | 18:27 |
| gmmaha | its just neutron where compute is the top of the host list | 18:27 |
| ntpttr | gmmaha: +1, this would actually make the site.yml more consistent | 18:27 |
| gmmaha | vhosakot: thanks for the link.. fixed the config, bulding the containers now and will test deploy | 18:28 |
| gmmaha | inc0: give me afew | 18:28 |
| sdake | gmmaha ok here is my problem and i wnat you to repeat it in your own words if you dont mind :) | 18:28 |
| gmmaha | sdake: :) sure.. | 18:29 |
| sdake | we don't know why delgate and run to when used in conjunction cause incorrect lookup of the inventory file | 18:29 |
| sdake | so if someone does something different then you have done, kolla might break in a different way because of the same bug | 18:29 |
| sdake | my proposal is to work around them by finding something that works and using that pattern instead | 18:30 |
| sdake | i reallly dont understand why my change didn't solve the problem | 18:31 |
| sdake | ntpttr did you look at my suggested change based upo nvhosakots idea? | 18:31 |
| gmmaha | sdake: let me phrase what i understood.. | 18:31 |
| vhosakot | which idea.. we tried many things last night | 18:31 |
| sdake | vhosakot i tried something new this morning | 18:31 |
| sdake | check rview queue | 18:32 |
| vhosakot | ah ok... will check | 18:32 |
| *** Marga_ has quit IRC | 18:32 | |
| gmmaha | delegate_to and run_once together isnt working as expected, instead of relying on a funky theory of the order of hosts will make a difference without understanding the full implications of it, you would rather have a change that is more reliable and quantifiable. | 18:33 |
| gmmaha | sdake: is that a fair understanding | 18:33 |
| sdake | gmmaha bingo | 18:33 |
| ntpttr | sdake: the patch you put up for review you mean? | 18:34 |
| sdake | if we had a root cause | 18:34 |
| sdake | ntpttr yes | 18:34 |
| sdake | rather then "shits broke" | 18:34 |
| ntpttr | sdake: I thought that you were getting the same issue with that | 18:34 |
| sdake | i could go with a reorder | 18:34 |
| sdake | ntpttr yes but i dont understand why | 18:34 |
| ntpttr | sdake: I think if we could contact some ansible experts, maybe some people who fixed this issue, we could get a clearer understanding of how to fix it | 18:35 |
| sdake | ntpttr which makes me think something else is wrong in our playbooks | 18:35 |
| vhosakot | sdake: ntpttr gmmaha: I like sdake's fix to ask neutron_bootstrap_server from the operator... was this fix tested ? | 18:35 |
| sdake | my wife said most ansibl bugs are related to incorrect " and { usage in 1.9 | 18:35 |
| ntpttr | vhosakot: yeah it was tested, but it didn't work | 18:35 |
| sdake | vhosakot that would clearly work | 18:35 |
| sdake | how about this | 18:35 |
| vhosakot | did not work... why ? same error.. ah | 18:35 |
| sdake | we define a new group | 18:35 |
| sdake | called [bootstrap] | 18:35 |
| sdake | vhosakot same error | 18:36 |
| vhosakot | ah.. | 18:36 |
| sdake | then bootstrap code always runs on the bootstrap node | 18:37 |
| ntpttr | sdake: that would be cool, worth trying | 18:37 |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Make keepalived virtual_router_id param configurable https://review.openstack.org/300143 | 18:37 |
| sdake | gmmaha can you try that out? | 18:37 |
| ntpttr | sdake: so a new role entirely? | 18:37 |
| gmmaha | sdake: sure.. | 18:38 |
| vhosakot | i dont think new role... a new group | 18:38 |
| dims_ | sdake : rhallisey : removed apparmour completely, rebooted, then kolla-build, kolla-ansible deploy and things are better with libvirt and nova-compute containers | 18:38 |
| sdake | i think our current working theory is the fact that network and controller have the same nodes confuses ansible | 18:38 |
| rhallisey | dims_, nice | 18:38 |
| dims_ | sdake : rhallisey : horizon seems to have tripped now http://paste.openstack.org/show/492644/ | 18:38 |
| dims_ | :) | 18:38 |
| sdake | ya horion is busted in my env too | 18:39 |
| sdake | dims are you running apache on your host? | 18:39 |
| vhosakot | it was fix it master... stale apache pid file.. 1 sec | 18:39 |
| dims_ | sdake : nope, i stopped apache on my host | 18:40 |
| sdake | vhosakot read dims log | 18:40 |
| sdake | i haven't seeen this one before | 18:40 |
| dims_ | sdake : quick start without haproxy | 18:41 |
| *** dave-mccowan has quit IRC | 18:41 | |
| vhosakot | no listening sockets available | 18:41 |
| sdake | the address is already in use | 18:41 |
| dims_ | this is all i have in my globals.yml http://paste.openstack.org/show/492648/ | 18:41 |
| inc0 | sdake, adding bootstrap group | 18:42 |
| sdake | Address already in use: AH00072 | 18:42 |
| vhosakot | https://bugs.launchpad.net/kolla/+bug/1557036 | 18:42 |
| openstack | Launchpad bug 1557036 in kolla "httpd (pid 1) already running when starting keystone or horizon container" [Critical,Fix released] - Assigned to Sam Yaple (s8m) | 18:42 |
| inc0 | would require us to either use some ansible magics to fill it out | 18:42 |
| inc0 | ot make it users job which isn't too good either | 18:42 |
| vhosakot | search for "Set the 'ServerName' directive globally to" in that bug | 18:42 |
| sdake | ntpttr not a new role a new group | 18:42 |
| vhosakot | ah, the bug does not have socket message... ok | 18:43 |
| sdake | i dont care how this neutron bug is fixed, i just dont want it to replicate to other parts of our code base where we reuse this same pattern | 18:45 |
| sdake | neutron-servers[0] is resolving to compute[0] | 18:45 |
| gmmaha | inc0: vhosakot: neutron issue doesnt get fixed iwth the manila enabled | 18:46 |
| vhosakot | I dont understand where and how manila came from :) | 18:46 |
| gmmaha | sdake: vhosakot: you still want me to test the bootstrap group? | 18:46 |
| gmmaha | and guessing you would like a nova_bootstrap, neutron_bootstrap etc? | 18:46 |
| sdake | gmmaha lets try that | 18:46 |
| vhosakot | I'm thinking about the new [bootstrap] group... code must be updated to use this new group also | 18:46 |
| gmmaha | vhosakot: that was an inc0 suggestion | 18:47 |
| vhosakot | how will it be _used_ in neutron bootstrap ? | 18:47 |
| sdake | delegates_to: bootstrap | 18:47 |
| sdake | bootstrap may onl yhve 1 entry | 18:48 |
| sdake | possible anible magic to populate it with controller[0] | 18:48 |
| sdake | i dont know how to do that last bit | 18:48 |
| ntpttr | sdake: +1 sounds good. Just add the bootstrap group in the inventory and maybe make it a child of control by default | 18:49 |
| ntpttr | then you can delegate to bootstrap[0] and it should work | 18:49 |
| sdake | its an ugly ugly hack but we are out off other options | 18:50 |
| gmmaha | sdake: let me see how it works | 18:50 |
| sdake | why does the ubuntu source deploy gate always fail | 18:51 |
| * sdake groans | 18:51 | |
| sdake | http://logs.openstack.org/91/299791/1/check/gate-kolla-dsvm-deploy-ubuntu-source/3cf5406/console.html#_2016-03-31_07_55_47_163 | 18:53 |
| vhosakot | sdake: becasue of http://logs.openstack.org/91/299791/1/check/gate-kolla-dsvm-deploy-ubuntu-source/3cf5406/console.html#_2016-03-31_07_55_26_985 | 18:53 |
| sdake | vhosakot is there a bug for that | 18:54 |
| huikang | sdake, I remember there is a PS for that | 18:54 |
| huikang | resolving the hostname for rabbitmq | 18:54 |
| sdake | huikang i think this is a separate issue | 18:55 |
| huikang | sdake, I see | 18:55 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla: Fix IP resolve check for RabbitMQ on AIO https://review.openstack.org/300152 | 18:55 |
| sdake | vhosakot if you llook at y link the failure is different | 18:55 |
| sdake | folks if you know how to fix these gat ebugs make it a prioirty to fix them | 18:57 |
| sdake | they slow down the review process significantly | 18:57 |
| sdake | i've got a 30 min meeting - bbiaf :) | 18:57 |
| vhosakot | let me check the queue for the PS proposed to fix it | 18:57 |
| sdake | some people may need to take over sam's reviews | 18:58 |
| sdake | he was working on alot of the ubuntu gating problems | 18:58 |
| *** openstackgerrit has quit IRC | 19:03 | |
| *** openstackgerrit has joined #openstack-kolla | 19:04 | |
| *** dave-mccowan has joined #openstack-kolla | 19:04 | |
| *** rajathagasthya has quit IRC | 19:05 | |
| *** Marga_ has joined #openstack-kolla | 19:09 | |
| *** alyson_ has joined #openstack-kolla | 19:12 | |
| *** dwalsh has quit IRC | 19:14 | |
| elmiko | sdake: posted the a message about threat analysis, but forget to add a [kolla] tag, sorry :/ | 19:20 |
| elmiko | fyi, http://lists.openstack.org/pipermail/openstack-dev/2016-March/091075.html | 19:21 |
| *** rajathagasthya has joined #openstack-kolla | 19:26 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Nova-compute should chown -R /var/lib/nova https://review.openstack.org/300180 | 19:29 |
| *** dwalsh has joined #openstack-kolla | 19:30 | |
| SiRiuS_ | nihilifer, ping | 19:30 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Add remove apparmor reminder to the quickstart https://review.openstack.org/300184 | 19:37 |
| *** mgoddard has quit IRC | 19:41 | |
| *** sdake_ has joined #openstack-kolla | 19:43 | |
| nihilifer | SiRiuS_: pong | 19:43 |
| sdake_ | hey folks | 19:44 |
| *** sdake has quit IRC | 19:44 | |
| sdake_ | I know this is an extremely bad time | 19:44 |
| sdake_ | but I have an emergency that I need to deal with thta will consume the next 24 hours of my life that i wont get back | 19:44 |
| SiRiuS_ | nihilifer, I tried to deploy kolla-mesos all-in-one, and now it deploys the apps | 19:44 |
| SiRiuS_ | nihilifer, but I get an error | 19:44 |
| sdake_ | i am going to drop off irc during that period | 19:45 |
| sdake_ | if something pressing needs attentin please use email | 19:45 |
| *** dwalsh has quit IRC | 19:45 | |
| sdake_ | please work together as we always do to get the job done on the remaining bugs where possible ;) | 19:45 |
| SiRiuS_ | nihilifer, http://paste.openstack.org/show/492661/ | 19:45 |
| SiRiuS_ | that is for mariadb | 19:45 |
| SiRiuS_ | I tried memchached as well, but I get the same error | 19:46 |
| nihilifer | SiRiuS_: you're doing bare metal deployment or vagrant? | 19:46 |
| SiRiuS_ | vagran | 19:46 |
| SiRiuS_ | the mesos cluster is deployed | 19:46 |
| SiRiuS_ | but the apps don't deploy on multi-node at all | 19:47 |
| SiRiuS_ | and an all-in-one i get the error above | 19:47 |
| SiRiuS_ | but at least they deploy | 19:47 |
| mlima_ | are gates ok? | 19:47 |
| SiRiuS_ | nihilifer, that log is from docker logs | 19:49 |
| nihilifer | SiRiuS_: what are your interfaces and ip addresses? | 19:49 |
| nihilifer | SiRiuS_: i mean, the output of "ip a" | 19:50 |
| SiRiuS_ | nihilifer, http://paste.fedoraproject.org/347983/14594538 | 19:51 |
| nihilifer | eth2 didn't receive ip address from bridge, that may be a problem | 19:51 |
| nihilifer | you're using libvirt or virtualbox? | 19:52 |
| SiRiuS_ | nihilifer, virtualbox, but isn't eth2 used for the neutron external network ? | 19:52 |
| SiRiuS_ | nihilifer, here is my cfgs | 19:53 |
| SiRiuS_ | http://paste.fedoraproject.org/347985/59453903/ | 19:53 |
| g3ek | hello, during build I have this : INFO:kolla.cmd.build:base:Step 3 : LABEL kolla_version "0.0.1" instead of INFO:kolla.cmd.build:base:Step 3 : LABEL kolla_version "2.0.0" | 19:53 |
| SiRiuS_ | and http://paste.fedoraproject.org/347987/45391014/ | 19:53 |
| g3ek | Do someone can help me to put it back to 2.0.0 | 19:53 |
| g3ek | ? | 19:53 |
| *** sdake_ has quit IRC | 19:56 | |
| nihilifer | SiRiuS_: yes, it is. but the start script inside containers wants to be aware both of private and public interface | 19:56 |
| SiRiuS_ | nihilifer, I could give it an address, but let me try something real quick | 19:57 |
| nihilifer | please try with assigning an addres, that should help | 19:57 |
| nihilifer | in general, the start script is fetching variables which are used for generating configuration files | 19:58 |
| nihilifer | so to generate i.e. neutron configs well, it has to be aware of all ifaces | 19:58 |
| gmmaha | vhosakot: the suggestion from sdake works.. | 20:06 |
| gmmaha | ntpttr: inc0: ^^ | 20:06 |
| gmmaha | shall i go ahead and update his patch? | 20:07 |
| gmmaha | nevermind | 20:07 |
| gmmaha | it was nova.. | 20:07 |
| gmmaha | bah! | 20:07 |
| *** salv-orlando has joined #openstack-kolla | 20:08 | |
| sbezverk | folks, I need a bit help from git expert who worked with patch stream, anybody out there? ;-) | 20:13 |
| manjeets | sdake kolla need warmup and test fires to get ready for bigger shot is a sure thing | 20:14 |
| *** inc0 has quit IRC | 20:20 | |
| vhosakot | sbezverk: what is patch stream ? | 20:21 |
| g3ek | hello, I found where the problem is, but I dont know where the default var is set : /usr/local/share/kolla/docker/base/Dockerfile.j2 : {{ kolla_version }} seems to be set to 0.0.1 instead of 2.0.0. How can I change it ? | 20:23 |
| sbezverk | vhosakot I have three patches sitting in the same kolla folder | 20:26 |
| vhosakot | three different branches ? | 20:26 |
| sbezverk | vhosakot nope I split iscsi patch into three, sdake walked me through the process.. but I forgot to ask him how to commit changes if there is change only in one of those patches.. | 20:27 |
| gmmaha | sbezverk: which patch has the change? lets say patch 2 has the change, this is what i would do | 20:28 |
| patchbot | gmmaha: https://review.openstack.org/#/c/2/ | 20:28 |
| gmmaha | cherry-pick 1st patch, take the new updated 2nd patch, commit it, cherry-pick 3rd patch and git review | 20:29 |
| gmmaha | the 1st patch should go without change and the other two patches will get updated | 20:29 |
| vhosakot | sbezverk: git branch <name of the branch that has the change>, git add *, git commit, git review.... is this what you're looking for ? | 20:30 |
| sbezverk | vhosakot but if the file I changed is already exist, I still need to git add {file}?? | 20:31 |
| vhosakot | yes... to add you changes to an already existing file in the repo | 20:31 |
| sbezverk | vhosakot thanks, let me try it | 20:32 |
| vhosakot | sure.. np | 20:32 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Add integration with Kolla infrastructure https://review.openstack.org/300004 | 20:34 |
| sbezverk | vhosakot, it looks like it worked :-) | 20:35 |
| vhosakot | sbezverk: ah ok.. cool.. great! :) will review when I get a chance! | 20:36 |
| sbezverk | vhosakot, please do and do not forget now there are three parts! | 20:36 |
| vhosakot | yep.. saw that :) | 20:36 |
| *** unicell has quit IRC | 20:42 | |
| *** ayoung has quit IRC | 20:43 | |
| SiRiuS_ | nihilifer, cool it works now (took some time because of buggy vbox networking), eth2 needed an IP address | 20:44 |
| SiRiuS_ | nihilifer, but here is something interesting | 20:44 |
| SiRiuS_ | nihilifer, in the globals.yml I should have multimode set to "no" | 20:45 |
| gmmaha | vhosakot: the suggestion from sdake didnt work | 20:46 |
| gmmaha | delegate_to is really busted in ansible 1.9.X | 20:47 |
| gmmaha | let me try one more thing | 20:47 |
| SiRiuS_ | nihilifer, because I'm deploying all-in-one | 20:50 |
| SiRiuS_ | nihilifer, but if I do that, it does not deploy the app | 20:50 |
| SiRiuS_ | nihilifer, the app remains in "waiting" state in marathon | 20:51 |
| ccesario | rhallisey, ping!? | 20:51 |
| *** dave-mccowan has quit IRC | 20:51 | |
| rhallisey | cbaesema, hey | 20:51 |
| rhallisey | ccesario, hey | 20:51 |
| SiRiuS_ | nihilifer, it's weird because the only way I can deploy an app (like mariadb) is if a deploy all-in-one but with the multinode set to "yes" | 20:52 |
| ccesario | rhallisey, I need an opinion .... https://review.openstack.org/#/c/300143/1/ansible/roles/haproxy/templates/keepalived.conf.j2@32 needs a diferent Id too.... when external address is set | 20:53 |
| patchbot | ccesario: patch 300143 - kolla - Make keepalived virtual_router_id param configurable | 20:53 |
| ccesario | rhallisey, what is the best way to do it ? create other variable too ? | 20:53 |
| ccesario | keepalived_virtual_router_id_internal and keepalived_virtual_router_id_external ? | 20:54 |
| rhallisey | ccesario, I'll comment | 20:55 |
| rhallisey | ya | 20:55 |
| rhallisey | ^ that would work | 20:55 |
| ccesario | rhallisey, thanks | 20:56 |
| rhallisey | no prob | 20:56 |
| dims_ | rhallisey : gmmaha : hurray, got a live one..deployed a cirros vm | 20:58 |
| rhallisey | dims_, nice! | 20:58 |
| dims_ | just have to doc the apparmor thing | 20:58 |
| g3ek | where can I found the code of this : version.cached_version_string() | 20:58 |
| dims_ | g3ek : http://codesearch.openstack.org/?q=def%20cached_version_string&i=nope&files=&repos= | 20:59 |
| g3ek | dims_: thanks, I dont know why the content of my cached version is 0.0.1 instead of 2.0.0. Someone know where can I change it? | 21:01 |
| rhallisey | dims_, https://review.openstack.org/#/c/300184/ | 21:02 |
| patchbot | rhallisey: patch 300184 - kolla - Add remove apparmor reminder to the quickstart | 21:02 |
| rhallisey | dims_, add to that if there's more to it | 21:03 |
| *** unicell has joined #openstack-kolla | 21:05 | |
| *** fbarilla has joined #openstack-kolla | 21:12 | |
| gmmaha | dims_: awesome.. guesssing it was apparmor | 21:15 |
| vhosakot | g3ek: if you use master, you should get 2.0.0 | 21:15 |
| gmmaha | vhosakot: none of my tries are working :( | 21:15 |
| gmmaha | and i have a feeling i am starting to get brain-dead on this issue.. | 21:16 |
| gmmaha | shall i throw up a patch on this for you to take a look at and see how to best re-structure it ? | 21:16 |
| vhosakot | kolla_version is not set by the user/operator I think | 21:16 |
| vhosakot | g3ek: ^^ | 21:16 |
| *** wuhg has joined #openstack-kolla | 21:16 | |
| g3ek | vhosakot : I know, the problem is that I have clone the master repo and I dont know why the version i get is now 0.0.1. I just found that my pip list show that kolla is 0.0.1-dev something | 21:18 |
| vhosakot | g3ek: my output | 21:19 |
| vhosakot | $ pip list | grep kolla | 21:19 |
| vhosakot | kolla (2.0.0.0rc2.dev151) | 21:19 |
| vhosakot | g3ek: ^^^ | 21:19 |
| vhosakot | g3ek: after cloning the master repo, did you install it as well ? | 21:19 |
| nihilifer | SiRiuS_: hmmm... can you post the exact configuration in marathon of one of these apps? | 21:19 |
| nihilifer | probably something is bad with "constraints" | 21:20 |
| g3ek | yes with pip install --upgrade . (from the clone folder) | 21:20 |
| g3ek | vhosakot: check above | 21:20 |
| vhosakot | g3ek: can you try this ? | 21:20 |
| vhosakot | pip uninstall kolla | 21:20 |
| vhosakot | clone master | 21:20 |
| vhosakot | pip install kolla | 21:20 |
| dims_ | gmmaha : yep it was apparmor | 21:21 |
| g3ek | It work, but I did a lot of modification to the clone repository and I cant lose them ... I will try to merge with caution... | 21:21 |
| gmmaha | dims_: coool.. thanks for confirming | 21:22 |
| dims_ | rhallisey : +1'ed | 21:22 |
| *** achanda has quit IRC | 21:22 | |
| dims_ | rhallisey : this was my super simple globals.yml https://gist.github.com/dims/04a046e7286f73a87274f2a7af37f752 | 21:22 |
| SiRiuS_ | nihilifer, http://picpaste.com/3WMuzaIR.png | 21:25 |
| SiRiuS_ | for memcached | 21:25 |
| SiRiuS_ | this is all-in-one but with the multimode variable in globals.yml set to "yes" | 21:26 |
| *** banix has quit IRC | 21:27 | |
| *** haplo37 has joined #openstack-kolla | 21:29 | |
| *** sdake has joined #openstack-kolla | 21:30 | |
| *** ccesario_ has joined #openstack-kolla | 21:34 | |
| sdake | vhosakot around? | 21:35 |
| vhosakot | yep\ | 21:36 |
| gmmaha | vhosakot: sdake: your suggestion worked.. let me update your patch | 21:37 |
| vhosakot | sdake: o/ | 21:37 |
| sdake | i dumped state in prep for emergency | 21:38 |
| sdake | so i'll be back when its over ;) | 21:38 |
| *** fbarilla has quit IRC | 21:38 | |
| *** ccesario_ has quit IRC | 21:40 | |
| *** haplo37 has quit IRC | 21:42 | |
| vhosakot | g3ek: you dont need to... installing master will install 2.0.0.0rc2.dev151 | 21:48 |
| openstackgerrit | Ganesh Maharaj Mahalingam proposed openstack/kolla: Workaround ansible bug related to delegate_to https://review.openstack.org/299803 | 21:48 |
| gmmaha | vhosakot: ^^ | 21:52 |
| gmmaha | with this patch it seems to work fine | 21:52 |
| gmmaha | i will try and test more tonight if possible | 21:52 |
| gmmaha | but this seems ok | 21:52 |
| sdake | gmmaha does that patch work with the neutron bootstrap thing you sent up | 22:03 |
| sbezverk | sdake what emergecy you are talking about? Something happened in the area? | 22:04 |
| g3ek | vhosakot : Thanks a lot! I finally make it work again! So I had to change the pip install like you suggest : pip install kolla==2.0.0.0rc2.dev151 instead of pip install . (from the folder). With that I have the good version and everything works again. I just dont understand why it was working before and not anymore (the pip install .) | 22:04 |
| sdake | sbezverk nah just personal thing | 22:04 |
| sdake | nothing to be concerned about | 22:04 |
| *** unicell has quit IRC | 22:04 | |
| sbezverk | sdake ok :-) I thought you have incoming avalanche or something | 22:04 |
| sdake | avalancheof heat in arizona incoming;) | 22:05 |
| vhosakot | g3ek: cool, cheers! :) | 22:07 |
| sbezverk | sdake sorry not strong on US geography ;-) | 22:10 |
| *** huikang has quit IRC | 22:11 | |
| *** rajathagasthya has quit IRC | 22:23 | |
| *** unicell has joined #openstack-kolla | 22:38 | |
| *** dave-mccowan has joined #openstack-kolla | 22:40 | |
| *** SiRiuS_ has quit IRC | 22:43 | |
| *** ayoung has joined #openstack-kolla | 22:45 | |
| *** dave-mccowan has quit IRC | 22:50 | |
| *** sdake has quit IRC | 22:57 | |
| *** weiyu has joined #openstack-kolla | 23:00 | |
| *** blahRus has quit IRC | 23:05 | |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Make keepalived virtual_router_id param configurable https://review.openstack.org/300143 | 23:08 |
| *** sdake has joined #openstack-kolla | 23:09 | |
| *** weiyu has quit IRC | 23:10 | |
| *** salv-orlando has quit IRC | 23:11 | |
| *** gm__ has joined #openstack-kolla | 23:11 | |
| *** sdake_ has joined #openstack-kolla | 23:12 | |
| *** sdake has quit IRC | 23:13 | |
| *** gm__ has quit IRC | 23:13 | |
| *** weiyu_ has joined #openstack-kolla | 23:16 | |
| *** ccesario_ has joined #openstack-kolla | 23:16 | |
| ccesario_ | vhosakot: ping? | 23:16 |
| vhosakot | ccesario_: pong | 23:17 |
| ccesario_ | vhosakot: about your comment https://review.openstack.org/#/c/300143/ | 23:17 |
| patchbot | ccesario_: patch 300143 - kolla - Make keepalived virtual_router_id param configurable | 23:17 |
| ccesario_ | why is not needed ? could you please explain ? | 23:17 |
| vhosakot | ccesario_: sure | 23:18 |
| ccesario_ | because, it is needed when we deploy two or more clouds using "single" kolla deployer | 23:18 |
| ccesario_ | rhallisey follow me in the tests today at morning... | 23:19 |
| gmmaha | sdake_: yes it does | 23:19 |
| vhosakot | "vrrp_instance kolla_internal_vip" already has "internal" in it which is different from "vrrp_instance kolla_external_vip" that has "external" in it... | 23:19 |
| gmmaha | worked on my local setup | 23:19 |
| gmmaha | we have been debugging the last gew days | 23:19 |
| sdake_ | gmmaha so does it work without set_facts? | 23:20 |
| vhosakot | ccesario_: so, why do you think vrrp_instance kolla_internal_vip must be vrrp_instance kolla_internal_vip_51 | 23:20 |
| vhosakot | internal and external are already differentiated | 23:20 |
| gmmaha | sdake_: hmm, dont think so.. i tried setting thew variable in other places and it wasnt playing along well | 23:20 |
| sdake_ | so set_facts is global | 23:22 |
| sdake_ | i think that makes sense | 23:23 |
| sdake_ | that is what I wanted in my original change | 23:23 |
| sdake_ | I woonder though if the other change is needed in your patch | 23:23 |
| openstackgerrit | Ganesh Maharaj Mahalingam proposed openstack/kolla: Workaround ansible bug related to delegate_to https://review.openstack.org/299803 | 23:24 |
| ccesario_ | vhosakot: weel, in my tests using the same network segment for both clouds, the deploy stop when the virtual_router_id and vrrp_instance it was set previously | 23:24 |
| gmmaha | sdake_: which other change? | 23:24 |
| gmmaha | where i am changing it from delegate_to to when? | 23:25 |
| gmmaha | yeah, without that i found those respective plays dont happen and skip like before | 23:25 |
| sdake_ | https://review.openstack.org/#/c/299803/3/ansible/roles/neutron/tasks/deploy.yml | 23:25 |
| patchbot | sdake_: patch 299803 - kolla - Workaround ansible bug related to delegate_to | 23:25 |
| gmmaha | i had to dump run_once and force it to do that | 23:25 |
| vhosakot | ccesario_: yes, I agree about the virtual_router_id being different for internal and external | 23:25 |
| vhosakot | ccesario_: here is what I mean | 23:25 |
| gmmaha | sdake_: hmmm.. we can try and test without it | 23:25 |
| gmmaha | but it worked this way for sure | 23:25 |
| vhosakot | ccesario_: first cloud - 51 and 52 | 23:25 |
| vhosakot | ccesario_: second cloud - 53 and 54 | 23:26 |
| sdake_ | delegate_to doesn't work? | 23:26 |
| sdake_ | I hadn't noticed that changed to a when | 23:26 |
| gmmaha | sdake_: yeah.. not in the play for sure | 23:26 |
| sdake_ | we need run_once: True | 23:26 |
| *** ayoung has quit IRC | 23:26 | |
| gmmaha | sdake_: if thats there. it still bails out with the first ocmpute node | 23:26 |
| vhosakot | so, the internal ID (51) of first cloud will not conflict with internal ID (53) of second cloud | 23:26 |
| ccesario_ | vhosakot: yes yes, but too when the vrrp_instance is equal it caused problem too.... | 23:27 |
| sdake_ | http://docs.ansible.com/ansible/playbooks_delegation.html | 23:27 |
| vhosakot | ccesario_: ah | 23:27 |
| ccesario_ | therefore I force the rename | 23:27 |
| *** weiyu_ has quit IRC | 23:27 | |
| ccesario_ | did you understand ? | 23:27 |
| vhosakot | ccesario_: after changing, did it work for both clouds ? | 23:28 |
| ccesario_ | the name make reference to the router id | 23:28 |
| ccesario_ | yes yes | 23:28 |
| ccesario_ | :) | 23:28 |
| gmmaha | sdake_: let me test by adding run_once back | 23:28 |
| gmmaha | but i doubt it works | 23:28 |
| ccesario_ | let me paste the keepalived content | 23:28 |
| vhosakot | please | 23:28 |
| sdake_ | If you want to perform a task on one host with reference to other hosts, use the ‘delegate_to’ keyword on a task. This is ideal for placing nodes in a load balanced pool, or removing them. It is also very useful for controlling outage windows. Using this with the ‘serial’ keyword to control the number of hosts executing at one time is also a good idea: | 23:28 |
| sdake_ | i dont understand why delegate_to is used instead of when in the current code | 23:29 |
| ccesario_ | vhosakot: http://pastebin.com/5beCtH0P | 23:31 |
| sdake_ | delegated facts are only available in 2.0 which is probably why set_facts with delegate_to doesnt work | 23:31 |
| sdake_ | By default, any fact gathered by a delegated task are assigned to the inventory_hostname (the current host) instead of the host which actually produced the facts (the delegated to host). In 2.0, the directive delegate_facts may be set to True to assign the task’s gathered facts to the delegated host instead of the current one.: | 23:32 |
| sdake_ | this documentation implies he issue is where the facts are produced | 23:32 |
| sdake_ | If you want to run a task that's associated with a host, but on different host, you should try delegate_to. | 23:34 |
| sdake_ | i think the issue is the facts are not delgated with delegate_to | 23:35 |
| sdake_ | but delegate_to seems more correct | 23:35 |
| sdake_ | which is why when wasn't used | 23:35 |
| sdake_ | true/false? | 23:35 |
| gmmaha | sdake_: with run_once included it fails.. http://paste.openstack.org/show/492684/ | 23:36 |
| sdake_ | see liue 2 and line 5 | 23:36 |
| sdake_ | same hostname | 23:36 |
| sdake_ | 06 is a compute node right? | 23:37 |
| gmmaha | sdake_: not using delegate_to here.. Just run_once with when | 23:37 |
| gmmaha | sdake_: yeah | 23:37 |
| sdake_ | why is it in the neutron server list? | 23:37 |
| gmmaha | so the facts are gathered given to the curren host once and done | 23:37 |
| sdake_ | or does when just run across all hosts? | 23:38 |
| gmmaha | i think for the plan it runs it through the list of hosts it created for it.. | 23:38 |
| gmmaha | and the ordering has to do somethign with it | 23:38 |
| gmmaha | so it triies on the first host there (compute -> 06) and tries once to run it.. | 23:38 |
| gmmaha | irrespective of the outcome its done | 23:38 |
| sdake_ | this is why delegate_to is running on 06 | 23:39 |
| gmmaha | soemone had created a feature request to add run_once_until_success for ansible | 23:39 |
| gmmaha | right | 23:39 |
| gmmaha | but its not delegating it to the variable outside | 23:39 |
| gmmaha | are you thinking if we use something otehr than set_fact delegate_to wil lwork? | 23:39 |
| sdake_ | no but i think we want delegte_to | 23:39 |
| gmmaha | sdake_: i have to hit the road to avoid traffic.. will be back online in half hour | 23:40 |
| sdake_ | sounds good | 23:40 |
| sdake_ | delegate_to to me seems like an optimization | 23:40 |
| sdake_ | what we really need here is delegate_facts | 23:40 |
| sdake_ | can you git review the patch that generated that last log? | 23:41 |
| *** vhosakot has quit IRC | 23:42 | |
| sbezverk | sdake_ I want to add a variale pointing to the location of custom rpms which kolla's user would like to add to the base? do you think it would be supported by community? | 23:42 |
| sdake_ | that already exissts | 23:43 |
| sdake_ | config option to build | 23:43 |
| sdake_ | bbi1 hr need quick powernap | 23:43 |
| *** sdake_ has quit IRC | 23:43 | |
| sbezverk | sdake_ ok | 23:43 |
| *** unicell has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
