| *** SiRiuS has joined #openstack-kolla | 00:04 | |
| *** SiRiuS has quit IRC | 00:05 | |
| *** banix has quit IRC | 00:13 | |
| *** timonwong has joined #openstack-kolla | 00:14 | |
| *** vhosakot has joined #openstack-kolla | 00:16 | |
| vhosakot | sdake: I see you already pushed the PS... cool... https://review.openstack.org/#/c/308073/1/tools/setup_gate.sh | 00:17 |
|---|---|---|
| patchbot | vhosakot: patch 308073 - kolla - Workaround broken nodepool to unblock gate | 00:17 |
| sdake | ya seeme easy enough | 00:17 |
| sdake | idont know if its corrct or not | 00:17 |
| vhosakot | sdake: ah cool.. | 00:17 |
| sdake | the grep worked for me in my testing | 00:18 |
| vhosakot | sdake: cool... | 00:18 |
| sdake | and it figures out valid ip addresses | 00:18 |
| vhosakot | sdake: I was thinking should we grep for ann non-empty files.. was working on that logic | 00:18 |
| *** huikang has joined #openstack-kolla | 00:18 | |
| vhosakot | ann=all | 00:18 |
| sdake | our gate scripts expect correct output from nodepool | 00:20 |
| sdake | this seems like a reasonale expectation | 00:20 |
| sdake | atm tho nodepool is putting out incorrect data | 00:20 |
| vhosakot | ah ok | 00:20 |
| *** sdake has quit IRC | 00:21 | |
| *** sdake has joined #openstack-kolla | 00:25 | |
| *** MarMat has quit IRC | 00:29 | |
| ccesario_ | vhosakot: ping ? | 00:30 |
| vhosakot | ccesario_: po | 00:30 |
| vhosakot | pong | 00:30 |
| ccesario_ | have you any idea or could you confirm if this happen with your deployment ? https://bugs.launchpad.net/kolla/+bug/1570550 | 00:30 |
| openstack | Launchpad bug 1570550 in kolla "elasticsearch.url param point to host address when haproxy it is enabled" [Medium,In progress] - Assigned to Carlos Cesario (ccesario) | 00:30 |
| *** ravig has quit IRC | 00:31 | |
| ccesario_ | I don 't get collect or found logs to identify this problem | 00:31 |
| *** ravig has joined #openstack-kolla | 00:31 | |
| ccesario_ | but I tested it 3 times and alwasy the problem happen when the env already is deployed | 00:32 |
| ccesario_ | in mitaka and master branch | 00:33 |
| *** banix has joined #openstack-kolla | 00:34 | |
| ccesario_ | vhosakot: ooops wrong bug link.... I meant this one https://bugs.launchpad.net/kolla/+bug/1571434 | 00:35 |
| openstack | Launchpad bug 1571434 in kolla "Kibana does not allow create index when enabled in an already deployed system" [Undecided,New] | 00:35 |
| ccesario_ | :P | 00:35 |
| *** ravig has quit IRC | 00:37 | |
| *** sdake has quit IRC | 00:38 | |
| vhosakot | ccesario_: I reviewed both.. will give my comments in review | 00:38 |
| *** sdake has joined #openstack-kolla | 00:38 | |
| vhosakot | sdake: https://review.openstack.org/#/c/307686/ passed gate | 00:39 |
| patchbot | vhosakot: patch 307686 - kolla - Fix iscsid & tgtd sections in inventory samples | 00:39 |
| ccesario_ | vhosakot: yes, I replied your comment | 00:39 |
| ccesario_ | vhosakot: did you see the last comment ? https://review.openstack.org/#/c/306443/ http://paste.openstack.org/show/494353/ | 00:41 |
| patchbot | ccesario_: patch 306443 - kolla - Make Kibana elasticsearch.url param point to VIP w... | 00:41 |
| *** huikang has quit IRC | 00:42 | |
| *** huikang has joined #openstack-kolla | 00:42 | |
| *** huikang has quit IRC | 00:47 | |
| *** haplo37 has joined #openstack-kolla | 00:49 | |
| vhosakot | sdake: https://review.openstack.org/#/c/306662/ also did not see nodepool issue is gate... | 00:53 |
| patchbot | vhosakot: patch 306662 - kolla - Make Cinder access glance in round robin fashion | 00:53 |
| vhosakot | is the gate fixed already ? | 00:54 |
| ccesario_ | time to sleep .... good night guys... cya tomorrow!!! | 00:54 |
| vhosakot | ccesario_: night! | 00:54 |
| *** ccesario_ has quit IRC | 00:55 | |
| sdake | vhosakot it sort of looks like it - ovh passed | 00:56 |
| vhosakot | ah ok.. | 00:57 |
| *** huikang has joined #openstack-kolla | 00:57 | |
| vhosakot | bluebox did not see nodepool issue https://review.openstack.org/#/c/307686/ | 00:58 |
| patchbot | vhosakot: patch 307686 - kolla - Fix iscsid & tgtd sections in inventory samples | 00:58 |
| sdake | internap failed here http://logs.openstack.org/76/307876/1/check/gate-kolla-dsvm-build-centos-binary/15893b2/console.html#_2016-04-19_15_34_13_020 | 00:58 |
| sdake | bluebox never has the nodepool issue because it has two interfaces | 00:58 |
| sdake | internap only has one | 00:58 |
| sdake | ovh failed here http://logs.openstack.org/76/307876/1/check/gate-kolla-dsvm-deploy-centos-binary/7b22446/console.html#_2016-04-19_15_46_38_356 | 01:00 |
| vhosakot | yea saw that | 01:00 |
| vhosakot | sdake: I see you are checking nodes_private instead of node_private in nodes_private | 01:07 |
| vhosakot | https://review.openstack.org/#/c/308073/1/tools/setup_gate.sh | 01:07 |
| patchbot | vhosakot: patch 308073 - kolla - Workaround broken nodepool to unblock gate | 01:07 |
| *** dave-mccowan has joined #openstack-kolla | 01:07 | |
| *** huikang has quit IRC | 01:08 | |
| *** huikang has joined #openstack-kolla | 01:08 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Workaround broken nodepool to unblock gate https://review.openstack.org/308073 | 01:09 |
| *** ayoung has joined #openstack-kolla | 01:09 | |
| sdake | vhosakot better? | 01:11 |
| vhosakot | sdake: cool... node_private is good :) let us wait for gate results... | 01:11 |
| *** huikang has quit IRC | 01:13 | |
| *** ccesario___ has quit IRC | 01:18 | |
| *** ccesario___ has joined #openstack-kolla | 01:19 | |
| *** weiyu has joined #openstack-kolla | 01:20 | |
| *** ravig has joined #openstack-kolla | 01:21 | |
| sdake | vhosakot have you seen this bug https://bugs.launchpad.net/kolla/+bug/1571612 | 01:21 |
| openstack | Launchpad bug 1571612 in kolla "Prechecks fails when checking if kolla_internal_vip_address is in the same network as network_interface on all nodes" [Medium,Triaged] | 01:21 |
| vhosakot | sdake: yes saw that | 01:23 |
| *** weiyu has quit IRC | 01:25 | |
| *** rhallisey has quit IRC | 01:26 | |
| vhosakot | sdake: this is what I came up with... to add all files in /etc/nodepool/ with valid IP's into ansible inventory...http://paste.openstack.org/show/494723/ | 01:26 |
| vhosakot | sdake: we could try if your PS still sees nodepool issue in ovh | 01:27 |
| vhosakot | sdake: I will reply to that bug | 01:27 |
| sdake | vhosakot the infra folks said he dont want public copied to private if private contains dta | 01:28 |
| sdake | dta/data | 01:28 |
| vhosakot | ah ok cool | 01:29 |
| *** weiyu has joined #openstack-kolla | 01:29 | |
| sdake | so i dont think yur approach would work correclty | 01:29 |
| sdake | alsot he privaet file is used elsewhere in thta setup script | 01:29 |
| sdake | just a fyi :) | 01:29 |
| *** weiyu has quit IRC | 01:30 | |
| vhosakot | ah ok :) | 01:30 |
| sdake | 335 jobs in the check queue | 01:32 |
| sdake | man openstack is backed up | 01:32 |
| * sdake wtb plunger | 01:32 | |
| vhosakot | yes, zuul is slow | 01:33 |
| sdake | its a capacity problem wrt not enogh vms to use | 01:33 |
| sdake | 335 jobs = 1-2k vms | 01:33 |
| sdake | i think ther is about 1k vms available to openstack | 01:33 |
| *** weiyu_ has joined #openstack-kolla | 01:34 | |
| vhosakot | wow... 1k vms in gate | 01:34 |
| sdake | clarkb says there are 500vms in the gate | 01:36 |
| sdake | but they lost a whole slew of capacity in the last couple of days | 01:36 |
| *** ravig has quit IRC | 01:53 | |
| *** ravig has joined #openstack-kolla | 01:53 | |
| *** ccesario___ has quit IRC | 01:55 | |
| *** ccesario___ has joined #openstack-kolla | 01:55 | |
| *** huikang has joined #openstack-kolla | 02:00 | |
| *** mwheckmann has joined #openstack-kolla | 02:01 | |
| *** tfukushima has joined #openstack-kolla | 02:07 | |
| *** unicell has quit IRC | 02:16 | |
| *** sdake_ has joined #openstack-kolla | 02:20 | |
| *** banix has quit IRC | 02:23 | |
| *** sdake has quit IRC | 02:23 | |
| *** huikang has quit IRC | 02:23 | |
| *** fragatin_ has quit IRC | 02:26 | |
| *** absubram has joined #openstack-kolla | 02:28 | |
| *** dave-mcc_ has joined #openstack-kolla | 02:31 | |
| *** dave-mccowan has quit IRC | 02:32 | |
| *** sdake_ has quit IRC | 02:36 | |
| *** tfukushima has quit IRC | 02:40 | |
| *** tfukushima has joined #openstack-kolla | 02:41 | |
| *** coolsvap has joined #openstack-kolla | 02:42 | |
| *** sekrit is now known as CIA | 02:44 | |
| *** fragatina has joined #openstack-kolla | 02:46 | |
| *** dave-mcc_ has quit IRC | 02:47 | |
| *** sdake has joined #openstack-kolla | 02:48 | |
| *** fragatina has quit IRC | 02:51 | |
| *** vhosakot has quit IRC | 02:55 | |
| *** yuanying has quit IRC | 02:56 | |
| *** vhosakot has joined #openstack-kolla | 03:01 | |
| coolsvap | vhosakot, please have a look at comment in https://review.openstack.org/#/c/304783/4/docker/base/Dockerfile.j2 | 03:02 |
| patchbot | coolsvap: patch 304783 - kolla - Update base dockerfile for formatting | 03:02 |
| vhosakot | coolsvap: yep, I will remove my -1.. thanks for the reply :) | 03:03 |
| coolsvap | vhosakot, thanks! | 03:03 |
| vhosakot | cool | 03:03 |
| coolsvap | sdake, vhosakot can you please have a look at https://review.openstack.org/#/c/305611/ | 03:04 |
| patchbot | coolsvap: patch 305611 - kolla - Remove check for config files to verify MountFlags | 03:04 |
| coolsvap | we need to get some conclusion on this | 03:04 |
| sdake | the gate has an 8 hour backup | 03:04 |
| coolsvap | there was one more bug opened regarding the same issue yesterdya | 03:04 |
| coolsvap | sdake, not approve but there is a discussion about the verification step | 03:05 |
| coolsvap | please have a look once | 03:05 |
| vhosakot | coolsvap: yes, will look at it | 03:05 |
| coolsvap | vhosakot, thanks | 03:06 |
| vhosakot | sdake: wow 8 hr backup... crazy | 03:06 |
| sdake | not having looked at jeffrey's comments | 03:06 |
| sdake | i think the issue with this patch is you are not producing an & operation | 03:06 |
| sdake | instead you are looking for a speciic value | 03:07 |
| sdake | whereas other mountflags may be desired | 03:07 |
| *** fragatina has joined #openstack-kolla | 03:07 | |
| sdake | is it possible to and the output with 2^20 and if != 0 proceed with a successful check? | 03:07 |
| *** ravig has quit IRC | 03:08 | |
| vhosakot | coolsvap: is MountFlags=1048576 seen _always_ and _only_ when MountFlags is set to shared | 03:09 |
| coolsvap | yes | 03:09 |
| sdake | yes but an't mountflags be multiple things? | 03:09 |
| vhosakot | coolsvap: 20th bit from where ? LSB MSB ? from left, right ? :) | 03:09 |
| coolsvap | vhosakot, 20th bit from LSB, 2^20 = 1048576 | 03:10 |
| sdake | so what if mountflgs have 19th bit st as well | 03:11 |
| sdake | this check will fail | 03:11 |
| coolsvap | yes | 03:11 |
| sdake | all we care is 20th bit is set | 03:11 |
| sdake | not if othe rbits are set in addition | 03:11 |
| coolsvap | sdake, thats what my question is | 03:11 |
| coolsvap | what are we concerned about validating | 03:11 |
| coolsvap | the steps which we have documented | 03:12 |
| sdake | that the 20th bit is set | 03:12 |
| coolsvap | or combinations | 03:12 |
| sdake | we dont care about other bits | 03:12 |
| coolsvap | if its not documented we shouldn't care | 03:12 |
| sdake | what ar ethe othe rbits that canbe set | 03:12 |
| *** fragatina has quit IRC | 03:12 | |
| coolsvap | i think there is shared|readonly thing | 03:12 |
| coolsvap | which Jeffrey mentioned | 03:12 |
| vhosakot | coolsvap: 2^20 = 1048576 in binary is 100000000000000000000 | 03:12 |
| *** phuongnh has joined #openstack-kolla | 03:13 | |
| coolsvap | vhosakot, yes | 03:13 |
| vhosakot | so, 20th bit is 1... yes | 03:13 |
| sdake | so just logical and the output with 2^20 | 03:13 |
| sdake | and if that is greater then 0 | 03:13 |
| sdake | you know 20th bit is set | 03:13 |
| sdake | and the check should pass | 03:13 |
| coolsvap | sdake, yes | 03:13 |
| coolsvap | we verified it on both centos and ubuntu | 03:14 |
| vhosakot | 1048571 is 11111111111111111011 which also has 20th bit 1... will 1048571 be seen anytime ? | 03:14 |
| coolsvap | and its in systemd code as well | 03:14 |
| coolsvap | vhosakot, no | 03:14 |
| coolsvap | if you have mountflags = shared | 03:14 |
| coolsvap | you will have 1048576 in systemctl show docker | 03:14 |
| coolsvap | you can verify | 03:15 |
| sdake | can mountflags = shared|somehtingelse? | 03:15 |
| vhosakot | https://www.freedesktop.org/software/systemd/man/systemd.exec.html#MountFlags= | 03:15 |
| vhosakot | I see it can be shared, slave or private | 03:16 |
| coolsvap | I am not sure where Jeffrey has used shared|readonly | 03:16 |
| coolsvap | thats what I need to confirm | 03:16 |
| sdake | rtfming it ooks like http://www.dsm.fordham.edu/cgi-bin/man-cgi.pl?topic=systemd.exec&sect=5 | 03:17 |
| sdake | only shared, slae, or private are valid flags | 03:17 |
| vhosakot | right coolsvap: so, if MountFlags is not 1048576, fail the prechecks as it is not shared.. is this logic correct ? | 03:18 |
| sdake | yellow like your cowardly selves! | 03:18 |
| vhosakot | hahahahahaha | 03:18 |
| openstackgerrit | Merged openstack/kolla: Update .gitreview for stable/liberty https://review.openstack.org/308064 | 03:18 |
| sdake | enders game is rockin | 03:19 |
| coolsvap | vhosakot, yes | 03:19 |
| sdake | ok lets do this | 03:19 |
| sdake | lets merge the patch as is | 03:19 |
| sdake | and revert the other patch which checks this stuff | 03:19 |
| sdake | and if someone comes along and says "I need more mountflags anded in" | 03:19 |
| sdake | we can tackle that problem i na zstream | 03:19 |
| sdake | no sense chsing ghosts | 03:19 |
| coolsvap | agree | 03:20 |
| sdake | that is why we have zstreams | 03:20 |
| vhosakot | me too.. right now, kolla nneds shared... | 03:20 |
| sdake | is a revert availlbe for the other stuff which is broken | 03:20 |
| coolsvap | which patch you want to revert? | 03:20 |
| sdake | not sure daneyon_ said the prechecks fail on systemd because of some /etc/systemd check | 03:20 |
| vhosakot | yes, daneyon_ reported it I think.. 1 sec | 03:20 |
| coolsvap | yes | 03:20 |
| sdake | which doesnt exist on centos | 03:20 |
| sdake | this heck is better | 03:21 |
| coolsvap | i marked it duplicate of current bug in progress | 03:21 |
| sdake | rather hen checking files on the filesystem lets check the results of the daemon | 03:21 |
| coolsvap | sdake, we can merge this | 03:21 |
| coolsvap | i will cherrypick and do required changes for mitaka | 03:21 |
| sdake | and liberty | 03:21 |
| coolsvap | and liberty | 03:21 |
| sdake | lets reveret the other patch first | 03:21 |
| coolsvap | I think this is it | 03:22 |
| coolsvap | https://review.openstack.org/#/c/304931/ | 03:22 |
| patchbot | coolsvap: patch 304931 - kolla - check /e/s/s/docker.service in ubuntu (MERGED) | 03:22 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Revert "check /e/s/s/docker.service in ubuntu" https://review.openstack.org/308122 | 03:23 |
| vhosakot | coolsvap: is this the bug you are talking about ? https://bugs.launchpad.net/kolla/+bug/1571281 | 03:24 |
| openstack | Launchpad bug 1571281 in kolla " Remove check for config files to verify MountFlags" [Critical,In progress] - Assigned to Swapnil Kulkarni (coolsvap) | 03:24 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Revert "check /e/s/s/docker.service in ubuntu" https://review.openstack.org/308124 | 03:24 |
| coolsvap | vhosakot, yes | 03:24 |
| coolsvap | sdake, we both did revert | 03:24 |
| coolsvap | should i abandon? | 03:25 |
| sdake | rockin huh :) | 03:25 |
| sdake | ya go ahead | 03:25 |
| sdake | and ack my change | 03:25 |
| sdake | nd vhosakot ack my revert | 03:25 |
| sdake | then if there are no conflicts with your change we can merge it | 03:25 |
| sdake | or you can rebase it | 03:25 |
| vhosakot | ok.. cool | 03:25 |
| *** tfukushima has quit IRC | 03:26 | |
| *** tfukushima has joined #openstack-kolla | 03:27 | |
| vhosakot | coolsvap: we're reverting becasue /lib/systemd/system/docker.service does not always exist ? | 03:27 |
| sdake | /etc/* doesent always exist | 03:28 |
| vhosakot | oh ok | 03:28 |
| vhosakot | ack'ed the revert | 03:29 |
| coolsvap | vhosakot, yes | 03:31 |
| coolsvap | I and Jeffrey discussed we need to have some way which does not rely on configuration files | 03:31 |
| coolsvap | we also tested systemctl cat docker | 03:31 |
| coolsvap | but it picks the configuration files alphabetically which cannot be validated for the value in system | 03:32 |
| vhosakot | cool | 03:33 |
| *** weiyu_ has quit IRC | 03:34 | |
| *** haplo37 has quit IRC | 03:39 | |
| *** ayoung has quit IRC | 03:40 | |
| *** weiyu_ has joined #openstack-kolla | 03:42 | |
| sdake | just tot be clear, prechecks fails 100% on centos because of htis issue coolsvap ? | 03:48 |
| *** yuanying has joined #openstack-kolla | 03:48 | |
| coolsvap | sdake, yes | 03:49 |
| coolsvap | once i update docker, the MountFlag is again set to slave | 03:49 |
| coolsvap | although its set to shared in kolla.conf in /e/s/s | 03:50 |
| vhosakot | coolsvap: does /lib/systemd/system/docker.service exist 100% of the time ? | 03:58 |
| coolsvap | vhosakot, we dont need to worry about /lib/systemd/system/docker.service | 03:59 |
| vhosakot | coolsvap: checking for MountFlags=1048576 in the output of "systemctl show docker" is good for both files ? /lib/systemd/system/docker.service or if it is overridden by /etc/systemd/system/docker.service ? | 04:01 |
| coolsvap | vhosakot, yes | 04:01 |
| vhosakot | coolsvap: got it... so, once the revert is merged, just rebase, and merge the PS that checks MountFlags=1048576 | 04:02 |
| coolsvap | vhosakot, yes | 04:02 |
| vhosakot | cool | 04:03 |
| vhosakot | sdake: your setup_workaround_broken_nodepool PS worked fine.. gate results came out for ovh and internap.. | 04:04 |
| vhosakot | sdake: do both ovh and internap have the nodepool issue ? | 04:04 |
| sdake | nice lets get it acked | 04:04 |
| sdake | internap only i think | 04:04 |
| daneyon_ | vhosakot sdake Latest bug I hit trying to ansible deploy: https://bugs.launchpad.net/kolla/+bug/1572395 | 04:04 |
| openstack | Launchpad bug 1572395 in kolla "Ansible Play Fail: keystone | Creating admin project" [Undecided,New] | 04:04 |
| vhosakot | daneyon_: json error ? | 04:04 |
| sdake | daneyon_ try https://review.openstack.org/#/c/307876/ | 04:05 |
| patchbot | sdake: patch 307876 - kolla - Disable tty for keystone register script | 04:05 |
| sdake | are you cleaning up between fialed deployed? | 04:05 |
| sdake | deploys? | 04:05 |
| sdake | cleanup-containers | 04:05 |
| sdake | cleanup-images -f | 04:05 |
| daneyon_ | i'm using ubuntu | 04:06 |
| vhosakot | daneyon_: so, all the images build fine now when you rebuild ? | 04:06 |
| sdake | are you using mater or stable/mitaka? | 04:06 |
| sdake | daneyon_ are you using a registry? | 04:07 |
| daneyon_ | vhosakot yes, using ubuntu and when I build image-by-image. not sure, maybe my lab has shotty Internet connectivity. sometimes I have to try building an image multiple times for the build to complete successfully. | 04:08 |
| daneyon_ | yes, local registry. | 04:08 |
| sdake | ok run cleanup-containers | 04:08 |
| sdake | run cleanup-images | 04:08 |
| sdake | with -f flag | 04:09 |
| *** fragatina has joined #openstack-kolla | 04:09 | |
| sdake | apply that patch linked above | 04:09 |
| sdake | you can click cherry-pick from the download menu | 04:09 |
| sdake | and just paste it in | 04:09 |
| sdake | then try a deploy | 04:09 |
| daneyon_ | ok | 04:09 |
| sdake | are you deploying on a centos os? | 04:09 |
| daneyon_ | ubuntu | 04:10 |
| sdake | you wont haveto rebuild since your images are in the registry | 04:10 |
| sdake | just apply patch after cleanup and kolla-ansible deploy | 04:10 |
| daneyon_ | i can not get all the req's centos images to build... filed a bug on it with the dets | 04:10 |
| sdake | one thing, run pip show olla | 04:10 |
| sdake | pip show kolla | 04:10 |
| vhosakot | daneyon_: since rebuild works and since intermittent network issues causing kolla build breakage is not issue in kolla code, can I move https://bugs.launchpad.net/kolla/+bug/1567657 to invalid for now, and can you re-open it if you see the issue again.. let me know what yout hink ? | 04:10 |
| openstack | Launchpad bug 1567657 in kolla mitaka "Build Fails Using Trunk" [Critical,Confirmed] | 04:10 |
| sdake | also please use stable/mitaka ;) | 04:11 |
| sdake | daneyon_ run pip show olla | 04:11 |
| daneyon_ | sdake, so stable/mitaka plus the ^ patch | 04:11 |
| sdake | kolla | 04:11 |
| sdake | right | 04:11 |
| sdake | but show me pip show kolla first | 04:12 |
| sdake | need tom kae sure you didn't pip install kolla | 04:12 |
| sdake | as the instructions recommend | 04:12 |
| sdake | that needs to go from the documentation | 04:12 |
| sdake | it causes more trobule then its worth | 04:12 |
| daneyon_ | sdake https://gist.github.com/danehans/b24c0b57b8e01c0f34359f9c0b52d4f5 | 04:12 |
| vhosakot | sdake: should I recheck nodepool PS ? | 04:12 |
| sdake | vhosakot nah lets just merge it | 04:12 |
| sdake | daneyon_ pip unisntall koll | 04:13 |
| vhosakot | cool | 04:13 |
| sdake | a | 04:13 |
| sdake | work from the olla owrking directory | 04:13 |
| *** fragatina has quit IRC | 04:13 | |
| daneyon_ | sdake even though i pip installed kolla. i have symlinked /usr/local/share/kolla -> /root/kolla/ (git dir) | 04:15 |
| sdake | dont do that!! | 04:15 |
| sdake | remove symlink | 04:15 |
| sdake | pip uninstall | 04:15 |
| sdake | work out of the git woring directory | 04:15 |
| sdake | pip install kolla is meant for peopel that don't plan to ever upgrade their koll install | 04:16 |
| vhosakot | daneyon_: let me know if I can move https://bugs.launchpad.net/kolla/+bug/1567657 to invalid and you can re-open it if you see the issue again ? | 04:20 |
| openstack | Launchpad bug 1567657 in kolla mitaka "Build Fails Using Trunk" [Critical,Confirmed] | 04:20 |
| sdake | nothign we can do about flakey mirrors | 04:20 |
| sdake | say i need to go eat | 04:20 |
| sdake | i'll bbl | 04:20 |
| daneyon_ | vhosakot ok | 04:25 |
| vhosakot | daneyon_: thanks | 04:25 |
| daneyon_ | yw | 04:25 |
| *** fragatina has joined #openstack-kolla | 04:26 | |
| *** timonwong has quit IRC | 04:28 | |
| *** fragatina has quit IRC | 04:31 | |
| *** timonwong has joined #openstack-kolla | 04:33 | |
| *** sdake has quit IRC | 04:36 | |
| openstackgerrit | Merged openstack/kolla: Revert "check /e/s/s/docker.service in ubuntu" https://review.openstack.org/308124 | 04:42 |
| daneyon_ | vhosakot or sdake kolla-build command is now hosed after pip uninstall, change to stable/mitaka and cherry-picked the keystone patch ^. See: https://gist.github.com/danehans/b24c0b57b8e01c0f34359f9c0b52d4f5 | 04:42 |
| vhosakot | yes, kolla-build will be removed after pip uninstall | 04:43 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Remove check for config files to verify MountFlags https://review.openstack.org/305611 | 04:45 |
| *** weiyu_ has quit IRC | 04:46 | |
| *** coolsvap is now known as coolsvap|away | 04:47 | |
| *** tfukushima has quit IRC | 04:50 | |
| *** mwheckmann has quit IRC | 05:01 | |
| *** daneyon has joined #openstack-kolla | 05:01 | |
| *** daneyon_ has quit IRC | 05:04 | |
| *** sdake has joined #openstack-kolla | 05:09 | |
| sdake | yo | 05:09 |
| vhosakot | yo! | 05:25 |
| *** tfukushima has joined #openstack-kolla | 05:29 | |
| sdake | any progress on meeges | 05:29 |
| vhosakot | meeges ? | 05:32 |
| *** weiyu_ has joined #openstack-kolla | 05:32 | |
| sdake | merges | 05:32 |
| vhosakot | yep, reviewing PS's in queue now | 05:34 |
| sdake | let me grab my laotoo | 05:35 |
| *** sdake_ has joined #openstack-kolla | 05:37 | |
| *** daneyon_ has joined #openstack-kolla | 05:37 | |
| sdake_ | i notice alot of gatte failures on ubuntu | 05:38 |
| sdake_ | http://logs.openstack.org/73/308073/2/check/gate-kolla-dsvm-deploy-ubuntu-source/2864e49/console.html#_2016-04-20_03_45_27_982 | 05:38 |
| sdake_ | we need to fix all the gate failures | 05:38 |
| sdake_ | after stable/liberty and stable/mitaka are tagged | 05:38 |
| vhosakot | ok cool | 05:39 |
| *** daneyon has quit IRC | 05:40 | |
| sdake_ | our prioriities from my perspetive are | 05:41 |
| sdake_ | 1. tag mitaka on 21st | 05:41 |
| sdake_ | 2. tag 1.1.0 as soon as feasible | 05:41 |
| sdake_ | 3. resolve all gate failures so our gates are passing consistently | 05:41 |
| sdake_ | 3. implement ipv6 support in kolla | 05:41 |
| sdake_ | 4. implement plugin support for atleast the compute kit projects | 05:42 |
| sdake_ | 5. port to ansible 2.0 | 05:42 |
| *** weiyu_ has quit IRC | 05:43 | |
| sdake_ | 6. make kola a powerful tool for -operating- openstack clouds | 05:44 |
| vhosakot | re: 4... does nova have plugins ? | 05:45 |
| sdake_ | yup | 05:45 |
| kklimonda | sdake_: does 4 mean support for neutron plugins? | 05:46 |
| sdake_ | neutron cinder and horizon are theo nes i'm mostly concerned with | 05:46 |
| vhosakot | sdake_: like hypervisors - kvm, qemu, xen... ? | 05:46 |
| sdake_ | we implement those 3 hypervisors | 05:46 |
| sdake_ | i am not sure what the nova plugins do | 05:47 |
| vhosakot | yes.. i can see neutorn plugins for $ | 05:47 |
| vhosakot | sdake_: is there a link for nova plugins ? | 05:47 |
| vhosakot | 4 | 05:47 |
| sdake_ | you would have to look at the code | 05:47 |
| vhosakot | ah ok | 05:47 |
| *** weiyu_ has joined #openstack-kolla | 05:47 | |
| sdake_ | we need to get dneyon unblocked so he can get magnum functinoal | 05:48 |
| kklimonda | or rather drivers, I've started looking into deploying contrail with kolla, and was wondering if you have any plans regarding that, or is it getting out of scope of kolla | 05:48 |
| sdake_ | what is contrail | 05:48 |
| sdake_ | one of our objectives is to deploy the big tent | 05:48 |
| kklimonda | neutron plugin/driver, replacement for ovs | 05:49 |
| sdake_ | `is that like ovn? | 05:49 |
| sdake_ | atm we implement ovs and linuxbridge | 05:50 |
| sdake_ | that covers 90% of the deployments | 05:50 |
| sdake_ | haven't heard of contrail | 05:50 |
| vhosakot | sdake_: is daneyon_ blocked by https://bugs.launchpad.net/kolla/+bug/1551992 ? | 05:50 |
| openstack | Launchpad bug 1551992 in kolla "Magnum failed to create trustee" [Critical,In progress] | 05:50 |
| sdake_ | but the more the marrier | 05:50 |
| sdake_ | vhosakot he is going to fix that | 05:50 |
| sdake_ | he is blocked because he ca't deploy kolla | 05:50 |
| kklimonda | yes, I think it's similar to ovn (haven't looked much into ovn though) | 05:50 |
| sdake_ | is it provided a a driver in the openstack git namespace? | 05:51 |
| vhosakot | sdake_: ok | 05:51 |
| sdake_ | moredirctly is it under neutron governance? | 05:51 |
| sdake_ | i would like to implement support for all vednor-specific pugins | 05:51 |
| *** rstarmer has joined #openstack-kolla | 05:54 | |
| *** yuanying_ has joined #openstack-kolla | 05:57 | |
| *** yuanying has quit IRC | 05:58 | |
| kklimonda | I don't think they are currently using openstack infrastructure, and they seem to be lagging behind with mitaka - Juniper in general seems to have problem grasping concept of building a community around that project. | 05:58 |
| nihilifer | sdake: kklimonda: from what i know, contrail provides neutron plugin (which should be fine to just put into separate container and run), but it also provides nova-vif driver, which should be installed alongside with nova-libvirt | 06:01 |
| nihilifer | to run this nova-vif driver in kolla, we should find some way to install 3rd party things in nova-libvirt container | 06:02 |
| nihilifer | https://github.com/Juniper/contrail-nova-vif-driver | 06:02 |
| *** fragatina has joined #openstack-kolla | 06:02 | |
| nihilifer | some jinja2 if conditional / macro / whatever should do the job imo | 06:03 |
| nihilifer | awww, s/nova-libvirt/nova-compute/g | 06:05 |
| kklimonda | contrail runs a tons of services, 4 extra databases, plugin that replaces ml2, and a kernel module for compute nodes that does routing etc. | 06:05 |
| *** weiyu_ has quit IRC | 06:05 | |
| kklimonda | and yes, apparently a vif drive too | 06:06 |
| kklimonda | driver* | 06:06 |
| *** fragatina has quit IRC | 06:07 | |
| nihilifer | yep, the kernel module couldn | 06:07 |
| *** weiyu_ has joined #openstack-kolla | 06:07 | |
| nihilifer | 't be installed by kolla in any way | 06:07 |
| kklimonda | how is openvswitch module provided? as part of host system? | 06:08 |
| nihilifer | oh, sorry, it looks like ovs is installed in kolla container | 06:09 |
| nihilifer | and modprobed in start script | 06:09 |
| nihilifer | so sorry, that seems to be doable too with contrail kernel module | 06:09 |
| nihilifer | https://github.com/openstack/kolla/blob/master/docker/openvswitch/openvswitch-base/Dockerfile.j2 https://github.com/openstack/kolla/blob/master/docker/openvswitch/openvswitch-vswitchd/extend_start.sh | 06:10 |
| kklimonda | I'm actually surprised you can load kernel modules from inside of containers. I guess it works though, being same system etc. | 06:11 |
| sdake_ | https://drive.google.com/file/d/0B_yCSDGnhIbzTDZTN1lwT3A1M1E/view | 06:11 |
| sdake_ | so contrail is maintained outside the openstack namespace in git? | 06:16 |
| sdake_ | to be considered for kolla i think it makes sense for the project to use openstack namespace and ci system | 06:17 |
| sdake_ | s | 06:17 |
| nihilifer | yes, it's maintained on github.com/Juniper | 06:19 |
| nihilifer | and review.opencontrail.org | 06:19 |
| nihilifer | so they have their own gerrit, own CI etc. | 06:19 |
| *** weiyu_ has quit IRC | 06:22 | |
| nihilifer | if we have a principle to not support things outside openstack infra, then the only way to support things like contrail is to implement possibility of making plugins for kolla, both with own dockerfiles and possibility of overriding the existing ones - but i have no concrete idea yet how such thing may be implemented | 06:22 |
| *** weiyu_ has joined #openstack-kolla | 06:28 | |
| sdake_ | nihilifer asalkeld heads up posting to mailing list about status of kolla-mesos | 06:36 |
| sdake_ | its t-3 days to summit and people need to know the truth | 06:36 |
| asalkeld | sure | 06:39 |
| *** mikelk has joined #openstack-kolla | 06:40 | |
| *** weiyu_ has quit IRC | 06:40 | |
| sdake_ | i have made a strong attempt not to throw you or your employer under the bus in the procees | 06:40 |
| sdake_ | tell me if I failed at that :) | 06:41 |
| sdake_ | the main reason is lots of people arecoming at me with requeests for kolla-kubeernetess | 06:41 |
| sdake_ | with smaller team sizes | 06:41 |
| *** weiyu_ has joined #openstack-kolla | 06:42 | |
| sdake_ | as in 1 or 2 people | 06:42 |
| sdake_ | anyway I had hpoed your emploeyr could have done this gracefully oon the ml but that didn't happen | 06:42 |
| sdake_ | so it is what it is | 06:43 |
| sdake_ | i'm tired of hiding the truth - and have hidden it for far too long for us to qualify as an open community | 06:43 |
| asalkeld | sdake_: no worries,all good | 06:45 |
| *** yuanying_ has quit IRC | 06:47 | |
| *** yuanying has joined #openstack-kolla | 06:48 | |
| *** weiyu_ has quit IRC | 06:53 | |
| *** banix has joined #openstack-kolla | 07:04 | |
| *** coolsvap|away has quit IRC | 07:06 | |
| sdake_ | asalkeld response looks good | 07:10 |
| sdake_ | asalkeld your welcome to stay on the cr team unless your reviews drop off in which cae i'll either send a email similar to harm's or you can step down ;) | 07:11 |
| asalkeld | ok | 07:11 |
| sdake_ | same story goes for your peers | 07:12 |
| sdake_ | and nihilfer was core before mirantis really got invvolved in kolla so i expect he may still continue on | 07:12 |
| sdake_ | we ren't making any immediate changes to the cr team as a result of this is the bottom line | 07:14 |
| sdake_ | if you decide to sty involved with kolla (which it doesn't sound like from your response) then your morthen welcome to stay invovled :) | 07:15 |
| sdake_ | the cr team trusts you or they wouldnt have voted you in in the first place | 07:15 |
| sdake_ | i hope we can keep yo uaround, but with heat, i made a clean break - undersstand if that is your approach | 07:16 |
| *** coolsvap|away has joined #openstack-kolla | 07:19 | |
| *** vhosakot has quit IRC | 07:21 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-mesos: Deprecate kola-mesos in the README https://review.openstack.org/308165 | 07:23 |
| openstackgerrit | Steven Dake proposed openstack/kolla-mesos: Deprecate kolla-mesos in the README https://review.openstack.org/308165 | 07:23 |
| *** mimizone has quit IRC | 07:24 | |
| *** mimizone has joined #openstack-kolla | 07:25 | |
| *** athomas has joined #openstack-kolla | 07:28 | |
| *** Serlex has joined #openstack-kolla | 07:32 | |
| sdake_ | asalkeld will you be at summit? | 07:32 |
| *** tfukushima has quit IRC | 07:35 | |
| *** shardy has joined #openstack-kolla | 07:36 | |
| asalkeld | sdake_: yip | 07:37 |
| *** tfukushima has joined #openstack-kolla | 07:37 | |
| sdake_ | cool well hope to see you there - iv'e got a super packed schedule | 07:37 |
| sdake_ | but we can likely catch a few beers :) | 07:37 |
| *** weiyu_ has joined #openstack-kolla | 07:40 | |
| *** weiyu_ has quit IRC | 07:40 | |
| *** weiyu_ has joined #openstack-kolla | 07:43 | |
| *** banix has quit IRC | 07:48 | |
| *** yuanying has quit IRC | 07:49 | |
| *** yuanying has joined #openstack-kolla | 07:52 | |
| *** tfukushima has quit IRC | 07:52 | |
| *** tfukushima has joined #openstack-kolla | 07:56 | |
| *** gfidente has joined #openstack-kolla | 07:59 | |
| *** yuanying has quit IRC | 08:05 | |
| *** mbound has joined #openstack-kolla | 08:07 | |
| *** coolsvap|away is now known as coolsvap | 08:15 | |
| *** jmccarthy has joined #openstack-kolla | 08:18 | |
| *** rstarmer has quit IRC | 08:38 | |
| *** shardy has quit IRC | 08:38 | |
| *** shardy has joined #openstack-kolla | 08:38 | |
| ccesario___ | morning guys _:) | 08:45 |
| coolsvap | ccesario, morning :) | 08:50 |
| *** stvnoyes has joined #openstack-kolla | 08:55 | |
| *** pmisiak has joined #openstack-kolla | 09:45 | |
| *** athomas has quit IRC | 09:54 | |
| *** dougs has joined #openstack-kolla | 09:55 | |
| *** mbound has quit IRC | 09:59 | |
| *** athomas has joined #openstack-kolla | 10:00 | |
| *** mbound has joined #openstack-kolla | 10:02 | |
| openstackgerrit | jackning proposed openstack/kolla: Added webroot config to dashboard https://review.openstack.org/306928 | 10:02 |
| *** mbound has quit IRC | 10:04 | |
| *** mbound has joined #openstack-kolla | 10:05 | |
| *** dougs has quit IRC | 10:13 | |
| *** dougs has joined #openstack-kolla | 10:14 | |
| *** dougs has quit IRC | 10:16 | |
| *** dougs has joined #openstack-kolla | 10:18 | |
| *** mbound has quit IRC | 10:19 | |
| *** dougs1 has joined #openstack-kolla | 10:19 | |
| *** mbound has joined #openstack-kolla | 10:20 | |
| *** dougs has quit IRC | 10:21 | |
| *** ccesario___ has quit IRC | 10:23 | |
| *** mbound has quit IRC | 10:23 | |
| *** mbound has joined #openstack-kolla | 10:25 | |
| *** weiyu_ has quit IRC | 10:28 | |
| *** dougs has joined #openstack-kolla | 10:37 | |
| *** dougs1 has quit IRC | 10:37 | |
| *** weiyu has joined #openstack-kolla | 10:40 | |
| *** dougs has quit IRC | 10:43 | |
| *** dougs has joined #openstack-kolla | 10:44 | |
| *** dougs has quit IRC | 10:45 | |
| *** dougs has joined #openstack-kolla | 10:46 | |
| *** gfidente has quit IRC | 10:46 | |
| *** weiyu has quit IRC | 10:49 | |
| *** weiyu has joined #openstack-kolla | 10:52 | |
| *** dougs has quit IRC | 10:55 | |
| *** dougs has joined #openstack-kolla | 10:55 | |
| *** ccesario___ has joined #openstack-kolla | 10:57 | |
| *** cfarquhar has quit IRC | 10:59 | |
| *** dougs1 has joined #openstack-kolla | 10:59 | |
| *** cfarquhar has joined #openstack-kolla | 11:00 | |
| *** cfarquhar has joined #openstack-kolla | 11:00 | |
| *** dougs has quit IRC | 11:01 | |
| *** weiyu has quit IRC | 11:07 | |
| *** dougs1 has quit IRC | 11:07 | |
| *** dougs has joined #openstack-kolla | 11:07 | |
| *** mikelk has quit IRC | 11:08 | |
| *** mikelk has joined #openstack-kolla | 11:09 | |
| *** dougs has quit IRC | 11:17 | |
| *** dougs has joined #openstack-kolla | 11:17 | |
| *** dougs has left #openstack-kolla | 11:19 | |
| *** weiyu has joined #openstack-kolla | 11:25 | |
| *** mlima has joined #openstack-kolla | 11:28 | |
| *** mlima has quit IRC | 11:36 | |
| *** rhallisey has joined #openstack-kolla | 11:39 | |
| *** phuongnh has quit IRC | 11:44 | |
| *** gfidente has joined #openstack-kolla | 11:46 | |
| *** SiRiuS has joined #openstack-kolla | 11:50 | |
| *** weiyu has quit IRC | 11:53 | |
| *** dwalsh has joined #openstack-kolla | 12:10 | |
| *** eal has joined #openstack-kolla | 12:17 | |
| *** mlima has joined #openstack-kolla | 12:19 | |
| *** tfukushima has quit IRC | 12:32 | |
| *** prithiv has joined #openstack-kolla | 12:35 | |
| *** coolsvap has quit IRC | 12:37 | |
| *** dave-mccowan has joined #openstack-kolla | 12:46 | |
| *** mwheckmann has joined #openstack-kolla | 12:47 | |
| *** salv-orlando has joined #openstack-kolla | 12:48 | |
| *** necuser has joined #openstack-kolla | 13:00 | |
| *** wznoinsk has joined #openstack-kolla | 13:01 | |
| *** wznoinsk has quit IRC | 13:02 | |
| *** wznoinsk has joined #openstack-kolla | 13:07 | |
| *** timonwong_ has joined #openstack-kolla | 13:16 | |
| *** ntpttr has quit IRC | 13:18 | |
| prithiv | has anyone faced issue with heka container in multi node | 13:19 |
| *** timonwong has quit IRC | 13:19 | |
| sdake_ | morning | 13:24 |
| sdake_ | wtb acks https://review.openstack.org/#/c/308073/2 | 13:24 |
| patchbot | sdake_: patch 308073 - kolla - Workaround broken nodepool to unblock gate | 13:24 |
| sdake_ | pbourke ? | 13:25 |
| sdake_ | any other core reviewers around? | 13:25 |
| mlima | morning sdake_ | 13:25 |
| sdake_ | morning dave-mccowan | 13:28 |
| dave-mccowan | good morning sdake_ | 13:28 |
| sdake_ | hey mlima | 13:29 |
| mlima | hey :0 | 13:29 |
| mlima | :) | 13:29 |
| sdake_ | prithiv what issue are you having | 13:30 |
| sbezverk | sdake_ morning, tried privilged: true for swift-rsyncd no go, now changing kolla_docker for capabilities | 13:30 |
| *** ravig has joined #openstack-kolla | 13:31 | |
| sdake_ | sbezverk https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities | 13:31 |
| prithiv | unix socket error for my other compute node while running heka container | 13:31 |
| prithiv | after that all other tasks are skipping | 13:33 |
| sdake_ | sbezverk its hard to tll rom the docker documentation if all capabilities are passed when --privieged is used | 13:39 |
| mlima | sdake_, ? | 13:40 |
| sdake_ | prithiv could you paste the socket error | 13:40 |
| openstackgerrit | Merged openstack/kolla: Workaround broken nodepool to unblock gate https://review.openstack.org/308073 | 13:42 |
| sbezverk | sdake_ I changed kolla-docker and now I see net-bind capability is set for that container, but still no go, I guess it gets blocked by --security-opt | 13:43 |
| sdake_ | are yu runnign with seinux? | 13:43 |
| *** necuser has quit IRC | 13:43 | |
| sdake_ | did you add net-cap-sysadmin as the capability? | 13:43 |
| sbezverk | sdake_ NET_BIND_SERVICEBind a socket to internet domain privileged ports (port numbers less than 1024). | 13:44 |
| sdake_ | sbezverk if you have added capabitlies to kolla_docker lets get that merged | 13:44 |
| sdake_ | you need netp-cap_sysdin to bind under 1024 | 13:44 |
| sbezverk | sdake_ not according to doc | 13:44 |
| sdake_ | run man capabilities on yur linux host | 13:44 |
| sbezverk | sdake_ CAP_NET_BIND_SERVICE | 13:45 |
| sbezverk | Bind a socket to Internet domain privileged ports (port numbers less than 1024). | 13:45 |
| sdake_ | what is net-cpa-sysadmin | 13:45 |
| sbezverk | sdake_ does not seem to be related: CAP_SYS_ADMIN | 13:46 |
| sbezverk | * Perform a range of system administration operations including: quotactl(2), mount(2), umount(2), swapon(2), swapoff(2), sethostname(2), and setdomainname(2); | 13:46 |
| prithiv | failed: [silpixa00394078] => {"changed": true, "failed": true} | 13:46 |
| prithiv | msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.23/images/create?tag=2.0.0&fromImage=10.237.214.35%3A4000%2Fkollaglue%2Fubuntu-source-heka',),) | 13:46 |
| sdake_ | prithiv need to see your docker logs please | 13:46 |
| sdake_ | if your on a system with sysemd run journalctl -xe > z | 13:46 |
| sdake_ | and paste z | 13:46 |
| sbezverk | sdake_ I will dig into --security-opt=seccomp=unconfined I think "default seccomp profile" blocks me here | 13:47 |
| sdake_ | sbezverk are you running with selinux enabled? | 13:47 |
| sbezverk | sdake_ permissive mode | 13:48 |
| sdake_ | sbezverk what error do y uoget back from the rsync process | 13:48 |
| sbezverk | sdake_ it fails to bind to the socket | 13:49 |
| sbezverk | but when I used setcap it worked | 13:49 |
| sbezverk | sdake_ it must be docker blocikng it | 13:49 |
| *** ayoung has joined #openstack-kolla | 13:50 | |
| sdake_ | are you sure your proceess has the right capbilities nside the container? | 13:51 |
| sdake_ | i'd verify that irst | 13:51 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Workaround broken nodepool to unblock gate https://review.openstack.org/308340 | 13:52 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Workaround broken nodepool to unblock gate https://review.openstack.org/308341 | 13:52 |
| sbezverk | sdake_ yes, I found how to setup sec profile for docker, will try it and et you knwo | 13:52 |
| *** prithiv has quit IRC | 13:54 | |
| sdake_ | the thing is if setpcap worked i dont understand why passing capabilities wouoldn't work | 13:55 |
| *** absubram has quit IRC | 13:56 | |
| sdake_ | sbezverk how did you verify your set cap kolla_docker module worked properly? | 14:00 |
| sbezverk | sdake_ I inspect the container | 14:02 |
| sbezverk | and it tells me exactl options used for thisspecific continer | 14:02 |
| sbezverk | sdake_ 2 minutes I am about to test with sec profile | 14:03 |
| sdake_ | options !+ capabiltieis the process has runnign in the container | 14:03 |
| sdake_ | ok | 14:03 |
| *** prithiv has joined #openstack-kolla | 14:05 | |
| *** stvnoyes has left #openstack-kolla | 14:10 | |
| *** v1k0d3n has joined #openstack-kolla | 14:12 | |
| *** ravig has quit IRC | 14:16 | |
| *** mbound has quit IRC | 14:20 | |
| *** rafiki has joined #openstack-kolla | 14:21 | |
| sdake_ | prithiv the docker logs would be helpful on a 500 error | 14:22 |
| prithiv | yes. i am building images so i will have to see the logs after i deploy. will update you in a while | 14:22 |
| sdake_ | ok | 14:24 |
| *** jmccarthy1 has left #openstack-kolla | 14:25 | |
| *** prithiv has quit IRC | 14:28 | |
| *** rafiki_ has joined #openstack-kolla | 14:29 | |
| *** inc0 has joined #openstack-kolla | 14:30 | |
| inc0 | hey, good mornign | 14:30 |
| *** absubram has joined #openstack-kolla | 14:32 | |
| sbezverk | sdake_ man I set up capabilities and security profile still it fails http://paste.openstack.org/show/494831/ | 14:38 |
| sdake_ | run a process in the container to print out the capabilities | 14:39 |
| sdake_ | make sure the process is actuallyg ettting thecapabilites you set | 14:39 |
| sdake_ | capsh --print | 14:39 |
| sbezverk | sdake_ cannot it is flapping | 14:40 |
| sdake_ | add it at start of extend_start.sh for the container | 14:40 |
| sdake_ | the nuse docker logs to display the output | 14:41 |
| sbezverk | sdake_ ok doing it | 14:41 |
| sdake_ | dropping off vpn brb | 14:42 |
| *** sdake_ has quit IRC | 14:42 | |
| *** sdake_ has joined #openstack-kolla | 14:45 | |
| *** athomas has quit IRC | 14:47 | |
| *** secman has joined #openstack-kolla | 14:48 | |
| *** ravig has joined #openstack-kolla | 14:48 | |
| sbezverk | sdake_ changes in kolla_docker, should I push them for review or drop? | 14:49 |
| sdake_ | once they are working keep them of coursse | 14:49 |
| sdake_ | we need capability setting | 14:49 |
| sbezverk | sdake_ they are wroking :-) just not helping in this specific case | 14:50 |
| sdake_ | what does capsh --print show? | 14:50 |
| sbezverk | sdake_ still waiting for rebuild to complete | 14:50 |
| *** athomas has joined #openstack-kolla | 14:52 | |
| sdake_ | [sdake@minime-03 swift-rsyncd]$ docker run -it --cap-add=NET_BIND_SERVICE 192.168.1.103:4000/kollaglue/centos-binary-swift-rsyncd:2.0.0 bash | 14:52 |
| sdake_ | [root@1cdad6c423d3 /]# | 14:52 |
| sdake_ | i would expect to be logged in as the swift user after this operation | 14:52 |
| *** ravig has quit IRC | 14:53 | |
| sdake_ | swift user has nologin set | 14:54 |
| sdake_ | this is odd when i run the swift rsync contner it doesnt drop root | 14:56 |
| sdake_ | which is not what i would expect | 14:56 |
| sbezverk | sdake_ you are on centos right? | 14:57 |
| sdake_ | yup | 14:58 |
| sdake_ | mybe my rsyncd is old | 14:58 |
| sbezverk | sdake_ in this case docker was built with setcap command | 14:58 |
| *** ravig has joined #openstack-kolla | 14:59 | |
| *** mlima has quit IRC | 15:00 | |
| *** prithiv has joined #openstack-kolla | 15:00 | |
| sdake_ | yo inc0 | 15:01 |
| sdake_ | i got your backportwork unblocked | 15:01 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Making Mitaka deploying liberty https://review.openstack.org/308390 | 15:04 |
| inc0 | really sdake_ ?;) | 15:04 |
| sdake_ | really | 15:04 |
| sdake_ | go wild | 15:04 |
| inc0 | look at patchset | 15:04 |
| inc0 | I just submitted | 15:04 |
| inc0 | like a minute ago | 15:04 |
| *** ravig has quit IRC | 15:06 | |
| *** ravig has joined #openstack-kolla | 15:07 | |
| *** ravig has quit IRC | 15:08 | |
| *** mlima has joined #openstack-kolla | 15:14 | |
| sbezverk | sdake_ http://paste.openstack.org/show/494833/ | 15:14 |
| sdake_ | it ooks like yur runnign with privielged | 15:15 |
| sbezverk | sdake_ I think there is something specific to rsyncd binary | 15:16 |
| sbezverk | which is fixed either by running as root or with setcap on the actual binary file | 15:17 |
| sdake_ | run it via strace | 15:23 |
| *** ravig has joined #openstack-kolla | 15:23 | |
| inc0 | sdake_, we have bug for backport? oO | 15:24 |
| sdake_ | file one | 15:24 |
| sdake_ | if we have one we can dupe it | 15:24 |
| *** pmisiak has quit IRC | 15:25 | |
| *** dwalsh has quit IRC | 15:25 | |
| *** salv-orlando has quit IRC | 15:26 | |
| *** Serlex has quit IRC | 15:27 | |
| *** pbourke has quit IRC | 15:27 | |
| *** pbourke has joined #openstack-kolla | 15:28 | |
| sdake_ | secure-keep-caps: no (unlocked) | 15:31 |
| sdake_ | what is this sbezverk ? | 15:31 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Changes needed to deploy liberty with mitaka code https://review.openstack.org/308390 | 15:32 |
| *** falanx has joined #openstack-kolla | 15:32 | |
| *** LiftedKilt has joined #openstack-kolla | 15:33 | |
| falanx | can Kolla use calico or opencontrail as its neutron plugin? | 15:33 |
| sbezverk | sdake_ I think it is good because I have the same on root user | 15:33 |
| sdake_ | falanx atm no | 15:33 |
| inc0 | but afair we had some mechanism to provide your own plugin | 15:34 |
| inc0 | let me check | 15:34 |
| sbezverk | sdake_ what about apparmor? I do not have much experience with this, do you know how to disable it on a container unconditionally? | 15:37 |
| sdake_ | your on ubuntu? | 15:38 |
| sdake_ | run dmesg | 15:38 |
| sdake_ | it will tell you if apparmor is complaining | 15:38 |
| sdake_ | outside teh container | 15:38 |
| inc0 | yeah, we can't really deploy contrail of calico now | 15:38 |
| sbezverk | sdake_ the issue with setcap is ONLY on ubuntu and I got strace, http://paste.openstack.org/show/494839/ | 15:39 |
| LiftedKilt | inc0: are contrail/calico on the roadmap to add support for? | 15:39 |
| sbezverk | sdake_ despite all permissions it fails to bind to a socket :-( | 15:39 |
| inc0 | don't we have meeting now guys? | 15:40 |
| sdake_ | EPRM? | 15:40 |
| sdake_ | meeting is in 50 minuts inc0 | 15:40 |
| inc0 | LiftedKilt, we need to figure out different plugins, and that is on roadmap | 15:40 |
| inc0 | probably won't write code to deploy calico (although I like it) | 15:41 |
| inc0 | but will make it easier | 15:41 |
| sbezverk | sdake_ we can rollback swift-rsyncd to use root until we figure out | 15:41 |
| inc0 | with a bit of hacking you could do this now I think | 15:41 |
| sbezverk | or Ubuntu fixes aufs attributes | 15:41 |
| *** ravig has quit IRC | 15:43 | |
| sdake_ | sbezverk can i see the full strace output please | 15:44 |
| LiftedKilt | inc0: ok thanks | 15:44 |
| LiftedKilt | inc0: unfortunately calico/contrail is an absolute must for us | 15:45 |
| inc0 | LiftedKilt, let me think | 15:45 |
| sbezverk | sdake_ I pasted it! did it get trimmed? | 15:45 |
| inc0 | I would expect to make it possible to deploy right now with a bit of legwork | 15:45 |
| sbezverk | sdake_ it is here and it is complete: http://paste.openstack.org/show/494839/ | 15:46 |
| inc0 | LiftedKilt, soo | 15:48 |
| inc0 | as for calico | 15:48 |
| inc0 | while we don't have calico in containers | 15:48 |
| inc0 | if you run this one on host, it might just work | 15:48 |
| inc0 | contrail might be trickier | 15:50 |
| LiftedKilt | running calico outside a container is fine, I just want to make sure the integration is tight enough that it stays manageable | 15:50 |
| sdake_ | bind(3, {sa_family=AF_INET, sin_port=htons(873), sin_addr=inet_addr("192.168.80.25")}, 16) = -1 EACCES (Permission denied) | 15:51 |
| inc0 | so if you run neutron services inside container, that shouldn't affect calico at all | 15:51 |
| sbezverk | sdake_ it is not supposed to hapen as we allow binding to lower ports | 15:51 |
| inc0 | and you can override your configs with kolla now | 15:51 |
| *** mikelk has quit IRC | 15:52 | |
| inc0 | as for contrail | 15:52 |
| LiftedKilt | inc0: you're saying to just integrate calico with docker itself? | 15:52 |
| inc0 | LiftedKilt, no | 15:52 |
| inc0 | so we don't run docker networking | 15:52 |
| inc0 | we run host networking | 15:52 |
| LiftedKilt | inc0: ok | 15:52 |
| sdake_ | EACCES The address is protected, and the user is not the superuser. | 15:53 |
| inc0 | neutron has access to host network stack in it's full capability | 15:53 |
| inc0 | so it won't be any different than running it on host from network perspective | 15:53 |
| inc0 | therefore calico shouldnt be affected | 15:53 |
| sdake_ | http://man7.org/linux/man-pages/man2/bind.2.html | 15:53 |
| sbezverk | sdake_ but we have this capability: cap_net_bind_service we do not need to superuser | 15:53 |
| inc0 | you'd need to deploy calico manually ofc but well | 15:54 |
| *** ccesario___ has quit IRC | 15:54 | |
| *** ccesario___ has joined #openstack-kolla | 15:54 | |
| inc0 | as for contrail, I never deployed contrail so it's hard to tell how much work would be involved | 15:54 |
| inc0 | we dont install opencontrail in our containers, but that one is doable | 15:54 |
| LiftedKilt | sorry yeah when I asked about contrail I primarily meant opencontrail | 15:55 |
| inc0 | well, same thing, we don't install it | 15:55 |
| inc0 | but you can | 15:55 |
| LiftedKilt | right | 15:55 |
| inc0 | so when you build your containers | 15:55 |
| inc0 | you can provide kolla-build -I your_dockerfile_part | 15:56 |
| inc0 | it will inject contents of your_dockerfile_part to our dockerfile | 15:56 |
| LiftedKilt | I'm definitely more interested in calico than opencontrail - I'll go back and talk to the calico team and see what they think about creating a playbook to complement kolla | 15:56 |
| inc0 | so if you add contrail code installation | 15:56 |
| dasm | \ | 15:56 |
| inc0 | LiftedKilt, for newton we have plugins on roadmap and fairly high on it | 15:56 |
| inc0 | we want kolla deployment code to be pluggable | 15:57 |
| inc0 | so you could hack around it for liberty and mitaka and make it proper in newton | 15:57 |
| *** mark-casey1 has joined #openstack-kolla | 15:57 | |
| inc0 | I would be happy to have calico plugin working with kolla:) | 15:57 |
| LiftedKilt | how are you guys doing with nova plugins? | 15:57 |
| inc0 | kvm | 15:58 |
| LiftedKilt | in order to make a go of this we need to get lxd working | 15:58 |
| inc0 | didn't do anything besides libvirt | 15:58 |
| *** salv-orlando has joined #openstack-kolla | 15:58 | |
| inc0 | haha running lxd inside docker?;) | 15:58 |
| inc0 | you dawg I heard you like containers | 15:58 |
| LiftedKilt | inc0: haha | 15:59 |
| inc0 | again, doable and hackable around | 15:59 |
| inc0 | would like to see it working;) | 15:59 |
| inc0 | thing is, if we support in kolla every possible way to deploy openstack, it will be hell to maintain | 15:59 |
| inc0 | having multiple distros is hard enough | 16:00 |
| sdake_ | sbezverk reading kernel soruce tree gie me a bit | 16:00 |
| inc0 | however what we can and will do is to enable pluggable architecture | 16:00 |
| inc0 | LiftedKilt, it you want we have session in Austin about that, so your input would be values | 16:00 |
| inc0 | valued | 16:00 |
| sbezverk | sdake_ sure | 16:01 |
| inc0 | in any case | 16:01 |
| LiftedKilt | inc0: unfortunately we won't be able to make it to Austin this time | 16:01 |
| sbezverk | sdake_ meanwhile I will git review changes to kolla-dicker | 16:01 |
| LiftedKilt | inc0: yeah I understand the code sprawl hell | 16:01 |
| inc0 | worst case scenerio is that you write your own ansible part | 16:01 |
| inc0 | at least part of it | 16:01 |
| inc0 | docker is really only runtime of services, we didn't focus on separating containers too much | 16:02 |
| inc0 | so everything should be doable with a bit of legwork | 16:02 |
| inc0 | and we're here to help | 16:02 |
| mark-casey1 | anyone seen this? multinode ubuntu launching a single VM gives libvirtError: Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory http://paste.openstack.org/show/494845/ | 16:03 |
| mark-casey1 | googling lead me to https://review.openstack.org/#/c/279910/ | 16:04 |
| patchbot | mark-casey1: patch 279910 - kolla - Use ubuntu Mitaka repo (MERGED) | 16:04 |
| *** vhosakot has joined #openstack-kolla | 16:05 | |
| LiftedKilt | inc0: cool - I'll keep playing with it | 16:05 |
| LiftedKilt | inc0: thanks for the help and tips | 16:05 |
| inc0 | if you hit any issue that kolla makes it impossible to use different neutron plugins | 16:05 |
| *** banix has joined #openstack-kolla | 16:05 | |
| inc0 | make sure to let us know | 16:05 |
| inc0 | we'll probably want to remove that obstacle | 16:06 |
| LiftedKilt | inc0: for sure | 16:06 |
| *** prithiv has quit IRC | 16:08 | |
| *** ccesario___ has quit IRC | 16:08 | |
| *** ccesario___ has joined #openstack-kolla | 16:08 | |
| *** prithiv has joined #openstack-kolla | 16:09 | |
| sdake_ | err = -EACCES; | 16:16 |
| sdake_ | if (snum && snum < PROT_SOCK && | 16:16 |
| sdake_ | !ns_capable(net->user_ns, CAP_NET_BIND_SERVICE)) | 16:16 |
| sdake_ | goto out; | 16:16 |
| sdake_ | thats the kernel code | 16:17 |
| vhosakot | meeting in 12 minutes | 16:18 |
| *** haplo37 has joined #openstack-kolla | 16:18 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Addining abbility to specify capabilities and security https://review.openstack.org/308447 | 16:18 |
| sdake_ | https://github.com/torvalds/linux/blob/master/net/ipv4/af_inet.c#L479 | 16:19 |
| sbezverk | sdake_ we have CAP_NET blah what is snum check? | 16:19 |
| sdake_ | sbezverk did you run dmesg | 16:19 |
| sbezverk | sdake_ yes, but there is nothing interesting ther | 16:20 |
| sdake_ | snum is the protocol service | 16:20 |
| sdake_ | what about the audit log | 16:20 |
| sdake_ | if snum and snum < prot-sOCK (which I assume is 1024) | 16:20 |
| sdake_ | is it not possible to specify the rsync port? | 16:21 |
| sdake_ | one tidy workaround woudl be specifign the rsync port of our own liking above 1024 | 16:21 |
| *** prithiv has quit IRC | 16:22 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Addining abbility to specify capabilities and security https://review.openstack.org/308447 | 16:22 |
| *** MarMat has joined #openstack-kolla | 16:23 | |
| *** ccesario___ has quit IRC | 16:23 | |
| sbezverk | sdake_ what I do nto get is what is the difference between running setcap and we setting up capabilities | 16:23 |
| sdake_ | that is why we are looing ath the kernel | 16:23 |
| sdake_ | this is probably a kernel bug | 16:23 |
| sdake_ | related to --net=host | 16:23 |
| sdake_ | since we are not using namespaced networking | 16:23 |
| *** ccesario___ has joined #openstack-kolla | 16:23 | |
| sdake_ | and clearl ythe kernel is doing something with namespacing here | 16:23 |
| sdake_ | if I culd find the ns_ function | 16:23 |
| sdake_ | it would be helpful | 16:24 |
| *** prithiv has joined #openstack-kolla | 16:24 | |
| *** prithiv has quit IRC | 16:24 | |
| sdake_ | https://github.com/torvalds/linux/blob/master/kernel/capability.c#L375 | 16:26 |
| *** sacharya has joined #openstack-kolla | 16:26 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Addining ability to specify capabilities and security https://review.openstack.org/308447 | 16:28 |
| sdake_ | smels like a kernel bug to me | 16:28 |
| sdake_ | sbezverk can you make rsync run no a different port above 1024? | 16:29 |
| vhosakot | meeting now | 16:30 |
| sbezverk | sdake_ will investigate it | 16:31 |
| sbezverk | sdake_ would be really cool to catch a bug in Linus' code ;-) | 16:32 |
| *** rajathagasthya has joined #openstack-kolla | 16:47 | |
| daneyon_ | sdake_ kolla-build does not work since doing the pip uninstall: https://gist.github.com/danehans/b24c0b57b8e01c0f34359f9c0b52d4f5 | 16:48 |
| sdake_ | team meeting | 16:48 |
| sdake_ | be with you in 40 minutes | 16:48 |
| daneyon_ | np, thx | 16:48 |
| *** hectaman has joined #openstack-kolla | 16:49 | |
| *** ccesario___ has quit IRC | 16:50 | |
| *** ccesario___ has joined #openstack-kolla | 16:50 | |
| *** jasonsb has joined #openstack-kolla | 16:56 | |
| *** thumpba has joined #openstack-kolla | 16:58 | |
| *** mbound has joined #openstack-kolla | 17:06 | |
| *** fragatina has joined #openstack-kolla | 17:08 | |
| *** ccesario___ has quit IRC | 17:09 | |
| *** v1k0d3n has quit IRC | 17:12 | |
| *** athomas has quit IRC | 17:13 | |
| *** unicell has joined #openstack-kolla | 17:15 | |
| *** ccesario___ has joined #openstack-kolla | 17:22 | |
| *** ravig has joined #openstack-kolla | 17:23 | |
| *** jtriley has joined #openstack-kolla | 17:23 | |
| *** ravig has quit IRC | 17:25 | |
| *** ravig has joined #openstack-kolla | 17:25 | |
| *** mbound has quit IRC | 17:26 | |
| *** jasonsb has quit IRC | 17:32 | |
| *** jasonsb has joined #openstack-kolla | 17:32 | |
| sbezverk | sdake_ I got rsyncd listening on a non-default port by there is a trick with configuration, so clients would be aware of it | 17:32 |
| *** ccesario___ has quit IRC | 17:33 | |
| sdake_ | need to automate that sbezverk | 17:33 |
| sbezverk | sdake_ let me get it working properly ;-) first | 17:35 |
| *** ccesario___ has joined #openstack-kolla | 17:35 | |
| *** jtriley has quit IRC | 17:37 | |
| *** jasonsb has quit IRC | 17:37 | |
| sdake_ | makesure port is configurble as othe services are done atm | 17:39 |
| *** sdake__ has joined #openstack-kolla | 17:41 | |
| *** fragatina has quit IRC | 17:42 | |
| *** fragatina has joined #openstack-kolla | 17:42 | |
| *** sdake__ has quit IRC | 17:43 | |
| *** sdake_ has quit IRC | 17:43 | |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add Kuryr Docker container https://review.openstack.org/298000 | 17:44 |
| *** sdake_ has joined #openstack-kolla | 17:45 | |
| *** SiRiuS has quit IRC | 17:46 | |
| *** SiRiuS has joined #openstack-kolla | 17:47 | |
| *** mwheckmann has quit IRC | 17:50 | |
| inc0 | sdake_ https://review.openstack.org/#/c/308390/ gates are green besides centos binary | 17:50 |
| patchbot | inc0: patch 308390 - kolla (stable/liberty) - Changes needed to deploy liberty with mitaka code | 17:50 |
| inc0 | which is expected | 17:51 |
| sdake_ | inc0 hot | 17:51 |
| sdake_ | brb need to connect to vpn | 17:51 |
| *** sdake__ has joined #openstack-kolla | 17:53 | |
| inc0 | sdake__, one thing tho | 17:56 |
| inc0 | it only has liberty versions of stuff we deployed in liberty | 17:57 |
| *** sdake_ has quit IRC | 17:57 | |
| inc0 | so if we introduces service in mitaka, it's mitaka in repo | 17:57 |
| sdake__ | i think all of the keys used for ubuntu need to use the liberty cloud repo versoins | 17:57 |
| *** gfidente has quit IRC | 17:57 | |
| sdake__ | so mariadb for eg is the liberty version | 17:58 |
| sdake__ | and so on | 17:58 |
| inc0 | mariadb is not liberty...mariadb is mariadb | 17:58 |
| inc0 | but well | 17:58 |
| inc0 | we didn't test mariadb with liberty | 17:58 |
| inc0 | I'd rather risk version mismatch between openstack - maria than incorrect maria deployment code because we didnt write it for this version | 17:59 |
| mark-casey1 | if I wanted to test a different value in nova.conf could I know I need to rebuild nova containers but could I only kill those containers in the deployment and then re-run deploy? or does that really need clean-containers and clean-images? | 18:10 |
| *** jtriley has joined #openstack-kolla | 18:14 | |
| inc0 | mark-casey1, neither | 18:14 |
| inc0 | you don't need to rebuild anything | 18:14 |
| *** sdake__ has quit IRC | 18:14 | |
| inc0 | change value in your /etc/kolla/configs | 18:15 |
| inc0 | and kolla-ansible reconfigure | 18:15 |
| inc0 | or even kolla-ansible reconfigure -t nova | 18:15 |
| mark-casey1 | at 10,000 feet view what does that do? single layer change to container and redeploy it? | 18:16 |
| inc0 | mark-casey1, so we don't embed configs into containers | 18:17 |
| *** ravig has quit IRC | 18:17 | |
| inc0 | instead we bindmount a directory | 18:17 |
| mark-casey1 | ah | 18:17 |
| rhallisey | mark-casey1, the layer doesn't change | 18:17 |
| inc0 | and on container start it copies over content from this to /etc | 18:17 |
| *** mlima has quit IRC | 18:17 | |
| inc0 | so what reconfigure does is to copy new files there and restarts container really | 18:17 |
| mark-casey1 | awesome. I knew there was a thing there in how that part worked but now it makes sense | 18:18 |
| mark-casey1 | thnx both | 18:18 |
| *** jtriley has quit IRC | 18:20 | |
| *** jtriley has joined #openstack-kolla | 18:21 | |
| *** ayoung has quit IRC | 18:25 | |
| *** ravig has joined #openstack-kolla | 18:25 | |
| *** mlima has joined #openstack-kolla | 18:29 | |
| *** ravig has quit IRC | 18:30 | |
| mark-casey1 | inc0: change the config in /etc/kolla on the deployer or all targets? | 18:30 |
| *** ravig has joined #openstack-kolla | 18:30 | |
| mark-casey1 | /etc/kolla only has 3 files on deployer | 18:31 |
| inc0 | mark-casey1, if you create /etc/kolla/config/nova.conf it will override config every option from your nova.conf | 18:32 |
| inc0 | so you can specify just one, and it will override only this one and rest will be there | 18:32 |
| inc0 | config/nova-conductor.conf will override just conductors | 18:33 |
| *** ravig has quit IRC | 18:33 | |
| mark-casey1 | inc0: if I've pip installed can I change stuff in /usr/share/kolla too and have reconfigure pick it up? or would that require rebuilding? | 18:33 |
| inc0 | config/nova/my_hostname/nova.conf will override just nova.conf on my_hostname | 18:33 |
| *** ravig has joined #openstack-kolla | 18:33 | |
| *** ravig has quit IRC | 18:35 | |
| inc0 | what's in /usr/share/kolla? you set up this as your node config directory? | 18:35 |
| mark-casey1 | I checked the repo out from git a pip installed on the deployer, which I thikn is what created it in /usr/share. but I was assuming that changes there still required rebuilding images | 18:36 |
| inc0 | ah | 18:36 |
| inc0 | I use virtualenvs | 18:36 |
| inc0 | but no, you should work on /etc/kolla/config regardless | 18:36 |
| *** mwheckmann has joined #openstack-kolla | 18:37 | |
| inc0 | if there is no config dir there, just create one | 18:37 |
| inc0 | for .conf changes that is | 18:37 |
| mark-casey1 | inc0: ok. last q hopefully. what is the path inside ../config based on? how would I place an override for /etc/libvirt/qemu.conf? | 18:38 |
| mark-casey1 | /etc/kolla/config/nova/qemu.conf? | 18:38 |
| inc0 | mark-casey1, https://github.com/openstack/kolla/blob/master/ansible/roles/nova/tasks/config.yml#L55 this is for merge_configs | 18:39 |
| inc0 | let me check if you can pull that off with libvirt as well | 18:39 |
| inc0 | this was designed for .ini format | 18:39 |
| inc0 | no, unfortunately you can't do that with qemu or libvirt | 18:40 |
| *** ravig has joined #openstack-kolla | 18:40 | |
| inc0 | you need to modift kolla templates | 18:40 |
| mark-casey1 | ok. I'll edit the template and rebuild. fyi qemu.conf is empty, so I think this is failing https://github.com/openstack/kolla/blob/master/docker/nova/nova-libvirt/Dockerfile.j2#L32 | 18:41 |
| *** SiRiuS has quit IRC | 18:41 | |
| mark-casey1 | so I'm getting libvirtError: Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory | 18:41 |
| *** ravig has quit IRC | 18:41 | |
| mark-casey1 | https://bugzilla.redhat.com/show_bug.cgi?id=1290357 | 18:41 |
| openstack | bugzilla.redhat.com bug 1290357 in systemd "virtlogd is not started/enabled on fresh libvirt install" [Urgent,New] - Assigned to systemd-maint | 18:41 |
| inc0 | mark-casey1, rebuild isn't needed regardless | 18:41 |
| mark-casey1 | oh ok | 18:41 |
| inc0 | do you run COPY_ONCE or COPY_ALWAYS? | 18:42 |
| mark-casey1 | then I'll reconfigure after editing template and see if that fixes it | 18:42 |
| mark-casey1 | default value on that one | 18:42 |
| inc0 | so if you chagne your globals to COPY_ALWAYS and redeploy | 18:42 |
| *** jtriley has quit IRC | 18:42 | |
| inc0 | https://github.com/openstack/kolla/blob/master/etc/kolla/globals.yml#L11 | 18:42 |
| inc0 | which is default | 18:43 |
| inc0 | all you need to do is change template | 18:43 |
| inc0 | and restart containers | 18:43 |
| inc0 | however | 18:43 |
| mark-casey1 | I uncommented it just in case. changed template and reconfigure is running | 18:43 |
| inc0 | if you go to node itself | 18:43 |
| inc0 | and kolla_config_directory | 18:44 |
| inc0 | and change file out there | 18:44 |
| inc0 | it will work as well | 18:44 |
| inc0 | there are ways:) | 18:44 |
| mark-casey1 | right... cause the configs are bind mounted from target not deployer(?) | 18:44 |
| inc0 | yup | 18:45 |
| inc0 | they're copied from deployer to target | 18:45 |
| inc0 | and then bindmouned and copied inside container to containers /etc | 18:45 |
| mark-casey1 | boom. active vms | 18:50 |
| inc0 | congrats:) | 18:50 |
| mark-casey1 | ty | 18:50 |
| *** rajathagasthya has quit IRC | 18:51 | |
| mark-casey1 | trying floating ip now but I'm pretty sure that's a bug I hit in stable/mitaka | 18:51 |
| mark-casey1 | inc0: | 18:51 |
| *** mwheckmann has quit IRC | 18:54 | |
| inc0 | mark-casey1, with floatings? care to share details? | 18:55 |
| *** SiRiuS has joined #openstack-kolla | 18:56 | |
| mark-casey1 | was trying a floating ip. not working but I"m pretty sure that's on me https://github.com/ropsoft/kolla_from_vagrant/blob/master/layout.png | 18:56 |
| *** v1k0d3n has joined #openstack-kolla | 18:56 | |
| mark-casey1 | the thing I thought was a bug is qemu.conf being empty making this fail (https://github.com/openstack/kolla/blob/master/docker/nova/nova-libvirt/Dockerfile.j2#L32) making libvirt look for /var/run/libvirt/virtlogd-sock and fail to start VMs. implemented here https://review.openstack.org/#/c/306671/ | 18:58 |
| patchbot | mark-casey1: patch 306671 - kolla - Use proper protocol scheme with nova | 18:58 |
| mark-casey1 | no floating ips though is probably just something I've screwed up with my vlans. gotta afk for a bit. thx a million though! been working on this for a bit now so active VM is exciting | 19:00 |
| *** salv-orlando has quit IRC | 19:03 | |
| *** mwheckmann has joined #openstack-kolla | 19:08 | |
| *** salv-orlando has joined #openstack-kolla | 19:26 | |
| *** SiRiuS has quit IRC | 19:30 | |
| *** ccesario___ has quit IRC | 19:36 | |
| *** rajathagasthya has joined #openstack-kolla | 19:49 | |
| *** mlima has quit IRC | 19:55 | |
| *** ravig has joined #openstack-kolla | 19:59 | |
| *** eal has quit IRC | 20:00 | |
| *** shardy has quit IRC | 20:03 | |
| *** ravig has quit IRC | 20:04 | |
| *** ravig has joined #openstack-kolla | 20:05 | |
| *** shardy has joined #openstack-kolla | 20:12 | |
| *** ntpttr_ is now known as ntpttr | 20:17 | |
| mark-casey1 | inc0: ping | 20:21 |
| inc0 | mark-casey1, whats up? | 20:21 |
| mark-casey1 | wanted to see if what I was saying earlier about qemu.conf being empty was making any sense, and if I should file something on it | 20:22 |
| *** thumpba has quit IRC | 20:23 | |
| inc0 | mark-casey1, thats strange | 20:37 |
| *** banix has quit IRC | 20:39 | |
| mark-casey1 | I'm on Ubuntu 15.10, not Trusty. If package choices are made based on parent OS and if Trusty had an older version of libvirt I could see it not being an issue for some others | 20:40 |
| inc0 | mark-casey1, do you have nova-libvirt/qemu.conf on host? | 20:40 |
| inc0 | I run 15.10 all the time | 20:40 |
| inc0 | but with 15.10 you still need to make shared /run | 20:41 |
| mark-casey1 | root@jovial-shop:~# docker exec nova_libvirt ls -la /etc/libvirt/qemu.conf | 20:41 |
| mark-casey1 | -rw-r--r-- 1 root root 23 Apr 20 18:49 /etc/libvirt/qemu.conf | 20:41 |
| inc0 | and you do it with systemd conf | 20:41 |
| inc0 | did you do it? | 20:41 |
| mark-casey1 | yes, took some figuring out but I found that | 20:41 |
| mark-casey1 | so that ls -la I posted deffinitely said the file was empty before I changed the template | 20:41 |
| mark-casey1 | and AFAICT the template really is empty in git | 20:42 |
| mark-casey1 | https://bugzilla.redhat.com/show_bug.cgi?id=1290357 I dunno if upstream is the right word in this case but it's being done for this | 20:42 |
| openstack | bugzilla.redhat.com bug 1290357 in systemd "virtlogd is not started/enabled on fresh libvirt install" [Urgent,New] - Assigned to systemd-maint | 20:42 |
| mark-casey1 | oh but I pasted the wrong review for kolla! oops. one sec | 20:45 |
| mark-casey1 | this was the one I meant to say was for the kolla side, to address the thing in the redhat bugzilla link | 20:46 |
| mark-casey1 | https://review.openstack.org/#/c/279910/ | 20:46 |
| patchbot | mark-casey1: patch 279910 - kolla - Use ubuntu Mitaka repo (MERGED) | 20:46 |
| inc0 | hmm, let me try to deploy this one | 20:52 |
| *** shardy has quit IRC | 20:55 | |
| *** rajathagasthya has quit IRC | 20:58 | |
| *** rafiki_ has quit IRC | 21:02 | |
| *** ccesario___ has joined #openstack-kolla | 21:03 | |
| *** SiRiuS has joined #openstack-kolla | 21:06 | |
| *** inc0 has quit IRC | 21:17 | |
| *** ccesario___ has quit IRC | 21:18 | |
| *** ccesario___ has joined #openstack-kolla | 21:18 | |
| *** rajathagasthya has joined #openstack-kolla | 21:21 | |
| *** ccesario___ has quit IRC | 21:21 | |
| *** ccesario___ has joined #openstack-kolla | 21:22 | |
| *** SiRiuS has quit IRC | 21:24 | |
| *** sbezverk has quit IRC | 21:25 | |
| *** ccesario___ has quit IRC | 21:26 | |
| *** ccesario___ has joined #openstack-kolla | 21:27 | |
| *** sbezverk has joined #openstack-kolla | 21:27 | |
| *** sdake_ has joined #openstack-kolla | 21:28 | |
| *** ccesario___ has quit IRC | 21:28 | |
| sdake_ | yo | 21:29 |
| *** ccesario___ has joined #openstack-kolla | 21:31 | |
| *** rhallisey has quit IRC | 21:37 | |
| daneyon_ | sdake_ kolla-build does not work since I did a pip uninstall. Here is the error: https://gist.github.com/danehans/b24c0b57b8e01c0f34359f9c0b52d4f5 | 21:37 |
| sdake_ | dont use kolla-build | 21:37 |
| sdake_ | use tools/buid.py | 21:37 |
| daneyon_ | ok | 21:38 |
| *** ccesario___ has quit IRC | 21:38 | |
| sdake_ | was that the prolemf from themorning | 21:39 |
| sdake_ | apologies i laid down andj ust owke up | 21:39 |
| daneyon_ | np | 21:39 |
| daneyon_ | looks like my ks image is building now, thx | 21:39 |
| sbezverk | sdake_ it looks like swift has confiugration for nondefault ports but it is so wierd | 21:40 |
| *** ccesario___ has joined #openstack-kolla | 21:40 | |
| sdake_ | cool so lets use that orkaround | 21:42 |
| sbezverk | sdake_ could you check this please: https://review.openstack.org/#/c/308447/ | 21:43 |
| patchbot | sbezverk: patch 308447 - kolla - Addining ability to specify capabilities and security | 21:43 |
| sbezverk | if you are not super busy of course | 21:43 |
| sdake_ | i just woke up | 21:44 |
| sdake_ | gie me 5 minutes to get warmed up | 21:44 |
| sbezverk | sdake_ another allnighter last night ;-) ?? | 21:44 |
| *** daneyon has joined #openstack-kolla | 21:44 | |
| *** sdake_ has quit IRC | 21:45 | |
| *** v1k0d3n has quit IRC | 21:45 | |
| *** sdake_ has joined #openstack-kolla | 21:47 | |
| sdake_ | sbezverk what cli do you pass to rsync? | 21:51 |
| *** salv-orl_ has joined #openstack-kolla | 21:56 | |
| sbezverk | sdake_ for rsync I specify port to listen for example 10873 | 21:56 |
| sdake_ | anyone know what the system call is that creaates a container? | 21:56 |
| sbezverk | sdake_ but the main issue is rings must be generated using new syntax | 21:56 |
| sbezverk | and each replicator container needs to be reconfiguredd. | 21:57 |
| sdake_ | whatis new about the syntax | 21:57 |
| sdake_ | cant the replicator container be preconfigured? | 21:57 |
| sdake_ | another ptoin is not to use net=host for rsync | 21:58 |
| sdake_ | i am teting if my assuption is correct abut --net=host incompatible with cap | 21:58 |
| sbezverk | sdake_ do not know yet I am still not satisfied by what I see in replication logs | 21:58 |
| *** salv-orlando has quit IRC | 21:59 | |
| sdake_ | note rsync wm on centos kernel | 22:02 |
| sdake_ | without --net=host | 22:03 |
| sdake_ | note rsync works or me with net=host | 22:03 |
| sdake_ | cli optoin i use is rsync --daemon | 22:04 |
| sbezverk | sdake --port 10873 | 22:04 |
| sdake_ | i thought it had to be run on a port uner 1000 for the bind error the happen | 22:04 |
| sdake_ | ()[swift@minime-03 /]$ ps -ef | grep rsync | 22:05 |
| sdake_ | swift 26 1 0 22:05 ? 00:00:00 rsync --port 500 --daemon | 22:05 |
| sdake_ | swift 34 1 0 22:05 ? 00:00:00 grep rsync | 22:05 |
| sdake_ | [sdake@minime-03 tools]$ docker run -it --cap-add=NET_BIND_SERVICE 192.168.1.103:4000/kollaglue/centos-binary-swift-rsyncd:2.0.0 bash | 22:06 |
| sbezverk | sdake_ it looks like net_bind is working properly in centos | 22:06 |
| sbezverk | if you do not get that error | 22:07 |
| sdake_ | [sdake@minime-03 tools]$ docker run -it --cap-add=NET_BIND_SERVICE --net=host 192.168.1.103:4000/kollaglue/centos-binary-swift-rsyncd:2.0.0 bash | 22:07 |
| sdake_ | ()[swift@minime-03 /]$ rsync --port 500 --daemon | 22:07 |
| sdake_ | ()[swift@minime-03 /]$ ps -ef | grep rsync | 22:07 |
| sdake_ | swift 19 1 0 22:07 ? 00:00:00 rsync --port 500 --daemon | 22:07 |
| sdake_ | arn't you on selinux? | 22:09 |
| sdake_ | rather ubuntuian? | 22:09 |
| sbezverk | sdake_ ubuntu does not have it by default | 22:09 |
| sdake_ | ok i am generating logs to report a bug | 22:09 |
| sdake_ | so these people will gte this shit fixed | 22:09 |
| sbezverk | sdake_ I am on centos in general, but since we hit this issue on ubuntu I use ubuntu now to debug it | 22:09 |
| sdake_ | can you replicate this log | 22:09 |
| sdake_ | wich is your kernel version? | 22:10 |
| sbezverk | 4.2.0.27 | 22:10 |
| sdake_ | http://paste.fedoraproject.org/357995/19065814/ -> http://paste.fedoraproject.org/357995/19065814 | 22:17 |
| sdake_ | replicate that typescript on ubuntu please | 22:17 |
| sbezverk | sdake_ http://paste.openstack.org/show/494898/ | 22:26 |
| sdake_ | try touch /etc/rsyncd.conf | 22:28 |
| sdake_ | then again run the command | 22:28 |
| sbezverk | sdake_ cannot do it, I run under swift user | 22:29 |
| sdake_ | tocuh a config file in your home dir | 22:30 |
| *** daneyon has quit IRC | 22:30 | |
| mark-casey1 | sdake_: thought you might like this http://paste.openstack.org/show/494899/ | 22:31 |
| openstackgerrit | Merged openstack/kolla: Use proper protocol scheme with nova https://review.openstack.org/306671 | 22:31 |
| mark-casey1 | (run from deployer, that is) | 22:32 |
| *** ayoung has joined #openstack-kolla | 22:33 | |
| *** ccesario___ has quit IRC | 22:34 | |
| sdake_ | sbezverk rebuild a container for ubuntu with the touch /etc/rsync.conf | 22:34 |
| sdake_ | or whatever file it is coplaining abut | 22:34 |
| sdake_ | mark-casey1 what does all that do | 22:35 |
| * sdake_ not a neutorn wizard | 22:35 | |
| *** mwheckmann has quit IRC | 22:35 | |
| mark-casey1 | oh trust me, me either. more time on that than I'd care to admit. eth1.163 is neutron_external_bridge. all that adds a br-xo for our second WAN and gets it into neutron so I can have multiple externals | 22:36 |
| mark-casey1 | adds eth1.162 to br-xo, and etc | 22:36 |
| sdake_ | mark-casey1 we have an advanced admin guide - that wuld be hepful theer if you could explain how its useful :) | 22:38 |
| *** haplo37 has quit IRC | 22:38 | |
| mark-casey1 | essentially I can have a VM with a floating IP on my LAN and then move it to a public IP on one ISP with a few clicks, and the to a public on another ISP with a few more. it isn't quite working yet... I have to add the extra floating pools. | 22:39 |
| mark-casey1 | but yes, I can write it up | 22:39 |
| sdake_ | cool sounds good ;) | 22:39 |
| sdake_ | sbezverk can you rebuild a continer with a rsync.conf in it plz | 22:40 |
| sdake_ | id like to rule out apparmor as a cause | 22:42 |
| sdake_ | and identify it squarly at the kernel | 22:42 |
| sbezverk | sdake_ strange thing is in normal rsyncd container rsyncd.conf should be there, I guess it is not there now since ansible is not used to start it | 22:44 |
| sdake_ | yes it gets copied in by ansible | 22:44 |
| sdake_ | so you nneed tom ake an empty file | 22:44 |
| sdake_ | reproducing through our call chain wont work for docker cats | 22:47 |
| sdake_ | they want to see it reproduced through their call chain (dockercli) | 22:47 |
| sbezverk | sdake_ if I start container with root priv to be able to create rsyncd.conf file, then exit it how I can start it (the same container) again but with normal user swift?? | 22:49 |
| sdake_ | yu cant yu need to rebuild a container as i said 10 minutes ago | 22:49 |
| sbezverk | sdake_ ok I must have missed it ;-) | 22:51 |
| sdake_ | mdoify the docker file to create an empty rsync.conf file | 22:51 |
| sdake_ | via touch | 22:51 |
| sbezverk | sdake_ even if I create empty file I will not be able to modify it, I copy good file to the right location | 22:55 |
| sdake_ | all we care is if rsync startsup | 22:56 |
| sdake_ | fro mwhat icould tell your rsyn wasn't starting previousy | 22:56 |
| sbezverk | sdake_ before trying to bind to a socket it cheks for config file | 22:57 |
| sdake_ | i know, so we are oging to create an empty one | 22:57 |
| sbezverk | I saw that before I was fighting with setcap | 22:57 |
| sdake_ | just do it - we are burning daylight and i hae to leave for trave lsooon | 22:58 |
| sdake_ | 2 hour drive to phoenix | 22:58 |
| sbezverk | sdake_ it is building, takes the same time if I touch the file or copy it ;-) | 22:59 |
| *** mark-casey1 has quit IRC | 22:59 | |
| sbezverk | sdake_ http://paste.openstack.org/show/494900/ | 23:01 |
| *** mark-casey has joined #openstack-kolla | 23:01 | |
| *** daneyon has joined #openstack-kolla | 23:05 | |
| *** mark-casey has quit IRC | 23:05 | |
| *** mark-casey has joined #openstack-kolla | 23:06 | |
| *** daneyon has quit IRC | 23:09 | |
| *** ravig has quit IRC | 23:13 | |
| *** mark-casey1 has joined #openstack-kolla | 23:18 | |
| *** mark-casey has quit IRC | 23:20 | |
| *** mark-casey1 has quit IRC | 23:23 | |
| *** MarMat has quit IRC | 23:25 | |
| *** MarMat_ has joined #openstack-kolla | 23:25 | |
| *** salv-orl_ has quit IRC | 23:26 | |
| *** ravig has joined #openstack-kolla | 23:27 | |
| *** thumpba has joined #openstack-kolla | 23:38 | |
| sdake_ | sbezverk any luck? | 23:40 |
| sdake_ | sbezverk was that with --net=host? | 23:43 |
| sbezverk | sdake_ have you check the paste? | 23:45 |
| sbezverk | http://paste.openstack.org/show/494900/ | 23:45 |
| sdake_ | sbezverk i hae ano ther idea | 23:45 |
| sdake_ | perhaps the rsync port is alrady in use by your host | 23:45 |
| sbezverk | sdake_ I do not think so as --net=host also ansible paramter | 23:45 |
| sdake_ | wel run with --net=host | 23:45 |
| sdake_ | look t my typescript | 23:45 |
| sdake_ | it runs the continer with --cap and --net | 23:45 |
| sdake_ | and anothe wiwth just --cap | 23:45 |
| sbezverk | sdake_ my bad it was with --net | 23:46 |
| sdake_ | so it does work | 23:47 |
| sdake_ | try without --cap and se if it fails | 23:47 |
| sbezverk | I copied and pasted exact command you gave just changed a few ubuntu related things | 23:47 |
| sdake_ | where is your kolla_docker change | 23:48 |
| sdake_ | sbezverk if it works rom the cli the kolla-docker integratin is wrong | 23:48 |
| sdake_ | use docker inspect to see i yu see any deltas | 23:49 |
| *** mwheckmann has joined #openstack-kolla | 23:50 | |
| *** sbezverk has quit IRC | 23:52 | |
| *** ozialien has joined #openstack-kolla | 23:54 | |
| *** sbezverk has joined #openstack-kolla | 23:56 | |
| *** ravig has quit IRC | 23:57 | |
| *** ravig has joined #openstack-kolla | 23:58 | |
| *** sdake__ has joined #openstack-kolla | 23:58 | |
| *** ravig has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
