| kfox1111 | well, I think the complication is going to be in the live upgrading workflows each of the openstack components come up with. | 00:00 |
|---|---|---|
| alanmeadows | is that it would be better if it could provide the gap as k8s functionality is missing to make this seamless with its own upgrade/rollback workflows | 00:00 |
| kfox1111 | the workflow's gotta be implementable in whatever workflow helm provides. | 00:00 |
| alanmeadows | that maybe naive | 00:00 |
| kfox1111 | or needs a k8s extention to do that logic. | 00:00 |
| alanmeadows | but thats where my thinking is until I get deeper into it | 00:00 |
| kfox1111 | yeah. | 00:00 |
| alanmeadows | right so ideally one says | 00:01 |
| kfox1111 | I've been coming from the, I want to put this into production as soon as I can. which means, | 00:01 |
| alanmeadows | i want to go fully native for as much k8s work as I can do for each component | 00:01 |
| kfox1111 | manual workflow. which I'm good with. :) | 00:01 |
| kfox1111 | cause I'm already manualy deploying openstack with packages. so k8s openstack is a huge step up. :) | 00:01 |
| alanmeadows | but there will be gaps and something has to fill it, hoping helm can (probably not right now) | 00:01 |
| * kfox1111 nods | 00:01 | |
| alanmeadows | and also whats going to help with a larger upgrade orchestration, again _maybe_ helm, or maybe thats asking too much | 00:01 |
| alanmeadows | don't know, being optimistic ;-) | 00:02 |
| kfox1111 | yeah. k8s seems to really have a good toolset for deploying webapps. and openstack's pretty much a web app. | 00:02 |
| kfox1111 | with the possible exeption of the sql db stuff. | 00:02 |
| kfox1111 | its just a hugely complicated webapp. :) | 00:02 |
| *** ayoung has quit IRC | 00:02 | |
| alanmeadows | right the devil is the orchestration because of the spaghetti ties | 00:02 |
| alanmeadows | and for a super robust approach | 00:02 |
| kfox1111 | I have no doubt helm will be able to orchestrate a web ap that complicated eventually. | 00:03 |
| alanmeadows | enough state saving so you _can_ rollback | 00:03 |
| kfox1111 | just not sure when. if it can now, or if it will take a while. | 00:03 |
| kfox1111 | yeah. | 00:03 |
| kfox1111 | having one superpackage for the whole deployment is cool for deployment. | 00:03 |
| kfox1111 | pretty scary for day2 when you have to fix one little thing in one k8s object. | 00:03 |
| alanmeadows | yeah based on the documentation i've absorbed one big superpackage for upgrades doesn't seem like something helm has really architected just yet | 00:04 |
| kfox1111 | you could tweak the super pakcage and upgrade. but it carries the resk of breaking everything. | 00:04 |
| *** msimonin1 has quit IRC | 00:04 | |
| alanmeadows | seems like you could stitch some magic together with the pre and post hooks too | 00:04 |
| kfox1111 | yeah. | 00:04 |
| alanmeadows | which I had to discover with the ceph chart | 00:04 |
| kfox1111 | jobs are really cool. :) | 00:04 |
| alanmeadows | and jobs ;-) | 00:05 |
| kfox1111 | I've yet to try out the cronjob type but it looks really nice too. | 00:05 |
| kfox1111 | I'm planning on giving that a try for fernet token rotation at some point. | 00:05 |
| alanmeadows | the concept of hooks is there | 00:05 |
| alanmeadows | https://github.com/att-comdev/aic-helm/blob/master/ceph/templates/pre-install-secret.yaml | 00:05 |
| alanmeadows | so that gives me hope | 00:05 |
| alanmeadows | just needs expansion | 00:06 |
| jascott1 | if we were to ugprade sub chart and then upgrade whole release, is helm smart enough to just run package that has version change? | 00:06 |
| kfox1111 | yeah. that was new since I looked at it last. that will go a long way. | 00:06 |
| kfox1111 | jascott1: not sure.... | 00:06 |
| alanmeadows | my adventures with upgrades is really thin | 00:06 |
| kfox1111 | jascott1: I also asked what the behavior was for upgrading a daemonset. | 00:06 |
| alanmeadows | but I plan to go there soon | 00:06 |
| kfox1111 | haven't heard back. | 00:06 |
| kfox1111 | alanmeadows: it seems to have been added only since alpha3 so very very new. :/ | 00:07 |
| alanmeadows | yup ;-) | 00:07 |
| kfox1111 | I'm worried it will delete daemonsets on upgrade. | 00:07 |
| *** asalkeld has joined #openstack-kolla | 00:07 | |
| *** Pavo has quit IRC | 00:07 | |
| alanmeadows | you'd think it'd be smarter with a state container | 00:07 |
| alanmeadows | but who knows :) | 00:07 |
| kfox1111 | yeah. being so new, I'd guess they have only covered the common cases. but not sure. | 00:08 |
| kfox1111 | which is why I've been playing it conservitive, and looking at packaging all the things individualy. | 00:08 |
| alanmeadows | i look at this way | 00:08 |
| kfox1111 | if I deploy it that way, I can upgrade precicely the things I want to, and leave the rest alone. | 00:08 |
| alanmeadows | by the time we chart everything properly | 00:08 |
| alanmeadows | in a way thats operationalized | 00:09 |
| alanmeadows | hopefully more superpackage work | 00:09 |
| alanmeadows | will have made it in ;-) | 00:09 |
| kfox1111 | and for those that want a super pakcage, we can include those ina an overarchign package with deps. | 00:09 |
| kfox1111 | yeah. I'm hoping that will be the case. :) | 00:09 |
| jascott1 | from a helm pov i could see an openstack install looking like several component releases | 00:09 |
| kfox1111 | though converting template number 2 took about 30 min. | 00:09 |
| alanmeadows | true, once youve got a pattern ... | 00:10 |
| kfox1111 | I think we could get all the existing kolla-kubernetes templates into helm in proably a few days. | 00:10 |
| alanmeadows | i also think even the idea of a superpackage | 00:10 |
| alanmeadows | is a bit of a hack in helm | 00:10 |
| *** zhurong has joined #openstack-kolla | 00:10 | |
| kfox1111 | yeah. | 00:10 |
| alanmeadows | it needs some more thought | 00:10 |
| kfox1111 | I was thinking something like: | 00:10 |
| kfox1111 | openstack package -> compute kit package -> keystone/nova/glance/cinder/neutron -> l3 package, openvswitch-agent, etc | 00:11 |
| alanmeadows | the fact that for e.g. openstack-helm has a middle layer of an uber openstack chart to keep the actual region overrides from having their requirements shifting is a bit kludgy | 00:11 |
| alanmeadows | it works | 00:11 |
| kfox1111 | coudl have a "advanced services package" with stuff like sahara,trove,magnum, | 00:11 |
| kfox1111 | etc. | 00:11 |
| srwilker | I was working on mariadb/keystone today to get familiar with Helm. I'm sure someone might beat me to the punch, but hoping to have it pushed this evening | 00:12 |
| *** asalkeld has quit IRC | 00:12 | |
| kfox1111 | srwilker: have a look at this: https://review.openstack.org/#/c/396296/ | 00:12 |
| *** v1k0d3n has joined #openstack-kolla | 00:12 | |
| alanmeadows | kfox1111: I think breaking it out that way may help upgrades later | 00:12 |
| *** Pavo has joined #openstack-kolla | 00:12 | |
| kfox1111 | I finished retooling it today. should make it easier to add more. | 00:12 |
| kfox1111 | alanmeadows: yeah. | 00:12 |
| srwilker | kfox1111: I was going off that after awhile. It's proved helpfup | 00:13 |
| srwilker | Helpful | 00:13 |
| kfox1111 | srwilker: cool. :) | 00:13 |
| rhallisey | kfox1111, what pieces of the helm work are covered in your patch? | 00:13 |
| rhallisey | looks like neutron | 00:13 |
| kfox1111 | rhallisey: a copule of templates from neutron to prove out the code sharing, and build system will work. | 00:13 |
| rhallisey | ok | 00:13 |
| rhallisey | I'll make bp's for each service so we can split that work up | 00:14 |
| kfox1111 | it does successfully share the common-lib between multiple templates. so should be easy to convert the rest. | 00:14 |
| rhallisey | doing the same for operators | 00:14 |
| rhallisey | ok | 00:14 |
| kfox1111 | cool. :) | 00:14 |
| srwilker | Going to snooze for the openstack meetup. Be back online in a few hours | 00:16 |
| kfox1111 | srwilker: cool. l8r. | 00:17 |
| *** v1k0d3n has quit IRC | 00:17 | |
| *** hfu has joined #openstack-kolla | 00:17 | |
| *** masterdubs has joined #openstack-kolla | 00:22 | |
| *** masterdubs has quit IRC | 00:23 | |
| *** zhurong has quit IRC | 00:24 | |
| *** asalkeld has joined #openstack-kolla | 00:27 | |
| *** eaguilar has quit IRC | 00:27 | |
| *** asalkeld has quit IRC | 00:32 | |
| *** yingjun has joined #openstack-kolla | 00:32 | |
| jascott1 | will we try to ship pvc resources? I read somewhere it was recommended not to | 00:33 |
| kfox1111 | unknown... | 00:33 |
| kfox1111 | they kind of feel like configmaps to me. | 00:34 |
| kfox1111 | I was kind of thinking of making pvc configurable for rabbit too. | 00:34 |
| kfox1111 | for rpc rabbits, being stateless is ok. (maybe even prefereed) | 00:35 |
| *** tonanhngo has quit IRC | 00:39 | |
| *** yingjun has quit IRC | 00:40 | |
| *** yingjun has joined #openstack-kolla | 00:40 | |
| *** sean-k-mooney has quit IRC | 00:46 | |
| *** sean-k-mooney has joined #openstack-kolla | 00:46 | |
| *** ayoung has joined #openstack-kolla | 00:50 | |
| kfox1111 | oh... I wonder if the dynamic pvc support for ceph landed in 1.5.... | 00:52 |
| *** tonanhngo has joined #openstack-kolla | 00:54 | |
| kfox1111 | https://github.com/kubernetes/kubernetes/issues/36470 | 00:55 |
| *** v1k0d3n has joined #openstack-kolla | 00:55 | |
| *** tonanhngo_ has joined #openstack-kolla | 00:56 | |
| *** severion has joined #openstack-kolla | 00:59 | |
| *** tonanhngo has quit IRC | 00:59 | |
| *** tonanhngo_ has quit IRC | 01:00 | |
| *** yingjun has quit IRC | 01:01 | |
| *** v1k0d3n has quit IRC | 01:02 | |
| *** jrich523 has quit IRC | 01:04 | |
| *** ayoung has quit IRC | 01:06 | |
| *** jrich523 has joined #openstack-kolla | 01:08 | |
| *** eaguilar has joined #openstack-kolla | 01:09 | |
| *** rhallisey has quit IRC | 01:24 | |
| *** jemcevoy has quit IRC | 01:33 | |
| *** zhurong has joined #openstack-kolla | 01:36 | |
| *** zhugaoxiao has quit IRC | 01:47 | |
| *** zhangyufei has joined #openstack-kolla | 01:48 | |
| *** newmember has joined #openstack-kolla | 01:48 | |
| *** inc0 has joined #openstack-kolla | 01:50 | |
| *** severion has quit IRC | 01:56 | |
| *** v1k0d3n has joined #openstack-kolla | 01:56 | |
| *** v1k0d3n has quit IRC | 01:59 | |
| *** f13o has joined #openstack-kolla | 02:00 | |
| *** v1k0d3n has joined #openstack-kolla | 02:00 | |
| *** f13o has quit IRC | 02:06 | |
| *** inc0 has quit IRC | 02:07 | |
| *** Pavo has quit IRC | 02:07 | |
| *** sbezverk_ has joined #openstack-kolla | 02:08 | |
| *** sbezverk has quit IRC | 02:10 | |
| *** Pavo has joined #openstack-kolla | 02:12 | |
| QuentinM | kfox1111: you might be interested in https://github.com/kubernetes/kubernetes/pull/33944 | 02:17 |
| QuentinM | kfox1111: direct usage of the Cinder/Keystone APIs to mount/dynamically provision Cinder volumes in Kubernetes without the need to use the OpenStack provider: credentials are provided by a secret. | 02:18 |
| *** zhangyufei has quit IRC | 02:19 | |
| *** adrian_otto has quit IRC | 02:19 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Use kolla-ansible for deployment https://review.openstack.org/398506 | 02:22 |
| *** zhangyufei has joined #openstack-kolla | 02:27 | |
| *** v1k0d3n has quit IRC | 02:29 | |
| *** v1k0d3n has joined #openstack-kolla | 02:29 | |
| *** yingjun has joined #openstack-kolla | 02:42 | |
| *** severion has joined #openstack-kolla | 02:42 | |
| *** v1k0d3n has quit IRC | 02:46 | |
| *** dave-mccowan has joined #openstack-kolla | 02:46 | |
| *** yingjun_ has joined #openstack-kolla | 02:52 | |
| *** yingjun has quit IRC | 02:55 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 02:57 |
| *** unicell has quit IRC | 02:57 | |
| *** fragatina has quit IRC | 03:03 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Use kolla-ansible for deployment https://review.openstack.org/398506 | 03:03 |
| *** fragatina has joined #openstack-kolla | 03:05 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Move glance precheck into its own role https://review.openstack.org/399354 | 03:09 |
| *** fragatin_ has joined #openstack-kolla | 03:09 | |
| *** fragatina has quit IRC | 03:09 | |
| *** zhurong_ has joined #openstack-kolla | 03:10 | |
| *** zhurong has quit IRC | 03:12 | |
| *** fragatin_ has quit IRC | 03:14 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Add chrony ansible role https://review.openstack.org/399355 | 03:14 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Fix neutron.conf.j2 metadata_workers spelling error https://review.openstack.org/399356 | 03:18 |
| *** zhangyufei has quit IRC | 03:18 | |
| *** yingjun has joined #openstack-kolla | 03:26 | |
| *** yingjun_ has quit IRC | 03:28 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 03:29 |
| *** awiddersheim has joined #openstack-kolla | 03:31 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Remove build related env in tox.ini https://review.openstack.org/399360 | 03:31 |
| *** tonanhngo has joined #openstack-kolla | 03:41 | |
| *** zhurong_ has quit IRC | 03:42 | |
| *** eaguilar has quit IRC | 03:42 | |
| *** tonanhngo has quit IRC | 03:42 | |
| *** zhurong has joined #openstack-kolla | 03:43 | |
| *** tovin07 has joined #openstack-kolla | 03:55 | |
| *** dave-mccowan has quit IRC | 03:56 | |
| *** g3ek has quit IRC | 03:59 | |
| *** haplo37_ has quit IRC | 04:00 | |
| *** tonanhngo has joined #openstack-kolla | 04:03 | |
| *** tonanhngo has quit IRC | 04:05 | |
| *** Pavo has quit IRC | 04:07 | |
| *** sp_ has joined #openstack-kolla | 04:11 | |
| *** Pavo has joined #openstack-kolla | 04:12 | |
| openstackgerrit | Surya Prakash Singh proposed openstack/kolla: Consistent home directory creation for all the services https://review.openstack.org/390179 | 04:12 |
| *** imcsk8 has quit IRC | 04:17 | |
| *** imcsk8 has joined #openstack-kolla | 04:18 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 04:29 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' to neccesary tasks (general roles) https://review.openstack.org/398682 | 04:34 |
| *** adrian_otto has joined #openstack-kolla | 04:35 | |
| *** coolsvap has joined #openstack-kolla | 04:37 | |
| *** ayoung has joined #openstack-kolla | 04:43 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 05:02 |
| *** unicell has joined #openstack-kolla | 05:03 | |
| *** severion has quit IRC | 05:07 | |
| *** v1k0d3n has joined #openstack-kolla | 05:08 | |
| *** v1k0d3n has quit IRC | 05:11 | |
| *** v1k0d3n has joined #openstack-kolla | 05:12 | |
| *** v1k0d3n has quit IRC | 05:17 | |
| *** v1k0d3n has joined #openstack-kolla | 05:18 | |
| *** srwilker has quit IRC | 05:37 | |
| *** akscram has quit IRC | 05:46 | |
| *** akscram has joined #openstack-kolla | 05:47 | |
| *** haplo37 has quit IRC | 05:52 | |
| *** prameswar has joined #openstack-kolla | 05:55 | |
| coolsvap | bmace: done | 05:55 |
| bmace | coolsvap: done? :) | 05:57 |
| coolsvap | bmace: updated the overview on kolla-ansible lp | 05:57 |
| bmace | coolsvap: aaah, ok, that message was really old. lost context. | 05:58 |
| coolsvap | sorry I think i was disconnected in between so my session started with that as the last message on the channel | 05:59 |
| *** zhangyufei has joined #openstack-kolla | 05:59 | |
| bmace | i moved the ansible stop changes over into the kolla-ansible repo and made a note on how you can easily use it against a single or list of hosts: https://review.openstack.org/#/c/399289/1 | 05:59 |
| *** v1k0d3n has quit IRC | 06:00 | |
| *** haplo37 has joined #openstack-kolla | 06:04 | |
| *** g3ek has joined #openstack-kolla | 06:05 | |
| *** Pavo has quit IRC | 06:07 | |
| *** markmcclain has quit IRC | 06:09 | |
| openstackgerrit | Cao Xuan Hoang proposed openstack/kolla-kubernetes: Remove white space between print () https://review.openstack.org/399381 | 06:10 |
| *** markmcclain has joined #openstack-kolla | 06:10 | |
| *** Pavo has joined #openstack-kolla | 06:11 | |
| *** haplo37 has quit IRC | 06:13 | |
| *** g3ek has quit IRC | 06:14 | |
| *** adrian_otto has quit IRC | 06:18 | |
| *** g3ek has joined #openstack-kolla | 06:26 | |
| *** haplo37 has joined #openstack-kolla | 06:26 | |
| *** msimonin has joined #openstack-kolla | 06:28 | |
| *** msimonin has quit IRC | 06:28 | |
| *** zhangyufei has quit IRC | 06:29 | |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-ansible: Add blank space to deploy.yml file https://review.openstack.org/399389 | 06:36 |
| *** pc_m has quit IRC | 06:38 | |
| *** zhangyufei has joined #openstack-kolla | 06:40 | |
| *** tonanhngo has joined #openstack-kolla | 06:58 | |
| *** tonanhngo has quit IRC | 07:00 | |
| *** haplo37_ has joined #openstack-kolla | 07:05 | |
| *** Satya_ has joined #openstack-kolla | 07:08 | |
| bjolo | morning | 07:15 |
| *** zhangyufei has quit IRC | 07:15 | |
| *** DTadrzak has joined #openstack-kolla | 07:18 | |
| *** zhangyufei has joined #openstack-kolla | 07:21 | |
| *** msimonin has joined #openstack-kolla | 07:22 | |
| *** msimonin has quit IRC | 07:22 | |
| openstackgerrit | Surya Prakash Singh proposed openstack/kolla: Closes-Bug: #1082248 https://review.openstack.org/399411 | 07:29 |
| openstack | bug 1082248 in kolla "Use uuidutils instead of uuid.uuid4()" [Wishlist,In progress] https://launchpad.net/bugs/1082248 - Assigned to Surya Prakash Singh (confisurya) | 07:29 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Allow neutron-fwaas to be enabled in Neutron https://review.openstack.org/398336 | 07:29 |
| *** sdake_ has joined #openstack-kolla | 07:32 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 07:34 |
| *** hfu has quit IRC | 07:38 | |
| *** hfu has joined #openstack-kolla | 07:39 | |
| openstackgerrit | Merged openstack/kolla-ansible: Remove build related env in tox.ini https://review.openstack.org/399360 | 07:40 |
| sdake_ | sup peeps | 07:49 |
| *** lukl has quit IRC | 07:49 | |
| *** f13o has joined #openstack-kolla | 07:52 | |
| *** imcsk8 has quit IRC | 07:56 | |
| *** imcsk8 has joined #openstack-kolla | 07:56 | |
| *** duonghq has joined #openstack-kolla | 07:57 | |
| *** v1k0d3n has joined #openstack-kolla | 08:00 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 08:02 |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** openstackgerrit has joined #openstack-kolla | 08:03 | |
| sdake_ | duonghq ok dude, i'm back :) | 08:03 |
| sdake_ | duonghq whats up | 08:03 |
| duonghq | sdake_, atm, gerrit link to bp is working, last few days, the link is broken | 08:05 |
| *** v1k0d3n has quit IRC | 08:05 | |
| sdake_ | you mean when you subit something to gerrit it does't go to t he blueprint? | 08:06 |
| sdake_ | for which repo/luanhcpad? | 08:06 |
| duonghq | when I clicked the blueprint link in commit message, it said that bp is not existed | 08:06 |
| duonghq | for kolla-ansible | 08:06 |
| duonghq | but atm, everything is ok | 08:06 |
| sdake_ | ok | 08:07 |
| sdake_ | either someone else fixed it | 08:07 |
| sdake_ | or it fixed itself :) | 08:07 |
| *** Pavo has quit IRC | 08:07 | |
| duonghq | I have believe that it fixed has self-healing ability | 08:07 |
| duonghq | (j/k) | 08:07 |
| coolsvap | sdake_: duonghq kolla-ansible was not part of openstack earlier | 08:08 |
| coolsvap | so the link was not working | 08:08 |
| duonghq | coolsvap, understood, | 08:08 |
| *** pc_m has joined #openstack-kolla | 08:09 | |
| duonghq | but kolla-ansible gate still broken? | 08:09 |
| duonghq | just to confirm | 08:09 |
| sdake_ | coolsvap ya that would do it | 08:09 |
| sdake_ | coolsvap thta was the first thingi was goin to check actually :) | 08:09 |
| coolsvap | i did that when i first encoutered the issue | 08:10 |
| duonghq | thank coolsvap | 08:10 |
| *** matrohon has joined #openstack-kolla | 08:12 | |
| *** Pavo has joined #openstack-kolla | 08:12 | |
| sdake_ | duonghq unknown - Jeffrey4l has been working on it | 08:12 |
| sdake_ | Jeffrey4l need any help to ge t the gate unblcoked? | 08:12 |
| sdake_ | I just woke up for 12 hrs of hibernation | 08:12 |
| sdake_ | here to help | 08:12 |
| sdake_ | for/from | 08:13 |
| Jeffrey4l | sdake_, no. it almost be done. | 08:13 |
| sdake_ | nice | 08:13 |
| Jeffrey4l | kolla gate is OK https://review.openstack.org/398506 | 08:13 |
| duonghq | sdake_, cool | 08:13 |
| sdake_ | Jeffrey4l yup i saw kolla gate is ok | 08:14 |
| *** duonghq has quit IRC | 08:15 | |
| *** hogepodge has quit IRC | 08:15 | |
| sdake_ | disagree re globals and passwords going in kolla-ansible | 08:18 |
| openstackgerrit | wangwei proposed openstack/kolla: Add the function of docker login before pushing to registry https://review.openstack.org/399426 | 08:18 |
| sdake_ | or atleast ithink its open to discussion | 08:18 |
| openstackgerrit | Merged openstack/kolla: Use kolla-ansible for deployment https://review.openstack.org/398506 | 08:18 |
| sdake_ | Jeffrey4l ^ | 08:19 |
| Jeffrey4l | thanks. | 08:21 |
| sdake_ | no i mean see above re disagree Jeffrey4l | 08:21 |
| Jeffrey4l | re globals and passwords: why? any reason? it is only used by ansible. | 08:22 |
| Jeffrey4l | sdake_, | 08:22 |
| openstackgerrit | Merged openstack/kolla: Fix missing libvirt python module in ceilometer-compute (v2) https://review.openstack.org/399145 | 08:22 |
| sdake_ | globals and passwords re used by kubernetes as well | 08:22 |
| sdake_ | the idea is to hvae one config file to rule them all | 08:22 |
| sdake_ | that way operators dont hve to learn two ways to config things | 08:22 |
| Jeffrey4l | kubernets also use config file generated by kolla-ansible. this is the root cause. | 08:23 |
| Jeffrey4l | sdake_, | 08:23 |
| *** zhubingbing has joined #openstack-kolla | 08:26 | |
| zhubingbing | hello guys | 08:26 |
| sdake_ | sup zhubingbing | 08:27 |
| zhubingbing | hi | 08:27 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 08:28 |
| *** Satya_ has quit IRC | 08:28 | |
| *** Satya_ has joined #openstack-kolla | 08:29 | |
| coolsvap | Jeffrey4l: did you get the release script running? | 08:29 |
| Jeffrey4l | coolsvap, yep. it works as expected ;) | 08:29 |
| Jeffrey4l | btw, i am using arch, and run this script in a python virtual env. | 08:29 |
| coolsvap | Jeffrey4l: cool i will try that once more maybe next time | 08:30 |
| coolsvap | i just saw a flurry of mails changing milestone, thought to ask you | 08:30 |
| Satya_ | Hi Jeffrey | 08:31 |
| Satya_ | regarding rabbitmq issue | 08:31 |
| Jeffrey4l | coolsvap, you can try it now. like move all unfinished bug into o2. | 08:31 |
| Satya_ | we have a solution now | 08:31 |
| Jeffrey4l | even though nothing will happen. but you can make sure the script runs successfully. | 08:31 |
| Satya_ | which can have rabbit run in a mgmt interface rather than api | 08:32 |
| Jeffrey4l | Satya_, realy? how? | 08:32 |
| *** zhangyufei has quit IRC | 08:32 | |
| *** msimonin has joined #openstack-kolla | 08:32 | |
| Satya_ | we changed the kolla code and added a new parameter as rabbit_interface | 08:32 |
| Jeffrey4l | coolsvap, yep. | 08:32 |
| Satya_ | can we push that to upstream? | 08:32 |
| sdake_ | dsyqupdytrsm pld | 08:32 |
| Jeffrey4l | i do not think so :( | 08:32 |
| Satya_ | Hi Sdake | 08:32 |
| sdake_ | damn | 08:32 |
| sdake_ | Satya_ can ou run a git review on it plz | 08:33 |
| sdake_ | Satya_ i really would like to see this happen | 08:33 |
| sdake_ | rabbitmq should not be running on the api interface | 08:33 |
| Satya_ | sure | 08:33 |
| Jeffrey4l | it is more a edge cause i think. | 08:33 |
| *** STOP_nwo has joined #openstack-kolla | 08:33 | |
| STOP_nwo | hi | 08:33 |
| Jeffrey4l | sdake_, all port running on the api interface, like mysql, mongodb. | 08:33 |
| sdake_ | Jeffrey4l i know | 08:34 |
| STOP_nwo | where u from ppl | 08:34 |
| sdake_ | the infra needs to not run on the api interface | 08:34 |
| sdake_ | STOP_nwo huh? | 08:34 |
| Jeffrey4l | there is no mgmt interface concept in kolla. but the api interface is. | 08:34 |
| Jeffrey4l | the really api_interface is the extenral api interface. | 08:34 |
| sdake_ | Jeffrey4l right - so lets add a mgmt interface | 08:34 |
| Satya_ | we changed all other components to use the mgmt interface (rabbit_interface) for transport URL | 08:35 |
| Satya_ | http://paste.openstack.org/show/589680/ | 08:35 |
| *** zhangyufei has joined #openstack-kolla | 08:35 | |
| Jeffrey4l | then what's the difference between api interface, mgmt interface and external vip interface? | 08:35 |
| sdake_ | api interface = internal interface on which api runs | 08:36 |
| *** egonzalez90 has joined #openstack-kolla | 08:36 | |
| sdake_ | mgmt interface = internal interface on which mgmt infrasturcture runs | 08:36 |
| sdake_ | external vip interface is the VIP for mapping external connections to internal VIP | 08:36 |
| sdake_ | is that right | 08:37 |
| Jeffrey4l | how to define the mgmt infra? non-openstack service? | 08:37 |
| sdake_ | just woke up and haven't had my dose of caffeine yet | 08:37 |
| Jeffrey4l | ;) | 08:37 |
| sdake_ | Jeffrey4l right, non-openstack service | 08:37 |
| Jeffrey4l | hmm, we set authencation on both openstack service and non-openstack service. i do not think we need separate this two kind of service. | 08:39 |
| sdake_ | its not about auth | 08:39 |
| sdake_ | its about network traffic | 08:39 |
| Jeffrey4l | ok. let met think | 08:40 |
| *** msimonin has left #openstack-kolla | 08:40 | |
| *** zhangyufei has quit IRC | 08:41 | |
| egonzalez90 | Jeffrey4l: morning, globals.yml is going to be at kolla-ansible repo. Is not going to be shared between ansible and k8s? | 08:45 |
| openstackgerrit | wangwei proposed openstack/kolla: Add the function of docker login before pushing https://review.openstack.org/399426 | 08:45 |
| *** tovin07_ has joined #openstack-kolla | 08:45 | |
| *** tovin07 has quit IRC | 08:45 | |
| Jeffrey4l | egonzalez90, the question is why kolla-k8s need such file. the answer is kolla-k8s need kolla-ansible's genconfig features. | 08:46 |
| Jeffrey4l | so it is nothing to do with where is the globals.yml and passwords.yml file. | 08:46 |
| Satya_ | https://bugs.launchpad.net/kolla/+bug/1642878 | 08:46 |
| openstack | Launchpad bug 1642878 in kolla "RabbitMQ should communicate through a different network rather than api_network" [Undecided,New] | 08:46 |
| *** zhubingbing has quit IRC | 08:48 | |
| *** zhangyufei has joined #openstack-kolla | 08:50 | |
| sdake_ | https://www.youtube.com/watch?v=NGOvH1T-dsA | 08:52 |
| sdake_ | Jeffrey4l so, where the file goes i am less interested in | 08:52 |
| sdake_ | Jeffrey4l what i'm intereted in is that is is consistent between deliverables | 08:53 |
| Jeffrey4l | sdake_, what's that means? more specific? | 08:53 |
| *** shardy has joined #openstack-kolla | 08:53 | |
| sdake_ | Jeffrey4l same config file for kolla-ansible as kolla-kubernetes | 08:54 |
| sdake_ | how that happens, idc :) | 08:54 |
| sdake_ | i'm ok with C&P in fact :) | 08:54 |
| sdake_ | although that is not idela | 08:54 |
| sdake_ | ideal | 08:55 |
| Jeffrey4l | hmm. hate C&P. we can not maintain two files in two repo. | 08:55 |
| sdake_ | Jeffrey4l right, not ideal | 08:55 |
| sdake_ | Jeffrey4l so then the question is where does th file go | 08:55 |
| sdake_ | we are talking about putting the default config in the kolla repo | 08:56 |
| Jeffrey4l | not the file. ( globals or passwords. ) but all the tempaltes files. | 08:56 |
| sdake_ | (for each service) | 08:56 |
| sdake_ | Jeffrey4l sorrry dude if I seem dense, i'm sort of exhausted | 08:56 |
| sdake_ | Jeffrey4l but i didn't grok tha lst sensense | 08:56 |
| sdake_ | sentence | 08:58 |
| bmace | fyi sdake_ i added you to a re-review of the stop changes. i got them all moved over into the new kolla-ansible repo | 08:59 |
| Jeffrey4l | the kolla-k8s and kolla-ansible hasn't the same configuration for certain service. there are lots if-else in the configuration file. if we put it in common place ( like kolla ), it will be full of if-else. | 08:59 |
| Jeffrey4l | but i have no better idea now ;( | 09:00 |
| *** narasimha_SV has joined #openstack-kolla | 09:00 | |
| portdirect | gate for kolla good to go? | 09:00 |
| Jeffrey4l | portdirect, yep. | 09:00 |
| *** zhangyufei has quit IRC | 09:00 | |
| portdirect | whoot! cheers Jeffrey4l :) | 09:00 |
| sdake_ | Jeffrey4l we need to abstract all that if-then-else crap out into one place | 09:00 |
| sdake_ | Jeffrey4l the reason ew ehae if else atm is because of things like ansible_* veriable | 09:00 |
| narasimha_SV | hi sorry to ask like this . I dont find ansible folder in master branch of kolla | 09:01 |
| sdake_ | narasimha_SV git checkout http://github.com/openstack/kolla-ansible | 09:01 |
| *** zhurong has quit IRC | 09:01 | |
| *** ChanServ sets mode: +o sdake_ | 09:01 | |
| *** sdake_ is now known as sdake | 09:01 | |
| *** ChanServ sets mode: +o sdake | 09:01 | |
| narasimha_SV | ok | 09:01 |
| narasimha_SV | thanks sdake | 09:02 |
| *** zhurong has joined #openstack-kolla | 09:02 | |
| *** sdake changes topic to "Looking for Ansible deployment? git checkout http://github.com/openstack/kolla-ansible; New to Kolla? Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA):" | 09:02 | |
| *** gfidente has joined #openstack-kolla | 09:03 | |
| *** gfidente has joined #openstack-kolla | 09:03 | |
| Jeffrey4l | sdake, that's will be optimal. | 09:04 |
| sdake | Jeffrey4l right | 09:04 |
| Jeffrey4l | need out. brb | 09:04 |
| *** liyifeng has quit IRC | 09:06 | |
| *** rmart04 has joined #openstack-kolla | 09:12 | |
| openstackgerrit | narasimha18sv proposed openstack/kolla-ansible: typo error of keyring spelling https://review.openstack.org/399455 | 09:12 |
| *** zhangyufei has joined #openstack-kolla | 09:15 | |
| *** zhangyufei has joined #openstack-kolla | 09:15 | |
| *** strigazi_AFK is now known as strigazi | 09:15 | |
| narasimha_SV | hi is there anyone who has worked on GNOCCHI here ? | 09:16 |
| narasimha_SV | I need some help | 09:16 |
| *** liyifeng has joined #openstack-kolla | 09:18 | |
| Satya_ | hi | 09:22 |
| openstackgerrit | wangwei proposed openstack/kolla: Add the function of docker login before pushing https://review.openstack.org/399426 | 09:24 |
| *** zhangyufei has quit IRC | 09:29 | |
| *** papacz has joined #openstack-kolla | 09:31 | |
| *** shardy has quit IRC | 09:34 | |
| *** shardy has joined #openstack-kolla | 09:34 | |
| *** yingjun has quit IRC | 09:38 | |
| *** yingjun has joined #openstack-kolla | 09:38 | |
| *** yingjun has quit IRC | 09:38 | |
| *** yingjun has joined #openstack-kolla | 09:39 | |
| *** papacz has quit IRC | 09:39 | |
| *** yingjun has quit IRC | 09:39 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: TrivialFix: Remove unused code https://review.openstack.org/368569 | 09:40 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Clean imports in code https://review.openstack.org/368592 | 09:40 |
| *** STOP_nwo has quit IRC | 09:42 | |
| *** hfu has quit IRC | 09:43 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Remove white space between print () https://review.openstack.org/399381 | 09:48 |
| *** papacz has joined #openstack-kolla | 09:49 | |
| *** derekjhyang has quit IRC | 09:51 | |
| *** zhurong has quit IRC | 09:59 | |
| *** Pavo has quit IRC | 10:07 | |
| *** Pavo has joined #openstack-kolla | 10:11 | |
| *** tovin07_ has quit IRC | 10:26 | |
| *** Satya_ has quit IRC | 10:27 | |
| *** hieulq has quit IRC | 10:28 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible: Ignore dom0 qemu processes during destroy https://review.openstack.org/399203 | 10:28 |
| *** fragatina has joined #openstack-kolla | 10:35 | |
| *** fragatina has quit IRC | 10:36 | |
| *** fragatina has joined #openstack-kolla | 10:37 | |
| *** ppalacios has joined #openstack-kolla | 10:40 | |
| *** senk has joined #openstack-kolla | 10:44 | |
| *** liyifeng has quit IRC | 10:45 | |
| *** sdake has quit IRC | 10:48 | |
| pbourke | bjolo: morning, bad news for you :p | 10:55 |
| pbourke | bjolo: seems to work with centos/newton also. have dumped lots of info on the bug for you | 10:55 |
| *** DTadrzak has quit IRC | 11:04 | |
| *** ppalacios1 has joined #openstack-kolla | 11:05 | |
| *** ppalacios1 has quit IRC | 11:06 | |
| *** ppalacios has quit IRC | 11:07 | |
| openstackgerrit | Javier Castillo AlcÃbar proposed openstack/kolla-ansible: Fix missing heka/heka-ceilometer.toml in ceilometer_compute https://review.openstack.org/399134 | 11:09 |
| openstackgerrit | Javier Castillo AlcÃbar proposed openstack/kolla-ansible: Included new temmplates https://review.openstack.org/399525 | 11:09 |
| *** sdake has joined #openstack-kolla | 11:16 | |
| *** prameswar has quit IRC | 11:24 | |
| sdake | rbergeron around? | 11:30 |
| bjolo | pbourke, wtf! | 11:30 |
| bjolo | you are supposed to FAIL :) | 11:31 |
| sdake | rbergeron thanks for teacing me about this youtube feature: https://youtu.be/mqT66AVml48?t=3m10s | 11:31 |
| bjolo | pbourke, tnx so much for taking the time to do this | 11:34 |
| bjolo | i will really read through your setup and see why we fail | 11:34 |
| pbourke | bjolo: np, sorry you're having so much trouble with it | 11:34 |
| *** khamtamtun has joined #openstack-kolla | 11:35 | |
| bjolo | off the top of my head you use real interfaces and i use vlan interfaces but that should not cause any problems | 11:35 |
| bjolo | aaaahhh maybe it does | 11:36 |
| bjolo | all vlan interfaces share the same mac address by default | 11:37 |
| bjolo | [root@eselde02u37 kolla]# ip l | grep -B1 bond0 | grep link | tail -n 3 | 11:40 |
| bjolo | link/ether e4:1d:2d:bc:cc:c0 brd ff:ff:ff:ff:ff:ff | 11:40 |
| bjolo | link/ether e4:1d:2d:bc:cc:c0 brd ff:ff:ff:ff:ff:ff | 11:40 |
| bjolo | maybe that throws off neutron agent | 11:40 |
| openstackgerrit | Pete Birley proposed openstack/kolla: Update Percona repo to current release https://review.openstack.org/399549 | 11:40 |
| bjolo | some of OF rules are based on mac addresses right | 11:41 |
| bjolo | pbourke, you still have you env up? | 11:41 |
| bjolo | if so, can you please dump the flows on br-int and both external bridges | 11:42 |
| pbourke | bjolo: i've gone back to the oraclelinux/master setup but that should be ok | 11:42 |
| bjolo | yes | 11:42 |
| pbourke | bjolo: one sec | 11:42 |
| bjolo | since it works in both setups | 11:42 |
| *** prameswar has joined #openstack-kolla | 11:50 | |
| *** eaguilar has joined #openstack-kolla | 11:52 | |
| sdake | Jeffrey4l around | 11:52 |
| *** hfu has joined #openstack-kolla | 11:59 | |
| *** yingjun has joined #openstack-kolla | 12:02 | |
| *** srwilkers has joined #openstack-kolla | 12:03 | |
| *** Pavo has quit IRC | 12:07 | |
| sdake | Jeffrey4l if you appear (and havn'et gone to bed) could you let me know what state of gate is. All reviews currently blocked in kolla-ansible, and want to know if there is anything I can do to help get that unblocked (including writing pc code) :) | 12:09 |
| *** Pavo has joined #openstack-kolla | 12:12 | |
| *** khamtamtun has quit IRC | 12:12 | |
| sdake | wow kolla #3 on reviews for ocata cycle atm | 12:13 |
| sdake | right behind neutron and nova ;) | 12:13 |
| sdake | #4 on commits | 12:13 |
| pbourke | bjolo: those logs are there now | 12:16 |
| *** sdake_ has joined #openstack-kolla | 12:16 | |
| openstackgerrit | Javier Castillo AlcÃbar proposed openstack/kolla-ansible: updated heka-ceilometer template file https://review.openstack.org/399566 | 12:18 |
| *** sdake has quit IRC | 12:20 | |
| mgoddard | hi, has anyone come across issues when using the kolla-ansible reconfigure command where containers are reconfigured unnecessarily because of permissions errors? | 12:21 |
| *** derekjhyang has joined #openstack-kolla | 12:21 | |
| mgoddard | in */tasks/reconfigure.yaml the kolla_set_configs command is run without sudo, whereas sudo -E is used in kolla_start. | 12:22 |
| mgoddard | (we have a modified config.json that copies in a non-root unreadable file) | 12:23 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Clean up loc https://review.openstack.org/399570 | 12:25 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Remove init-runonce from docker repo https://review.openstack.org/399572 | 12:28 |
| *** prameswar has quit IRC | 12:29 | |
| *** hogepodge has joined #openstack-kolla | 12:31 | |
| *** rhallisey has joined #openstack-kolla | 12:33 | |
| Jeffrey4l | sdake_, https://review.openstack.org/398501 patch set 13 works expect oracle related jobs. i am debugging the oracle jobs. | 12:36 |
| * Jeffrey4l is taking care of son. | 12:36 | |
| Jeffrey4l | should fix this today. | 12:36 |
| pbourke | Jeffrey4l: let me know if I can help | 12:38 |
| pbourke | Jeffrey4l: looking at PS 13 but all still seem to be failed | 12:38 |
| pbourke | including centos | 12:38 |
| Jeffrey4l | https://review.openstack.org/#/c/398501/ ps 13 works except oracle. | 12:41 |
| *** tonanhngo has joined #openstack-kolla | 12:41 | |
| Jeffrey4l | did u open the correct url? | 12:42 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Clean up README.rst in docker repo https://review.openstack.org/399582 | 12:42 |
| Jeffrey4l | it is weird now. oracle jobs say ' OSError: [Errno 2] No such file or directory' http://logs.openstack.org/01/398501/13/check/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/06cda4a/console.html#_2016-11-18_07_09_25_236923 | 12:43 |
| Jeffrey4l | i have no idea what file he required. | 12:43 |
| *** tonanhngo has quit IRC | 12:43 | |
| Jeffrey4l | if u have any idea. it will be very helpful. | 12:43 |
| sdake_ | Jeffrey4l let me have a look at tthe review and see if anthing pops out | 12:44 |
| sdake_ | Jeffrey4l ps13 is red on all deploy gates | 12:44 |
| Jeffrey4l | sdake_, patch 13 is not the latest ps. | 12:44 |
| sdake_ | Jeffrey4l yup i know | 12:45 |
| Jeffrey4l | click toggle ci button at the bottom. | 12:45 |
| sdake_ | i toggle'd ci | 12:45 |
| sdake_ | and i see it is passing some gates | 12:45 |
| sdake_ | wierd how it is showing in main gate screen failures | 12:45 |
| Jeffrey4l | sdake_, because it only show the last PS's result. ( i added some test lines now. so the last/latest ps all failed. ) | 12:46 |
| sdake_ | http://logs.openstack.org/01/398501/13/check/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/06cda4a/console.html#_2016-11-18_07_09_25_233810 | 12:46 |
| sdake_ | Jeffrey4l i used /13 at end of url... so that shouldn't show latest patchset | 12:47 |
| sdake_ | but ok got it :) | 12:47 |
| *** Jeffrey4l has quit IRC | 12:47 | |
| sdake_ | pbourke I see r/home/jenkins | 12:47 |
| sdake_ | that seems wrong? | 12:48 |
| pbourke | sdake_: -r | 12:48 |
| sdake_ | ok well there needs to be a spce in there | 12:48 |
| pbourke | its subprocess that's raising the exception so I think it may be trying to execute a binary that's not there | 12:48 |
| sdake_ | subprocess finds the process | 12:49 |
| sdake_ | or there would not be that huge backtrace | 12:49 |
| sdake_ | what it isn't finding is the requirements.txt file and test-requirements.txt | 12:49 |
| sdake_ | is -rrequirements.txt an acceptable usage? | 12:49 |
| sdake_ | my best guess at this point is wrong version of pip | 12:50 |
| sdake_ | lemme pull that repo and see what tox.ini looks like | 12:50 |
| pbourke | python -c 'import subprocess; subprocess.check_output("foo")' | 12:51 |
| pbourke | same stacktrace | 12:51 |
| sdake_ | pbourke ok | 12:52 |
| *** srwilkers has quit IRC | 12:52 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 12:52 |
| sdake_ | next best guess is lack of pip in the system | 12:52 |
| sdake_ | as in centos ships it, oracle linux doesn't | 12:53 |
| sdake_ | lemme pull down jeffrey's patch now | 12:53 |
| openstackgerrit | Pete Birley proposed openstack/kolla: DO NOT MERGE!: Remove unneeded PythonMySQL libs from OpenStack Base Image https://review.openstack.org/399042 | 12:54 |
| *** liyifeng has joined #openstack-kolla | 12:55 | |
| openstackgerrit | Pete Birley proposed openstack/kolla: DO NOT MERGE!: Remove unneeded PythonMySQL libs from OpenStack Base Image https://review.openstack.org/399042 | 12:55 |
| *** yingjun has quit IRC | 12:57 | |
| *** yingjun has joined #openstack-kolla | 12:57 | |
| *** yingjun has quit IRC | 13:01 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Clean up loc https://review.openstack.org/399570 | 13:02 |
| *** magicboiz has quit IRC | 13:05 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Allow neutron-fwaas to be enabled in Neutron https://review.openstack.org/398336 | 13:06 |
| *** senk has quit IRC | 13:07 | |
| *** Jeffrey4l has joined #openstack-kolla | 13:07 | |
| Jeffrey4l | pbourke, about https://review.openstack.org/398336 the releasenote is parsed to {"Add support for neutron-fwaas. Set 'enable_neutron_fwaas": "yes' to enable."} | 13:08 |
| sdake_ | Jeffrey4l quick q, what is current speculation on why oraclelinux is failing? | 13:10 |
| sdake_ | ls | 13:11 |
| Jeffrey4l | sdake_, no idea. | 13:11 |
| Jeffrey4l | i suspend the some software version is different. but oracle and centos are using the same base diso. | 13:12 |
| sdake_ | right | 13:12 |
| sdake_ | i just put that together ;) | 13:12 |
| Jeffrey4l | in the latest ps. i print all rpm version and pip freeze. | 13:12 |
| sdake_ | looking at tox.ini from patchset 13 | 13:13 |
| sdake_ | - testr run test_build.DeployTestCentosBinary | 13:13 |
| sdake_ | oh nm thats there for oraclelinux as well | 13:14 |
| sdake_ | brain still booting :) | 13:14 |
| sdake_ | Jeffrey4l did you use pip to print out the software versions of the python stuff too? | 13:14 |
| sdake_ | how about selinux? | 13:15 |
| Jeffrey4l | rpm will print the python version. | 13:15 |
| sdake_ | rpm doesn't print python version if pip is used to do the install | 13:15 |
| *** lamt has joined #openstack-kolla | 13:15 | |
| sdake_ | it prints the rpm version of the package not the pip version fof the package | 13:15 |
| sdake_ | is audit log captured anywhere? | 13:16 |
| sdake_ | (re selinux idea) | 13:16 |
| Jeffrey4l | latest logs http://logs.openstack.org/01/398501/17/check/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/fc6dcf9/console.html | 13:16 |
| Jeffrey4l | sdake_, no ;( | 13:17 |
| sdake_ | can you turn off selinux so we can try that line of thinking | 13:17 |
| Jeffrey4l | ok. i can try in next patch. | 13:17 |
| sdake_ | and eliminate it as a cause | 13:17 |
| sdake_ | cool | 13:17 |
| Jeffrey4l | anything else u want to catch? | 13:17 |
| *** dave-mccowan has joined #openstack-kolla | 13:18 | |
| sdake_ | Jeffrey4l http://logs.openstack.org/01/398501/17/check/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/fc6dcf9/console.html#_2016-11-18_13_06_18_605075 | 13:19 |
| sdake_ | what is the $ about at the end of the tox command? | 13:19 |
| sdake_ | wather in the middle | 13:19 |
| sdake_ | rather in the middle | 13:19 |
| sdake_ | looks super suspicious | 13:20 |
| Jeffrey4l | may be. | 13:20 |
| sdake_ | /home/jenkins/workspace/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/.tox$ | 13:20 |
| Jeffrey4l | it should the current workspace. | 13:21 |
| *** zhubingbing has joined #openstack-kolla | 13:21 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Remove docker from kolla-ansible https://review.openstack.org/398320 | 13:21 |
| Jeffrey4l | it created the deploy-oraclelinux-binary virtualenv successfully. | 13:21 |
| Jeffrey4l | but when calling pip command with failed. | 13:21 |
| sdake_ | Jeffrey4l check this out | 13:22 |
| sdake_ | http://logs.openstack.org/01/398501/17/check/gate-kolla-ansible-dsvm-deploy-centos-source-centos-7-nv/1b617ca/console.html#_2016-11-18_13_05_08_272878 | 13:22 |
| sdake_ | centos source | 13:23 |
| sdake_ | is doing something with oraclelinux? | 13:23 |
| Jeffrey4l | sdake_, it is work. in the patch, i write `tox -e oraclexxxx for all gate` | 13:23 |
| Jeffrey4l | it is ok. | 13:23 |
| Jeffrey4l | check the changed i made in the latest patch. | 13:24 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 13:25 |
| *** msimonin has joined #openstack-kolla | 13:26 | |
| sdake_ | i have a better speculatiion now | 13:26 |
| sdake_ | https://review.openstack.org/#/c/398501/17/tools/setup_gate.sh | 13:26 |
| sdake_ | see line 32 | 13:26 |
| Jeffrey4l | invocation failed (errno 2), args: ['/home/jenkins/workspace/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/.tox/deploy-oraclelinux-binary/bin/pip', 'install', '-U', '-r/home/jenkins/workspace/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/requirements.txt', '-r/home/jenkins/workspace/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/test-requirements.txt'], cwd: /home/jenkins/workspace/gate-kolla-ansib | 13:26 |
| Jeffrey4l | le-dsvm-deploy-oraclelinux-binary-centos-7-nv | 13:26 |
| sdake_ | sudo only works in predefined paths | 13:27 |
| sdake_ | Jeffrey4l throwing ideas at you :) | 13:28 |
| Jeffrey4l | but line 32 works. | 13:28 |
| sdake_ | https://review.openstack.org/#/c/398501/17/tox.ini | 13:30 |
| sdake_ | see line 156 | 13:31 |
| sdake_ | compare to orcle linux implementation of tox.ini | 13:31 |
| Jeffrey4l | hmm isn't it the same? | 13:32 |
| sdake_ | see line 168 | 13:33 |
| Jeffrey4l | left or right? | 13:33 |
| sdake_ | left | 13:33 |
| Jeffrey4l | we have sudo in whitelist_extennals | 13:33 |
| sdake_ | in oraclelinux yes, in ubuntu no | 13:33 |
| Jeffrey4l | so? | 13:33 |
| Jeffrey4l | yes. but centos has sudo. | 13:34 |
| Jeffrey4l | and centos and ubuntu works. | 13:34 |
| *** sdake has joined #openstack-kolla | 13:36 | |
| sdake | Jeffrey4l i'll take out sudo and try that in a different changeset with your work | 13:36 |
| sdake | can you try selinux approach? | 13:37 |
| Jeffrey4l | ok. | 13:37 |
| sdake | (ie diable) | 13:37 |
| Jeffrey4l | sorry? | 13:37 |
| sdake | disable selinux on all centos | 13:37 |
| Jeffrey4l | i am check the tox source code, and try to see when it may raise such exception. | 13:37 |
| Jeffrey4l | sdake, ok. | 13:37 |
| sdake | that way we got two jobs going in the gates at once | 13:37 |
| *** sdake_ has quit IRC | 13:38 | |
| Jeffrey4l | yep. | 13:39 |
| *** sdake_ has joined #openstack-kolla | 13:40 | |
| sdake_ | centos has sudo in the whilelist externals | 13:40 |
| sdake_ | which is odd | 13:40 |
| sdake_ | wonder why ubuntu even works ;) | 13:40 |
| *** sdake has quit IRC | 13:43 | |
| *** srwilker has joined #openstack-kolla | 13:44 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 13:48 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Remove bandit references from tox.ini https://review.openstack.org/399597 | 13:50 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Remove Ansible references from tox.ini https://review.openstack.org/399597 | 13:50 |
| *** senk has joined #openstack-kolla | 13:51 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-ansible: Remove docker reference related to bandit from tox.ini https://review.openstack.org/399600 | 13:52 |
| *** dave-mcc_ has joined #openstack-kolla | 13:53 | |
| sdake_ | dave-mcc_ around? | 13:54 |
| Jeffrey4l | sdake_, http://logs.openstack.org/01/398501/19/check/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/1818a09/console.html#_2016-11-18_13_52_31_897571 | 13:55 |
| dave-mcc_ | sdake_ good morning! | 13:55 |
| sdake_ | dave-mcc_ sup dave | 13:55 |
| Jeffrey4l | need more debugging. | 13:55 |
| *** dave-mccowan has quit IRC | 13:55 | |
| sdake_ | dave-mcc_ jeffrey and i have a question | 13:55 |
| sdake_ | dave-mcc_ in kolla, mariadb and rabbitmq bind to the api interface | 13:55 |
| sdake_ | i would htink it makes more sense to bind to the management interface | 13:56 |
| sdake_ | kolla has no management interface | 13:56 |
| sdake_ | thoughts on that? | 13:56 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 13:56 |
| dave-mcc_ | are you asking a question on nomenclature? | 13:56 |
| dave-mcc_ | or on design? | 13:56 |
| *** krtaylor has quit IRC | 13:57 | |
| Jeffrey4l | dave-mcc_, design | 13:58 |
| sdake_ | dave-mcc_ design | 14:00 |
| sdake_ | our api interface is what the apis bind to | 14:00 |
| dave-mcc_ | Jeffrey4l sdake_ kolla calls the internal interface "api_interface". so, the design is as you expect: maria and rabbit are on the internal network. | 14:00 |
| dave-mcc_ | sdake_ it's what the internal apis bind to | 14:01 |
| sdake_ | dave-mcc_ so its secure? (main concern here) | 14:01 |
| sdake_ | Jeffrey4l i see you turned off selinux and it looks like it still doesn't work properly | 14:01 |
| Jeffrey4l | sdake_, yep. | 14:01 |
| sdake_ | from that log | 14:02 |
| dave-mcc_ | sdake_ a prudent operator will isolate the internal network (and thus all the api_interfaces). no one should talk to those except internal services. | 14:02 |
| sdake_ | dave-mcc_ cool | 14:02 |
| sdake_ | so api_interface should really be called internal_interface | 14:02 |
| dave-mcc_ | sdake_ the external network is the one humans talk to. those endpoints are protected by TLS. | 14:02 |
| dave-mcc_ | sdake_ yea. that would save the confusion. it's really a question of nomenclature. | 14:03 |
| sdake_ | ok well i guess thatship hsa sailed | 14:03 |
| sdake_ | as long as its secure | 14:03 |
| sdake_ | ;) | 14:03 |
| *** fguillot has joined #openstack-kolla | 14:03 | |
| Jeffrey4l | sdake_, check this line http://logs.openstack.org/01/398501/20/check/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/fd95181/console.html#_2016-11-18_14_03_49_727371 | 14:04 |
| Jeffrey4l | hmm interesting. the interpreter is cut. | 14:06 |
| Jeffrey4l | pip: /home/jenkins/workspace/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-cent: bad interpreter: No such file or directory | 14:06 |
| sdake_ | Jeffrey4l did project config change? | 14:06 |
| dave-mcc_ | sdake_ Jeffrey4l i was just reading haproxy.cfg.j2 to double check. it's still secure, but i describe it wrong. | 14:06 |
| Jeffrey4l | i do not think so. and centos works. centos and oraclelinux using the same base image, which is 'cento-7 | 14:07 |
| sdake_ | if project-config was changed incorrectly (someone may have fatfingerered) | 14:07 |
| *** Pavo has quit IRC | 14:07 | |
| sdake_ | antoher possibility is the length is too long for either zuul or tox | 14:07 |
| sdake_ | count the characters :) | 14:07 |
| dave-mcc_ | sdake_ Jeffrey4l api_interface is internal only, but no one should send to it directly. all API traffic goes to either the internal VIP or external VIP. HAProxy unwraps TLS and send the external traffic to the internal network. | 14:08 |
| sdake_ | dave-mcc_ that sounds correct | 14:08 |
| sdake_ | thanks | 14:08 |
| Jeffrey4l | but the weird things is centos works. | 14:08 |
| sdake_ | oraclelinux > centos in terms of length | 14:09 |
| pbourke | http://stackoverflow.com/questions/10813538/shebang-line-limit-in-bash-and-linux-kernel | 14:09 |
| pbourke | his example works on ubuntu but not on oraclelinux for me | 14:09 |
| dave-mcc_ | sdake_ Jeffrey4l the only access an external user should have to an api_interface is through HAProxy, and there is no route for maria or rabbit to go from external_vip to api_interface. | 14:09 |
| Jeffrey4l | why test oraclelinux, it is using centos image. Linux centos-7-rax-iad-5452306 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | 14:10 |
| pbourke | Jeffrey4l: its more to test the docker images | 14:10 |
| Jeffrey4l | pbourke, it is prepare test virtual env on host os. | 14:11 |
| pbourke | hmm | 14:11 |
| pbourke | what sdake said, len(oraclelinux) > len(centos) | 14:11 |
| dave-mcc_ | sdake_ Jeffrey4l when most devs use a single network or all-in-one, this nuance might get lost. but, the internal network should a non-routable on a different interface than the external network. | 14:12 |
| Jeffrey4l | ok. anyway. need fix this at first. then we can try to find why centos works but oracle linux not. | 14:12 |
| sdake_ | Jeffrey4l can you make some kind of hack to convert oraclelinux to "ol" in tox.ini | 14:12 |
| sdake_ | ot test out the limit theory | 14:12 |
| *** Pavo has joined #openstack-kolla | 14:12 | |
| Jeffrey4l | dave-mcc_, if he only use one network, add `mgmt` network ( or something else ) won't help. | 14:13 |
| sdake_ | it will probably require a change to tox.ini and setup_gate.sh | 14:13 |
| sdake_ | Jeffrey4l right i think dave_mcc agrees :) | 14:13 |
| Jeffrey4l | fyi, the shebang is cut at 80 character ;( | 14:14 |
| sdake_ | which shebang do you speak of | 14:14 |
| Jeffrey4l | pip. | 14:15 |
| pbourke | Jeffrey4l: create an alias or symlink | 14:15 |
| *** senk has quit IRC | 14:15 | |
| pbourke | then sed that into setup.sh | 14:15 |
| Jeffrey4l | check this and the following two lines http://logs.openstack.org/01/398501/20/check/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/fd95181/console.html#_2016-11-18_14_03_49_727371 | 14:15 |
| *** v1k0d3n has joined #openstack-kolla | 14:16 | |
| *** zhurong has joined #openstack-kolla | 14:18 | |
| sdake_ | pbourke symlink for which? | 14:20 |
| pbourke | sdake_: for the tox pip | 14:20 |
| dave-mcc_ | Jeffrey4l sdake_ if an operator puts the external VIP on the same network as the internal interfaces, he's going to have a bad day. maybe he could add iptables commands to secure it? | 14:21 |
| sdake_ | pbourke don't totally follow other then its a rift on my suggestoin of working around the limit via some kind of hacking :) | 14:21 |
| pbourke | pretty much :) | 14:21 |
| Jeffrey4l | pbourke, it is caused by the content of pip ( first line, shebang), not the location of pip | 14:21 |
| pbourke | ln -s <big-long-path/pip> /tmp/pip | 14:21 |
| pbourke | sed -i s#<big-long-path/pip>#/tmp/pip#g setup.sh | 14:22 |
| sdake_ | pbourke pip produces a shell script it appears | 14:23 |
| sdake_ | that pip produced shell script is defective | 14:23 |
| *** msimonin has quit IRC | 14:23 | |
| pbourke | then sed that | 14:23 |
| sdake_ | sed it where? | 14:23 |
| sdake_ | rather not pip, tox | 14:24 |
| Jeffrey4l | a possible solution is change the pip shebang to : #!/usr/bin/env python | 14:24 |
| pbourke | same idea I just got the scripts wrong | 14:24 |
| Jeffrey4l | since the venv path is in the first location. | 14:24 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Remove docker from kolla-ansible https://review.openstack.org/398320 | 14:24 |
| pbourke | that sounds good too Jeffrey4l | 14:24 |
| Jeffrey4l | it should works. | 14:24 |
| sdake_ | see working together - found the root cause ;) | 14:25 |
| Jeffrey4l | but now, i curiosity why it won't work in oracle ;( | 14:25 |
| *** severion has joined #openstack-kolla | 14:25 | |
| sdake_ | oraclelinux > centos | 14:25 |
| sdake_ | centos probably right at the very limit | 14:25 |
| Jeffrey4l | i do not think we found the root cause. | 14:25 |
| *** goldyfruit has joined #openstack-kolla | 14:25 | |
| Jeffrey4l | oraclelinux and centos are using the same base image (centos-7) | 14:25 |
| sdake_ | yes but oracleinux passed in the schebang taks up more space | 14:26 |
| sdake_ | then centos does | 14:26 |
| Jeffrey4l | sdake_, there is no oraclelinux thing during that issue happen. | 14:26 |
| sdake_ | this causes kernel not to understand the command | 14:26 |
| Jeffrey4l | the oraclelinux is only used in docker image/container. | 14:26 |
| sdake_ | Jeffrey4l http://logs.openstack.org/01/398501/20/check/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/fd95181/console.html#_2016-11-18_14_03_49_729198 | 14:26 |
| sdake_ | oraclelinux is right ther ein that schebang | 14:26 |
| Jeffrey4l | it is the python virtualenv name. | 14:27 |
| sdake_ | gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/ | 14:27 |
| Jeffrey4l | generated by tox -edeploy-oraclelinux | 14:27 |
| Jeffrey4l | gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv this is generated by CI scripts. automatically. | 14:27 |
| sdake_ | look at hte log that i posted | 14:28 |
| *** v1k0d3n has quit IRC | 14:28 | |
| sdake_ | its got the schebang command from pip right in it | 14:28 |
| *** jheroux has joined #openstack-kolla | 14:28 | |
| sdake_ | oraclelinux = 11 charts, centos = 6 | 14:28 |
| sdake_ | each used twice | 14:28 |
| Jeffrey4l | yes. but this is in *centos-7* image, still. it is nothing to do with oraclelinux. | 14:28 |
| sdake_ | charts/chars | 14:28 |
| Jeffrey4l | sorry? | 14:28 |
| Jeffrey4l | 11 6? | 14:29 |
| sdake_ | characaters | 14:29 |
| sdake_ | ok... | 14:29 |
| sdake_ | #!/home/jenkins/workspace/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/.tox/deploy-oraclelinux-binary/bin/python | 14:29 |
| sdake_ | do you se eabove how oracleinux is used in the schebang? | 14:29 |
| Jeffrey4l | hmm. let count. seem so. | 14:29 |
| *** severion has quit IRC | 14:30 | |
| *** zhurong has quit IRC | 14:30 | |
| Jeffrey4l | ➤ echo -n ' #!/home/jenkins/workspace/gate-kolla-ansible-dsvm-deploy-centos-binary-centos-7-nv/.tox/deploy-centos-binary/bin/python' | wc | 14:30 |
| Jeffrey4l | 0 1 120 | 14:30 |
| sdake_ | right less then 128 chars | 14:30 |
| Jeffrey4l | there are 120 char for in centos case. | 14:30 |
| sdake_ | ole > 128 chars | 14:30 |
| *** derekjhyang has quit IRC | 14:31 | |
| Jeffrey4l | but the interpreate is cut at 80 chart, not 128 chat | 14:31 |
| *** duonghq has joined #openstack-kolla | 14:31 | |
| Jeffrey4l | echo -n '#!/home/jenkins/workspace/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-cent' |wc <--- 80 | 14:31 |
| sdake_ | according to that stackoverflow link above its cut at 128 chars | 14:31 |
| *** inc0 has joined #openstack-kolla | 14:32 | |
| inc0 | gmorning | 14:32 |
| sdake_ | where are yo getting the schebang with 80 chars from jeffrey4l, which log? | 14:32 |
| sbezverk_ | rhallisey: ping | 14:33 |
| rhallisey | yo | 14:33 |
| inc0 | how do we do with voting gates?:) | 14:33 |
| Jeffrey4l | sdake_, this line http://logs.openstack.org/01/398501/20/check/gate-kolla-ansible-dsvm-deploy-oraclelinux-binary-centos-7-nv/fd95181/console.html#_2016-11-18_14_03_49_727371 | 14:33 |
| sbezverk_ | rhallisey: building helm for openvswitch. Should I create bp for this or it is done under one common bp for all helm charts? | 14:34 |
| rhallisey | sbezverk_, create a bp per service | 14:34 |
| rhallisey | its easier to track and distribute the work | 14:34 |
| sdake_ | Jeffrey4l that is not the schebang in the ctual pip file | 14:34 |
| inc0 | rhallisey, can you please help me with puppet? | 14:34 |
| sbezverk_ | rhallise: cool, will do thanks | 14:34 |
| sdake_ | that is the output | 14:34 |
| sdake_ | of zuul | 14:34 |
| rhallisey | if someone want to create bps for each helm piece, I'd super appreciate it | 14:34 |
| sdake_ | the reason its cut at 80 chars, who knows ;) | 14:34 |
| *** zhurong has joined #openstack-kolla | 14:35 | |
| rhallisey | inc0, what's the issue ur having? | 14:35 |
| inc0 | https://review.openstack.org/#/c/399221/ <- we need to make this work | 14:35 |
| Jeffrey4l | sdake_, i test on centos-7 just now. it do cut at 80 char. ;( | 14:35 |
| inc0 | to enable reasonable gates for both kolla-ansible and k8s | 14:35 |
| portdirect | egonzalez90: you about? | 14:37 |
| sdake_ | inc0 lets get the gates working first to unblock reviews in general ;) | 14:37 |
| egonzalez90 | portdirect: yes | 14:37 |
| inc0 | so registry would make it a lot easier | 14:38 |
| sdake_ | not really | 14:38 |
| sdake_ | it would make it more reliable | 14:38 |
| sdake_ | easier = incremental | 14:38 |
| inc0 | easier too | 14:38 |
| inc0 | because zuul cloner won't be requirement | 14:39 |
| portdirect | egonzalez90: trying to get desgniate up, and have a few qs (I'll message you to leave these cats alone :) ) | 14:39 |
| Jeffrey4l | inc0, how to trigger the deployment gate by using registry? | 14:39 |
| egonzalez90 | portdirect: ok | 14:39 |
| inc0 | Jeffrey4l, trigger? | 14:39 |
| inc0 | what do you mean? | 14:39 |
| *** liyifeng has quit IRC | 14:39 | |
| *** sdake has joined #openstack-kolla | 14:39 | |
| sdake | inc0 zuul cloner is a 1 liner and is already implemented | 14:40 |
| Jeffrey4l | inc0, u do not want to use zuul cloner, right? | 14:40 |
| inc0 | yeah, for cross-repo gates | 14:40 |
| sdake | Jeffrey4l lets just get the ansible gets unblocked plz so revies can start rolling | 14:40 |
| Jeffrey4l | how to test kolla-ansible when kolla is changed? | 14:40 |
| sdake | Jeffrey4l you do that by putting kolla-ansible gates in kolla itself... | 14:41 |
| sdake | this is call cross repo gating | 14:41 |
| Jeffrey4l | how to solve such case: dockerfile change and only when the kolla ansible is changed. | 14:41 |
| sdake | fungi mentioned this in a previous thread | 14:41 |
| Jeffrey4l | sdake, yep. but this is implemented by using zuul-cloner. if not, how we implement such by using registry? inc0 | 14:42 |
| inc0 | Jeffrey4l, not this | 14:42 |
| sdake | Jeffrey4l inc0 is talking about going the toher way | 14:42 |
| *** msimonin has joined #openstack-kolla | 14:42 | |
| inc0 | what we can do with registry is ansible gates voting and easier to manage | 14:42 |
| sdake | when a commit hits kolla-asnible, a registry makes things more reliable | 14:42 |
| inc0 | as we won't need to build images in ansible | 14:42 |
| *** hfu has quit IRC | 14:43 | |
| Jeffrey4l | inc0, sdake got. sorry for misunderstand ;( | 14:43 |
| sdake | Jeffrey4l no reason to apologieze | 14:43 |
| Jeffrey4l | and yes. it will be helpful :) | 14:43 |
| sdake | Jeffrey4l still, lets get ansible unblocked ;) | 14:43 |
| sdake | PLZ | 14:43 |
| sdake | i can't review a damn thing atm | 14:43 |
| *** sdake_ has quit IRC | 14:43 | |
| sdake | so back on topic | 14:43 |
| sdake | this 80 char thing vs 128 char thing | 14:43 |
| Jeffrey4l | fyi sdake shebang issue in pip source code. https://github.com/pypa/pip/issues/1773 | 14:43 |
| *** msimonin has quit IRC | 14:43 | |
| openstackgerrit | Merged openstack/kolla: Clean up loc https://review.openstack.org/399570 | 14:44 |
| *** hfu has joined #openstack-kolla | 14:45 | |
| sdake | Jeffrey4l its a kernel limit | 14:45 |
| Jeffrey4l | ye. | 14:45 |
| Jeffrey4l | yes. | 14:45 |
| *** lamt has quit IRC | 14:46 | |
| Jeffrey4l | OK. i got the solution. | 14:46 |
| Jeffrey4l | will push a patch to test. | 14:46 |
| sdake | nice | 14:46 |
| sdake | off to breakfast bbl | 14:47 |
| Jeffrey4l | but still care about why centos works not oraclelinux not. | 14:47 |
| *** msimonin has joined #openstack-kolla | 14:49 | |
| *** berendt has joined #openstack-kolla | 14:49 | |
| berendt | first customer site deployed with kolla :) | 14:49 |
| Jeffrey4l | berendt, cool. | 14:49 |
| inc0 | congrats:) | 14:50 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 14:51 |
| *** lamt has joined #openstack-kolla | 14:51 | |
| openstackgerrit | Merged openstack/kolla: Remove init-runonce from docker repo https://review.openstack.org/399572 | 14:52 |
| *** jtriley has joined #openstack-kolla | 14:53 | |
| pbourke | berendt: nice | 14:54 |
| pbourke | berendt: how many nodes? | 14:54 |
| berendt | at the beginning about 20 nodes | 14:55 |
| *** zhurong has quit IRC | 14:55 | |
| *** v1k0d3n has joined #openstack-kolla | 14:56 | |
| *** adrian_otto has joined #openstack-kolla | 14:57 | |
| *** Satya_ has joined #openstack-kolla | 14:58 | |
| Satya_ | Hi sdake_ you there? | 14:58 |
| sdake | berendt sweet news | 14:59 |
| sdake | berendt imo send a cut :) | 15:00 |
| sdake | Satya_ sup | 15:00 |
| Satya_ | Please triage "https://bugs.launchpad.net/kolla-ansible/+bug/1642878" | 15:01 |
| openstack | Launchpad bug 1642878 in kolla-ansible "RabbitMQ should communicate through a different network rather than api_network" [Undecided,New] - Assigned to Satya Sanjibani Routray (satroutr) | 15:01 |
| sdake | Satya_ i am out of the triaging business for awhile, sorry | 15:01 |
| inc0 | Satya_, that's not a bug... | 15:01 |
| Satya_ | ohh ok... | 15:01 |
| sdake | Satya_ we have a whole slew of core revieweres and other folks that can triage ping them :) | 15:02 |
| inc0 | and we don't have anything called "management network" | 15:02 |
| inc0 | this is called api_network in Kolla | 15:02 |
| sdake | Satya_ check logs of this discsussion with dave_mcc today | 15:03 |
| Satya_ | ok i will edit for secondary ID | 15:03 |
| sdake | Satya_ the problem is simple - api_network is a misnomer | 15:03 |
| sdake | it should be called "internal_network" | 15:03 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Remove tools/init-runonce in setup.cfg https://review.openstack.org/399629 | 15:03 |
| sdake | which is confusing to everyone | 15:03 |
| Jeffrey4l | sdake, inc0 pbourke check this please, it blocks the gate https://review.openstack.org/399629 | 15:04 |
| Satya_ | i added a piece as rabbitmq_interface | 15:04 |
| sdake | Jeffrey4l hrm i guess i messed that up | 15:04 |
| sdake | sorry about that - +2ed your change | 15:04 |
| Satya_ | so it will be flexible for api_network or any other internal network... | 15:04 |
| sdake | Jeffrey4l the worst part is I knew I needed to remove it from setup.cfg too | 15:04 |
| sdake | and then forgot about it 30 seconds later | 15:05 |
| Jeffrey4l | thanks ;) never mind | 15:05 |
| *** narasimha_SV has quit IRC | 15:05 | |
| pbourke | Jeffrey4l: will I approve? | 15:05 |
| sdake | pbourke i did | 15:05 |
| sdake | pbourke but ya it would hav ebeen fine for you to do so i think | 15:06 |
| Jeffrey4l | the fastest patch :) | 15:06 |
| sdake | obviously an error that i introduced about 30 mins ago | 15:06 |
| sdake | sorry guys tired from 36 hrs of bullshit | 15:06 |
| sdake | got 12 hrs sleep last night tho :) | 15:06 |
| sdake | yay | 15:06 |
| inc0 | Satya_, what's the use case for rabbitmq on dedicated network? | 15:06 |
| inc0 | rabbitmq traffic can't really saturate any relevant datacenter network, it won't add any security as every node has to have access to it anyway | 15:07 |
| Satya_ | usually in production everyone wants the traffic segrigation | 15:07 |
| inc0 | yes... | 15:07 |
| sdake | ya i get segration of traffic... for a reason :) | 15:07 |
| inc0 | api_interface exists | 15:07 |
| Satya_ | and the rabbitmq traffic should go with a different network rather than the api network | 15:08 |
| inc0 | why? | 15:08 |
| sdake | that is what api_interfae is for | 15:08 |
| sdake | Satya_ if we did that, then things like heat-engine would have to bind to it | 15:08 |
| Satya_ | yeah all communication is going through the api network | 15:08 |
| *** TxGirlGeek has joined #openstack-kolla | 15:08 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix ci gate https://review.openstack.org/398501 | 15:08 |
| sdake | so halfof openstack woudl have to communicate with some kind of wierd internal/external vip | 15:08 |
| inc0 | Satya_, all openstack service-to-service communication | 15:08 |
| sdake | which might reduce security | 15:08 |
| sdake | Satya_ security is ensured on kolla by the external_vip_network | 15:09 |
| sdake | Satya_ please read the logs from this morning | 15:09 |
| inc0 | yeah, if you're afraid that api_network has to be exposed externally - it doesn't | 15:09 |
| sdake | your questions asked and answered there :) | 15:09 |
| Satya_ | yep reading... | 15:09 |
| sdake | api_network shoudl be called internal_network | 15:09 |
| sdake | why is it called api_network? Unknown. | 15:09 |
| rhallisey | inc0, https://review.openstack.org/399221 | 15:09 |
| sdake | can we call it internal_network now, not sure | 15:10 |
| inc0 | w00t rhallisey | 15:10 |
| inc0 | thanks! | 15:10 |
| rhallisey | that should help get it going | 15:10 |
| rhallisey | that's closer | 15:10 |
| sdake | it sure would be alot simpler on everyone if it wasn't misnamed | 15:10 |
| rhallisey | you don't need too much | 15:10 |
| sdake | rhallisey knows the puppets? | 15:11 |
| Satya_ | ok we should have flexibility to use any network for rabbit "https://bugs.launchpad.net/kolla/+bug/1583655" | 15:11 |
| openstack | Launchpad bug 1583655 in kolla newton "rabbitmq setup from secondary IP" [Critical,Confirmed] | 15:11 |
| rhallisey | sdake, ye | 15:11 |
| sdake | Satya_ what youa re talking about is creating two internal interfaces | 15:11 |
| Satya_ | yes | 15:11 |
| sdake | one for the api internal interface, one for the servers internal interface | 15:11 |
| Satya_ | yes | 15:12 |
| inc0 | Satya_, tell me reason why anyone would want rabbitmq on separate iface please | 15:12 |
| Satya_ | where one is used for pxe booting the system and system gets the name from dns | 15:12 |
| Satya_ | and other is the network where openstack communicates | 15:12 |
| inc0 | well what you don't want to do is to put rabbitmq on pxe network | 15:12 |
| inc0 | because pxe *can* saturate network easily | 15:13 |
| Satya_ | yes | 15:13 |
| sdake | Satya_ ok, the problem with that model you just described is the api servers communicate with the actual servers servers :) | 15:13 |
| inc0 | and it will break your running openstack | 15:13 |
| Jeffrey4l | Satya_, did my solution work? enable the extend_interface. | 15:13 |
| sdake | Satya_ so those two networks would HAVE to be connected | 15:13 |
| Satya_ | yes | 15:13 |
| Jeffrey4l | you can move all your api_interface into your mgmt network. | 15:13 |
| Satya_ | jeffrey the solution worked after changing the hostname | 15:13 |
| sdake | ok cats - time for a smoke | 15:14 |
| sdake | bbl | 15:14 |
| Jeffrey4l | Satya_, hmm. my solution is move api interface, not changing hostname ;( | 15:14 |
| Satya_ | with that the problem will be if i am going for a periodic monitoring over the mgmt network | 15:14 |
| Satya_ | then api request will get slower | 15:14 |
| Satya_ | that didnt worked :( | 15:15 |
| Jeffrey4l | inc0, the root cause is rabbitmq hostname must be resolvable for the ip it listening. | 15:15 |
| Satya_ | jerrrey | 15:15 |
| Satya_ | yes... | 15:15 |
| inc0 | and hostname is set up for pxe network...? | 15:16 |
| Jeffrey4l | Satya_, if you just move the rabbitmq traffic to mgmt network, doesn't it become slow? | 15:17 |
| rhallisey | inc0, are you in #openstack-infra? | 15:17 |
| Satya_ | nope | 15:17 |
| inc0 | forgot to join | 15:17 |
| rhallisey | k | 15:17 |
| Satya_ | as it will be used only for pxe booting | 15:18 |
| Satya_ | may get the monitoring also but monitoring can be on a different network also | 15:18 |
| Jeffrey4l | ( so many limitation are show together in you env ;( ) | 15:18 |
| Satya_ | so rabbit network will be only dedicated for messaging and pxe boot | 15:19 |
| Satya_ | DavidTurner can give us more details also :) | 15:20 |
| Satya_ | DavidTurner_ there? | 15:20 |
| Jeffrey4l | Satya_, check this https://www.rabbitmq.com/configure.html | 15:20 |
| Jeffrey4l | search 'env hsotname' | 15:20 |
| Jeffrey4l | seems the rabbitmq can use a different hostname rather than the default one. | 15:21 |
| Jeffrey4l | if so, you can change the hostname to `*-api` without changing the real hostname. | 15:21 |
| Satya_ | we already tried that.. :( nodename dont take any effect... | 15:22 |
| Jeffrey4l | it is HOSTNAME, not NODENAME Satya_ | 15:22 |
| Jeffrey4l | RABBITMQ_NODENAME and HOSTNAME are two different variables. Satya_ | 15:23 |
| inc0 | Jeffrey4l, it will take ansible_hostname | 15:23 |
| *** prameswar has joined #openstack-kolla | 15:23 | |
| inc0 | what you can do is to force ansible_hostname to x in inventory | 15:23 |
| Satya_ | NODENAME=bunny@myhost | 15:23 |
| inc0 | but let me say that again, rabbitmq on pxe network is asking for troubles | 15:23 |
| Jeffrey4l | NameDefaultDescription | 15:24 |
| Jeffrey4l | HOSTNAME | 15:24 |
| Jeffrey4l | Unix, Linux: `env hostname` | 15:24 |
| Jeffrey4l | MacOSX: `env hostname -s` | 15:24 |
| Jeffrey4l | The name of the current machine | 15:24 |
| Jeffrey4l | sorry for multi line. | 15:24 |
| Jeffrey4l | but could u check this here https://www.rabbitmq.com/configure.html | 15:24 |
| Jeffrey4l | i think it should work. ( tbh, i never tried this ) | 15:24 |
| *** jtriley has quit IRC | 15:25 | |
| Satya_ | it says rabbitmq depends on hostname | 15:25 |
| Satya_ | but not specifying we can have a entry in the config so it will change the hostname of our wish... | 15:26 |
| Satya_ | am i right or reading that in a different way not sure :( | 15:27 |
| *** adrian_otto has quit IRC | 15:27 | |
| inc0 | Satya_, so hold on rabbitmq doesn't use api_interface IP really | 15:28 |
| Jeffrey4l | it says rabbitmq depends on HOSTNAME, and its default value is `env hostname`. | 15:28 |
| *** prameswar has quit IRC | 15:28 | |
| Satya_ | nope | 15:29 |
| Jeffrey4l | then i think it can be change. ( if not, that is no *default* concept ) | 15:29 |
| Satya_ | it uses the hostname configured | 15:29 |
| DaveTurner | Satya_ I'm here | 15:29 |
| Satya_ | Hi Dave :) | 15:29 |
| inc0 | it uses hostname | 15:30 |
| inc0 | exactly, and it will use hostname | 15:30 |
| Satya_ | yes | 15:30 |
| inc0 | and your hostname points to different interface than api | 15:30 |
| inc0 | so rabbitmq will work on it's own | 15:30 |
| inc0 | as for openstack services | 15:31 |
| inc0 | just use config override to point to rabbitmq you want | 15:31 |
| inc0 | and that should just work | 15:31 |
| Satya_ | and if a dns is configured and pxe boot will be done from that network the hostname will be a different than api if someone wants to use the api network different than the pxe network | 15:31 |
| inc0 | this somone is openstack services | 15:31 |
| Satya_ | nope | 15:32 |
| inc0 | so you can change configuration of these to use pxe address | 15:32 |
| Satya_ | yes | 15:32 |
| Satya_ | Dave can give more details about the situation we are in... | 15:33 |
| Satya_ | smoke time for me :) will be back in a bit... | 15:33 |
| inc0 | http://docs.openstack.org/developer/kolla/advanced-configuration.html#openstack-service-configuration-in-kolla | 15:33 |
| inc0 | Satya_, ^ | 15:33 |
| inc0 | use this override mechanism and override transport_url for rabbitmq to point to nodes over pxe | 15:34 |
| inc0 | and it should do what you want | 15:34 |
| DaveTurner | Satya_ This dependency on hostname is a concern for use, since we typically install our corporate monitoring tool on all hosts, which also has a dependency on host name- but we do not want the monitoring traffic running on same vlan as rabbit traffic. | 15:34 |
| DaveTurner | for *us | 15:34 |
| inc0 | DaveTurner, that's issue of rabbitmq tho, unfortunately it doesn't work with IPs | 15:35 |
| inc0 | nothing we can do about it | 15:35 |
| Satya_ | We have a working code which can help us actually.. | 15:35 |
| Satya_ | just want to check if we can put that for upstream for review... | 15:36 |
| Satya_ | that introduce a new interface which can be configured to the primary interface from which the hostname is configured | 15:38 |
| portdirect | I've worked round rabbit is issues similar to this by bind mounting the containers /etc/hosts to somewhere in /var/run and injectsing a custom config into the container - dont now if this may help you? | 15:39 |
| inc0 | yeah, that would actually work, override in-container /etc/hosts | 15:41 |
| *** hfu has quit IRC | 15:41 | |
| *** senk has joined #openstack-kolla | 15:41 | |
| Satya_ | That will actually help but that need a lot of manual intervention for changing the transport url for all the services... | 15:41 |
| Satya_ | :( | 15:41 |
| *** hfu has joined #openstack-kolla | 15:41 | |
| *** berendt has quit IRC | 15:41 | |
| *** hfu has quit IRC | 15:42 | |
| inc0 | yeah, best we can do atm | 15:42 |
| *** hfu has joined #openstack-kolla | 15:42 | |
| *** hfu has quit IRC | 15:43 | |
| *** hfu has joined #openstack-kolla | 15:43 | |
| Satya_ | atm? i know that as all time money/all time medicine/ a time machine | 15:43 |
| Satya_ | :P | 15:43 |
| *** hfu has quit IRC | 15:43 | |
| *** sdake has quit IRC | 15:44 | |
| *** hfu has joined #openstack-kolla | 15:44 | |
| *** senk has quit IRC | 15:44 | |
| *** hfu has quit IRC | 15:44 | |
| inc0 | at this time | 15:44 |
| inc0 | or at the moment | 15:44 |
| inc0 | rather | 15:44 |
| *** hfu has joined #openstack-kolla | 15:45 | |
| Satya_ | :) | 15:45 |
| *** senk has joined #openstack-kolla | 15:45 | |
| *** hfu has quit IRC | 15:45 | |
| *** senk has quit IRC | 15:45 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Mariadb Kubernetes Operator https://review.openstack.org/399263 | 15:45 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Kubernetes Operator base https://review.openstack.org/399262 | 15:45 |
| portdirect | with newer versions of docker it can get quite ugly, as the OCI runtime expects to be able to play with /etc/hosts - though you can work round this by bind mounting somewhere else and then cating the correct contents of the hosts file into place before starting rabbit - so yeah it's a horrible hack.. | 15:46 |
| portdirect | but works | 15:46 |
| Satya_ | and changing the transport url for all components | 15:47 |
| portdirect | sorry can't help there - only just starting to use kolla containers - I'm not using kolla-ansible I'm afraid :( | 15:49 |
| Satya_ | hmm | 15:49 |
| inc0 | changing transport url would be your best bet | 15:51 |
| inc0 | on the bright not it's not THAT much | 15:51 |
| inc0 | just similar file for every openstack service you're running | 15:51 |
| inc0 | so one for glance, one for neutron | 15:51 |
| *** sdake has joined #openstack-kolla | 15:51 | |
| inc0 | there are only so many services available | 15:51 |
| sdake | i wonder if you could run the univerise in a docker container | 15:52 |
| sdake | ;-) | 15:52 |
| inc0 | you can run vm in docker container and universe in vm | 15:52 |
| sdake | how did Jeffrey4l 's workaround go? | 15:53 |
| *** adrian_otto has joined #openstack-kolla | 15:54 | |
| *** tonanhngo has joined #openstack-kolla | 15:54 | |
| *** msimonin has quit IRC | 15:54 | |
| *** tonanhngo has quit IRC | 15:54 | |
| *** msimonin has joined #openstack-kolla | 15:54 | |
| *** msimonin has quit IRC | 15:56 | |
| Satya_ | inc0 i see transport url is in lot many places | 15:57 |
| *** srwilker has quit IRC | 15:57 | |
| Satya_ | http://paste.openstack.org/show/589726/ | 15:57 |
| *** zhubingbing has quit IRC | 15:57 | |
| inc0 | yeah..once per every service | 15:57 |
| inc0 | but do you run every service? | 15:57 |
| Satya_ | yes | 15:58 |
| Satya_ | including trying over gnocchi :P | 15:58 |
| *** jtriley has joined #openstack-kolla | 15:58 | |
| inc0 | ok, well, grep -ir transport_url | cut -d/ -f1 | 15:58 |
| inc0 | and do cp override file based on output | 15:59 |
| *** eaguilar has quit IRC | 15:59 | |
| *** msimonin has joined #openstack-kolla | 15:59 | |
| inc0 | every problem in the world can be solved by shell one-liner | 15:59 |
| *** eaguilar has joined #openstack-kolla | 15:59 | |
| Satya_ | lol yeah | 15:59 |
| Satya_ | but i have to do on all servers | 16:00 |
| inc0 | ansible then | 16:00 |
| inc0 | and no | 16:00 |
| inc0 | you don't | 16:00 |
| inc0 | just deployment node | 16:00 |
| Satya_ | currently i have 8 nodes | 16:01 |
| Satya_ | but i am scared if i say this solution to the ops guys they will get mad over us... | 16:01 |
| inc0 | Satya_, you do it only for deployment node | 16:04 |
| inc0 | so just tell them to copy this directory to etc and done:) | 16:04 |
| *** Pavo has quit IRC | 16:07 | |
| Satya_ | what about add a new interface as rabbitmq_interface and give the flexibility for user to change it anyways and default it to api_interface | 16:09 |
| *** Pavo has joined #openstack-kolla | 16:12 | |
| inc0 | it's not really bad change, but we'd need to do this across the board | 16:13 |
| Jeffrey4l | sdake, it should work. but wait for the init-once fix to move on. https://review.openstack.org/398501 | 16:13 |
| inc0 | still, to be honest I'd stick to what's available today | 16:13 |
| macsz | did anyone try to use gpu passthrough with a kolla-provisioned cloud? there is this bug -> https://bugs.launchpad.net/nova/+bug/1642419 and I am wandering if this is docker-related issue or sth is wrong with nova | 16:13 |
| openstack | Launchpad bug 1642419 in OpenStack Compute (nova) "GPU Passthrough isn't working" [Undecided,New] | 16:13 |
| sdake | macsz could be either | 16:14 |
| sbezverk_ | kfox1111: ping | 16:14 |
| sdake | i'd speculate gpu passthrough requires some type of host bindmount | 16:14 |
| sdake | i'd speculate nova gpu passthrough is not all that solid | 16:14 |
| sdake | just speculations :) | 16:14 |
| Satya_ | i have the change i can push | 16:15 |
| inc0 | I added that it also affects us | 16:15 |
| inc0 | I bet nova-compute somehow doesn't see gpu | 16:16 |
| macsz | sdake: that might be true :) i hoped to push the responsibility on other project (if, for instance you knew that gpu passthrough was not working in kolla deployments), but I guess it stays in it's home town for now :D | 16:16 |
| *** tonanhngo has joined #openstack-kolla | 16:16 | |
| openstackgerrit | Merged openstack/kolla: Remove tools/init-runonce in setup.cfg https://review.openstack.org/399629 | 16:16 |
| sdake | macsz i dont know anyone has ever tried | 16:17 |
| *** tonanhngo has quit IRC | 16:17 | |
| sdake | macsz what is your use case btw | 16:17 |
| inc0 | I know Sam had some fun with PCI passthru | 16:17 |
| inc0 | but he got it working | 16:18 |
| *** msimonin has quit IRC | 16:18 | |
| macsz | sdake: i am just doing a bug triage | 16:18 |
| sdake | macsz roger | 16:18 |
| Satya_ | macsz please triage "https://bugs.launchpad.net/kolla-ansible/+bug/1642878" | 16:19 |
| openstack | Launchpad bug 1642878 in kolla-ansible "RabbitMQ should communicate through a different network rather than api_network" [Undecided,New] - Assigned to Satya Sanjibani Routray (satroutr) | 16:19 |
| kfox1111 | morning | 16:19 |
| kfox1111 | sbezverk_: ping | 16:20 |
| sbezverk_ | kfox1111: hey, any chance you could separate common macro in a separate ps and push it I need it to complete openvswitch | 16:21 |
| *** msimonin has joined #openstack-kolla | 16:21 | |
| kfox1111 | sbezverk_: I think it would be better to do it as a dependency on my ps. that way, you can reuse the build scripts and test it with the gate script more easily. | 16:22 |
| macsz | Satya_: i am working on nova right now, I would love to help with everything, but my time is limited and thus I am afraid I can't help with this bug :) | 16:23 |
| kfox1111 | should be able to just add an entry in tools/build_packages.py to get it to build your stuff. and tweak tests/bin/ceph_workflow.sh to swithout the openvswitch kollakube res call with the helm install one. | 16:24 |
| *** tonanhngo has joined #openstack-kolla | 16:24 | |
| Satya_ | NP macsz :) | 16:24 |
| *** tonanhngo has quit IRC | 16:25 | |
| *** adrian_otto has quit IRC | 16:26 | |
| *** tonanhngo has joined #openstack-kolla | 16:26 | |
| sbezverk_ | kfox1111: it is not very convenient as you keep changing other things in your ps, also bunch of other folks we need to do the same and this approach does not look too good.. | 16:27 |
| kfox1111 | I think I'm done changing plumbing, unless others have objections to it. | 16:28 |
| kfox1111 | I'm thinking its close to commitable actually. | 16:28 |
| portdirect | kfox1111: I'll have a look over - but so far LGTM | 16:29 |
| sbezverk_ | kfox1111: second question, what happened to namespace in spec? it is gone now. | 16:30 |
| kfox1111 | sbezverk_: just commented on that. | 16:30 |
| *** eaguilar has quit IRC | 16:31 | |
| *** duonghq has quit IRC | 16:32 | |
| portdirect | kfox1111/sbezverk: I think thats the right approach with helm (no ns in chart) but we should double check - I've gone through helm's source as the docs seem pretty thin. | 16:32 |
| sbezverk_ | portdirect: I think having helm variable with namespace consistent accross kolla kubernetes look better imho | 16:35 |
| *** rmart04 has quit IRC | 16:36 | |
| sbezverk_ | it is one point to change if you want using another namespace | 16:36 |
| kfox1111 | lets hold the conversation in the spec for posterity. | 16:37 |
| portdirect | +! | 16:38 |
| portdirect | +1 (whoops) | 16:38 |
| kfox1111 | (I think having the conversation though is a good idea. I'm not too tied to any particular solution. trying to figure it out myself) | 16:38 |
| *** aric49 has joined #openstack-kolla | 16:39 | |
| sbezverk_ | kfox1111: sounds good.. but helm not having something global and common is bad :-( Before we had a common kubernetes.yaml file with all variables, now we are going to have bunch of per-microservice config/value files. it looks like a config nightmare to me.. | 16:40 |
| kfox1111 | sbezverk_: I see it as a building block. we probably still want the overarching common config, as thats super nice. | 16:40 |
| kfox1111 | so we may want to do something like have kollakube continue to be a thing, | 16:40 |
| aric49 | Hi Everyone --- quick question about deploying Kolla. | 16:41 |
| kfox1111 | and just have kollakube res create ... call helm install? | 16:41 |
| aric49 | According to the docs, it looks like Ubuntu 14.04 is supported. Does anyone know if Ubuntu 16.04 Xenial will work? | 16:41 |
| kfox1111 | at least until some other workflow bits pop up, such as operators/(helm/entrypoint orchestration) | 16:41 |
| sbezverk_ | kfox1111: ok, hopefully it will be addressed somehow or by somebody ;-) | 16:43 |
| aric49 | Or if there is anything to watch out for when deploying on Xenial? | 16:43 |
| kfox1111 | sbezverk_: yeah. :) | 16:43 |
| kfox1111 | I think the architecture spec has it covered in at least two different ways. | 16:43 |
| kfox1111 | sbezverk_: ryan's already started working on operators too. so it may be covered there soon. | 16:44 |
| *** TxGirlGeek has quit IRC | 16:45 | |
| *** severion has joined #openstack-kolla | 16:46 | |
| *** TxGirlGeek has joined #openstack-kolla | 16:46 | |
| *** sdake has quit IRC | 16:47 | |
| *** egonzalez90 has quit IRC | 16:48 | |
| *** v1k0d3n has quit IRC | 16:49 | |
| *** TxGirlGeek has quit IRC | 16:50 | |
| *** TxGirlGeek has joined #openstack-kolla | 16:51 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible: Ignore dom0 qemu processes during destroy https://review.openstack.org/399203 | 16:53 |
| *** TxGirlGeek has quit IRC | 16:53 | |
| *** TxGirlGeek has joined #openstack-kolla | 16:54 | |
| *** TxGirlGeek has quit IRC | 17:00 | |
| *** Serlex has joined #openstack-kolla | 17:01 | |
| *** TxGirlGeek has joined #openstack-kolla | 17:01 | |
| portdirect | throwing it out there - but it would be very easy to chain helm by having a chart containing jobs that simply launched other helm charts | 17:03 |
| *** TxGirlGeek has quit IRC | 17:03 | |
| *** adrian_otto has joined #openstack-kolla | 17:04 | |
| *** TxGirlGeek has joined #openstack-kolla | 17:04 | |
| *** msimonin has quit IRC | 17:05 | |
| openstackgerrit | Merged openstack/kolla-ansible: Fix neutron.conf.j2 metadata_workers spelling error https://review.openstack.org/399356 | 17:08 |
| *** dasm is now known as ferros | 17:09 | |
| *** ferros is now known as dasm | 17:09 | |
| inc0 | can I get ack on this? https://review.openstack.org/#/c/399082/ | 17:10 |
| *** bmace has quit IRC | 17:10 | |
| *** bmace has joined #openstack-kolla | 17:11 | |
| kfox1111 | portdirect: True. | 17:13 |
| *** matrohon has quit IRC | 17:13 | |
| kfox1111 | that may solve the issue I came up with where you may want one package, but multiple subpackage instantiations of the same package. | 17:13 |
| mliima | done inc0 | 17:16 |
| inc0 | thanks | 17:17 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Tacker NFV Ansible support https://review.openstack.org/399179 | 17:21 |
| *** eaguilar has joined #openstack-kolla | 17:26 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Horizon custom policies https://review.openstack.org/399707 | 17:27 |
| sbezverk_ | kfox1111: when I have two yaml in template and run helm install, helm picks up olny first file in template and ignores second, am I missing something or I need to dump all specs in the same yaml file? | 17:27 |
| kfox1111 | so... | 17:29 |
| kfox1111 | in regular helm build, it should grab everything in the templates directory. | 17:29 |
| kfox1111 | for our stuff, I've been making it one object per package. that way you have nice fine grained upgradability. | 17:30 |
| kfox1111 | we can then make upper level packages that combine the subpackages. | 17:30 |
| *** severion has quit IRC | 17:34 | |
| sbezverk_ | kfox1111: that part I figured out, I am doing one chart for ovsdb and one chart for vswitchd | 17:34 |
| kfox1111 | can I please get another kola-kube core for this: https://review.openstack.org/#/c/396812/ | 17:34 |
| *** v1k0d3n has joined #openstack-kolla | 17:34 | |
| sbezverk_ | but along with ovsdb I have config map | 17:35 |
| sbezverk_ | helm pick up configmap but ignores yaml for daemonset | 17:35 |
| sbezverk_ | strange :-( | 17:35 |
| kfox1111 | I think we want to not put the configmaps in the the packages. | 17:35 |
| kfox1111 | otherwise, the user will have to get into the buisness of customizing packages to do config updates. | 17:36 |
| kfox1111 | or alternatively for some workflow's, we put configmaps in their own packages. | 17:37 |
| sbezverk_ | kfox1111: configmap is temporary just to test ovs and not bother with generating config maps | 17:37 |
| kfox1111 | ah. | 17:37 |
| kfox1111 | there's the gate tests too. | 17:37 |
| kfox1111 | which is why I was recommending basing on my ps. | 17:38 |
| kfox1111 | I've been running helm install --dry-run --debug testpackage, | 17:38 |
| kfox1111 | looking to see if it renders fine, then git review; check experimental | 17:38 |
| openstackgerrit | Merged openstack/kolla-kubernetes: External openvswitch https://review.openstack.org/396812 | 17:38 |
| kfox1111 | sbezverk_: thanks | 17:39 |
| kfox1111 | meeting time. bbiab | 17:40 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Add custom policies in service.json https://review.openstack.org/399712 | 17:40 |
| *** rmart04 has joined #openstack-kolla | 17:43 | |
| inc0 | pbourke, still around? | 17:44 |
| pbourke | inc0: yeah | 17:44 |
| inc0 | https://review.openstack.org/#/c/398501/ <- care to take a look | 17:45 |
| inc0 | ? | 17:45 |
| inc0 | oracle gates are bad, maybe you'll spot what's wrong | 17:46 |
| *** rmart04_ has joined #openstack-kolla | 17:46 | |
| *** schwicht has joined #openstack-kolla | 17:47 | |
| openstackgerrit | Merged openstack/kolla: Dependency init container https://review.openstack.org/399082 | 17:48 |
| *** rmart04_ has quit IRC | 17:48 | |
| *** rmart04 has quit IRC | 17:48 | |
| inc0 | yay merge with voting gates \o/ | 17:49 |
| pbourke | inc0: myself Jeffrey4l and sdake were looking at this earlier | 17:52 |
| pbourke | we had an idea so I need to check if Jeffrey4l tried that yet or not | 17:53 |
| inc0 | ok:) | 17:53 |
| Jeffrey4l | pbourke, inc0 still struggle on this issue ;( | 17:53 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add a section regarding share migration https://review.openstack.org/399715 | 17:53 |
| pbourke | Jeffrey4l: did you try hacking the shebang? | 17:53 |
| Jeffrey4l | i am trying to remove the `tox -e xxxx` right now. | 17:53 |
| Jeffrey4l | it is useless after split. | 17:53 |
| Jeffrey4l | pbourke, hacking do not work. before run tox -e, the virtual env and the pip file need to be hack is not exist. | 17:54 |
| pbourke | :( | 17:55 |
| pbourke | ffs | 17:55 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add a section regarding share migration https://review.openstack.org/399715 | 17:56 |
| *** adrian_otto has quit IRC | 17:58 | |
| pbourke | Jeffrey4l: python -m also not working? | 18:05 |
| Jeffrey4l | pbourke, yes. tox call `pip freeze` there is no place to overwrite this. | 18:05 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix ci gate https://review.openstack.org/398501 | 18:06 |
| *** unicell has quit IRC | 18:06 | |
| *** Pavo has quit IRC | 18:07 | |
| Jeffrey4l | pbourke, ^& this make life easier. but it won't work until this https://review.openstack.org/#/c/399694/ is merged. | 18:07 |
| mliima | Jeffrey4l, we need merge it https://review.openstack.org/#/c/399694/ ? | 18:08 |
| Jeffrey4l | mliima, yep.. | 18:08 |
| Jeffrey4l | mliima, https://review.openstack.org/398501 this need that change. | 18:09 |
| mliima | oh god | 18:09 |
| inc0 | Jeffrey4l, I'll ask infra cores to review it | 18:09 |
| Jeffrey4l | mliima, it make test easier. there is no need to call tox -e xxx to generated the images we need. | 18:09 |
| Jeffrey4l | and it won't work too. ( we need other way solution ) | 18:10 |
| Jeffrey4l | current, i simple use `kolla-build -p gate || true ` :( | 18:10 |
| Jeffrey4l | inc0, thanks. | 18:11 |
| Jeffrey4l | hrm, maybe run `testr run test_build.DeployTestUbuntuSOurce` in kolla folder is better . | 18:11 |
| *** adrian_otto has joined #openstack-kolla | 18:11 | |
| *** Pavo has joined #openstack-kolla | 18:12 | |
| *** pbourke has quit IRC | 18:12 | |
| *** pbourke has joined #openstack-kolla | 18:12 | |
| mliima | Jeffrey4l, should have a better way to do `kolla-build -p gate || true ` | 18:14 |
| Jeffrey4l | maybe run `testr run test_build.DeployTestUbuntuSOurce` in kolla folder is better . | 18:14 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Initial Helm support https://review.openstack.org/396296 | 18:14 |
| Jeffrey4l | mliima, more work should be done at this point. | 18:14 |
| mliima | :) | 18:14 |
| Jeffrey4l | or once the infra registry is done, we won't need this. | 18:14 |
| jascott1 | checkout dude's handheld k8s cluster :) https://hackernoon.com/diy-kubernetes-cluster-with-x86-stick-pcs-b0b6b879f8a7#.o2mvf79u0 | 18:15 |
| Jeffrey4l | just pull the image from infra'd registry. | 18:15 |
| mliima | then this is temporary? | 18:15 |
| mliima | `kolla-build -p gate || true ` | 18:15 |
| kfox1111 | Jeffrey4l: wow. fun. :) | 18:15 |
| Jeffrey4l | i hope so. i hope infra can set the docker registry up. inc0 is working on this. | 18:16 |
| inc0 | yeah, this review https://review.openstack.org/#/c/399221/ | 18:16 |
| mliima | I do not know if it is worth spending time improving it if it will change after registration get ready Jeffrey4l | 18:16 |
| mliima | registry | 18:17 |
| mliima | infra registry*** hehe | 18:17 |
| Jeffrey4l | mliima, wait for day. the registry PS seems will be merged soon,. | 18:17 |
| mliima | no problem Jeffrey4l, i trust in your work | 18:19 |
| Jeffrey4l | mliima, it is inc0's work. ;) | 18:19 |
| inc0 | Jeffrey4l, we still need to write solid puppet | 18:19 |
| *** senk has joined #openstack-kolla | 18:19 | |
| mliima | haha | 18:19 |
| mliima | okay | 18:19 |
| inc0 | but it will be super beneficial for out gates | 18:20 |
| Jeffrey4l | yep. | 18:20 |
| inc0 | this will allow us voting deploy gates | 18:20 |
| Jeffrey4l | it will speed up the gate. | 18:20 |
| mliima | inc0, why not use 16.04? https://review.openstack.org/#/c/399221/2 | 18:22 |
| mliima | any special reason? | 18:22 |
| inc0 | infra runs Trusty mostly on mirror nodes afaik | 18:22 |
| mliima | oh | 18:23 |
| mliima | make sense | 18:23 |
| *** Serlex has quit IRC | 18:29 | |
| *** jtriley has quit IRC | 18:33 | |
| *** unicell has joined #openstack-kolla | 18:36 | |
| *** krtaylor has joined #openstack-kolla | 18:46 | |
| *** Satya_ has quit IRC | 18:55 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix ci gate https://review.openstack.org/398501 | 18:58 |
| *** lamt has quit IRC | 18:58 | |
| *** unicell has quit IRC | 19:00 | |
| *** sdake has joined #openstack-kolla | 19:02 | |
| *** senk has quit IRC | 19:09 | |
| *** schwicht has quit IRC | 19:10 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Initial Helm support https://review.openstack.org/396296 | 19:15 |
| *** gfidente has quit IRC | 19:16 | |
| *** jtriley has joined #openstack-kolla | 19:17 | |
| sdake | hi peeps | 19:22 |
| kfox1111 | hi | 19:22 |
| *** mliima has quit IRC | 19:23 | |
| *** unicell has joined #openstack-kolla | 19:32 | |
| *** unicell has quit IRC | 19:32 | |
| *** unicell has joined #openstack-kolla | 19:32 | |
| *** schwicht has joined #openstack-kolla | 19:38 | |
| *** jtriley has quit IRC | 19:51 | |
| kfox1111 | sbezverk_: thanks by the way, for reviewing and pushing/asking questions about the helm archetecture stuff. | 19:58 |
| kfox1111 | its kind of weird the way I did it, and its good to figure out why I did what I did and if its really the right aproach. | 19:58 |
| sbezverk_ | kfox1111: sorry Kevin, but it feels a bit uncomfortable using the process as it is now, well at least to me. | 20:06 |
| sdake | which process, the review process? | 20:07 |
| sbezverk_ | kfox1111: I send emails to some helm experts asking for some input, will share if anything comes back from them.. | 20:07 |
| sbezverk_ | sdake: no, the way things rendered with helm | 20:07 |
| *** jtriley has joined #openstack-kolla | 20:07 | |
| *** Pavo has quit IRC | 20:07 | |
| kfox1111 | sbezverk_: I think we should write down the issues we have discovered, | 20:07 |
| kfox1111 | and run the set by them and see what they think. | 20:08 |
| sdake | sound fantastic | 20:08 |
| sdake | i'd like to know to o:) | 20:08 |
| sbezverk_ | kfox1111: +2 | 20:09 |
| kfox1111 | I really need to get some other work done today though, so will have to wait till at least tonight. :/ | 20:10 |
| sbezverk_ | kfox1111: no rush :-) | 20:12 |
| *** schwicht has quit IRC | 20:12 | |
| *** Pavo has joined #openstack-kolla | 20:12 | |
| kfox1111 | semi rush. :/ holds up doing anything other then prototyping. | 20:12 |
| *** schwicht has joined #openstack-kolla | 20:13 | |
| kfox1111 | sbezverk_: with helm and nesting, the vars are the api for the package. a lot of things we can decide upon later, but it will be painful to change an api. so we should invest the most time into figuring out what that part should be. | 20:15 |
| sbezverk_ | kfox1111: ok, if you do not mind I want to try nesting approach, just to see if I hit some roadblock | 20:25 |
| sbezverk_ | and if it works as helm people suggesting | 20:25 |
| inc0 | the world ends, got 500 from gmail | 20:26 |
| *** eaguilar has quit IRC | 20:28 | |
| sdake | inc0 internet not been woroking well last couple of days | 20:39 |
| inc0 | yeah | 20:39 |
| inc0 | on that note, I'm off | 20:39 |
| inc0 | need to see Houston before I leave TX;) | 20:40 |
| inc0 | have a good weekend folk | 20:40 |
| inc0 | s | 20:40 |
| *** inc0 has quit IRC | 20:40 | |
| sdake | kfox1111 possibly a dumb question | 20:44 |
| sdake | re https://review.openstack.org/#/c/396296/27/helm/src/neutron/templates/l3_agent_daemonset.yaml | 20:44 |
| sdake | why are VolumeMounts json in a yaml file? | 20:44 |
| *** ipsecguy has joined #openstack-kolla | 20:45 | |
| sbezverk_ | sdake: it is the way how init containers are implemented in kube these days | 20:48 |
| sdake | init containers not implemented in yaml because??? | 20:48 |
| sdake | just curious if you know | 20:48 |
| jascott1 | sdake you are talking about init container? | 20:48 |
| sdake | jascott1 apparently :) | 20:49 |
| sbezverk_ | sdake: because it is still alpha feature, will be moved to yaml when stop being alpha | 20:49 |
| sdake | cool | 20:49 |
| sdake | thanks sbezverk_ | 20:49 |
| jascott1 | that was my guess too :) | 20:49 |
| sdake | so the alpha apis use json, the non-alpha apis use yaml? | 20:49 |
| sbezverk_ | sdake: yep | 20:49 |
| *** ipsecguy_ has quit IRC | 20:49 | |
| sdake | will upstream kubernetes definately move that to yaml later? | 20:50 |
| sdake | helm serve is a rest API? | 20:51 |
| jascott1 | helm serve creates local chart repo | 20:51 |
| sdake | and exposes via which type of api jascott1 ? | 20:52 |
| sdake | or it just on disk? | 20:52 |
| jascott1 | i think it is just expected to have index file with all packages listed in json | 20:52 |
| jascott1 | dont know that it has an api | 20:52 |
| jascott1 | basically a simple webserver | 20:53 |
| sdake | when you get helm packages | 20:53 |
| sdake | you are doing a GET operation on some rest endpoint ? | 20:53 |
| jascott1 | ya so you can do 'helm package' | 20:53 |
| jascott1 | on a local repo | 20:53 |
| sdake | jascott1 my understanding of helm after reading zero docs and just obsorbing the last couple of weeks | 20:54 |
| sdake | is that it has a build process | 20:54 |
| sdake | with the current implemenetaiton we got, can we build it once and forget about it? | 20:54 |
| jascott1 | when you use helm to retrieve package from repo, it looks at that repo's index file, finds the chart/version you are lookin for and returns that tgz | 20:54 |
| sdake | much like we do with docker containers now | 20:54 |
| jascott1 | pretty much | 20:55 |
| sdake | ok, but there is config in these things? | 20:55 |
| sdake | i know there is the --set operation | 20:55 |
| sdake | thats super clunky to me - for the same reason ENV variables are clunky | 20:55 |
| sdake | you gotta know em all ahead of time | 20:55 |
| jascott1 | the repo is pretty dumb, they have examples of using github pages to serve one | 20:56 |
| sdake | the build is via rest api? | 20:56 |
| jascott1 | tiller has api for releases | 20:56 |
| jascott1 | but the repo doesnt afaik | 20:56 |
| kfox1111 | sdake: alpha api's are things k8s doesn't want to commit too 100% by puting them in their yaml spec. | 20:56 |
| kfox1111 | so they cram them into key/value pairs in attributes until they stabalize. | 20:57 |
| kfox1111 | serialized to a string, json's a little cleaner then yaml for doing that. | 20:57 |
| sdake | kfox1111 ahh, got it | 20:57 |
| sdake | kfox1111 thanks for the background | 20:57 |
| sdake | i'm after a specific peice of information jascott1 | 20:57 |
| kfox1111 | init containers are now listed as beta in 1.4 | 20:58 |
| sdake | when we implement operators, they will use helm charts to deploy | 20:58 |
| jascott1 | ya i forgot to mention its a beta annotation now | 20:58 |
| sdake | does that mean operators also need to build the helm charts? | 20:58 |
| kfox1111 | sdake: no. | 20:58 |
| sdake | cool | 20:58 |
| jascott1 | if the operator is crafting an update maybe | 20:58 |
| sdake | not cool :) | 20:59 |
| kfox1111 | I want to be able to ship binary prebuilt packages just like we ship binary prebuilt containers. | 20:59 |
| jascott1 | imo depdends | 20:59 |
| *** shardy has quit IRC | 20:59 | |
| sdake | ok which one is it :) | 20:59 |
| kfox1111 | so users can not have to worry about building packages at all, unless they are using them for orchestration. | 20:59 |
| sdake | need to know up front to implement operators | 20:59 |
| jascott1 | idk kfox1111 seems to have designs on it so maybe he has thought it through :) | 20:59 |
| kfox1111 | and then they only need to build the orchestrator packages. | 20:59 |
| kfox1111 | so operators wouldn't use that. | 20:59 |
| sdake | what is an orchestor package | 21:00 |
| sdake | you mean the master package? | 21:00 |
| kfox1111 | sdake: the entrypoint/helm thing. | 21:00 |
| kfox1111 | making a "neutron" package with all the microservices packages in it, and using helm to orchestrate the deployment. | 21:00 |
| sdake | so my assumption would be the operator would do a helm install neutron | 21:01 |
| kfox1111 | no. the operator would helm install neutron-l3-agent, neutron-openvswitch-agent, etc. | 21:01 |
| kfox1111 | as its orchestrating the deployment of the microservices. | 21:01 |
| sdake | ok | 21:01 |
| kfox1111 | "one of these, two of these, etc.." | 21:01 |
| sbezverk_ | kfox1111: it becomes tricky if you want to deploy just 1 microservice | 21:01 |
| jascott1 | my understanding of the existing operator is that they watch TPRs for changes and then invoke some action (as well as doing other non TPR related stuff) | 21:01 |
| sdake | kfox1111 i like that alot | 21:01 |
| kfox1111 | sbezverk_: can always just use the package directly in that case. | 21:01 |
| sdake | kfox1111 i said i wouldn't harass you for a week :) | 21:02 |
| kfox1111 | sdake: so, the neturon operator would do the logic in ceph_workflow.sh from the corisponding neutron stuff. | 21:02 |
| kfox1111 | jascott1: right. | 21:02 |
| kfox1111 | jascott1: so the tpr has the config description for "neutron", and the operator decides what microservices are needed, then instantiates the helm microservices packages related to the desired config. | 21:03 |
| sdake | kfox1111 cool - so the config does entirely from configmaps, is that accurate? | 21:03 |
| sdake | I would think there would be other config options needed as well in the long term future | 21:03 |
| kfox1111 | semitrue. | 21:03 |
| kfox1111 | the microservices themselves have some config bits too. | 21:04 |
| kfox1111 | there's config for openstack stuff. nova.conf kinds of things. container level. | 21:04 |
| kfox1111 | and then theres a little bit of config for the micrservices themselves. | 21:04 |
| sdake | that config is what? | 21:04 |
| kfox1111 | enable_kube_logger=true #slide in a fluentd into the pod so it will work with elasticsearch... kinds of things. | 21:04 |
| jascott1 | we might need a flowchart ;) | 21:05 |
| kfox1111 | +1 for flowchart. :) | 21:05 |
| sdake | so happy to make a diagram | 21:05 |
| sdake | do we have a full list of the configs the microservices need? | 21:05 |
| sdake | or is there a way to find that out | 21:05 |
| sdake | what i'm really after is - does helm have some specific config bits related to openstack itself | 21:06 |
| kfox1111 | no. :/ we'd have to crawl the services subdirectory and find out whats there now. | 21:06 |
| kfox1111 | no. | 21:06 |
| kfox1111 | openstack configs themselves just come in through configmaps. | 21:06 |
| sdake | cool so its got a bunch of undefined config options | 21:06 |
| sdake | that are part of the services directory | 21:06 |
| sdake | services directory contains what - k8s stuff or helm stuff? | 21:06 |
| kfox1111 | microservices have some config stuff that tweak the k8s objects to change/add features. like enabling/disabling entrypoint, selecting log agent, that kind of thing. | 21:07 |
| kfox1111 | those come through helm vars. | 21:07 |
| kfox1111 | service discovery's an overloaded term. which definition are you using? | 21:07 |
| sdake | are those standardized? | 21:07 |
| sdake | i didn't say service discovery | 21:07 |
| kfox1111 | sdake: kind of. | 21:07 |
| sdake | i said services DIRectory | 21:07 |
| kfox1111 | oh. | 21:08 |
| kfox1111 | ok. | 21:08 |
| kfox1111 | right. | 21:08 |
| kfox1111 | services directory is the current jinja2 based k8s objects | 21:08 |
| kfox1111 | those will be converted to helm golang based templates. | 21:08 |
| sdake | will be converted where? | 21:08 |
| sdake | when and by who :) | 21:09 |
| kfox1111 | the helm subdir | 21:09 |
| kfox1111 | I've got 2 converted in the prototype here: | 21:09 |
| kfox1111 | https://review.openstack.org/#/c/396296 | 21:09 |
| sdake | i see, so we are dumping k8s objects entirely? | 21:09 |
| kfox1111 | once we come to an agreement on how the build system should work, we can do the rest. | 21:09 |
| sdake | i am looking at that review right now | 21:09 |
| sdake | thats what triggered these qs :) | 21:09 |
| kfox1111 | no. just switching out jinja2 templates for golang templates | 21:10 |
| jascott1 | the k8s objects are just being transformed from jinja to gotpl | 21:10 |
| kfox1111 | and reparenting them to the helm dir. | 21:10 |
| jascott1 | sorry replacing is better choice than transforming | 21:10 |
| jascott1 | (of words) | 21:10 |
| *** jheroux has quit IRC | 21:11 | |
| *** fragatina has quit IRC | 21:11 | |
| jascott1 | if we have 'master' values file with everything in it, how would we be able to use that to install only specific charts? | 21:12 |
| sdake | so sounds like helm as per spec of being optional is not optional | 21:12 |
| kfox1111 | jascott1: we can't have master values in helm. | 21:12 |
| kfox1111 | :/ | 21:12 |
| jascott1 | I could see parsing it for the 'neutron' key and then applying those values to neutron chart | 21:12 |
| kfox1111 | sdake: I can't quite parse that. | 21:12 |
| sdake | is helm no longer optional after a reparenting and reimplementation in gotpl? | 21:13 |
| kfox1111 | sdake: oh. you mean, the templates? | 21:13 |
| kfox1111 | yes/no... | 21:13 |
| sdake | maybe? | 21:13 |
| sdake | which one of the 3 :) | 21:13 |
| kfox1111 | the templates are pure gotpl. so could be loaded into a webserver or something to render. | 21:13 |
| kfox1111 | rather hten helm. | 21:13 |
| sdake | ok so let me be more precise | 21:13 |
| kfox1111 | so, yes, more generic then just helm. | 21:13 |
| sbezverk_ | jascott1: has supports for global variables but it is kinbd of ugly | 21:13 |
| kfox1111 | though not without a bit of code. | 21:13 |
| sbezverk_ | I meant helm | 21:14 |
| sdake | are we picking a winner with helm and ignoring the kubernetes api for things like creating pods and whatnot | 21:14 |
| kfox1111 | sdake: for some things, yes. | 21:14 |
| sdake | which classification of things? | 21:14 |
| sdake | are in vs out | 21:14 |
| kfox1111 | ie, we're using helm to package up templates that render to k8s objects and helm passes to k8s. | 21:14 |
| kfox1111 | for other things, like configmaps, secrets, etc, | 21:14 |
| kfox1111 | no | 21:14 |
| sdake | ok so the yes = picking a winner | 21:15 |
| sdake | which is fine | 21:15 |
| sdake | i think helm is a natural choice here | 21:15 |
| kfox1111 | yeah. mostly picking a winner. | 21:15 |
| kfox1111 | agreed. | 21:15 |
| sdake | i am really keen to keep build separate from deploy | 21:16 |
| kfox1111 | it wouldn't be hard to switch to a different pm though if things changed. | 21:16 |
| kfox1111 | +1 | 21:16 |
| sdake | and to do that, need to keep config separate from build | 21:16 |
| kfox1111 | which is why the build_package.py thing is a thing. | 21:16 |
| kfox1111 | that too. yeah. | 21:16 |
| sdake | so, we have some config that gets built into the package | 21:17 |
| sdake | here is where i'm stuck if build separate from deploy and deploy depends on config | 21:18 |
| kfox1111 | not really. | 21:18 |
| sdake | (to configure the helm package) | 21:18 |
| kfox1111 | we have default vars for templates built into the packages. | 21:18 |
| kfox1111 | which anything is free to override. | 21:18 |
| sdake | hrm, going back on my word about not harassing you for a week :) | 21:18 |
| kfox1111 | :) | 21:18 |
| *** dave-mcc_ has quit IRC | 21:18 | |
| kfox1111 | its important to get right. no worries. | 21:19 |
| sdake | well this is a keen lesson kolla learned early on | 21:19 |
| sdake | and i want to make sure we reuse it | 21:19 |
| rhallisey | yo | 21:19 |
| sdake | is to not have a slew of config variables for everything | 21:19 |
| sdake | but instead provide a way to override custom stuff | 21:19 |
| sdake | we learned this with docker's ENV pattern | 21:19 |
| sdake | which sucks from my experience | 21:20 |
| kfox1111 | right. | 21:20 |
| sdake | so keen not to repeat that | 21:20 |
| rhallisey | everything is layered like we said | 21:20 |
| sdake | got it rhallisey | 21:20 |
| kfox1111 | helm follows the same docker env mispattern | 21:20 |
| sdake | focused on implementing operators :) | 21:20 |
| kfox1111 | if you put configmaps in them. | 21:20 |
| rhallisey | :D | 21:20 |
| rhallisey | operators are an essentials piece | 21:20 |
| sdake | rhallisey not focused on rewritting the spec :) | 21:20 |
| rhallisey | ya I'm just reading bacl | 21:21 |
| rhallisey | sdake, did you see latest patch? | 21:21 |
| rhallisey | I added the dockerfiles | 21:21 |
| sdake | rhallisey i have not | 21:21 |
| rhallisey | kk | 21:21 |
| sdake | rhallisey i just woke up about an hr ago | 21:21 |
| rhallisey | np | 21:21 |
| sdake | after crashing again around 8am this morning | 21:21 |
| jascott1 | ah the mad scientist schedule :) | 21:22 |
| sdake | jascott1 pretty much | 21:24 |
| *** adrian_otto has quit IRC | 21:24 | |
| sdake | kfox1111 mind pointing me at an example helm variable | 21:24 |
| jascott1 | love it! I have to stick a modified maker schedule (late morning) since I have to show up to Intel quite a bit | 21:24 |
| sdake | so i Ican figure out how to launch helm correctly in the operator | 21:25 |
| kfox1111 | yup. sec | 21:25 |
| jascott1 | i miss being able to work as late as I want :/ | 21:25 |
| kfox1111 | https://review.openstack.org/#/c/396296/27/tests/bin/ceph_workflow.sh | 21:25 |
| kfox1111 | the operator would be replacing the workflow in that script. | 21:26 |
| kfox1111 | so would be calling helm in the same way. | 21:26 |
| sdake | ok thats cool | 21:26 |
| sdake | but I'd like to understand a level deeper ;) | 21:26 |
| sdake | for personal growth | 21:26 |
| kfox1111 | grab the ps, do a: tools/build_packages.py /tmp/repo | 21:26 |
| *** msimonin has joined #openstack-kolla | 21:26 | |
| kfox1111 | then: | 21:26 |
| kfox1111 | helm install /tmp/repo/neutron-openvswitch-agent.* --set enable_kube_logger=false,type=compute,selector_key=kolla_compute,tunnel_interface=docker0 \ --namespace kolla --name neutron-openvswitch-agent-compute --dry-run --debug | 21:27 |
| jascott1 | so we dont think a top level values file will work? wont it be unruly if they are all spread out? | 21:27 |
| kfox1111 | then you canmess around with the --set flags. | 21:27 |
| kfox1111 | jascott1: yup, and yup. | 21:28 |
| kfox1111 | jascott1: thats one reason I'm building the common lib and build bits | 21:28 |
| kfox1111 | jascott1: so the code is at least shared, and defaults are common. | 21:28 |
| jascott1 | keys should be common too | 21:29 |
| jascott1 | service_port over mariadb_service_port for instance | 21:29 |
| jascott1 | much of it can be generic that way | 21:29 |
| kfox1111 | services should too. yeah. | 21:30 |
| kfox1111 | what do you mean by keys? | 21:30 |
| jascott1 | in the yaml | 21:30 |
| jascott1 | i still have master values file in my head | 21:30 |
| kfox1111 | oh. "helm install kolla/neutron-openvswitch-agent --version 2.0.2-1 \ --set enable_kube_logger=false,type=compute,selector_key=kolla_compute,tunnel_interface=docker0 \ --namespace kolla --name neutron-openvswitch-agent-compute | 21:30 |
| kfox1111 | bad paste.... | 21:30 |
| kfox1111 | so, "service_port over mariadb_service_port for instance" | 21:30 |
| jascott1 | it has keys: global, nuetron, nova etc | 21:30 |
| kfox1111 | got it. yeah. | 21:30 |
| jascott1 | pass that subtree to respective chart | 21:30 |
| sdake | i'm fairly keen to have one set of global config options between platform tools | 21:31 |
| kfox1111 | yeah. the operators will do some kind of passing those config bits through to the right packages. | 21:31 |
| jascott1 | so we only need one or two common template for something like readinessProbes | 21:31 |
| jascott1 | and the common template can just reference {{ service_port }} | 21:31 |
| kfox1111 | jascott1: we have common-deployments with all that in it. | 21:31 |
| sbezverk_ | kfox1111: hehe I got ovsdb running using helm chart :-) | 21:32 |
| kfox1111 | sbezverk_: nice. | 21:32 |
| sbezverk_ | kfox1111: but I had to use nested approach for now.. | 21:32 |
| kfox1111 | sbezverk_: yeah. shouln't be much different really from the fully nested, and the aproach I've been using. | 21:33 |
| sbezverk_ | kfox1111: for vswitchd I will add entrypoint to wait for ovs socket | 21:33 |
| *** Jeffrey4l has quit IRC | 21:33 | |
| kfox1111 | cool. | 21:33 |
| kfox1111 | I think we should add a common lib thingy for that. | 21:34 |
| kfox1111 | put a bare init-container in each object, then include the common_lib.init-containers in it. | 21:34 |
| jascott1 | and push deps for init container through values? | 21:34 |
| kfox1111 | yeah. | 21:34 |
| jascott1 | thats what I have been assuming | 21:34 |
| jascott1 | cool | 21:34 |
| sbezverk_ | kfox1111: sounds good | 21:35 |
| sdake | ok so where in the code is an example of a config value for helm kfox1111 ? | 21:35 |
| sdake | kfox1111 so I know what to look for :) | 21:35 |
| sdake | not the ceph workflow | 21:36 |
| sdake | but the helm package | 21:36 |
| kfox1111 | for the defaults, or the consumption of them? | 21:36 |
| kfox1111 | and in default helm building, or the build system I came up with? | 21:36 |
| jascott1 | there is nothing helm specific except the version of the packages and its deps | 21:36 |
| sdake | not helm sepcific, package specific | 21:36 |
| jascott1 | the values.yaml? | 21:37 |
| kfox1111 | sdake: so, like heres an example of config values that are shared betwen all packages: https://review.openstack.org/#/c/396296/27/helm/src/common_values.yaml | 21:37 |
| kfox1111 | heres some that are specific to l3agent: | 21:38 |
| kfox1111 | https://review.openstack.org/#/c/396296/27/helm/src/neutron/values/l3_agent_daemonset.yaml | 21:38 |
| sdake | is common_values.yaml a magic file? | 21:38 |
| sdake | fileame that is | 21:38 |
| *** fguillot has quit IRC | 21:38 | |
| jascott1 | no | 21:38 |
| jascott1 | values.yaml is tho | 21:38 |
| sdake | k | 21:38 |
| jascott1 | magic meaning autmatically picked up? | 21:38 |
| kfox1111 | build_packages.py combines the common and l3 agent ones together into the final values.yaml for the final l3agent package. | 21:38 |
| kfox1111 | so helm just sees it as one unified set of values that a user can set. | 21:39 |
| sdake | if I could make one request, it would be to see you one more time... :) | 21:39 |
| kfox1111 | this was done so that we idn't have to manually maintain the common stuff in every packages values.yaml | 21:39 |
| sdake | no really, what I'm sort of after is to keep the second example at a minimum | 21:39 |
| *** fragatina has joined #openstack-kolla | 21:41 | |
| sdake | for example on the second example, line #1: | 21:42 |
| sdake | type: network | 21:42 |
| kfox1111 | yeah. what about it? | 21:42 |
| sdake | what else could type be there? | 21:42 |
| kfox1111 | anything really. | 21:43 |
| kfox1111 | the idea in that specific case, | 21:43 |
| kfox1111 | is that you have different sets of hardware. such as network nodes, or compute nodes, or some other set of nodes. | 21:43 |
| kfox1111 | you can override the selector_key/value variables to make that daemonset land on a subset of your nodes. | 21:44 |
| kfox1111 | you then may need multiple helm package instances of that package to land multipe to cover your cluster with them. | 21:44 |
| jascott1 | node selector then | 21:44 |
| sdake | ok i totaly didn't understand that | 21:45 |
| sdake | let me use a different example :) | 21:45 |
| sdake | logger_configmap_name: neutron-l3-agent-logger | 21:45 |
| sdake | why configmap name needs configuration? | 21:45 |
| kfox1111 | cause you might want to do something different. | 21:45 |
| kfox1111 | ok.. so... | 21:45 |
| kfox1111 | back slightly to the previous exmapl, but more concrete.. | 21:45 |
| kfox1111 | multinode, we launch two instances of openvswitch-agent package. | 21:46 |
| kfox1111 | with setteings type=network, and type=compute | 21:46 |
| kfox1111 | by default, they share the same neutron-openvswitch-agent-logger configmap. | 21:46 |
| kfox1111 | but. if you wanted, you could make a second configmap, neutron-openvswitch-agent-compute, and: | 21:47 |
| kfox1111 | with setteings type=network, and type=compute,logger_configmap_name=neutron-openvswitch-agent-compute | 21:47 |
| kfox1111 | and then they could be configurable seperately. | 21:47 |
| sdake | ok so sanity out of he box defaults | 21:47 |
| sdake | this is for flexibility? | 21:48 |
| kfox1111 | yeah. sane defaults, and customizability for those that need it. | 21:48 |
| kfox1111 | yeah. | 21:48 |
| sdake | ok, well lets not abuse it plz :) | 21:48 |
| kfox1111 | as little as possible. :) | 21:48 |
| sdake | because it makes life seriously hard to document config options | 21:48 |
| kfox1111 | yeah. | 21:48 |
| sdake | thats another thign kolla decidedo n early on | 21:48 |
| kfox1111 | +1 | 21:49 |
| sdake | very few config options | 21:49 |
| sdake | although obviously things have bloated | 21:49 |
| kfox1111 | thats the api I was taking about. | 21:49 |
| sdake | which I dont like but I understand | 21:49 |
| jascott1 | I think its a clever solution but also worried about config complexity | 21:49 |
| kfox1111 | packages have an api. which is the values they support. | 21:49 |
| kfox1111 | the config complexity is mostly in the openstack config files themselves, and totally outside this piece of the architecture. | 21:49 |
| sdake | jascott1 that is enabled by placing configmaps outside the helm package itself | 21:50 |
| jascott1 | i was talking about your 'type' | 21:51 |
| kfox1111 | jascott1: ah. yeah. we need to minimize that sort of thing, but its critical to have that sort of thing too. :/ | 21:51 |
| jascott1 | its another thing to learn in order to understand system | 21:51 |
| jascott1 | no doubt | 21:51 |
| sdake | if it works out of the box, who cares :) | 21:51 |
| kfox1111 | yeah. what we need maybe is a document or a few reference archetectures. | 21:51 |
| jascott1 | true dat sir | 21:51 |
| kfox1111 | or something. | 21:51 |
| kfox1111 | like, "if you want to have a scaled out system, with one rabbit per service, heres an example of that..." | 21:52 |
| sdake | kfox1111 ya we went down that road during compose of documeitng all the config options | 21:52 |
| sdake | it quickly gerw totally out of control | 21:52 |
| jascott1 | would need a set of golden examples | 21:52 |
| kfox1111 | helm install --name rabbit-neutron --set configname=neutron-rabbit,state=stateless | 21:52 |
| jascott1 | but if it dosnt exist for a use case it might be hard on operator | 21:53 |
| kfox1111 | I think there will probably be just a few common deployment types. | 21:53 |
| kfox1111 | all in one. one controller + computes. ha controllers 2 compute host aggregates. | 21:53 |
| kfox1111 | so shouldn't be too hard to document a few reference deployments. | 21:54 |
| jascott1 | refresh me plz on current thoughts on upgrade? mostly operator based or are we thinking helm upgrade/rollback hooks etc will be involved? | 21:54 |
| jascott1 | asking because we dont want to config ourselves into some corner before we understand that piece | 21:54 |
| kfox1111 | jascott1: both. :/ | 21:54 |
| kfox1111 | some folks want one way, some another. | 21:55 |
| kfox1111 | so parallel implementations of that part. | 21:55 |
| kfox1111 | but not mixed. | 21:55 |
| kfox1111 | so either operator based upgrades, or helm/entrypoint, but not both together. | 21:55 |
| jascott1 | ok my real question was do we think helm upgrade can do the job | 21:55 |
| jascott1 | ah | 21:55 |
| sdake | i struggle to see how helm can do upgrades correctly :) | 21:55 |
| *** msimonin has quit IRC | 21:55 | |
| kfox1111 | I think helm upgrade can do minor, rolling upgrades of a subset of packages. | 21:55 |
| kfox1111 | otherwise, I'm unconvinced. | 21:55 |
| kfox1111 | I think there are 3 types of upgrades really. | 21:56 |
| *** schwicht has quit IRC | 21:56 | |
| *** dave-mcc_ has joined #openstack-kolla | 21:56 | |
| kfox1111 | "zero downtime". ie, k8s deployment object types that can rolling upgrade minor version N to miner verion N. | 21:56 |
| *** jtriley_ has joined #openstack-kolla | 21:56 | |
| kfox1111 | helm can do that. | 21:56 |
| *** schwicht has joined #openstack-kolla | 21:56 | |
| sdake | do you mean x.y.z = upgrade z? | 21:57 |
| sdake | n is not semver ;) | 21:57 |
| kfox1111 | "zero downtime major upgrade" this is the crown jewels... totally don't think helm can do it personally. | 21:57 |
| kfox1111 | sdake: yeah. z upgrade I mean. | 21:57 |
| kfox1111 | "downtime upgrade of just control plane". might be able to do with helm. | 21:57 |
| kfox1111 | "downtime of control + data plane" (I lied, gues there are 4). helm can do it for sure. pretty undesirable though. | 21:58 |
| kfox1111 | I was planning on writing up a spec or document with those spelled out and some thoughts on them. | 21:59 |
| *** schwicht has quit IRC | 21:59 | |
| sdake | please not another spec | 21:59 |
| *** jtriley has quit IRC | 21:59 | |
| kfox1111 | doc then. | 21:59 |
| sdake | wfm | 21:59 |
| *** dave-mcc_ has quit IRC | 21:59 | |
| jascott1 | helm has concept of release and subsets will either be in the single master release or there will be multiple releases at all times right? | 21:59 |
| sbezverk_ | kfox1111: was there an agreement how to name entry point containers inside of pod? | 22:00 |
| kfox1111 | jascott1: not sure... again, depends on if we're talking about just microservice pakages, or uberpackages for helm based deployjent/orchestration | 22:00 |
| jascott1 | we were talking about this before but think we said we werent sure of granularity of upgrades | 22:00 |
| kfox1111 | sbezverk_: the term inc0 came up with I think. | 22:01 |
| sbezverk_ | kfox1111: vswitchd-entrypoint would do? | 22:01 |
| *** jtriley_ has quit IRC | 22:01 | |
| kfox1111 | sbezverk_: "dependencies" ? | 22:01 |
| sbezverk_ | oh that crap name that does not mean too much?!?!?!? | 22:01 |
| kfox1111 | the "dependencies" init container. | 22:02 |
| sbezverk_ | kfox1111: sounds great to me | 22:02 |
| kfox1111 | then you can kubectl logs foo -c dependencies. :) | 22:02 |
| jascott1 | so we have release 1.0.0 with everything. I want to jut uprade nuetron so I bump that sub chart and roll master release 1.0.1 and do upgrade <- this is how we are thinking? | 22:02 |
| rhallisey | dependency init container is the name | 22:02 |
| kfox1111 | jascott1: for those wanting entrypoint. most of us are thinking operators though. | 22:02 |
| kfox1111 | for operators, we just make a new openvswitch-agent package and upgrade only that. | 22:03 |
| sbezverk_ | kfox1111: but even with operator you still need it to wait for depending service to come up no? | 22:03 |
| kfox1111 | so no reason to involve all the rest of neutron if nothing changes there. | 22:03 |
| kfox1111 | sbezverk_: yeah. in a few cases. | 22:03 |
| sbezverk_ | like vswitchd waits for ovsdb socket | 22:03 |
| jascott1 | makes sense for operator but even for entrypoint im concerned about what helm actually does with master release strategy | 22:03 |
| kfox1111 | there's entrypoint for pre deps, and there's entrypoint for workflow. | 22:04 |
| kfox1111 | s/pre/pure/ | 22:04 |
| kfox1111 | jascott1: agreed. honestly, I'm not convinced helm orchestration will wor kwell at all. but a bunch of folks do, so they can do it taht way if it works for them. | 22:04 |
| sbezverk_ | I think entrypoint is excellent and lighweight for dependency check | 22:05 |
| kfox1111 | I personally think operators are a better aproach to tackling the problem, at least for now. | 22:05 |
| jascott1 | we are using entrypoint here to describe non-operator deployment | 22:05 |
| kfox1111 | sbezverk_: it is. but it is used (abused?) by others to orchestrate things too, not just handing deps. | 22:05 |
| kfox1111 | yeah. | 22:06 |
| kfox1111 | I think it could work for a bunch of things and even full deployment, | 22:06 |
| kfox1111 | if upgrades are not considered. | 22:06 |
| sbezverk_ | kfox1111: it is theor problem, if somebody buys hammer and starts hammering his head, it is not a hammer problem | 22:06 |
| jascott1 | lol | 22:06 |
| kfox1111 | sbezverk_: right. | 22:06 |
| kfox1111 | kubernetes-entrypoint is a great tool. | 22:07 |
| jascott1 | sir id like to return this hammer, it is broken. | 22:07 |
| kfox1111 | I'm not sure its a great tool for some of the jobs its currently being employed to tackle. :) | 22:07 |
| *** Pavo has quit IRC | 22:07 | |
| jascott1 | agree | 22:07 |
| kfox1111 | same with helm. | 22:07 |
| kfox1111 | its a great tool. | 22:07 |
| jascott1 | since its not a strategy per-se we sholuld prob call it non-operator | 22:08 |
| kfox1111 | it might not be the best tool for solving every single openstack deployment task. | 22:08 |
| jascott1 | casue operators will likely need to use entrypoint init conatiners (for jsut deps) | 22:08 |
| kfox1111 | there are lots of non-operator strategies. | 22:08 |
| kfox1111 | ansible/chef being others. | 22:08 |
| kfox1111 | I'd be somewhat specific. | 22:08 |
| kfox1111 | I've been calling it helm/entrypoint | 22:08 |
| jascott1 | imean it sounds cool :) | 22:09 |
| kfox1111 | as its orchetration by using just those two tools. | 22:09 |
| *** Pavo has joined #openstack-kolla | 22:11 | |
| jascott1 | are we expecting people to abuse init conatiners in that model? | 22:12 |
| kfox1111 | but some of what I've been trying to do with the helm build scripts, is to allow packages to be created for both workflow styles. | 22:12 |
| kfox1111 | jascott1: absolutely. they don't see it as an abuse, but a prefered, good pattern. | 22:13 |
| *** schwicht has joined #openstack-kolla | 22:13 | |
| kfox1111 | so, if it works for them, I'm ok with it. | 22:13 |
| kfox1111 | I don't think it will work for me though. | 22:13 |
| kfox1111 | but, they can prove me wrong. I'd be happy to be proven wrong. :) | 22:13 |
| sbezverk_ | kfox1111: building packages is great for people who deploy, but I think for development, people should be able to deploy one specific service just using helm chart | 22:13 |
| jascott1 | appreciate answering the hundred questions. trying to understand upgrade path and overlap between operators and helm/entrypoint | 22:13 |
| kfox1111 | its simpler in a lot of ways. | 22:14 |
| kfox1111 | sbezverk_: semi agree.... | 22:14 |
| kfox1111 | what you said is the helm/entrypoint orchestration strategy. | 22:14 |
| kfox1111 | the other way is: | 22:14 |
| kfox1111 | helm install kolla/neutron-operator | 22:15 |
| kfox1111 | kubectl create -f neutron-config-for-service.yaml | 22:15 |
| kfox1111 | then the neutron-operator code kicks in, and launches all the helm micropackages. | 22:15 |
| jascott1 | is that create a TPR? | 22:16 |
| kfox1111 | jascott1: yes. | 22:16 |
| jascott1 | got it | 22:16 |
| jascott1 | but to deliver works out of the box, wont we have to abuse init containers (or do someting else)? | 22:17 |
| kfox1111 | there would be an overarching "compute-kit-operator" too. | 22:17 |
| kfox1111 | helm install kolla/compute-kit-operator | 22:17 |
| kfox1111 | kubectl create -f mycloud.yaml | 22:18 |
| sbezverk_ | kfox1111: but that deploys all of nuetron things!! it is not what we tried to achive with jinja templates, we broke to microservice to achive the best granularity, I still do not get why we need to step back to inferior solution | 22:18 |
| kfox1111 | which would trigger compute-kit operator to deploy neutron,glance,nova,etc. | 22:18 |
| kfox1111 | sbezverk_: right. for microservices, | 22:18 |
| kfox1111 | you can just deploy those directly with the microservice helm packages. | 22:18 |
| kfox1111 | helm install kolla/neutron-l3-agent | 22:19 |
| kfox1111 | that granularity isn't going away. | 22:19 |
| sbezverk_ | kfox1111: but you need to build it, there is not way currently to deploy microservice from the source chart.. | 22:19 |
| kfox1111 | you don't deploy from source charts. | 22:20 |
| kfox1111 | you deploy from built charts. | 22:20 |
| kfox1111 | helm install only takes tgz's. | 22:20 |
| kfox1111 | or entryies from a binary repo. | 22:20 |
| kfox1111 | or am I wrong? | 22:21 |
| sbezverk_ | kfox1111: hm, it is not what I see.. unless when I run helm install openvswitch it builds package under the hood | 22:21 |
| kfox1111 | hmm... so helm can install from an uncompressed chart dir? | 22:21 |
| kfox1111 | interesting.... | 22:21 |
| sbezverk_ | yep | 22:21 |
| sbezverk_ | that is what I do | 22:21 |
| kfox1111 | but I don't see a way for us to really do that. :/ | 22:22 |
| kfox1111 | we don't want to make a values.yaml and Chart.yaml for every microservice by hand, so its static | 22:22 |
| kfox1111 | and keep it up to date. | 22:22 |
| kfox1111 | so we have to build them. | 22:22 |
| jascott1 | ya i have been installing from local dir | 22:22 |
| jascott1 | remove the tar if its there or it will use the tar over local tho | 22:22 |
| jascott1 | brb | 22:23 |
| kfox1111 | long term, my strategy is to have a build pipeline that builds kolla containers and helm microservice packages that point to them. bumping versions as minimally nessisary. | 22:24 |
| kfox1111 | one binary released package update if any of the containers in the k8s object get updated. | 22:24 |
| *** schwicht has quit IRC | 22:25 | |
| kfox1111 | that way, to minor upgrades forwards or backwoards, its just upgrading/downgrading helm packages at that level. | 22:26 |
| *** schwicht has joined #openstack-kolla | 22:26 | |
| kfox1111 | and either a helm/entrypoint uberpackage or an orchestrator does the actual orchestration of that. | 22:26 |
| *** schwicht has quit IRC | 22:29 | |
| *** schwicht has joined #openstack-kolla | 22:29 | |
| kfox1111 | so, I think we need to continue to build values.yaml and Chart.yaml for now. and then create a helm issue to see if they can make those files templatable. | 22:31 |
| kfox1111 | then the build step can be skipped for those that want to try and skip it. | 22:31 |
| *** adrian_otto has joined #openstack-kolla | 22:33 | |
| kfox1111 | I think this is kind of a helm infancy problem. 2.0 is really 1.0 in a way. very little in common with helm classic. so its bound to have some lacking features that it will get by 2.2. | 22:33 |
| kfox1111 | adrian_otto: hi | 22:34 |
| adrian_otto | hi kfox1111 | 22:34 |
| kfox1111 | adrian_otto: https://review.openstack.org/#/c/392257/ | 22:35 |
| adrian_otto | cool, tx kfox1111 | 22:35 |
| kfox1111 | I think we settled on the common bits being very granular, and supporting both operator or helm/entrypoint orchestration for those that want to go that route. | 22:35 |
| kfox1111 | people who what operators can develop that code. those that want helm/entrypoint can there. | 22:36 |
| kfox1111 | and a year or so from now, maybe there will be a clear winner, or both will show their own aproach is valid. | 22:36 |
| kfox1111 | (or everyone's wrong and there will be anothe way. :) | 22:36 |
| sbezverk_ | kfox1111: do you have an example handy how to pass dependency env parameter to entrypoint container? | 22:39 |
| sbezverk_ | if I just do env/name/value do you think it should be the way they expect? | 22:40 |
| kfox1111 | sbezverk_: just the examples here: https://github.com/stackanetes/kubernetes-entrypoint | 22:41 |
| *** schwicht has quit IRC | 22:42 | |
| kfox1111 | DEPENDENCY_SERVICE=mariadb,keystone-api | 22:42 |
| kfox1111 | DEPENDENCY_JOBS=nova-init,neutron-init | 22:42 |
| kfox1111 | etc. | 22:42 |
| sbezverk_ | kfox1111: I saw that one, I was not sure about syntax in init contianer.. | 22:44 |
| kfox1111 | what do you mean? | 22:44 |
| kfox1111 | oh. | 22:44 |
| kfox1111 | they are docker env vars. | 22:45 |
| *** liyifeng has joined #openstack-kolla | 22:46 | |
| kfox1111 | sbezverk_: http://stackoverflow.com/questions/33478555/kubernetes-equivalent-of-env-file-in-docker | 22:49 |
| kfox1111 | sbezverk_: the way I see it, there is not too much difference between requiring packages to be built for helm then for requring a build step for containers. | 22:52 |
| kfox1111 | there isn't really a way to do a docker buildandrun . -it --rm . /bin/bash | 22:53 |
| *** Pavo has quit IRC | 23:12 | |
| jascott1 | kfox1111 a missing piece for me wrt to helm/entrypoint strategy, what are we delivering for kolla-k8s? | 23:15 |
| sdake | jascott1 i dont follow | 23:22 |
| *** Pavo has joined #openstack-kolla | 23:23 | |
| jascott1 | kfox1111 has been saying we have two strategies, operators and helm/entry-point | 23:23 |
| jascott1 | Im trying to understand if/how we will abuse entry-point in kolla-k8s repo | 23:24 |
| jascott1 | I like the idea of init containers being just being that and not abusing them | 23:24 |
| jascott1 | abusing=making them do more than just wait for deps | 23:25 |
| kfox1111 | jascott1: there are those that think kolla-kubernetes should not use operators but instead use helm/entrypoint. | 23:26 |
| kfox1111 | if those folks want to contribute/maintain that code, I'm ok with it. :) | 23:27 |
| kfox1111 | there are those that think thats not a good idea nad instead want to implement/use operators. | 23:27 |
| kfox1111 | if those folks want to contribute/maintain that code, I'm ok with it. :) | 23:27 |
| *** aric49 has quit IRC | 23:27 | |
| kfox1111 | my personal crystalball thinks the operator route is the more likely to produce success longterm. but thats my own personal opinion. I have been wrong before. :) | 23:28 |
| *** Pavo has quit IRC | 23:29 | |
| kfox1111 | I won't push away those that think the helm/entrypoint way actually can be made to successfully fly. maybe it can. | 23:29 |
| jascott1 | ah I guess I am looking for 'official goals' for ocata for the kk8s release. is that the wrong way to think? | 23:29 |
| kfox1111 | jascott1: its an opensource project. its powerd by those willing to write the code. | 23:29 |
| kfox1111 | so, whatever they want to work on will be the stuff that gets done. | 23:30 |
| kfox1111 | or whatever their managers do if they are paid to work on it. | 23:30 |
| *** Jeffrey4l has joined #openstack-kolla | 23:31 | |
| jascott1 | for sure but thought we also had some collective agreed upon roadmap somewhere | 23:31 |
| kfox1111 | I can only speak for my own motivations/managers goals. and they really want a solid base. so mostly the helm microservice package layer and below, and manual orchestration for now. | 23:32 |
| kfox1111 | jascott1: the spec called for a layerd aproach so that everyone would have a place to contribute. | 23:32 |
| kfox1111 | it layed out a plan for operators, | 23:33 |
| kfox1111 | and had some details about entrypoint. | 23:33 |
| sbezverk_ | Entrypoint WARNING: 2016/11/18 23:31:50 entrypoint.go:40: NAMESPACE env not set, using default | 23:34 |
| sbezverk_ | Entrypoint INFO: 2016/11/18 23:31:50 entrypoint.go:61: Resolving /var/run/openvswitch/db.sock | 23:34 |
| sbezverk_ | Entrypoint INFO: 2016/11/18 23:31:52 entrypoint.go:70: Dependency /var/run/openvswitch/db.sock is resolved | 23:34 |
| sbezverk_ | it looks ok | 23:34 |
| jascott1 | kfox1111 thanks | 23:34 |
| kfox1111 | but the exact solution for helm/entrypoint will have to come from those that know how it works. I don't really understand how it could work at openstack scale. | 23:34 |
| *** Pavo has joined #openstack-kolla | 23:34 | |
| kfox1111 | sbezverk_: probably need to set NAMESPACE to whatever helm is using. probably a var in the .Chart | 23:35 |
| kfox1111 | sbezverk_: but your right, looks good. :) | 23:35 |
| kfox1111 | sbezverk_: well, except maybe one thing... | 23:35 |
| kfox1111 | does it check for the existance of the socket, or its operation? | 23:36 |
| kfox1111 | it could still pass even if openvswitch wasnt running, depending on how it checks. | 23:36 |
| kfox1111 | in which case, it may be better to keep the actual socket off of /dev/shm or something, so it disapears at reboot. | 23:36 |
| sbezverk_ | kfox1111: good question, no idea, that is probably question to entrypoint developers | 23:37 |
| kfox1111 | sbezverk_: i'm guessing its per service. | 23:38 |
| sdake | jascott1 ya - we dont have "official goals" - people work on what they want to work on | 23:38 |
| kfox1111 | nscd for example just ignores calls if the socket exists but the daemon isn't listening yet. | 23:38 |
| kfox1111 | whre as most socket based things block until the server picks up. | 23:38 |
| kfox1111 | so only the socket's existince is important. | 23:38 |
| sdake | does anyone know if Jeffrey4l fixed the ansible gate? | 23:39 |
| sbezverk_ | kfox1111: who was the guy you submitted issue to for entrypoint? | 23:40 |
| sbezverk_ | I think he could answer your concern. | 23:40 |
| sdake | its not sufficient to check for the presence of the socket | 23:41 |
| sdake | the socket must be inspected via its API to make sure things are rolling | 23:41 |
| jascott1 | thats all k8s socket probes do afaik | 23:41 |
| sdake | or badness ensues | 23:41 |
| sdake | we suffered this with either mariadb or haproxy or rabbitmq | 23:41 |
| sdake | i forget which | 23:42 |
| *** TxGirlGeek has quit IRC | 23:42 | |
| kfox1111 | sbezverk_: I just did to the issue tracker for it. | 23:42 |
| sdake | we had to add alot of work to actually check the service was rolling before using it in bootstrap mode | 23:42 |
| sdake | because sockets get created before sockets provide service | 23:42 |
| kfox1111 | sdake: yeah. depends on the service. if it opens the unix socket blocking or nonblocking. | 23:42 |
| jascott1 | other projects like os-helm or stackanetes using either just socket or http for most probes and exec for the ones you just mentioned sdake | 23:42 |
| kfox1111 | if blocking, all is well. it will wait for the server on the other end to launch. | 23:43 |
| kfox1111 | nonblocking, things just break. :/ | 23:43 |
| kfox1111 | readyness probes are a good way to handle that too. | 23:43 |
| sdake | probably one workaround would be to change the socket from blocking to nonblocking after the service is up then | 23:43 |
| sdake | although that requires code changes in dependent software | 23:43 |
| kfox1111 | make an exec readyness hook that calls mariadb client inside the mariadb server, and ensure its respoinsive. | 23:44 |
| jascott1 | fyi a little survey of what others are doing https://gist.github.com/jascott1/a65b787af0750bafd161d47ea48b17dd | 23:44 |
| kfox1111 | jascott1: that list is incomplete. | 23:44 |
| jascott1 | for kk8s? | 23:45 |
| jascott1 | yeah there is a common template right | 23:45 |
| kfox1111 | yeah. most of our services use the following template: services/common/common-deployment.yml.j2 | 23:45 |
| kfox1111 | yeah. | 23:45 |
| sdake | kfox1111 by incomplete do you mean the other 7 or 8 implementations of openstack on kubernetes? :) | 23:46 |
| kfox1111 | sdake: no. just our list. | 23:46 |
| kfox1111 | sdake: that just makes me sad. :/ | 23:46 |
| kfox1111 | thanks for that... :) | 23:46 |
| sdake | kfox1111 pretend like you didn't hear it | 23:47 |
| sdake | I do that alot ;) | 23:47 |
| sbezverk_ | do you guys can recommend any IDE for go lang for windows? | 23:47 |
| kfox1111 | humans are really good at reinventing the wheel/not playing nice with each other. | 23:48 |
| sdake | sbezverk_ windows ide golang | 23:48 |
| sdake | oxymoron? :) | 23:48 |
| sdake | kfox1111 there is no DSL for human beings :) | 23:48 |
| sbezverk_ | sdake: go lang works on windows | 23:48 |
| sbezverk_ | just in cli | 23:48 |
| sdake | sbezverk_ i was talking about all 3 together | 23:48 |
| jascott1 | vim-go | 23:48 |
| jascott1 | in fancy winbashhell | 23:48 |
| kfox1111 | sdake: I know.. sucks. :/ | 23:49 |
| sdake | kfox1111 actually it is what creates greatness in humans | 23:49 |
| kfox1111 | sbezverk_: ubuntu for windows + vim? :) | 23:49 |
| jascott1 | sbezverk_ if you like jetbrains stuff they have a nice golang plugin | 23:49 |
| kfox1111 | sdake: yeah. mixed bag. :) | 23:49 |
| sdake | kfox1111 ya curse and blessing at same time | 23:49 |
| kfox1111 | sdake: you read the dune series? | 23:49 |
| sbezverk_ | jascott1: thanks it is much better option than others were suggesting ;-) | 23:50 |
| sdake | when i was a kid about 30 years ago | 23:50 |
| sdake | i dont remember it | 23:50 |
| *** fragatin_ has joined #openstack-kolla | 23:50 | |
| kfox1111 | golden path man... | 23:50 |
| kfox1111 | if you havent read the prequals, preprequals, and the final book, its worth it I think. | 23:50 |
| kfox1111 | the stuff newer then that, not so much. | 23:51 |
| sdake | ya i remember that part | 23:51 |
| kfox1111 | the final book explains the golden path finally. | 23:51 |
| kfox1111 | but so much better when you have all the prequal material too. | 23:51 |
| kfox1111 | ties it all together with a nice bow on top. :) | 23:51 |
| jascott1 | sbezverk_ atom is not bad but kinda slow and alittle buggy | 23:53 |
| *** fragatina has quit IRC | 23:53 | |
| jascott1 | tell me this, does Muadib take over or even live very long? | 23:55 |
| kfox1111 | not very long. | 23:56 |
| jascott1 | cool | 23:56 |
| kfox1111 | his son though.... | 23:56 |
| *** schwicht has joined #openstack-kolla | 23:56 | |
| jascott1 | imean loved the character but was hoping it was epic scale story | 23:56 |
| kfox1111 | yeah. it totally is epic. | 23:57 |
| *** srwilker has joined #openstack-kolla | 23:57 | |
| *** sdake_ has joined #openstack-kolla | 23:57 | |
| jascott1 | i read one of Franks books in Dune universe about 25 years ago.. they were on a water planet.. no idea what its called now | 23:57 |
| kfox1111 | the writer didn't get to totally finish it, but the son did, and did a fair job. | 23:57 |
| kfox1111 | arakis | 23:57 |
| kfox1111 | oh. | 23:57 |
| jascott1 | i know that one from the movie :) | 23:57 |
| rhallisey | no clue what dune is | 23:58 |
| jascott1 | it may have been but they lived on these islands made of orgranic rock | 23:58 |
| srwilker | Oh man, Frank Herbert talk? What a conversation to join in | 23:58 |
| kfox1111 | I think. caladan | 23:58 |
| kfox1111 | sdake: hehe. | 23:58 |
| sbezverk_ | I still keep dvd version of old Dune movie :-) | 23:58 |
| jascott1 | that they 'fed' waste to make them grow like coral | 23:58 |
| kfox1111 | rhallisey: big scifi universe | 23:59 |
| srwilker | sbezverk_: I have it too | 23:59 |
| jascott1 | I made myself wait to get 5.1 sound before watching it with the kids | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!