| *** TxGirlGeek has quit IRC | 00:00 | |
| sdake | well i'll assume you have - but in case you haven't, its brilliant work by a brilliant team that is committed to solving this problem the right way | 00:00 |
|---|---|---|
| sdake | but dont take my word for it, go read it | 00:00 |
| sdake | and if you have read it, comment on what you dont like | 00:01 |
| sdake | rather then create a parallel implementation | 00:01 |
| Pavo | well finally figured out why none of you guys can get to my horizon | 00:04 |
| Pavo | using tls | 00:04 |
| Pavo | my stupid ISP blocks 443 | 00:04 |
| rhallisey | git review is hanging :/ | 00:04 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture https://review.openstack.org/392257 | 00:04 |
| rhallisey | there we go | 00:04 |
| rhallisey | it skipped patch 10 O.o | 00:05 |
| srwilkers | go home gerrit, youre drunk | 00:08 |
| *** dave-mccowan has joined #openstack-kolla | 00:17 | |
| *** inc0 has quit IRC | 00:24 | |
| *** inc0 has joined #openstack-kolla | 00:24 | |
| *** schwicht has joined #openstack-kolla | 00:32 | |
| kfox1111 | Pavo: bummer. | 00:48 |
| *** zhubingbing has joined #openstack-kolla | 00:50 | |
| *** awiddersheim has quit IRC | 00:54 | |
| sdake | rhallisey i complained more on your spec | 00:54 |
| sdake | rhallisey can you fix it quickly | 00:54 |
| kfox1111 | I'm commenting too right now. | 00:55 |
| sdake | rhallisey also Duong wants to contribute as another member of your small army ;) | 00:55 |
| sdake | and Qin Wang | 00:55 |
| sdake | QIn Wang (qnang) is what the comment says | 00:55 |
| sdake | Duang Ha-Quang (duonghq) | 00:56 |
| sdake | sorry duong Ha-Quang (duonghq) | 00:56 |
| sdake | with a capital D :) | 00:56 |
| sdake | keyboard blows | 00:56 |
| sdake | pavo you can run it on a port other then 443 | 00:58 |
| sdake | rhallisey ^^ | 01:00 |
| sdake | i guess ryan went to walk his dog or something ;) | 01:00 |
| sdake | bbl | 01:00 |
| *** duonghq has joined #openstack-kolla | 01:00 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: add new line in cleanup-containers https://review.openstack.org/397454 | 01:04 |
| rhallisey | kk | 01:05 |
| *** tovin07_ has joined #openstack-kolla | 01:05 | |
| zhubingbing | -) | 01:08 |
| sdake | zhubingbing just whats needed, another newline | 01:09 |
| sdake | you realize those newlines take up precious disk space | 01:09 |
| sdake | -) | 01:09 |
| zhubingbing | But other scripts are written in this way | 01:09 |
| *** inc0 has quit IRC | 01:10 | |
| zhubingbing | I think the script format should be unified | 01:11 |
| zhubingbing | so..... | 01:11 |
| zhubingbing | -) | 01:11 |
| duonghq | morning | 01:12 |
| zhubingbing | morning | 01:12 |
| sdake | sup duonghq | 01:12 |
| zhubingbing | sup what's | 01:12 |
| *** schwicht has quit IRC | 01:12 | |
| zhubingbing | mean | 01:12 |
| zhubingbing | -) | 01:12 |
| sdake | sup = whats up | 01:12 |
| zhubingbing | wow， | 01:13 |
| duonghq | zhubingbing, do we have any convention on it? | 01:13 |
| zhubingbing | -) ok | 01:14 |
| *** harbor has joined #openstack-kolla | 01:14 | |
| *** harbor is now known as harbor_ | 01:14 | |
| *** harbor_ is now known as portdirect_ | 01:14 | |
| duonghq | I think your idea it's nice , but it should be in some docs. | 01:14 |
| tovin07_ | morning | 01:14 |
| zhubingbing | https://review.openstack.org/#/c/397454/ | 01:15 |
| zhubingbing | Do you think there's any problem with that？ | 01:16 |
| duonghq | no problem | 01:17 |
| duonghq | I mean that it's coding convention-related | 01:17 |
| sdake | inc0 was there a conclusion to the vote on trivialfix | 01:17 |
| sdake | hrm | 01:17 |
| duonghq | so, it should be in some documentation or guideline or .... (IMO) | 01:17 |
| duonghq | do we have any? | 01:17 |
| duonghq | maybe not really need for this kind of format | 01:18 |
| duonghq | do not sure | 01:18 |
| zhubingbing | I think the document is one of our major defects | 01:18 |
| duonghq | I do not think so, convention maybe very small thing | 01:19 |
| duonghq | I'm no sure | 01:19 |
| duonghq | *not | 01:19 |
| zhubingbing | need to slowly improve | 01:19 |
| duonghq | one more think you can consider: standardize shebang | 01:19 |
| *** schwicht has joined #openstack-kolla | 01:19 | |
| duonghq | I prefer /usr/bin/env bash over sheer /bin/bash | 01:19 |
| *** g3ek has quit IRC | 01:20 | |
| *** haplo37 has quit IRC | 01:20 | |
| duonghq | also /usr/bin/env python... | 01:21 |
| sdake | say dumb q | 01:21 |
| sdake | kfox1111 rhallisey - after reading spec i'm confused about this job operator thing | 01:21 |
| sdake | rather job pod | 01:21 |
| sdake | is that a shell script that runs and does its thing only once? | 01:21 |
| kfox1111 | sdake: kind of. k8s has a job object type. | 01:22 |
| sdake | kfox1111 i got that part ;) | 01:22 |
| kfox1111 | an ormal pod will automatically restart if the container fails. | 01:22 |
| sdake | i'm wondering how it works | 01:22 |
| kfox1111 | a job stops if it successfully exit's 0. | 01:22 |
| sdake | if pod restarts, does job restart? | 01:22 |
| kfox1111 | so you can use it for tasks. | 01:22 |
| sdake | i asked this a few days ago and the answer was yes | 01:23 |
| kfox1111 | for a job? yes. | 01:23 |
| kfox1111 | so, like, | 01:23 |
| kfox1111 | say you ahve a job, and the k8s node its running on dies half way through. | 01:23 |
| kfox1111 | the job will just get restarted on another node. | 01:23 |
| sdake | ok, but what if the pod fails after the job has finished? | 01:24 |
| zhubingbing | K8s this work mechanism is mainly dependent on etcd | 01:24 |
| kfox1111 | the job is done if the container exits 0. | 01:24 |
| sdake | and never starts again? | 01:24 |
| kfox1111 | so if the job completes, it won't get rescheduled. | 01:24 |
| kfox1111 | right. it just sticks around for output. | 01:25 |
| kfox1111 | you are free to delete it after you figure out that it finished. | 01:25 |
| kfox1111 | but you can leave it around too I think. | 01:25 |
| sdake | where does its config come from | 01:25 |
| kfox1111 | I always delete them though. | 01:25 |
| kfox1111 | wherever. | 01:25 |
| *** haplo37 has joined #openstack-kolla | 01:25 | |
| kfox1111 | a job is just like a normal pod, with that one slight difference. | 01:25 |
| sdake | where will it come from in our architecture | 01:25 |
| kfox1111 | configmaps and or secrets. | 01:26 |
| *** g3ek has joined #openstack-kolla | 01:26 | |
| sdake | and those come from the operator that lods in the configmap from the thirdparty config? | 01:26 |
| kfox1111 | ('ve not been talking about secrets so far. to not confuse the conversation. think of them as mostly the same thing though) | 01:26 |
| kfox1111 | either they come from an operator loaded through a thirdparty config, | 01:27 |
| sdake | secrets / config maps seem like similiar things to me | 01:27 |
| sdake | so thats fine | 01:27 |
| kfox1111 | or directly by a human if they want to go that route. | 01:27 |
| kfox1111 | yeah. its mostly bookeeping. | 01:27 |
| sdake | so we need to implement a job pod as well? | 01:27 |
| kfox1111 | we have jobs already. | 01:28 |
| sdake | mind showing me one that does the db registration | 01:28 |
| kfox1111 | sure. | 01:28 |
| sdake | db creation that is | 01:28 |
| kfox1111 | all of our k8s objects are currently under services/ | 01:28 |
| kfox1111 | I've been making things as generic as possible, so there are some common jobs now. | 01:29 |
| kfox1111 | but | 01:29 |
| sdake | if a job is a container, then it contains some kind of code to create a db, right? | 01:29 |
| *** awiddersheim has joined #openstack-kolla | 01:30 | |
| kfox1111 | actually, the createdb code isn't as clean as I'd like it. there are probably better examples. | 01:30 |
| sdake | mind showing me what you got so I don't have to hunt for it ;) | 01:31 |
| kfox1111 | https://github.com/openstack/kolla-kubernetes/blob/master/services/common/common-create-keystone-user.yml.j2 | 01:31 |
| kfox1111 | here's one that creates a services keystone user. | 01:31 |
| sdake | how does it know where to get its configmap? | 01:32 |
| kfox1111 | note, it does use the kolla-toolbox to create the user. though I think that woudl be easy/desirable to just switch out with the unified openstack cli. | 01:32 |
| kfox1111 | in this case, its using a secret. | 01:32 |
| kfox1111 | line 49 | 01:32 |
| sdake | wfm | 01:33 |
| sdake | i think the spec is wrong tho | 01:33 |
| sdake | around lines 5-8 | 01:33 |
| sdake | and possibly above | 01:33 |
| portdirect_ | nice: didn't realise you could use downward api to expose secrets | 01:33 |
| kfox1111 | portdirect_: some really awesome things you can do with k8s. :) | 01:34 |
| kfox1111 | sdake: not sure I follow. what you thinking? | 01:34 |
| kfox1111 | gota head out in 5 min. | 01:34 |
| portdirect_ | moving so fast - couldnt do that with 1.2... | 01:34 |
| sdake | kfox1111 what i'm getting at is the spec is incorrect since the operator itself wont be doing db registration or role creation or db user creation | 01:35 |
| kfox1111 | portdirect_: yeah. init containers are really awesome too. | 01:35 |
| sdake | instead the job does it | 01:35 |
| sdake | rhallisey ^^ | 01:36 |
| kfox1111 | sdake: kind of, yeah. an operator just runs k8s/helm. | 01:36 |
| sdake | and does dependency management | 01:36 |
| sdake | and handles complex actions | 01:36 |
| kfox1111 | and orchestrates the order of initial creations of things. | 01:36 |
| rhallisey | k | 01:36 |
| sdake | bootstrapping is not a complex action | 01:36 |
| kfox1111 | portdirect_: if you haven't played with init containers, I'd recommend doing so. they requre some ugly json encoding for now, but really are super useful. | 01:37 |
| sdake | makes alot of sense now - i was wondering how you were goin to do #2 without operators since there was no db around | 01:37 |
| sdake | i think we need a special container just to handle the init jobs | 01:37 |
| portdirect_ | kfox1111: been using them for ages :) https://github.com/portdirect/harbor/blob/latest/kubernetes/templates/keystone/controllers.yaml#L301 | 01:37 |
| rhallisey | ya there's no point of a job | 01:37 |
| sdake | so that it can detect duplicate dbs and whatnot | 01:37 |
| kfox1111 | sdake: for reference, have alook at: | 01:37 |
| kfox1111 | https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/ceph_workflow.sh | 01:37 |
| kfox1111 | there's our current deployment workflow in gate. | 01:38 |
| *** Pavo has quit IRC | 01:38 | |
| sdake | kfox1111 cool so thats what we need operate | 01:38 |
| sdake | rather automate | 01:38 |
| kfox1111 | sdake: yes. :) | 01:38 |
| sdake | the problem with that workflow is it lacks actions (such as reconfigure and upgrade) | 01:39 |
| *** Pavo has joined #openstack-kolla | 01:39 | |
| sdake | not that we need to handle it now, but it needs to be handled | 01:39 |
| kfox1111 | if you can take that one file and make it one kolla operator command, we're all set. :) | 01:39 |
| sdake | i think the implementation you ahve is fine | 01:39 |
| sdake | if ansible crashes midstream, bad things could happen ;-) | 01:39 |
| kfox1111 | yeah. I'm jsut saying, thats ultimately what the spec's going to do in operators. | 01:39 |
| kfox1111 | with better error handling. :) | 01:40 |
| sdake | oh you mean take that out of jobs and do it in operators? | 01:40 |
| sdake | that doesn't seem to make sense ;) | 01:40 |
| kfox1111 | that script just creates a set of k8s objects in a particular order, and waits for things to spawn handeling deps. | 01:40 |
| sdake | i mean the other one | 01:40 |
| kfox1111 | it is an implementation of a deployment workflow. | 01:40 |
| sdake | the first one you linked | 01:40 |
| kfox1111 | the job is staying | 01:41 |
| kfox1111 | the ceph_workflow launches it. :) | 01:41 |
| kfox1111 | https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/ceph_workflow.sh#L36 here for example. | 01:42 |
| kfox1111 | search for bootstrap in the ceph-workflow. those are all jobs I think. | 01:42 |
| sdake | right - that seems correct, so why put that logic in an operator? | 01:43 |
| kfox1111 | gotta head out. talk to you later. | 01:43 |
| sdake | later | 01:43 |
| kfox1111 | which logic? | 01:43 |
| sdake | the keystone-manage-db logic for example | 01:43 |
| sdake | the docker exec logic | 01:43 |
| kfox1111 | the logic of how to manage the db is in the job, so a human can manually run the job if they need. | 01:43 |
| sdake | right | 01:44 |
| sdake | so why have that in an operator? | 01:44 |
| kfox1111 | the k8s operator just runs things like a human would if the human doesn't want to. | 01:44 |
| sdake | oh, i thought it was down a layer | 01:44 |
| sdake | rather down a couple layers | 01:44 |
| kfox1111 | the job is packaged in helm. the operator launches the job through helm. | 01:44 |
| kfox1111 | so, kind of. | 01:45 |
| sdake | right, so the job always launched | 01:45 |
| *** dave-mccowan has quit IRC | 01:45 | |
| sdake | launches | 01:45 |
| sdake | so why put that in an operator at all | 01:45 |
| sdake | because it always happens | 01:45 |
| kfox1111 | someone's got to launch the job. | 01:45 |
| kfox1111 | the idea of the operators is so that the humans don't have to. | 01:45 |
| sdake | ok so operator has to launch the job then? | 01:45 |
| *** hfu has joined #openstack-kolla | 01:45 | |
| kfox1111 | the job has to be run to completion before the next step in the workflow can work too. | 01:46 |
| *** fragatin_ has joined #openstack-kolla | 01:46 | |
| kfox1111 | so an ideal spot for the operator. | 01:46 |
| kfox1111 | yeah. | 01:46 |
| portdirect_ | the operator needs to know if the job has finished i think? | 01:46 |
| sdake | jobs aren't linked to their objects? | 01:46 |
| kfox1111 | yeah. look at the ceph-workflow logic. | 01:46 |
| kfox1111 | there are places where it waits until jobs complate. | 01:46 |
| kfox1111 | thats orchestration logic. | 01:46 |
| sdake | i see | 01:46 |
| sdake | so jobs are parallel launched | 01:46 |
| kfox1111 | some are. | 01:46 |
| kfox1111 | there are esentially bariors in that workflow though. | 01:46 |
| sdake | the issue in the spec is a conflict in coherence | 01:47 |
| kfox1111 | I did as much in parallel as I could, batching up things that could be in parallel together. | 01:47 |
| kfox1111 | so it has implicit deps coded into it. | 01:47 |
| *** fragatin_ has quit IRC | 01:47 | |
| kfox1111 | which kind of sucks. but works. | 01:47 |
| sdake | in one place we say we are registering dbs via operators in another part we say during init containers | 01:47 |
| sdake | if the answer is "both depending on circumstance" I guess that should be written down to reduce confusion | 01:48 |
| kfox1111 | we init all db's in jobs laucnhed via operators | 01:48 |
| kfox1111 | never in an init-container. | 01:48 |
| *** fragatin_ has joined #openstack-kolla | 01:48 | |
| sdake | rhallisey ^^ | 01:48 |
| kfox1111 | init containers are intended to init an instance of a new pod. | 01:48 |
| kfox1111 | not for initing a distributed system. | 01:48 |
| *** fragatina has quit IRC | 01:49 | |
| kfox1111 | jobs are better for initing a distributed system, as it can be limited to 1 run. | 01:49 |
| sdake | i see 206-208 | 01:49 |
| sdake | so operator launches a job, not initializes the database | 01:49 |
| kfox1111 | right. | 01:50 |
| kfox1111 | it launches a job, and that job initializes the db. | 01:50 |
| sdake | 52-57 should be clarified "An operator launches a job which initializes a db" | 01:50 |
| kfox1111 | and the operator waits until completion. | 01:50 |
| *** fragatin_ has quit IRC | 01:50 | |
| kfox1111 | sdake: yeah. 52-57 looks wrong from that perspective. the other secion I think is the correct one. | 01:51 |
| sdake | kfox1111 right | 01:51 |
| portdirect_ | yeah | 01:51 |
| sdake | kfox1111 you understand where i'm coing form now :) | 01:51 |
| kfox1111 | ok. getting mroe in trouble. bbl. :) | 01:51 |
| kfox1111 | sdake: yeah. :) | 01:51 |
| *** kfox1111 is now known as kfox1111_away | 01:51 | |
| portdirect_ | it should be at 53 if that section remains | 01:51 |
| portdirect_ | whoops between 55 and 56 | 01:51 |
| openstackgerrit | lijing proposed openstack/kolla: Word choice for back end https://review.openstack.org/390788 | 01:53 |
| portdirect_ | sdake: i missed a load of discussion - i'll try catch up tomorrow, but just wanted to clarify at line 192 in the spec, it refers to existing config files - does this refer to file from a previous config, or default config files for a service? | 01:55 |
| sdake | portdirect_ i just responded to your comment | 01:56 |
| sdake | in the review | 01:56 |
| sdake | basically teh answer is either option works | 01:56 |
| sdake | we want kolla to be dead simple to use | 01:57 |
| sdake | or atlast I want that | 01:57 |
| sdake | and having operators jerking around with kubectl commands doesn't meet that test | 01:57 |
| sdake | but on the other hand, if they really want to they can | 01:57 |
| sdake | operators provide workflow where the warm blooded kind don't want to do their own workflow | 01:57 |
| portdirect_ | sdake ++, that sounds like how it should be :) | 01:58 |
| rhallisey | k back | 01:59 |
| *** zhurong has joined #openstack-kolla | 02:08 | |
| *** zhurong has quit IRC | 02:08 | |
| *** portdirect_ is now known as portdirect_away_ | 02:10 | |
| *** schwicht has quit IRC | 02:19 | |
| *** portdirect_away_ is now known as portdirect_ | 02:21 | |
| portdirect_ | one more thing... | 02:22 |
| portdirect_ | I've been thinking - it would be possible to run the current kolla entrypoint as an init container, and then launch the kolla container with a read-only rootfs in kolla-kubernetes | 02:23 |
| portdirect_ | which would offer some advantages from a security PoV | 02:24 |
| sdake | how is read-only elimited without entrypoint? | 02:30 |
| *** Pavo_ has joined #openstack-kolla | 02:34 | |
| *** Pavo has quit IRC | 02:34 | |
| *** Pavo_ is now known as Pavo | 02:34 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture https://review.openstack.org/392257 | 02:36 |
| rhallisey | sdake, check that out | 02:37 |
| rhallisey | I'll check in on the comments tmr | 02:37 |
| *** rhallisey has quit IRC | 02:37 | |
| *** Pavo has quit IRC | 02:38 | |
| portdirect_ | sdake: just hunted though - i think your right, there would be no need - the only place it would be requred (i think) with the current implementation is for the few serives that run behind apache - but that could be sorted by delacering the apache config dirs as emptydir volumes in the pod spec. | 02:40 |
| *** portdirect_ is now known as portdirect_away_ | 02:42 | |
| *** Pavo has joined #openstack-kolla | 02:43 | |
| *** yuanying has quit IRC | 02:51 | |
| *** unicell1 has quit IRC | 03:12 | |
| *** v1k0d3n has quit IRC | 03:27 | |
| *** fragatina has joined #openstack-kolla | 03:29 | |
| *** sdake has quit IRC | 03:36 | |
| *** Pavo has quit IRC | 03:38 | |
| *** yuanying has joined #openstack-kolla | 03:39 | |
| *** Pavo has joined #openstack-kolla | 03:40 | |
| *** srwilkers has quit IRC | 03:41 | |
| *** sdake has joined #openstack-kolla | 03:44 | |
| sdake | portdirect_away how did you interpret no need from that? | 03:44 |
| sdake | :) | 03:44 |
| sdake | i like security | 03:45 |
| sdake | its a critical weakness of mine :( | 03:45 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' to neccesary tasks (general roles) https://review.openstack.org/358539 | 03:45 |
| *** yuanying has quit IRC | 03:45 | |
| *** yuanying has joined #openstack-kolla | 03:49 | |
| *** sdake has quit IRC | 04:03 | |
| *** negronjl has quit IRC | 04:03 | |
| *** chas_ has joined #openstack-kolla | 04:13 | |
| *** chas_ has quit IRC | 04:17 | |
| *** v1k0d3n has joined #openstack-kolla | 04:19 | |
| *** sdake has joined #openstack-kolla | 04:22 | |
| *** severion has joined #openstack-kolla | 04:23 | |
| *** v1k0d3n has quit IRC | 04:25 | |
| *** v1k0d3n has joined #openstack-kolla | 04:27 | |
| *** severion has quit IRC | 04:29 | |
| *** coolsvap has joined #openstack-kolla | 04:43 | |
| sdake | sup coolsvap | 05:00 |
| coolsvap | sdake: hey | 05:01 |
| *** khamtamtun has joined #openstack-kolla | 05:04 | |
| *** khamtamtun has quit IRC | 05:21 | |
| coolsvap | sdake: submitted review to add kolla-ansible to global requirements https://review.openstack.org/397532 | 05:23 |
| sdake | for which project? | 05:23 |
| sdake | did you mean to add kolla to global requirements? | 05:24 |
| coolsvap | kolla-ansible | 05:24 |
| coolsvap | no, to sync with g-r | 05:24 |
| sdake | oh got it | 05:24 |
| sdake | thanks, didn't know that was a thing | 05:24 |
| sdake | i thought you meant requirements.txt | 05:25 |
| coolsvap | so are we planning to git pull for building images in kolla-ansible? | 05:25 |
| *** fragatina has quit IRC | 05:30 | |
| *** fragatina has joined #openstack-kolla | 05:31 | |
| *** fragatina has quit IRC | 05:35 | |
| *** chas_ has joined #openstack-kolla | 05:36 | |
| *** Pavo has quit IRC | 05:38 | |
| *** unicell has joined #openstack-kolla | 05:38 | |
| *** chas_ has quit IRC | 05:40 | |
| *** Pavo has joined #openstack-kolla | 05:41 | |
| *** berendt has joined #openstack-kolla | 05:43 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Revert "corrects invalid logrotate option maxsize" https://review.openstack.org/397214 | 05:53 |
| *** msimonin has joined #openstack-kolla | 05:58 | |
| openstackgerrit | Merged openstack/kolla: add new line in cleanup-containers https://review.openstack.org/397454 | 06:01 |
| *** Jeffrey4l has quit IRC | 06:03 | |
| *** unicell1 has joined #openstack-kolla | 06:04 | |
| *** unicell has quit IRC | 06:05 | |
| *** khamtamtun has joined #openstack-kolla | 06:17 | |
| *** msimonin has quit IRC | 06:22 | |
| *** khamtamtun has quit IRC | 06:35 | |
| *** mdnadeem has joined #openstack-kolla | 06:36 | |
| *** mdnadeem has quit IRC | 06:40 | |
| *** Jeffrey4l has joined #openstack-kolla | 06:41 | |
| openstackgerrit | zhongshengping proposed openstack/kolla: Deprecate scheduler_max_attempts option in nova https://review.openstack.org/397547 | 06:42 |
| duonghq | if we run deploy on existing system, does the mariadb is replace/erase? | 06:42 |
| *** khamtamtun has joined #openstack-kolla | 06:48 | |
| berendt | duonghq i do not think so | 06:48 |
| duonghq | thank berendt, I think so, but cannot get a cluster up for test now | 06:49 |
| berendt | my test environment is broken at the moment :( | 06:50 |
| duonghq | how is it broken? | 06:55 |
| berendt | i am restructuring my CI at the moment, broken means the systems are not installed at the moment | 06:56 |
| duonghq | ah, ok | 06:56 |
| *** nadeem_ has joined #openstack-kolla | 06:59 | |
| *** nadeem_ has quit IRC | 07:06 | |
| *** chas_ has joined #openstack-kolla | 07:07 | |
| *** YuYangWang has joined #openstack-kolla | 07:18 | |
| *** msimonin has joined #openstack-kolla | 07:19 | |
| *** msimonin has quit IRC | 07:23 | |
| *** nadeem_ has joined #openstack-kolla | 07:32 | |
| *** Pavo has quit IRC | 07:38 | |
| duonghq | which is deadline of kolla-k8s specs? | 07:42 |
| *** tonanhngo has quit IRC | 07:42 | |
| *** Pavo has joined #openstack-kolla | 07:43 | |
| *** nadeem_ has quit IRC | 07:47 | |
| openstackgerrit | Jeremy Liu proposed openstack/kolla: Word choice for back end https://review.openstack.org/390788 | 07:48 |
| *** sdake_ has joined #openstack-kolla | 07:49 | |
| *** sdake has quit IRC | 07:53 | |
| *** tonanhngo has joined #openstack-kolla | 08:01 | |
| *** tonanhngo has quit IRC | 08:02 | |
| *** neilus has joined #openstack-kolla | 08:03 | |
| *** nadeem_ has joined #openstack-kolla | 08:04 | |
| *** matrohon has joined #openstack-kolla | 08:10 | |
| *** magicboiz has quit IRC | 08:11 | |
| *** magicboiz has joined #openstack-kolla | 08:11 | |
| *** neilus has quit IRC | 08:20 | |
| *** tovin07_ has quit IRC | 08:24 | |
| *** Mech422 has quit IRC | 08:28 | |
| *** Mech422 has joined #openstack-kolla | 08:28 | |
| *** egonzalez90 has joined #openstack-kolla | 08:30 | |
| *** shardy has joined #openstack-kolla | 08:35 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Update the default value for Heat https://review.openstack.org/397577 | 08:38 |
| *** YuYangWang has quit IRC | 08:50 | |
| *** Serlex has joined #openstack-kolla | 08:55 | |
| *** athomas has joined #openstack-kolla | 09:03 | |
| *** gfidente has joined #openstack-kolla | 09:13 | |
| *** DTadrzak has joined #openstack-kolla | 09:15 | |
| *** openstackgerrit has quit IRC | 09:18 | |
| *** openstackgerrit has joined #openstack-kolla | 09:18 | |
| *** clsacramento has joined #openstack-kolla | 09:21 | |
| *** msimonin has joined #openstack-kolla | 09:23 | |
| *** sdake_ has quit IRC | 09:23 | |
| *** msimonin has left #openstack-kolla | 09:23 | |
| *** Serlex has quit IRC | 09:28 | |
| *** msimonin1 has joined #openstack-kolla | 09:28 | |
| *** neilus has joined #openstack-kolla | 09:31 | |
| *** msimonin1 has quit IRC | 09:32 | |
| *** msimonin has joined #openstack-kolla | 09:34 | |
| *** neilus has quit IRC | 09:36 | |
| *** Pavo has quit IRC | 09:38 | |
| berendt | where can i find the vagrant environment for kolla-kubernetes? | 09:41 |
| berendt | https://github.com/att-comdev/halcyon-vagrant-kubernetes | 09:42 |
| berendt | this is not for kolla-kubernetes, but that is the environment i looked for ;) | 09:43 |
| *** Pavo has joined #openstack-kolla | 09:43 | |
| *** hfu has quit IRC | 09:44 | |
| *** neilus has joined #openstack-kolla | 09:45 | |
| *** neilus has quit IRC | 09:48 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Move the "enable_destroy_images" into configure file https://review.openstack.org/397633 | 09:56 |
| *** Serlex has joined #openstack-kolla | 09:58 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Move the "enable_destroy_images" into configure file https://review.openstack.org/397633 | 09:58 |
| openstackgerrit | caoyuan proposed openstack/kolla: Move the "enable_destroy_images" into configure file https://review.openstack.org/397633 | 10:00 |
| *** sdake has joined #openstack-kolla | 10:02 | |
| *** papacz has joined #openstack-kolla | 10:02 | |
| *** neilus has joined #openstack-kolla | 10:04 | |
| sdake | evening folks | 10:05 |
| sdake | Jeffrey4l you about | 10:05 |
| egonzalez90 | eveing sdake | 10:05 |
| sdake | sup egonzalez90 | 10:05 |
| openstackgerrit | zhubingbing proposed openstack/kolla: change network_bw_out to network.bw.out in cloudkitty.conf.j2 https://review.openstack.org/397641 | 10:06 |
| *** sdake_ has joined #openstack-kolla | 10:08 | |
| *** neilus has quit IRC | 10:09 | |
| *** sdake has quit IRC | 10:11 | |
| *** tonanhngo has joined #openstack-kolla | 10:18 | |
| openstackgerrit | Merged openstack/kolla: Revert "corrects invalid logrotate option maxsize" https://review.openstack.org/397214 | 10:18 |
| *** tonanhngo has quit IRC | 10:18 | |
| openstackgerrit | Merged openstack/kolla: Use check_mode no instead of always_run https://review.openstack.org/397094 | 10:23 |
| openstackgerrit | Merged openstack/kolla: update dispatcher configurations for database backend https://review.openstack.org/397089 | 10:23 |
| *** neilus has joined #openstack-kolla | 10:26 | |
| *** duonghq has quit IRC | 10:27 | |
| *** fragatina has joined #openstack-kolla | 10:31 | |
| *** neilus has quit IRC | 10:32 | |
| *** fragatina has quit IRC | 10:36 | |
| *** khamtamtun has quit IRC | 10:39 | |
| *** sdake_ is now known as sdake | 10:40 | |
| coolsvap | pbourke: will you update the contributing doc for policy change related to TrivialFix? | 10:44 |
| pbourke | coolsvap: submitting a patch now | 10:44 |
| coolsvap | (y) | 10:44 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Update docs around TrivialFix https://review.openstack.org/397672 | 10:50 |
| sdake | pbourke nitpick on your doc change | 10:55 |
| sdake | https://bugs.launchpad.net/kolla/+bugs?search=Search&field.status=New | 10:56 |
| sdake | 40 new bugs | 10:56 |
| *** portdirect_away_ is now known as portdirect_ | 10:57 | |
| portdirect_ | morning :) | 10:57 |
| sdake | fwiw, I'm taking a break from bug triage for oh - about 3 or 4 months | 10:57 |
| sdake | i hope folks here pick up the slack;) | 10:57 |
| sdake | sup portdirect_ | 10:57 |
| sdake | egonzalez90 coolsvap pbourke if you like the lubernetes spec, +2 it plz, or leave a -1 and commentary | 10:58 |
| sdake | https://review.openstack.org/#/c/392257/ | 10:58 |
| portdirect_ | sdake: going back to last night (for me), I meant no need to move the kolla entrypoint - i was jumping the gun, we should be able to run kolla containers read-only in their current state i think | 10:59 |
| pbourke | sdake: I +1'ed a couple of times, but the patchsets just keep on rolling | 10:59 |
| portdirect_ | so we should try it :) | 10:59 |
| sdake | pbourke i think the patchsets are done | 10:59 |
| pbourke | kk | 10:59 |
| sdake | ryan set deadline of today for merging | 10:59 |
| sdake | and ryan running that shit show ;) | 11:00 |
| *** sbezverk has joined #openstack-kolla | 11:00 | |
| sdake | sbezverk ure alive | 11:00 |
| sdake | sbezverk check out our masterpeice: https://review.openstack.org/#/c/392257/ | 11:00 |
| sbezverk | sdake: yep, finally landed last night.. | 11:00 |
| sdake | well before you get locked into email have a read | 11:00 |
| sbezverk | sdake: sure thing. | 11:01 |
| *** khamtamtun has joined #openstack-kolla | 11:01 | |
| *** mgoddard_ has joined #openstack-kolla | 11:01 | |
| sdake | portdirect_ - i think the current thinking on entrypoint is it will be considered if it can be made optional default to off and not jammed into every container | 11:02 |
| sdake | as in, if you want entrypoint you can have it, if yo udont, it isn't forced upon you | 11:03 |
| portdirect_ | sorry: again we are hit by too many things with the same name - i was refering to the current (docker?) entrypoint to the kolla container, nothing to do with the k8s-entrypoint :) | 11:03 |
| sdake | i am definately not satisified with an entrypoint implementation that changes around all our containers | 11:03 |
| zhubingbing | sdake, do you know trove support v3? | 11:03 |
| sdake | zhubingbing v3 o wot | 11:03 |
| zhubingbing | keystone v3 | 11:04 |
| sdake | no clue | 11:04 |
| sdake | v2 is pretty much deprecated | 11:04 |
| portdirect_ | "i am definately not satisified with an entrypoint implementation that changes around all our containers" quoted for truth. | 11:04 |
| sdake | so i hope its moved on | 11:04 |
| sdake | its being trove | 11:04 |
| *** mgoddard has quit IRC | 11:04 | |
| sdake | proposition 205 failed by 2% in arizona | 11:05 |
| * sdake groans | 11:05 | |
| sdake | the same thing passed in 4 or 5 other states | 11:05 |
| sdake | ok guys, the dark knight rises or 300 | 11:06 |
| sdake | what shall it be | 11:07 |
| portdirect_ | dark knight | 11:08 |
| portdirect_ | easy | 11:08 |
| sdake | i believed in harvy dent | 11:09 |
| sdake | the best part of this movie is batman saves the world and escapes being batman :) | 11:10 |
| sdake | portdirect_ https://www.gliffy.com/go/share/ssmhcjkog50xyl3gensq | 11:13 |
| *** ChanServ sets mode: +o sdake | 11:13 | |
| *** ChanServ sets mode: -o sdake | 11:14 | |
| portdirect_ | that looks good | 11:14 |
| sdake | thanks | 11:15 |
| sdake | gotta be good at something | 11:15 |
| sdake | i can push around boxes on gliffy pretty well | 11:15 |
| portdirect_ | +1 | 11:16 |
| sdake | sbezverk ^ https://www.gliffy.com/go/share/ssmhcjkog50xyl3gensq | 11:16 |
| portdirect_ | the only q i have is where does "keystone default configuration" come from? | 11:16 |
| sdake | portdirect_ i think an open question we can answer later | 11:17 |
| portdirect_ | though thats out of scope for that diagram.. | 11:17 |
| portdirect_ | I'm down with that | 11:17 |
| sdake | we do have a genconfig feature of kolla | 11:17 |
| sdake | but obviously that is suboptimal | 11:17 |
| portdirect_ | lets get it working, then we can open up the rabbit warren :) | 11:17 |
| *** mgoddard_ has quit IRC | 11:17 | |
| sdake | i had a massive yak shaving session today | 11:17 |
| sdake | it all started with this diagram | 11:18 |
| portdirect_ | I'll look through the logs when I get into the office, but It looks like things became a lot clearer while I slept :) | 11:18 |
| *** mgoddard has joined #openstack-kolla | 11:18 | |
| sdake | portdirect_ what is the analog for a bag o cats | 11:19 |
| sdake | portdirect_ clearer to me atleast | 11:19 |
| sdake | portdirect_ i think the spec is pretty close to spot on | 11:19 |
| sdake | it has a few typos | 11:19 |
| portdirect_ | yup - it looks pretty solid from my perspective now | 11:20 |
| portdirect_ | that diagram of your is worth a thousand words thou - is it possible to get it referenced? | 11:20 |
| sdake | probably ends in more yak shaving | 11:21 |
| portdirect_ | sdake_: bag o cats <-> wheelbarrow of frogs | 11:21 |
| portdirect_ | you gonna be about in 30? | 11:21 |
| sdake | i dunno my kid woke me up at 3am | 11:22 |
| sdake | so may try to go back to bed but its hard for me to do | 11:22 |
| portdirect_ | :/ | 11:22 |
| sdake | when i'm awake, i'm awake | 11:22 |
| sbezverk | sdake: will you have some time today/tomorrow to chat about "operator" and its origins ;-)? | 11:22 |
| portdirect_ | I know that dude | 11:22 |
| sdake | sbezverk some shit the cats at coreos invented | 11:22 |
| sdake | silver bullet for workflow | 11:23 |
| sbezverk | sdake: got it | 11:23 |
| sdake | so thats the origins | 11:24 |
| sdake | origin | 11:24 |
| *** ntpttr has quit IRC | 11:30 | |
| *** ntpttr has joined #openstack-kolla | 11:31 | |
| *** Pavo has quit IRC | 11:38 | |
| sdake | sbezverk are you going to +2 that review or -1 it - ryan has set a deadline for today for it to merge | 11:39 |
| *** portdirect_ has quit IRC | 11:43 | |
| *** Pavo has joined #openstack-kolla | 11:43 | |
| sbezverk | sdake: I will do it by the end of day today. | 11:43 |
| sdake | well off to bed | 11:45 |
| *** sdake has quit IRC | 11:45 | |
| *** neilus has joined #openstack-kolla | 11:46 | |
| *** zhurong has joined #openstack-kolla | 11:51 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Update docs around TrivialFix https://review.openstack.org/397672 | 11:52 |
| *** eaguilar has joined #openstack-kolla | 11:52 | |
| *** zhurong has quit IRC | 11:59 | |
| *** hfu has joined #openstack-kolla | 12:01 | |
| bjolo_ | morning | 12:04 |
| *** zhurong has joined #openstack-kolla | 12:05 | |
| *** hfu has quit IRC | 12:07 | |
| pbourke | bjolo_: hey | 12:08 |
| pbourke | bjolo_: what was the latest on your network issue? | 12:08 |
| bjolo_ | pbourke, hey | 12:08 |
| bjolo_ | we are still troubleshooting it | 12:08 |
| bjolo_ | still not working | 12:08 |
| pbourke | could you paste your ml2_conf.ini at some stage? | 12:09 |
| bjolo_ | sure | 12:09 |
| pbourke | as well as relevant globals | 12:09 |
| bjolo_ | give me a few, deploying right now | 12:09 |
| pbourke | cool | 12:10 |
| bjolo_ | the take right now is that it is related to openvswitch and neutron-openvswitch-agent communication. As soon as you add a new external bridge, neutron agent keeps disconnecting openvswitch after 1s | 12:10 |
| bjolo_ | openvswitch reconnects, just to get disconnected again. | 12:11 |
| bjolo_ | and this is only happening for external bridges | 12:11 |
| bjolo_ | br-int and br-tun are just fine. | 12:11 |
| *** hfu has joined #openstack-kolla | 12:12 | |
| pbourke | can some cores please review https://review.openstack.org/#/c/392583/ | 12:13 |
| berendt | pbourke reading... | 12:13 |
| *** tonanhngo has joined #openstack-kolla | 12:13 | |
| pbourke | berendt: its simple enough I just keep looking for that registry script when I clone :) | 12:13 |
| *** zhurong has quit IRC | 12:14 | |
| *** tonanhngo has quit IRC | 12:15 | |
| *** zhurong has joined #openstack-kolla | 12:15 | |
| bjolo_ | http://paste.openstack.org/show/589275/ | 12:16 |
| bjolo_ | pbourke, ^^ | 12:17 |
| Jeffrey4l | sup sd | 12:17 |
| openstackgerrit | Merged openstack/kolla: Update docs around TrivialFix https://review.openstack.org/397672 | 12:18 |
| berendt | pbourke review done | 12:20 |
| bjolo_ | pbourke, here is the neutron-openvswitch-agent log as well http://paste.openstack.org/show/589276/ | 12:20 |
| openstackgerrit | Martin André proposed openstack/kolla: Update docs around TrivialFix https://review.openstack.org/397712 | 12:22 |
| pbourke | thanks bjolo_, few issues with it here also so will keep you updated | 12:23 |
| pbourke | thanks berendt | 12:23 |
| openstackgerrit | zhubingbing proposed openstack/kolla: change network_bw_out to network.bw.out in cloudkitty.conf.j2 https://review.openstack.org/397641 | 12:31 |
| pbourke | bjolo_: have you tried setting network_vlan_ranges | 12:34 |
| *** srwilkers has joined #openstack-kolla | 12:39 | |
| *** berendt has quit IRC | 12:39 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH https://review.openstack.org/326307 | 12:42 |
| *** schwicht has joined #openstack-kolla | 12:42 | |
| *** rhallisey has joined #openstack-kolla | 12:42 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Load murano dashbaord dynamic https://review.openstack.org/395957 | 12:47 |
| *** nadeem_ has quit IRC | 12:48 | |
| *** coolsvap has quit IRC | 12:50 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add neutron vpnaas code into neutron-server container https://review.openstack.org/397015 | 12:51 |
| openstackgerrit | zhubingbing proposed openstack/kolla: change network_bw_out to network.bw.out in cloudkitty.conf.j2 https://review.openstack.org/397641 | 12:52 |
| *** schwicht has quit IRC | 12:53 | |
| *** schwicht has joined #openstack-kolla | 12:55 | |
| bjolo_ | pbourke, no we have not tried vlan ranges since bond0 is not exclusive for neutron. bond0 is network_interface, bond0.820 is tunnel interface etc | 12:55 |
| openstackgerrit | Javier Castillo Alcíbar proposed openstack/kolla: Closes-Bug: #1641113 https://review.openstack.org/397729 | 12:55 |
| openstack | bug 1641113 in kolla "Error in Task: Creating Ceilometer MongoDB database" [Undecided,Confirmed] https://launchpad.net/bugs/1641113 | 12:55 |
| *** zhubingbing has quit IRC | 12:56 | |
| pbourke | maybe neutron still needs to know about the tags though | 12:56 |
| bjolo_ | pbourke, few issues at your place? i.e. you can confirm the bug? | 12:56 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Clean up docs around local registry https://review.openstack.org/392583 | 12:57 |
| *** schwicht has quit IRC | 12:57 | |
| *** dave-mccowan has joined #openstack-kolla | 13:03 | |
| *** eaguilar_ has joined #openstack-kolla | 13:04 | |
| *** eaguilar has quit IRC | 13:05 | |
| openstackgerrit | Javier Castillo Alcíbar proposed openstack/kolla: Fix ceilometer bootstrap https://review.openstack.org/397729 | 13:06 |
| *** fguillot has joined #openstack-kolla | 13:12 | |
| *** zhurong has quit IRC | 13:14 | |
| *** hfu has quit IRC | 13:15 | |
| *** sdake has joined #openstack-kolla | 13:16 | |
| *** khamtamtun has quit IRC | 13:16 | |
| *** lamt has joined #openstack-kolla | 13:16 | |
| *** lamt has quit IRC | 13:16 | |
| srwilkers | good morning | 13:17 |
| sdake | sup srwilkers | 13:18 |
| *** lamt has joined #openstack-kolla | 13:18 | |
| sdake | srwilkers https://www.gliffy.com/go/share/ssmhcjkog50xyl3gensq | 13:19 |
| *** stvnoyes has quit IRC | 13:20 | |
| srwilkers | yes | 13:20 |
| srwilkers | thank you for this sdake | 13:20 |
| *** hfu has joined #openstack-kolla | 13:21 | |
| srwilkers | diagrams help me much more than walls of scrollback | 13:21 |
| sdake | happy to help | 13:21 |
| *** stvnoyes has joined #openstack-kolla | 13:21 | |
| sdake | "You will have to imagine the heat." | 13:22 |
| sdake | best batman ever | 13:22 |
| srwilkers | agreed | 13:23 |
| *** lamt has quit IRC | 13:24 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Map host /proc folder into neutron-l3-agent container https://review.openstack.org/397738 | 13:25 |
| sdake | it was a little cheesy being around the time of the occupy wallstreet movement | 13:25 |
| sdake | but other then that it rocked :) | 13:26 |
| srwilkers | lol yeah | 13:26 |
| srwilkers | call me a fan boy, but Bane was my boy | 13:26 |
| Jeffrey4l | sup sdake | 13:27 |
| sdake | Jeffrey4l sup dude | 13:27 |
| sdake | Jeffrey4l inc0 mentioned you are taking over release liason responsibilities? | 13:27 |
| Jeffrey4l | yes. but need some train ;) | 13:27 |
| sdake | Jeffrey4l just a reminder rlease team likes to see releases on wednesday of release week | 13:28 |
| sdake | which is tomorrow in the us | 13:28 |
| sdake | srwilkers the best part about this movie | 13:29 |
| sdake | is batman saves the world | 13:29 |
| sdake | yet escapes being batman | 13:29 |
| Jeffrey4l | inc0 said we can release latter, after we have finished our repo split. | 13:29 |
| sdake | we have to definatley release this week | 13:29 |
| Jeffrey4l | OK. | 13:30 |
| sdake | i will do what i can to get the repo split merged prior to wed | 13:30 |
| *** portdirect_away is now known as portdirect | 13:30 | |
| *** neilus has quit IRC | 13:31 | |
| Jeffrey4l | so, what i need to to is push a PS witch the latest commit hash into openstack/release project, right? sdake | 13:32 |
| sdake | check out openstack/releases | 13:32 |
| sdake | it should be obvious what to do | 13:33 |
| Jeffrey4l | got. | 13:33 |
| sdake | 4.0.0.0b1 | 13:33 |
| sdake | wait and see if i can get repo split done tho | 13:33 |
| Jeffrey4l | np. | 13:33 |
| bjolo_ | Jeffrey4l, about https://review.openstack.org/#/c/397738/1 | 13:33 |
| bjolo_ | what are the symptoms of this? | 13:34 |
| Jeffrey4l | sdake, what's other responsibilities of release liason? | 13:34 |
| sdake | just tagging releases | 13:34 |
| bjolo_ | i have some weird neutron behavior, just wondering if all/part of it could be related to your PS | 13:34 |
| *** hfu has quit IRC | 13:34 | |
| sdake | hopefully not releasing stuff thats doa | 13:34 |
| Jeffrey4l | bjolo_, check the bug description. | 13:34 |
| Jeffrey4l | bjolo_, https://bugs.launchpad.net/kolla/+bug/1641946 | 13:34 |
| openstack | Launchpad bug 1641946 in kolla newton "l3 ha router need access /proc filesystem" [Undecided,New] | 13:34 |
| bjolo_ | l3 agent needs access to /proc | 13:35 |
| bjolo_ | ok, but why? what happens if it does not have access? | 13:35 |
| bjolo_ | nm | 13:36 |
| bjolo_ | read the bug on LP | 13:36 |
| Jeffrey4l | need not sure. it makes some configuration for the devices. let me check the sysctl: cannot stat /proc/sys/net/ipv6/conf/qg-79ecb185-50/accept_ra | 13:36 |
| bjolo_ | not related to my issue | 13:36 |
| Jeffrey4l | what u got? bjolo_ | 13:36 |
| *** zhubingbing_ has joined #openstack-kolla | 13:37 | |
| bjolo_ | trying to have multiple external flat networks | 13:37 |
| bjolo_ | http://paste.openstack.org/show/589275/ | 13:37 |
| bjolo_ | have not open bug yet cause it feels like we have not narrowed it down what the issue is | 13:38 |
| *** Pavo has quit IRC | 13:38 | |
| Jeffrey4l | 2016-11-15T12:15:51.586Z|01155|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connection closed by peer | 13:38 |
| *** neilus has joined #openstack-kolla | 13:38 | |
| Jeffrey4l | ^^ this ip address is wrong. | 13:38 |
| Jeffrey4l | need use a api_interface_address. | 13:39 |
| bjolo_ | hmm | 13:39 |
| bjolo_ | one external flat works great | 13:39 |
| *** neilus has quit IRC | 13:39 | |
| bjolo_ | as soon as you have 2 external flat or more, it breaks | 13:39 |
| *** neilus has joined #openstack-kolla | 13:39 | |
| Jeffrey4l | hmm weird | 13:41 |
| bjolo_ | where does br-vlan805 get its IP from? | 13:41 |
| *** jheroux has joined #openstack-kolla | 13:41 | |
| bjolo_ | we dont have any config for openvswitch really | 13:41 |
| bjolo_ | http://paste.openstack.org/show/589276/ | 13:42 |
| bjolo_ | the log from neutron-openvswitch-agent | 13:42 |
| sdake | srwilkers you know what i don't get about tihs batman series | 13:43 |
| *** Pavo has joined #openstack-kolla | 13:43 | |
| sdake | is they didn't follow it up with a robin spinoff | 13:43 |
| *** neilus has quit IRC | 13:44 | |
| Jeffrey4l | what 6633 port is ? | 13:45 |
| bjolo_ | manger port on neutron-openvswitch-agent | 13:46 |
| bjolo_ | all bridges connect via loopback 127.0.0.1 | 13:46 |
| bjolo_ | http://paste.openstack.org/show/589286/ | 13:46 |
| bjolo_ | br-tun and br-int does not get disconnected | 13:46 |
| *** neilus has joined #openstack-kolla | 13:46 | |
| bjolo_ | the weird thing is that in the neutron code, it should disconnect a client after 10s if it does not get a heartbeat | 13:47 |
| bjolo_ | but for us it is done instantly | 13:47 |
| Jeffrey4l | yep. no idea ;( , i will try to deploy neutron with two flat network later. | 13:50 |
| *** chas_ has quit IRC | 13:52 | |
| *** chas_ has joined #openstack-kolla | 13:52 | |
| *** zhubingbing_ has quit IRC | 13:53 | |
| *** chas__ has joined #openstack-kolla | 13:54 | |
| *** neilus has quit IRC | 13:54 | |
| *** chas_ has quit IRC | 13:56 | |
| *** khamtamtun has joined #openstack-kolla | 13:58 | |
| *** neilus has joined #openstack-kolla | 14:03 | |
| Jeffrey4l | sdake, what the last release date in utc? Nov 18, 00:00? | 14:03 |
| sdake | Jeffrey4l its not super strict | 14:03 |
| *** berendt has joined #openstack-kolla | 14:05 | |
| *** coolsvap has joined #openstack-kolla | 14:05 | |
| Jeffrey4l | ok | 14:05 |
| openstackgerrit | caoyuan proposed openstack/kolla: Add tacker doc link into README.rst https://review.openstack.org/397758 | 14:08 |
| *** papacz has quit IRC | 14:11 | |
| *** tonanhngo has joined #openstack-kolla | 14:12 | |
| *** tonanhngo has quit IRC | 14:13 | |
| *** eaguilar_ has quit IRC | 14:14 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Map host /proc folder into neutron-l3-agent container https://review.openstack.org/397738 | 14:15 |
| *** awiddersheim has quit IRC | 14:15 | |
| srwilkers | sdake: yes, im very surprised/disappointed about that | 14:15 |
| srwilkers | they set it up perfectly | 14:15 |
| srwilkers | and just let it go | 14:15 |
| *** schwicht has joined #openstack-kolla | 14:15 | |
| *** stvnoyes has left #openstack-kolla | 14:16 | |
| *** khamtamtun has quit IRC | 14:16 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: neutron-l3-agent needs access host /proc filesystem https://review.openstack.org/397738 | 14:17 |
| *** awiddersheim has joined #openstack-kolla | 14:19 | |
| sdake | Jeffrey4l pid_mode: host for neutron ? | 14:20 |
| Jeffrey4l | sdake, for neutron-l3-agent service only. | 14:21 |
| sdake | Jeffrey4l you need host proc for what? | 14:21 |
| Jeffrey4l | check the bug descript. | 14:21 |
| Jeffrey4l | sdake, https://launchpad.net/bugs/1641946 | 14:21 |
| openstack | Launchpad bug 1641946 in kolla newton "l3 ha router use host proc filesystem" [Undecided,New] | 14:21 |
| sdake | Jeffrey4l is there no better way to map the host proc? | 14:22 |
| Jeffrey4l | i have no idea what this /proc/sys/net/ipv6/conf/qg-79ecb185-50/accept_ra is used for. but it raise error, so.. | 14:22 |
| Jeffrey4l | direct use `docker run -v /proc:/proc` won't work. | 14:22 |
| Jeffrey4l | so i think no. ;( | 14:22 |
| sdake | Jeffrey4l did you reference google? | 14:23 |
| Jeffrey4l | docker run -v /proc:/hostproc works. but neutron do not read/change kernel configuration from that folder. | 14:23 |
| Jeffrey4l | no. | 14:23 |
| *** sdake has quit IRC | 14:23 | |
| Jeffrey4l | let's me check it | 14:23 |
| *** chas__ has quit IRC | 14:24 | |
| *** sdake has joined #openstack-kolla | 14:24 | |
| *** chas_ has joined #openstack-kolla | 14:25 | |
| *** eaguilar has joined #openstack-kolla | 14:25 | |
| sdake | Jeffrey4l https://github.com/docker/docker/issues/13398 | 14:25 |
| sdake | try bindmount of proc with --net=host | 14:25 |
| sdake | --net=host far less damaging then --net=pid | 14:26 |
| sbezverk | sdake: I am reading about operators, a bit confused. Is the app specific logic coded in a special image they run as in example of etcd operator? | 14:26 |
| Jeffrey4l | sdake, we are using --net=host for all containers. | 14:26 |
| Jeffrey4l | it won't be helpful for this case. | 14:27 |
| sdake | sbezverk seems like i've answered that q alot ;) | 14:27 |
| sdake | sbezverk there is a thing called a controller | 14:27 |
| Jeffrey4l | l3-agent need access the /proc file directly. | 14:27 |
| sdake | a controller is python intelligent agent | 14:27 |
| sdake | the controller is loaded into a docker container | 14:28 |
| sbezverk | sdake: do you have an example how they extend "kind" and what is exactly inside of this special image? | 14:28 |
| Jeffrey4l | and current docker do not support mount /proc:/proc | 14:28 |
| Jeffrey4l | docker run --net host -v /proc:/proc --privileged centos:7 bash | 14:28 |
| Jeffrey4l | docker: Error response from daemon: oci runtime error: rootfs_linux.go:53: mounting "/proc" to rootfs "/home/docker/overlay2/41a66251017eaff9b7d05450562cd038c6939e38d5f4eed127c5d95e996370be/merged" caused "\"/home/docker/overlay2/41a66251017eaff9b7d05450562cd038c6939e38d5f4eed127c5d95e996370be/merged/proc\" cannot be mounted because it is located inside \"/proc\"". | 14:28 |
| sdake | how about proc:/proc_host | 14:29 |
| Jeffrey4l | but neutron will still read the file in /proc | 14:29 |
| sdake | symlink? | 14:29 |
| *** inc0 has joined #openstack-kolla | 14:29 | |
| Jeffrey4l | he hasn't no idea about the /proc_host folder, unless this is configurable in neutron. | 14:29 |
| inc0 | good morning | 14:29 |
| sdake | sup inc0 | 14:29 |
| Jeffrey4l | hmm symlink? let me think.. | 14:29 |
| Jeffrey4l | inc0, morning | 14:29 |
| sdake | Jeffrey4l pid mode host is special sauce | 14:30 |
| inc0 | what news? | 14:30 |
| srwilkers | hey inc0 | 14:30 |
| *** chas_ has quit IRC | 14:30 | |
| Jeffrey4l | btw, we use pid mode in several containers. | 14:30 |
| sdake | should only be used in libvirt | 14:30 |
| *** lrensing has joined #openstack-kolla | 14:30 | |
| inc0 | Jeffrey4l, which ones besides libvirt? | 14:31 |
| Jeffrey4l | ceph-osd telegraf and libvirt | 14:31 |
| sdake | sbezverk extend "kind"? | 14:32 |
| sdake | sbezverk could you define king | 14:32 |
| sdake | sbezverk inside special image is a python shell script | 14:32 |
| sdake | sbezverk or some other type of binray | 14:32 |
| sdake | language is unimportant | 14:32 |
| sdake | sbezverk check this out, maybe it will help https://www.gliffy.com/go/share/ssmhcjkog50xyl3gensq | 14:33 |
| sbezverk | sdake: here is what they have in etcd operator cluster example: kind: "EtcdCluster" | 14:33 |
| *** papacz has joined #openstack-kolla | 14:33 | |
| sbezverk | sdake: kind etcdcluster does not exist by default | 14:33 |
| sdake | EtcdCluster is a thirdparty resource iiuc | 14:33 |
| sbezverk | so they extended it some how, I thought you might know how. | 14:33 |
| sdake | check out the diagram | 14:34 |
| Jeffrey4l | inc0, we are talking this issue and related pid_mode https://bugs.launchpad.net/kolla/+bug/1641946 | 14:34 |
| openstack | Launchpad bug 1641946 in kolla newton "l3 ha router use host proc filesystem" [Undecided,New] | 14:34 |
| sbezverk | sdake: sorry diagramm looks more like a marketing slide to me. I wanted to see code example :-) | 14:34 |
| inc0 | Jeffrey4l, this looks like issue with shared netns isn't it? | 14:35 |
| *** papacz has quit IRC | 14:35 | |
| Jeffrey4l | inc0, no. netns is related to /run/netns only. in this issue, l3-agent try to change some kernel parameter for certain devices. | 14:35 |
| Jeffrey4l | it is different. | 14:35 |
| Jeffrey4l | fyi: If you don’t want a host to automatically configure an address and route then you could disable this behaviour by writing “0″ to /proc/sys/net/ipv6/conf/*/accept_ra. | 14:36 |
| *** eaguilar_ has joined #openstack-kolla | 14:37 | |
| inc0 | Jeffrey4l, so we need to make l3 agent run with pid=host? | 14:37 |
| *** eaguilar has quit IRC | 14:37 | |
| *** jtriley has joined #openstack-kolla | 14:37 | |
| Jeffrey4l | yep only in this way, the l3-agent can touch this file, imo. | 14:37 |
| sdake | sbezverk are you familiar with the thirdpartyresource kind? | 14:38 |
| *** fguillot has quit IRC | 14:39 | |
| *** schwicht has quit IRC | 14:40 | |
| inc0 | http://stackoverflow.com/questions/23840737/how-to-remount-the-proc-filesystem-in-a-docker-as-a-r-w-system Jeffrey4l can you try this remount | 14:40 |
| inc0 | ? | 14:40 |
| sbezverk | sdake: in terms of kubernetes no, have not seen it before.. | 14:40 |
| inc0 | maybe it has issues with creating this file in the first place | 14:40 |
| inc0 | or rather...what does create this file? ovs-agent? | 14:40 |
| sdake | sbezverk https://github.com/kubernetes/kubernetes/blob/master/docs/design/extending-api.md | 14:41 |
| Jeffrey4l | it should be created by kernel when this device is created. | 14:41 |
| inc0 | have you tried to -v /run/netns? | 14:41 |
| sdake | inc0 i think that remount looks good with --net=host | 14:41 |
| Jeffrey4l | inc0, neutron-l3 is started with -v /run/netns parameter. | 14:42 |
| Jeffrey4l | i will run the remount solution. | 14:42 |
| *** lrensing has quit IRC | 14:42 | |
| Pavo | morning | 14:43 |
| *** lrensing has joined #openstack-kolla | 14:43 | |
| sdake | sbezverk that is where that resource originates from | 14:44 |
| v1k0d3n | morning guys. | 14:45 |
| srwilkers | morning v1k0d3n | 14:45 |
| inc0 | o/ | 14:45 |
| *** khamtamtun has joined #openstack-kolla | 14:45 | |
| inc0 | ehh sdake you want resource "keystone" just instead of well...pod? | 14:46 |
| sdake | to control and manage the pod dynamically it is mandatory | 14:47 |
| sdake | if we just want workflow, we got a shell script for that | 14:48 |
| sdake | that we could run from the cli | 14:48 |
| inc0 | you are aware that there is more to it right? | 14:49 |
| inc0 | upgrades can be solved systematically, without need of writing dedicated blob per service | 14:49 |
| *** khamtamtun has quit IRC | 14:50 | |
| sdake | https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/ceph_workflow.sh | 14:50 |
| inc0 | really sdake? you show me shellscript that works on aio and it's meant for aio? | 14:50 |
| sdake | its multinode | 14:50 |
| sdake | point being there is workflow without operators | 14:51 |
| inc0 | but it's racey, it's not HA | 14:51 |
| sdake | i agree, the workflow in shell is terrible | 14:51 |
| inc0 | and has many different issues single shellscript has | 14:51 |
| inc0 | let's do this, just write a damn keystone operator | 14:51 |
| sdake | working on it | 14:52 |
| inc0 | and start talking based on substance instead of misunderstandings | 14:52 |
| inc0 | ok, so let's shelve this discussion for when we have something to show for | 14:52 |
| *** eaguilar_ has quit IRC | 14:52 | |
| inc0 | in the meantime me, v1k0d3n and others who agree with entrypoint'ish solution will work on it ok? | 14:53 |
| sdake | inc0 training people, validating ideas | 14:53 |
| inc0 | I think idea is invalid, but prove me wrong please | 14:53 |
| sdake | sweet two parallel implementations | 14:53 |
| inc0 | yes, PoCs | 14:53 |
| sdake | divsive | 14:53 |
| inc0 | you won't be swayed | 14:53 |
| inc0 | it's Elemental over and over again | 14:54 |
| sdake | your actually wrong about the swayed part | 14:54 |
| inc0 | I hope so | 14:54 |
| sdake | i was swayed on elemental | 14:54 |
| inc0 | after I've shown you example | 14:54 |
| sdake | and kfox1111_away has offered up a compromise | 14:54 |
| inc0 | which is? | 14:54 |
| sdake | for cats that really tihnk they want to jerk around with entrypoint | 14:54 |
| sdake | make entrypoint init containers, but not pollute the entire container repo with entrypoint | 14:55 |
| sdake | make it optional and default to off | 14:55 |
| inc0 | thats fine | 14:55 |
| inc0 | well | 14:55 |
| sdake | i'm not sure where rhallisey stands on that | 14:55 |
| *** lamt has joined #openstack-kolla | 14:55 | |
| inc0 | at the end let's have single solution | 14:55 |
| inc0 | whichever is best | 14:55 |
| sdake | that would be a single solution | 14:56 |
| inc0 | but I want solution that will allow us to deploy prod-ready and tested by ovata | 14:56 |
| sdake | you choose between entrypoint and non-entrypoint | 14:56 |
| inc0 | single solution... | 14:56 |
| sdake | since all the complaints seem to be about entrypoint | 14:56 |
| inc0 | name them please | 14:56 |
| sdake | name which? | 14:56 |
| sdake | its in the spec | 14:56 |
| sdake | nonstop bitching about how we should choose tech X "because" | 14:57 |
| *** neilus has quit IRC | 14:57 | |
| v1k0d3n | inc0: yes, i agree with entrypoint. | 14:58 |
| sdake | v1k0d3n do you not agree with operator? | 14:58 |
| v1k0d3n | operator...i'm a huge fan, and this is a great thing (this is why i brought up), but i kind of wish i didn't now because the amount of work to get right is very large. | 14:59 |
| v1k0d3n | i dont think we fully understand exactly how much work this will take to do correctly. | 14:59 |
| inc0 | unless I miss something there aren't really any issues with entrypoint if that's external to actual container | 14:59 |
| inc0 | we need someting to deploy, upgrade, reconfigure | 14:59 |
| *** fguillot has joined #openstack-kolla | 14:59 | |
| rhallisey | ya we can have entrypoint as long as it doesn't interupt any of the other options | 14:59 |
| inc0 | rhallisey, it won't, it never would | 15:00 |
| v1k0d3n | entrypoint, when done correctly (current stackanetes entrypoint can be improved on) is the way for us forward. and honestly, we need to put this to rest...because at the moment, there's a lot of back and forth. | 15:00 |
| v1k0d3n | rhallisey: it won't | 15:00 |
| rhallisey | can you guys look at the argument in the spec | 15:00 |
| v1k0d3n | inc0: and i talked and it can be done. | 15:00 |
| inc0 | I wouldn't allow it to affect ansible or tripleo in any way | 15:00 |
| sdake | i think entrypoint is completely unnecessary, but am open to compromise on the point as mentioned above | 15:00 |
| rhallisey | I laid out an argument against entrypoint | 15:00 |
| v1k0d3n | i'll be back. have to reboot guys. | 15:00 |
| v1k0d3n | one sec | 15:00 |
| *** v1k0d3n has quit IRC | 15:00 | |
| rhallisey | inc0, check the spec comments | 15:01 |
| rhallisey | would like to know more thoughts | 15:01 |
| inc0 | checking comments | 15:01 |
| *** neilus has joined #openstack-kolla | 15:01 | |
| inc0 | rhallisey, so here's thing, there is significant difference between k8s and compose | 15:02 |
| inc0 | one that we'll leverage and will fix issues from compose | 15:02 |
| rhallisey | no there isn't | 15:02 |
| inc0 | we do have signle source of truth | 15:02 |
| inc0 | we can query k8s about state of dependencies | 15:02 |
| *** neilus has quit IRC | 15:02 | |
| inc0 | not look at containers and crap | 15:02 |
| *** v1k0d3n has joined #openstack-kolla | 15:02 | |
| rhallisey | I'm not talking about the deps | 15:02 |
| sdake | inc0 this needs your ack: https://review.openstack.org/396903 | 15:02 |
| inc0 | add that to proper readiness probes and it's no longer racey | 15:02 |
| rhallisey | deps are 1/1000 of the lifecycle problem | 15:03 |
| sdake | it is blocking Jeffrey4l 's tag | 15:03 |
| inc0 | so trust me (as someone who wrote upgrades in ansible) you don't want to write python code to solve this complexity | 15:03 |
| inc0 | there are better way | 15:03 |
| inc0 | ways | 15:03 |
| sdake | the idea that adding readiness probes removes raciness is an oxymoron | 15:04 |
| inc0 | sdake, upstream: https://github.com/sdake/kolla.git <- what this line does? | 15:04 |
| sdake | inc0 that is a snapshot of the kolla repo on saturday | 15:05 |
| *** eaguilar has joined #openstack-kolla | 15:05 | |
| sdake | with all tags and branches removed | 15:05 |
| inc0 | kk | 15:05 |
| inc0 | and in project config it will copy it then line becomes redundant right? | 15:05 |
| sdake | i'd update it but it took me couple hours to make that repo in the first place | 15:05 |
| sdake | then infra removes it | 15:05 |
| inc0 | and no, readness probes aren't oxymoron, it's logic that will tell k8s that pod is up | 15:06 |
| sdake | we had that in compose just not in readiness probes | 15:06 |
| rhallisey | did you read the whole argument? | 15:06 |
| rhallisey | there are multiple points | 15:06 |
| sbezverk | sdake: are you working on "operator" PoC? I would like to participate as I think this approach is more "kubernetes" like.. | 15:07 |
| sdake | sbezverk welcome to the party | 15:07 |
| inc0 | rhallisey, debugging and other operational stuff is done by proper toolset around | 15:07 |
| sdake | sbezverk lets get together today on webex and you can bring me up to speed on how to use my cluster ;) | 15:07 |
| v1k0d3n | btw, i am on a meeting with Quentin from CoreOS about entrypoint. | 15:07 |
| inc0 | v1k0d3n, maybe you could ask one of their guys to join here and clear up misunderstandings one for all? | 15:08 |
| *** chas_ has joined #openstack-kolla | 15:08 | |
| inc0 | since they invented thing | 15:08 |
| sdake | the only one that has a misunderstadning is you inc0 :) | 15:08 |
| *** tonanhngo has joined #openstack-kolla | 15:08 | |
| inc0 | and yet I am in agreement with v1k0d3n who brought up this topic and spent lots of time talking with coreos about it | 15:09 |
| v1k0d3n | inc0: i am suggesting this now. | 15:09 |
| inc0 | thank you | 15:09 |
| *** tonanhngo has quit IRC | 15:09 | |
| sdake | inc0 if it makes you feel any better tripleo is never going to use entrypoint, it wont work with their arch | 15:09 |
| sbezverk | sdake: sounds good, I hope it is still alive, I mean the cluster | 15:10 |
| inc0 | tripleo shouldn't use kolla-k8s at all, unless they really want to just shoot and forget deployment | 15:10 |
| *** chas__ has joined #openstack-kolla | 15:10 | |
| inc0 | without adding any orchiestration around | 15:10 |
| rhallisey | they could if we layer properly | 15:10 |
| sdake | if we jam entrypoint into all the images with readiness probes | 15:10 |
| inc0 | I've never heard TripleO saying that they plan to deploy kolla-k8s | 15:10 |
| inc0 | sdake, not-in-images | 15:10 |
| sdake | they plan to deploy the containers | 15:11 |
| inc0 | we can do it outside of images | 15:11 |
| rhallisey | I haven't either o.o | 15:11 |
| inc0 | you're not listening to me | 15:11 |
| sdake | inc0 obviously i am, there is lag | 15:11 |
| sdake | inc0 what I dont understand is your extremely strong desire for entrypoint, ok i don't understand it, but if you want it, we make it available | 15:12 |
| inc0 | sdake, I want to have deployment done by Ocata | 15:12 |
| rhallisey | I think we will | 15:12 |
| inc0 | and we just spent 2 weeks without solving any issue | 15:12 |
| sdake | we solved a whoel bunch of issues | 15:12 |
| rhallisey | we did solve a lot of things | 15:12 |
| inc0 | and I'm afraid you greatly underestimate complexity of operators | 15:12 |
| *** chas_ has quit IRC | 15:13 | |
| inc0 | ok, not an dependency issue | 15:13 |
| inc0 | while we have working tested solution | 15:13 |
| sdake | define *WE* | 15:13 |
| inc0 | that CoreOS likes, (again, they invented operators) | 15:13 |
| sdake | is this *WE* the kolla community? | 15:13 |
| rhallisey | tested working solution does not mean it's the best solution | 15:13 |
| inc0 | WE as people trying to deploy openstack on k8s | 15:13 |
| sdake | or *WE* some other magical entity? | 15:13 |
| inc0 | and yes, partially kolla community | 15:14 |
| inc0 | as I hope I'm still part of it | 15:14 |
| *** lrensing has quit IRC | 15:14 | |
| sdake | frankly I'm not out to solve othe people's problems | 15:14 |
| rhallisey | doesn't matter who brought up operators, we were toying with the concept before this. Now we have a name for it | 15:14 |
| sdake | except those that accept our mission statement | 15:14 |
| *** QuentinM has joined #openstack-kolla | 15:14 | |
| inc0 | rhallisey, but never as thing to solve all our deployment problems | 15:14 |
| *** chas__ has quit IRC | 15:15 | |
| inc0 | fencing pod is an operator almost | 15:15 |
| *** lrensing has joined #openstack-kolla | 15:15 | |
| inc0 | and it's great, let's use operators for hard stuff like fencing | 15:15 |
| inc0 | but not for *everything* | 15:15 |
| rhallisey | we were using the same concept with ansible | 15:15 |
| inc0 | that's my argument | 15:15 |
| rhallisey | no we were using the same concept | 15:16 |
| inc0 | you mean with kolla-toolbox? | 15:16 |
| rhallisey | it just wasn't in a pod | 15:16 |
| *** schwicht has joined #openstack-kolla | 15:16 | |
| inc0 | yes, but k8s is not ansible, we can just dump ansible into container and that's operator for you | 15:16 |
| inc0 | just not really k8s native way of doing things | 15:16 |
| sdake | inc0 the problem with that is it completely lacks any form of control | 15:17 |
| inc0 | no | 15:17 |
| sdake | an operator provides control | 15:17 |
| inc0 | I promise you it there is lots of control | 15:17 |
| inc0 | by modyfying end-state | 15:17 |
| rhallisey | this is where we have a disagreement. I don't believe we need to make openstack fit perfect to the kubernetes way of doing things. OpenStack should not have to go 100% of the way there | 15:17 |
| inc0 | rhallisey, aside from databases, I think we can | 15:18 |
| inc0 | well with few exceptions | 15:18 |
| inc0 | but most of it, yes | 15:18 |
| *** fguillot has quit IRC | 15:18 | |
| inc0 | and yes I am talking about upgrades and reconfigure too | 15:18 |
| *** igordcar1 has quit IRC | 15:19 | |
| rhallisey | so my opinion, going to do that ends in major disaster or openstack re writes itself to fit the kube model | 15:19 |
| inc0 | ok...what's wrong with it then? | 15:19 |
| inc0 | in your opinion? | 15:19 |
| *** igordcard has joined #openstack-kolla | 15:19 | |
| inc0 | what prevents us to run is close to k8s native model? | 15:19 |
| *** fguillot has joined #openstack-kolla | 15:19 | |
| sdake | a lack of operators ;) | 15:19 |
| rhallisey | solving the complex problem beyond deploy | 15:19 |
| rhallisey | problems* | 15:20 |
| inc0 | rhallisey, that's true in every application | 15:20 |
| inc0 | and still, I promise you can be done | 15:20 |
| sdake | yup that is why operators are a silver bullet | 15:20 |
| rhallisey | ya except every application is not openstack | 15:20 |
| sdake | for orchestration of complex applications | 15:20 |
| rhallisey | not every application is a complex application | 15:20 |
| inc0 | but there are complex applicaitons | 15:20 |
| rhallisey | not every application has 1million ways to be configured | 15:20 |
| inc0 | much more of them | 15:20 |
| inc0 | cassandra for example - waaaaay more complex than openstack | 15:21 |
| sdake | prometheus is an example of a complex application | 15:21 |
| rhallisey | not nevery application spans all the way from the bottom to the top of the stack | 15:21 |
| inc0 | in terms of management | 15:21 |
| sdake | it uses..... oeprators. | 15:21 |
| inc0 | ehh, I give up, you write a poc of operator | 15:21 |
| rhallisey | we are :) | 15:21 |
| inc0 | and I'll be happy to be proven wrong that it's not hyper work intensive | 15:21 |
| rhallisey | roger | 15:21 |
| rhallisey | I laid out the argument in the spec | 15:22 |
| rhallisey | I'm going to add it to why in my next patch | 15:22 |
| inc0 | yes, 1. not a compose, nto same problems | 15:22 |
| inc0 | 2. debugging is done by tools outside of deployment | 15:22 |
| Serlex | Anyone having problems setting sysctl values during deploy? | 15:22 |
| rhallisey | 3. client vs server side | 15:22 |
| *** neilus has joined #openstack-kolla | 15:23 | |
| rhallisey | I'll lay it out in the spec and you can comment | 15:23 |
| *** neilus has quit IRC | 15:23 | |
| inc0 | ok, thank you | 15:23 |
| rhallisey | cool :) | 15:23 |
| *** neilus has joined #openstack-kolla | 15:24 | |
| sdake | inc0 this needs an ack: https://review.openstack.org/396903 | 15:24 |
| inc0 | acked | 15:24 |
| *** eaguilar has quit IRC | 15:26 | |
| *** TxGirlGeek has joined #openstack-kolla | 15:27 | |
| *** coolsvap has quit IRC | 15:29 | |
| *** tonanhngo has joined #openstack-kolla | 15:32 | |
| *** bmace has quit IRC | 15:33 | |
| *** papacz has joined #openstack-kolla | 15:37 | |
| *** Pavo has quit IRC | 15:38 | |
| v1k0d3n | so guys, i talked with QuentinM from coreos for a while about stackanetes, entrypoint, operators, etc...and i think everyone wants to collaborate here. | 15:38 |
| *** Pavo has joined #openstack-kolla | 15:38 | |
| v1k0d3n | i think there's a misunderstanding about operators vs entrypoint. in fact, the conversation shouldn't be "vs" per say...but along with in the distant future. | 15:39 |
| v1k0d3n | entrypoint is the logic needed to make containers/applications self-aware, and ready for scheduling in a distributed system. | 15:39 |
| v1k0d3n | operator is different logic; this is how the daily operator would upgrade, scale, and resolve upgrade issues (if i understand correctly). that's not exactly the same thing. | 15:40 |
| v1k0d3n | QuentinM: can you share your thoughts in working with stackanetes? | 15:40 |
| sdake | v1k0d3n i disagree with the distant future thing :) | 15:41 |
| v1k0d3n | ok. well, can we determine what the greater group agrees on? | 15:42 |
| *** adrian_otto has joined #openstack-kolla | 15:43 | |
| sdake | v1k0d3n yes we have a rollcall vote for exactly this problem | 15:43 |
| inc0 | can we do PoCs of solutons please? | 15:43 |
| v1k0d3n | or, can we start collaborating on PoC's to move this forward? | 15:43 |
| sdake | v1k0d3n typically specs require a rollcall vote from the core reviewers | 15:43 |
| v1k0d3n | lol | 15:43 |
| v1k0d3n | same thought there inc0 | 15:43 |
| inc0 | I mena really, no plan ever survived facing the enemy | 15:43 |
| *** gagehugo has joined #openstack-kolla | 15:43 | |
| sdake | inc0 i don't get the analogy | 15:44 |
| v1k0d3n | ok, so i have a question sdake | 15:44 |
| inc0 | if we can't deliver mariadb and keystone co-deployment PoC quality soon, that's no-go for Feb anyway | 15:44 |
| v1k0d3n | has anyone reached out to folks at CoreOS to fully understand operator logic? | 15:44 |
| sdake | v1k0d3n i can't speak for others, but I haven't | 15:45 |
| pbourke | awiddersheim: hey andrew question, how sure are you that 'gather_facts: false' is required in site.yml to prevent ansible gathering twice | 15:45 |
| v1k0d3n | and since they (along with Intel) are using entrypoint...it seems logical that they would understand the differences. | 15:45 |
| sdake | instead I quizzed 75-100 people at CNCF about the model | 15:45 |
| sdake | so i got a varied lot of answers rather then one co's viewpoint | 15:45 |
| sdake | or two co's viewpoints | 15:45 |
| sdake | and none of these people i quizzed had horses in the race | 15:46 |
| QuentinM | Operators are here to help managing complex stateful applications such as etcd or galera. | 15:46 |
| sdake | (as in they didn't have a COI) | 15:46 |
| inc0 | o/ QuentinM thanks for joining us | 15:46 |
| QuentinM | They are designed to help Kubernetes taking smart actions on failures | 15:46 |
| QuentinM | To re-establish quorum properly for example | 15:46 |
| awiddersheim | pbourke: pretty sure | 15:47 |
| sdake | QuentinM what about application control? | 15:47 |
| QuentinM | They aren't meant to be used for every single application. | 15:47 |
| pbourke | awiddersheim: ok I'll check it out further, just not seeing that in any examples online | 15:47 |
| pbourke | awiddersheim: not a huge deal | 15:47 |
| awiddersheim | ok | 15:47 |
| awiddersheim | there are some other ways to do that which might be a little cleaner looking but | 15:47 |
| QuentinM | Say keystone-api, what do you need Kubernetes to do? Run a healthcheck, kill and restart the container if it fails. Pretty simple, can already do it. | 15:47 |
| awiddersheim | that is what we decided on | 15:47 |
| pbourke | awiddersheim: have a small change I'll prob submit soon that would appreciate you having a look | 15:47 |
| awiddersheim | sure | 15:48 |
| pbourke | awiddersheim: (around the fact gathering stuff again) | 15:48 |
| inc0 | QuentinM, more than this, for example nova upgrades are multi-step process | 15:48 |
| QuentinM | What else do you need to do with keystone-api? Maybe check if rabbitmq is available first and where it is | 15:48 |
| *** openstackgerrit has quit IRC | 15:48 | |
| inc0 | rather than just rolling code | 15:48 |
| QuentinM | inc0: init container | 15:48 |
| *** openstackgerrit has joined #openstack-kolla | 15:48 | |
| QuentinM | inc0: jobs | 15:48 |
| *** chas_ has joined #openstack-kolla | 15:48 | |
| sdake | QuentinM keystone contianer -> reconfigure and upgrade | 15:48 |
| sdake | QuentinM we also do config merging in kolla | 15:49 |
| inc0 | QuentinM, are operator meant to for example call jobs in correct order? | 15:49 |
| sdake | QuentinM we expect the operator to handle that | 15:49 |
| QuentinM | swap configmap and change image? | 15:49 |
| sdake | QuentinM ok lets take something more complex like nova+neutron | 15:49 |
| sdake | how precisely would you upgrade those by swapping images | 15:49 |
| *** absubram has joined #openstack-kolla | 15:50 | |
| QuentinM | for ordering jobs, containers and such, we use an entrypoint that queries the kubernetes API and the required resources | 15:50 |
| QuentinM | each app is made self-aware | 15:50 |
| sdake | that doesn't solve upgrades for nova does it inc0? | 15:51 |
| inc0 | it can | 15:51 |
| sdake | QuentinM the only way that would work viably is if entrypoint had some logic customized for each service | 15:51 |
| inc0 | with a bit tweaked entrupoint model | 15:51 |
| sdake | why do it in the container when it can be done outside the container | 15:51 |
| inc0 | in fact, we can do whatever with this model as every workflow can be done by set of self-dependand jobs | 15:51 |
| inc0 | I promise you that I can handle upgrades with entrypoint model on speed that I'm working on | 15:52 |
| sdake | QuentinM i'd suggest having this conversation with kfox1111_away and rhallisey - they are the two cats leading this merry bunch | 15:52 |
| openstackgerrit | caoyuan proposed openstack/kolla: Move the "enable_destroy_images" into configure file https://review.openstack.org/397633 | 15:53 |
| sdake | QuentinM i'm an innocent bystander ;) | 15:53 |
| *** chas_ has quit IRC | 15:53 | |
| sdake | same msg for you inc0 | 15:53 |
| inc0 | QuentinM, if you don't mind hanging out here, I'll point Ryan and Kevin (2 other cores from kolla-k8s) to this irc log and we'll have more questions | 15:54 |
| sdake | one question I do have is, what is all the pushback on operators about? | 15:54 |
| inc0 | I think I have fair grasp about operator idea, but we have some misunderstandings here | 15:54 |
| sdake | is it about the ocata schedule? | 15:54 |
| inc0 | sdake, it's about solving problem with tool not meant for it | 15:55 |
| inc0 | increasing complexity tremendously while at it | 15:55 |
| sdake | any other complaints? | 15:55 |
| inc0 | well, there will be when we write I bet | 15:56 |
| sdake | all the cats i spoke to at CNCF said workflow was never going to happen in kubernetes the only viable solution was controllers | 15:56 |
| inc0 | because we'll need to reinvent multiple wheels | 15:56 |
| sdake | same cats also said workflow was never going to happen in helm | 15:56 |
| inc0 | so I don't think anybody created perfect solution for it | 15:56 |
| inc0 | because operators I've seen - not a workflow | 15:57 |
| inc0 | and yes, I potentially can see operator'ish soluton that will deal with workflows | 15:57 |
| sdake | QuentinM could you explain to inc0 how resize in the etcd operator is a workflow? | 15:57 |
| * rhallisey is reading | 15:57 | |
| sdake | TIA :) | 15:57 |
| inc0 | and in fact, that's what I'm writing as we speak | 15:57 |
| rhallisey | I can't write an email | 15:57 |
| rhallisey | because the log blows up every 10 min | 15:57 |
| rhallisey | O.o | 15:57 |
| inc0 | rhallisey, yeah. | 15:57 |
| rhallisey | :) | 15:57 |
| sdake | so ryan set a deadline of today to merge the spec | 15:58 |
| inc0 | rhallisey, QuentinM is from CoreOS, I'm pretty sure we've met in BCN | 15:58 |
| sdake | lets either get it merged or abandon it | 15:58 |
| inc0 | no, let's come up with good solution | 15:58 |
| sdake | feel free to chime in on the spec | 15:58 |
| inc0 | I will | 15:58 |
| *** dave-mccowan has quit IRC | 15:59 | |
| inc0 | once Ryan adds issues with entrypoint | 15:59 |
| rhallisey | to many things going on at once :) | 15:59 |
| rhallisey | I'll finish the spec real quick | 15:59 |
| rhallisey | next iteration* | 15:59 |
| sdake | may I point people at the following: | 16:00 |
| *** TxGirlGeek has quit IRC | 16:00 | |
| sdake | https://github.com/openstack/kolla-kubernetes/blob/master/specs/README.rst | 16:00 |
| *** TxGirlGeek has joined #openstack-kolla | 16:01 | |
| Pavo | sdake typo | 16:02 |
| Pavo | refelcts | 16:02 |
| Pavo | should be reflects | 16:02 |
| *** unicell has joined #openstack-kolla | 16:03 | |
| *** mgiles has joined #openstack-kolla | 16:04 | |
| *** unicell1 has quit IRC | 16:05 | |
| QuentinM | inc0: I agree with you | 16:06 |
| QuentinM | inc0: Workflow should not depend on external services but each piece should be aware of its surroundings. | 16:06 |
| inc0 | well, there is value of having external mgmt | 16:07 |
| QuentinM | inc0: Let me see if I can pull one of the original operator author so he can discuss a bit deeper the pros/cons. | 16:07 |
| QuentinM | inc0: there is | 16:07 |
| inc0 | that would be awesome, thanks | 16:07 |
| *** zhubingbing_ has joined #openstack-kolla | 16:08 | |
| zhubingbing_ | hello guys | 16:08 |
| sdake | rhallisey debate - which is better diamonds or gold? | 16:12 |
| rhallisey | that's a complex q | 16:12 |
| rhallisey | or are you asking me to debate it | 16:12 |
| rhallisey | or want to know my opinion | 16:13 |
| sdake | asking your opinion | 16:13 |
| Pavo | diamonds last forever | 16:13 |
| sdake | pavo so does gold :) | 16:13 |
| srwilkers | gold is malleable and can change if we need it to | 16:13 |
| inc0 | jewels gold+diamonds;) | 16:13 |
| srwilkers | diamonds dont | 16:13 |
| rhallisey | sdake, I would say gold | 16:14 |
| sdake | why rhallisey | 16:14 |
| inc0 | diamond can cut gold, not the opposite | 16:14 |
| rhallisey | sdake, the market | 16:14 |
| zhubingbing_ | I'm starting to contribute to k8s next week. －） | 16:14 |
| sdake | srwilkers you analyzd correctly, but what is your opinion on better ;) | 16:14 |
| inc0 | zhubingbing_, prepare your brain for bending;) | 16:15 |
| srwilkers | sdake: gold, for that reason | 16:15 |
| openstackgerrit | Merged openstack/kolla: Ansible-ize OpenStack Designate https://review.openstack.org/353261 | 16:15 |
| zhubingbing_ | K8s used before. | 16:15 |
| sdake | anyone else care to hazard a guess? | 16:16 |
| inc0 | I need to get to markets | 16:16 |
| inc0 | but I'm afraid of my hot headiness | 16:17 |
| sdake | http://www.minecraftforum.net/forums/minecraft-discussion/survival-mode/283336-diamond-vs-gold | 16:17 |
| *** dave-mccowan has joined #openstack-kolla | 16:17 | |
| sdake | my kids are fans of minecraft | 16:17 |
| inc0 | minecraft is cool | 16:17 |
| sdake | i think those cats did a good job of modeling diamonds and gold - worth a read ;) | 16:18 |
| srwilkers | my first exposure to containers was making an image to spin up a minecraft server i still use at home | 16:18 |
| inc0 | when they create ALU in it, you won at parenting | 16:18 |
| sdake | parenting is a lifelong thing | 16:18 |
| sdake | you never win at parenting | 16:18 |
| sdake | you die first trying | 16:18 |
| inc0 | so is knowledge of ALUs | 16:19 |
| *** chas_ has joined #openstack-kolla | 16:19 | |
| sdake | srwilkers my first exposure was starting kolla ;-) | 16:19 |
| srwilkers | sdake: you win | 16:20 |
| sdake | srwilkers literally didn't know a thing about docker prior to spinning up the project | 16:20 |
| srwilkers | thats actually pretty encouraging | 16:20 |
| Pavo | I know the feeling sdake | 16:20 |
| portdirect | hey guys, i think on the issue of diamonds vo gold i can help, following esensive research i've concluded that | 16:20 |
| *** zhubingbing_ has quit IRC | 16:21 | |
| srwilkers | ill remember that the next time i feel like my heads swimming and i dont know wtf im doing | 16:22 |
| *** chas_ has quit IRC | 16:23 | |
| *** fguillot_ has joined #openstack-kolla | 16:27 | |
| kfox1111_away | monring. | 16:28 |
| sdake | sup kfox1111_away | 16:28 |
| *** fguillot has quit IRC | 16:28 | |
| inc0 | kfox1111_away, howdy | 16:28 |
| *** kfox1111_away is now known as kfox1111 | 16:28 | |
| sdake | you missed the diamonds wo gold discussion | 16:28 |
| inc0 | read the log plz;) | 16:28 |
| inc0 | not about this one | 16:28 |
| sdake | rather vo gold discussion | 16:28 |
| kfox1111 | seems long. how far back? | 16:28 |
| inc0 | around when QuentinM showed up:) | 16:29 |
| sdake | whenever inc0 woke up ;) | 16:29 |
| kfox1111 | about 10:14? | 16:30 |
| kfox1111 | reading... | 16:30 |
| *** mgoddard_ has joined #openstack-kolla | 16:30 | |
| inc0 | operators | 16:30 |
| inc0 | disclaimer QuentinM is from CoreOS, and I really think we have some disconnect on what operators are meant to be | 16:30 |
| sdake | planning paralysis rocks | 16:31 |
| sdake | i'm experiencing this at work | 16:32 |
| sdake | and also at work upstream | 16:32 |
| sdake | i guess that is a nick in diamond's favor | 16:32 |
| QuentinM | inc0: let's wait for the appropriate engineer to show up and discuss more technically with you guys what it means. while I am experienced with Kubernetes, the original authors are fairly more experienced on that particular area! | 16:33 |
| QuentinM | inc0: west side timezone | 16:33 |
| *** mgoddard has quit IRC | 16:33 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Load murano dashboard dynamic https://review.openstack.org/395957 | 16:34 |
| sdake | sup adrian_otto | 16:34 |
| sdake | been awhile since i've seen you here | 16:34 |
| qwang | good morning | 16:35 |
| adrian_otto | hey there. I am on a video conference and running my team meeting in #openstack-meeting-alt right now, so attention very divided. | 16:35 |
| sdake | roger | 16:35 |
| sdake | sup qwang | 16:35 |
| adrian_otto | ok, so is there any additional input we should consider on 395957 before going through the merge process? | 16:35 |
| adrian_otto | ack, see now I am distracted, using the wrong channel | 16:36 |
| sdake | ;) | 16:36 |
| sdake | sorry about that | 16:36 |
| inc0 | QuentinM, no problem, just getting Kevin up to speed, appreciate that | 16:36 |
| sdake | QuentinM what would help most is if those folks commented on teh specification | 16:37 |
| *** egonzalez90 has quit IRC | 16:37 | |
| portdirect | just looking at it 395957, does horizon support a db cache backend- it looks abit confisuing from the config? murano-dashboard needs this for complex topologies. | 16:38 |
| inc0 | sdake, ;et | 16:38 |
| inc0 | sdake, let's not add too much work to this discussion | 16:38 |
| inc0 | gerrit workflow is not something people are used to normally | 16:38 |
| inc0 | but regardless, let's have discussion | 16:39 |
| kfox1111 | almost caught up.... | 16:39 |
| kfox1111 | ok. not caught up with the spec chagnes (please lets get the spec right rather then push something through for a tight deadline?) | 16:39 |
| kfox1111 | will look at those in a bit. | 16:39 |
| kfox1111 | so... entrypoint vs operators. | 16:40 |
| kfox1111 | I think they are mostly ortoganal but have some overlap. | 16:40 |
| inc0 | well, not exactly orthogonal (I agree they are) but question is, what will solve deployment | 16:40 |
| inc0 | and upgrades | 16:40 |
| inc0 | and reconfigure | 16:41 |
| inc0 | so more universal tasks that aren't project specific | 16:41 |
| kfox1111 | I'm ok with self aware containers for some types of healing. but I also really am nervious about workflow being scattered about where a human operator can't easily debug / override it, or where unexpected results happen. | 16:41 |
| inc0 | kfox1111, no argument there, but I think that's something that can be solved by proper debugging methodology/external tooling | 16:42 |
| kfox1111 | the more embeded the workflow is in the containers thoguh, the harder it is to debug. or disable when it fights you. | 16:42 |
| inc0 | also API to "entrypoint" is not defined yet, we can figure out something easily debugabble | 16:42 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture https://review.openstack.org/392257 | 16:43 |
| rhallisey | review ^ | 16:43 |
| rhallisey | I need a quick bite | 16:43 |
| inc0 | but init-containers based workflow is not that bad | 16:43 |
| kfox1111 | inc0: distributed is always harder to debug then centeralized. | 16:43 |
| inc0 | but distributed is more robust | 16:43 |
| kfox1111 | init containers are great for initing pods. | 16:43 |
| inc0 | to stuff like failure scenerios and whatnot | 16:43 |
| kfox1111 | for things like mysql/keystone endpoints/etc, no init-containers for init. | 16:43 |
| rhallisey | kfox1111, I added some arguments in there. Worth a read | 16:43 |
| kfox1111 | rhallisey: will do. | 16:44 |
| portdirect | inc0: distributed in k8s context can actually cause failures - I've seen this with heavy load on api server | 16:44 |
| kfox1111 | so something needs to kick off bootstrap/upgrade type jobs. | 16:44 |
| QuentinM | Would it make sense to move this discussion over e-mail to have a better / long-lived / referencable discussion? | 16:44 |
| inc0 | portdirect, I'm not talking about every container ddosing API | 16:44 |
| sdake | QuentinM this channel is logged - see topic | 16:45 |
| inc0 | I'm talinkg actually about external pod to handle dependencies, pod will run single-node but with ability to be distributed | 16:45 |
| sdake | QuentinM the best place to have the discussion is in the specification itself | 16:45 |
| kfox1111 | yeah. another disadvantage of using the distributed container logic is that it may put a lot of load on the kube-apiserver in times of failure and it requires security credentials out in a lot more places. | 16:45 |
| inc0 | QuentinM, issue is, we want this thing to be done asap;) mail is long | 16:45 |
| kfox1111 | it can be mmitigated by using it where its really needed and not just everywhere. | 16:46 |
| inc0 | so ok, I'm not talking about every container ramming the API | 16:46 |
| portdirect | inc0: ah ok | 16:46 |
| portdirect | sry | 16:46 |
| inc0 | np | 16:46 |
| inc0 | this is common misunderstanding, because that's kinda how entrypoint works today | 16:46 |
| inc0 | but I'm talking of evolved version | 16:46 |
| kfox1111 | I think entrypoint in an init-container would be pretty good for a lot of things. | 16:47 |
| kfox1111 | I think operators solve the bootstrap/upgrade worflow issue. | 16:47 |
| QuentinM | sdake: https://review.openstack.org/#/c/392257/ right? | 16:47 |
| inc0 | that's right QuentinM | 16:48 |
| *** TxGirlGeek has quit IRC | 16:48 | |
| QuentinM | Thanks, forwarded | 16:48 |
| kfox1111 | I think the distributed workflow of stuff like, block container until systems ready, (passive workflow) fits nicely in entrypoint. | 16:48 |
| portdirect | I'm goint to sit this out beasue I have to head In 5, but my instsict relfects kfox1111' last two lines exactly at this stage. | 16:48 |
| kfox1111 | the active workflow, see if db has X, if not, create X, | 16:48 |
| *** portdirect is now known as portdirect_away | 16:48 | |
| *** TxGirlGeek has joined #openstack-kolla | 16:48 | |
| kfox1111 | is better handled by k8s jobs and driven by operators. | 16:48 |
| inc0 | kfox1111, agree | 16:48 |
| inc0 | just let's not reimplement ansible tbh | 16:49 |
| pbourke | awiddersheim: just verified it is needed | 16:49 |
| QuentinM | kfox1111: that's what we do with Stackanetes | 16:49 |
| *** chas_ has joined #openstack-kolla | 16:49 | |
| kfox1111 | QuentinM: awesome. :) | 16:49 |
| inc0 | QuentinM, upgrades done with operators? | 16:49 |
| inc0 | and deploy with entrypoint? (in our case entrypoint-initcontainer) | 16:49 |
| kfox1111 | inc0: not quite what I said. deploy is also done with operators, but the blocking of things till they come up is entrypoint. | 16:50 |
| sbezverk | inc0: https://coreos.com/blog/introducing-operators.html they talk about upgrades with operators | 16:50 |
| kfox1111 | sbezverk: welcome back. :) | 16:50 |
| sbezverk | kfox1111: thanks :-) you guys are way ahead now | 16:51 |
| inc0 | kfox1111, well, with entrypoint you just literally start EVERYTING up | 16:51 |
| inc0 | and it will resolve itself | 16:51 |
| inc0 | can be done with helm | 16:51 |
| kfox1111 | inc0: no. | 16:51 |
| inc0 | why? | 16:51 |
| kfox1111 | I don't want 300 pods trying to create an entyr in keystone. | 16:51 |
| sdake | asalkeld said this model was bad as well | 16:52 |
| QuentinM | why would it happen? | 16:52 |
| sdake | shame he isn't around atm | 16:52 |
| inc0 | so you want to deploy one service at time? | 16:52 |
| QuentinM | you've one job that creates the entry in keystone | 16:52 |
| kfox1111 | the workflow of creating an openstack service (deployment) is kind of a one time thing. | 16:52 |
| QuentinM | you can have the job wait for the keystone-api container to be up and answer healthchecks too. | 16:52 |
| inc0 | QuentinM, every service has to have entrypoint in ks - nova has its own, neutron too...and so on and so forth | 16:53 |
| QuentinM | I know | 16:53 |
| kfox1111 | that logic sholdn't be distributed or put in lots of different places. | 16:53 |
| inc0 | well, I think it can tho | 16:53 |
| kfox1111 | a job for creating a keystone entry is the solution. | 16:53 |
| inc0 | it's like...10 entrypoints total? | 16:53 |
| QuentinM | We are using jobs for Stackanetes | 16:53 |
| inc0 | and if you're using job, it wont' be THAT distributed | 16:53 |
| QuentinM | We don't have 300 pods trying to do it | 16:53 |
| QuentinM | Only one container for the job is running | 16:53 |
| *** srwilkers has quit IRC | 16:53 | |
| inc0 | yeah that ^ | 16:53 |
| kfox1111 | entrypoint should be passive. blcok container until deps are ready. | 16:53 |
| QuentinM | It waits for mariadb to be answer healthchecks (it simply asks the kubernetes API) | 16:54 |
| kfox1111 | exactly. | 16:54 |
| QuentinM | then run | 16:54 |
| inc0 | kfox1111, job can be a dependency too | 16:54 |
| QuentinM | it's a really simple design. | 16:54 |
| *** chas_ has quit IRC | 16:54 | |
| kfox1111 | dependency, yes. should it spawn, no. | 16:54 |
| kfox1111 | pasive workflow. | 16:54 |
| *** fragatina has joined #openstack-kolla | 16:54 | |
| QuentinM | what do you mean: should it spawn? | 16:54 |
| inc0 | "don't start nova-api unltil nova-keystone-bootstrap-job is ready" | 16:54 |
| QuentinM | yeah | 16:54 |
| kfox1111 | inc0: yes. that. | 16:54 |
| *** fragatina has quit IRC | 16:54 | |
| QuentinM | that's what we have | 16:54 |
| inc0 | can be done with entrypoint easily | 16:55 |
| kfox1111 | but nova-api for example should never try and launch a nova-register-keystone-endpoint job. | 16:55 |
| DTadrzak | Maybe we can out tomorrow PoC about kuberenetes entrypoint and init-container with How Stackanetes works? | 16:55 |
| inc0 | just...job is yet another dependency | 16:55 |
| inc0 | kfox1111, no..not nova-api | 16:55 |
| QuentinM | the nova-api container runs, but the nova-api binary doesn't, until kubernetes tells us that mariadb and the initialization job are OK. | 16:55 |
| DTadrzak | extend * | 16:55 |
| inc0 | something else will start all the jobs | 16:55 |
| kfox1111 | the nova-register-keystone-endpoint job shoudl only ever be run by either a person operator or a k8s operator. | 16:55 |
| inc0 | and pods in that matter | 16:55 |
| QuentinM | Hey DTadrzak | 16:55 |
| inc0 | and it will resolve itself | 16:55 |
| DTadrzak | Hello QuentinM:] | 16:56 |
| QuentinM | kfox1111: on Stackanetes, everything is deployed by kpm (just like helm) | 16:56 |
| QuentinM | kfox1111: at once | 16:56 |
| QuentinM | then, the entrypoint on each container, make sure that no openstack binary starts before their deps are OK | 16:56 |
| *** magicboiz has quit IRC | 16:56 | |
| inc0 | QuentinM, can job have an init-container tho? | 16:56 |
| kfox1111 | QuentinM: I'm a little concerned with that. I've got some oddball clusters, and workflows tend to be more hinderence to my usage then a help. | 16:56 |
| DTadrzak | I think that we have solved a lot of problem in Stackanetes we can show how it works exactly any maybe something will be worth to move it to kolla-k8s? | 16:56 |
| inc0 | we need deps in jobs as well | 16:56 |
| QuentinM | so the keystone boostrap job will execute once mariadb answers healthcheck | 16:56 |
| kfox1111 | I totally get some people want really sijmple deployment. and we should support that. | 16:56 |
| QuentinM | and then keystone api starts when job is finished | 16:57 |
| kfox1111 | so I want to be able to manually deploy everything rather then have orchestration do it. | 16:57 |
| QuentinM | it's a "drop all the manifests in one command and let the rest happen" pattern | 16:57 |
| inc0 | DTadrzak, it can't be exact copy as we don't really want entrypoint in every container | 16:57 |
| kfox1111 | the "magic" stuff bites me. | 16:57 |
| QuentinM | it's not magic? | 16:57 |
| inc0 | kfox1111, totally doable too | 16:57 |
| QuentinM | every container queries kubernetes for statuses | 16:58 |
| inc0 | I mean, it can be single command, it can be set of commands | 16:58 |
| QuentinM | is mariadb healthy? | 16:58 |
| inc0 | it'll all work | 16:58 |
| QuentinM | is the boostrap job finished? | 16:58 |
| QuentinM | if yes, then start the binary | 16:58 |
| kfox1111 | QuentinM: what if I don't bootstap? | 16:58 |
| Pavo | Stackanetes demo video looks interesting | 16:58 |
| inc0 | one command is just set of subcommands | 16:58 |
| kfox1111 | I have an existing cluster, I'm going to migrate. | 16:58 |
| kfox1111 | it already has a bootstraped db. | 16:58 |
| kfox1111 | no reason to have a job. | 16:58 |
| kfox1111 | that kind of stuff. | 16:58 |
| sdake | pavo would appreciate your honest eval feedback ;-) | 16:58 |
| *** portdirect_away is now known as portdirect | 16:58 | |
| QuentinM | kfox1111: well then just set the job status as finished | 16:59 |
| kfox1111 | the orchestration stuff either needs to be excluded, | 16:59 |
| kfox1111 | or really not assume things. :/ | 16:59 |
| QuentinM | or don't set the dep | 16:59 |
| QuentinM | https://www.youtube.com/watch?v=fbEmoxxO3Oc the video Pavo is mentioning | 16:59 |
| Pavo | from just watching the video seems pretty neat but also doesn't look to customizable | 16:59 |
| *** portdirect is now known as portdirect_away | 16:59 | |
| QuentinM | Pavo: it is actually | 16:59 |
| QuentinM | Pavo: you can drop in any configuration files or images you want | 16:59 |
| *** fguillot_ has quit IRC | 16:59 | |
| *** portdirect_away is now known as portdirect | 17:00 | |
| inc0 | yeah, this dependency model can be modified pretty neatly | 17:00 |
| kfox1111 | QuentinM: one goal I have, is to make packages/containers easily shippable. | 17:00 |
| inc0 | anything external - dont put it as dependency | 17:00 |
| kfox1111 | so it shoudl be easily customizable, without having to rebuild a package/container. | 17:00 |
| QuentinM | that's why we wrote kpm | 17:00 |
| QuentinM | to ship kubernetes packages | 17:00 |
| inc0 | QuentinM, fyi it'll be helm for us most likely | 17:00 |
| inc0 | but same stuff | 17:00 |
| QuentinM | it is kinda like helm, but gives more freedom on variables | 17:00 |
| QuentinM | yeah | 17:01 |
| Pavo | QuentinM how hard would it be to add Provider VLANs and separate all the different type of traffic, only reason I ask is because I haven't tried anything kubernetes yet | 17:01 |
| inc0 | Pavo, that's beyond k8s, that's neutron configuration | 17:01 |
| DTadrzak | +1 | 17:01 |
| QuentinM | for example, on Stackanetes, if you don't provide Ceph configuration, then the cinder Cependency is completely left out from any other services because Cinder won't be deployed. | 17:01 |
| kfox1111 | inc0: relavent though, as its a config thing the packages have to deal with. | 17:01 |
| *** fguillot has joined #openstack-kolla | 17:01 | |
| inc0 | kfox1111, we need to inject configmaps externally | 17:01 |
| inc0 | one way or another | 17:01 |
| QuentinM | inc0: +1 | 17:01 |
| kfox1111 | inc0: agreed. | 17:02 |
| Pavo | inc0 yeah I get that but doesn't neutron config have to relate to k8s connections? | 17:02 |
| inc0 | we can deal with this problem with tool designed specifically for it | 17:02 |
| *** g3ek has quit IRC | 17:02 | |
| kfox1111 | inc0: but then helm packages cant' stand completely on their own and need something like operator to help with the workflow. | 17:02 |
| inc0 | not really, from neutron perspective it'll look just as ansible deployed version | 17:02 |
| Pavo | ah | 17:02 |
| sdake | part of the workflow is configuration.. ;-) | 17:02 |
| kfox1111 | Pavo: yeah. it uses net=host, which ignores most of k8s networking. | 17:02 |
| *** haplo37 has quit IRC | 17:03 | |
| kfox1111 | so oprators are needed, and packages alone arn't sufficient. | 17:03 |
| *** haplo37_ has quit IRC | 17:03 | |
| Pavo | ok I think I am understanding it now | 17:03 |
| inc0 | kfox1111, deployment will be hadnled with mixture of entrypoint-init+helm | 17:03 |
| QuentinM | https://github.com/stackanetes/stackanetes/blob/master/neutron/templates/openvswitch.yaml.j2#L32 | 17:03 |
| kfox1111 | entrypoint-init+helm+operators | 17:03 |
| Pavo | this whole ansible, docker and kubernetes is all really new to me | 17:03 |
| QuentinM | https://github.com/stackanetes/stackanetes/blob/master/neutron/templates/init.yaml.j2#L26 | 17:03 |
| QuentinM | etc | 17:03 |
| inc0 | kfox1111, operators will not do deployment itself | 17:03 |
| Pavo | I am getting an understanding of docker pretty well though since I have started playing with kolla | 17:03 |
| inc0 | we'll put them wherever you need to do crazy stuff | 17:04 |
| Pavo | and starting to understand ansible really well also | 17:04 |
| kfox1111 | QuentinM: thats going to be alot tricker for me. | 17:04 |
| kfox1111 | QuentinM: I need multiple mariadb's, multiple rabbits, etc. | 17:04 |
| inc0 | Pavo, k8s is next level;) I suggest you take a look at it later, it's this new way of doing stuff;) | 17:04 |
| kfox1111 | can be templated I think though. | 17:04 |
| inc0 | but still early in the game tbh | 17:04 |
| QuentinM | we have | 17:04 |
| QuentinM | we have HA galera | 17:04 |
| QuentinM | HA rabbitmq | 17:05 |
| kfox1111 | yeah, no. I mean multiple ha things. | 17:05 |
| QuentinM | why not namespaces? | 17:05 |
| kfox1111 | like, ceilometer rabbit seperate from nova's rabbit, | 17:05 |
| QuentinM | it is designed to avoid this kind of collisions | 17:05 |
| kfox1111 | but woven into the config of nova-api. | 17:05 |
| QuentinM | kfox1111: right, we do that for elastic-search | 17:05 |
| Pavo | and I apologize to all of you since I am so newbish to all this | 17:05 |
| portdirect | QuentinM: there are reasons other than HA, we need seperation for security for many deployents | 17:05 |
| QuentinM | kfox1111: we have an elasticsearch for searchlight and another for something else | 17:05 |
| kfox1111 | Pavo: no worries. we're all learning. :) | 17:05 |
| QuentinM | kfox1111: we template the name of the service/deployment using the freedom that kpm gives us with JSONNET | 17:06 |
| QuentinM | portdirect: y | 17:06 |
| kfox1111 | QuentinM: so we need a way to launch multiple/differently named rabbits/mariadb's/elasticsearches. | 17:07 |
| kfox1111 | so my plan was to make a package for that, so it was launchable several times. | 17:07 |
| inc0 | kfox1111, but helm can do that right? | 17:09 |
| QuentinM | kfox1111: yeah, that's exactly what we did | 17:09 |
| QuentinM | kfox1111: we then inject certain things like the name of the service for discovery through variables | 17:09 |
| QuentinM | we have two deployments of the same elasticsearch package, but different parameters given | 17:09 |
| kfox1111 | yeah. | 17:09 |
| *** bmace has joined #openstack-kolla | 17:10 | |
| *** magicboiz has joined #openstack-kolla | 17:10 | |
| QuentinM | but it's quite hard to do with helm currently because of their limitation around variable manipulation unfortunately | 17:10 |
| kfox1111 | I think I've gotten that bit to work. | 17:10 |
| portdirect | Quentin: we would need this to allow for services that are not in the same k8s deplyment, or even deployed with k8s (should be possible with any method but worth bearing in mind) | 17:10 |
| QuentinM | cool | 17:10 |
| kfox1111 | portdirect: right. | 17:11 |
| inc0 | portdirect, can be done with config manipulation and dependency manipulation | 17:11 |
| kfox1111 | I want to be able to take clouds deployed with something else, | 17:11 |
| QuentinM | yeap | 17:11 |
| QuentinM | easy | 17:11 |
| kfox1111 | and a piece at a time "upgrade" to k8s managed openstack. | 17:11 |
| inc0 | you just...dont specify mariadb as dependency and hardcode connection to it in configs | 17:11 |
| kfox1111 | so workflow bits really can't get in the way. | 17:11 |
| rhallisey | time to read back while lunching :) | 17:11 |
| QuentinM | yeap inc0 | 17:11 |
| QuentinM | our openstack package are agnostic | 17:11 |
| QuentinM | you pass them the adress to your mariadb server for example | 17:11 |
| kfox1111 | so, those deps have to be templated out to make them configurable. | 17:11 |
| rhallisey | the community is active last few days :) | 17:12 |
| * rhallisey reads back | 17:12 | |
| QuentinM | so it can be either an internal mariadb, or an external, doesn't matter | 17:12 |
| QuentinM | same with keystone-api and such | 17:12 |
| inc0 | so QuentinM only reservation I have regarding entrypoint is that I don't think it should be embedded in every container | 17:12 |
| QuentinM | kfox1111: true | 17:12 |
| inc0 | and without that we will have issues with init containers for jobs... | 17:12 |
| QuentinM | kfox1111: in your case, they have to be | 17:12 |
| QuentinM | kfox1111: which is not a big deal to do | 17:12 |
| kfox1111 | so who runs the create-db job though? | 17:12 |
| *** matrohon has quit IRC | 17:13 | |
| kfox1111 | is that in a seperate package? | 17:13 |
| QuentinM | inc0: are you concerned about the size? | 17:13 |
| kfox1111 | I'm more of a fan of fine grained packages. | 17:13 |
| kfox1111 | gives more control. | 17:13 |
| inc0 | QuentinM, I'm concerned about kolla images being reusable | 17:13 |
| inc0 | they're not ansible-specifiv, we don't want them to be k8s specific too | 17:13 |
| QuentinM | kfox1111: we've made a "keystone" package for instance, it carries the job, now you can decide to avoid running it if you want. | 17:14 |
| inc0 | kfox1111, can package call a package? | 17:14 |
| *** tonanhngo has quit IRC | 17:14 | |
| QuentinM | kfox1111: yeah you can create meta-packages | 17:14 |
| kfox1111 | https://github.com/sapcc/openstack-helm/tree/master/neutron this large kind of package makes me thinkg it will be hard to work with. | 17:14 |
| QuentinM | inc0: yeah you can create meta-packages | 17:14 |
| kfox1111 | inc0: sort of. | 17:14 |
| *** TxGirlGeek has quit IRC | 17:14 | |
| sdake | kfox1111 is this accurate: https://www.gliffy.com/go/share/ssmhcjkog50xyl3gensq | 17:14 |
| kfox1111 | in helm, you can have subpackages, that are bundled wit hit. | 17:14 |
| kfox1111 | let me give you an example | 17:14 |
| inc0 | kfox1111, then you can make finer granularity | 17:14 |
| QuentinM | yeah that is gross | 17:14 |
| kfox1111 | https://review.openstack.org/#/c/396296/ | 17:15 |
| *** fguillot has quit IRC | 17:15 | |
| QuentinM | the issue is that helm is pretty bad at passing and modifying variables on the fly | 17:15 |
| kfox1111 | Ive converted one of our templates to helm. | 17:15 |
| QuentinM | so it makes doing small packages quite hard | 17:15 |
| *** TxGirlGeek has joined #openstack-kolla | 17:15 | |
| QuentinM | for no reason | 17:15 |
| QuentinM | just like the fact that currently they don't allow creating subfolders under the template folder | 17:15 |
| *** adrian_otto has quit IRC | 17:15 | |
| kfox1111 | QuentinM: my plan there was to maybe build the packages with tooling. | 17:15 |
| QuentinM | or allow you to do conditionals | 17:15 |
| kfox1111 | automate the building of the packages. | 17:15 |
| QuentinM | like we do: if this variable is true, then don't deploy this job, or don't deploy this package, or this conf | 17:16 |
| kfox1111 | so not so much redundant code, and easier to deal with some of the limitations. | 17:16 |
| QuentinM | and that's pretty important to us | 17:16 |
| kfox1111 | QuentinM: I found a workaroudn to that. | 17:16 |
| kfox1111 | with helm. | 17:16 |
| *** fguillot has joined #openstack-kolla | 17:16 | |
| kfox1111 | they launch all templates not starting with _ | 17:16 |
| kfox1111 | but you can put stuff in _ in a macro, and then call it conditionally in a main.yaml | 17:17 |
| sdake | kfox1111 woudl you mind taking a breather and checking my diagram plz | 17:17 |
| kfox1111 | sdake: sure | 17:17 |
| QuentinM | damn | 17:17 |
| sdake | https://www.gliffy.com/go/share/ssmhcjkog50xyl3gensq | 17:17 |
| inc0 | so I'd separate discussion about that really - helm/kpm is meant to render yamls for resources, something else has to deal with deployment/operations | 17:18 |
| kfox1111 | inc0: the issue is helm/kpm has some orchestration bits in it too. | 17:18 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Tacker NFV Ansible support https://review.openstack.org/389790 | 17:18 |
| kfox1111 | so we're discussing whether some bits can be done in the package system, or belongs in operators. | 17:18 |
| kfox1111 | I still am partial to logic in operators and not in packages. seperation of concerns. | 17:19 |
| kfox1111 | sdake: yeah, thats kind of what I was thinking. | 17:19 |
| *** chas_ has joined #openstack-kolla | 17:19 | |
| QuentinM | kfox1111: kpm has no orchestration bit | 17:19 |
| QuentinM | kfox1111: what kind of | 17:19 |
| kfox1111 | QuentinM: like, putting bootstrap jobs in the package that automaticaly get luanched. thats orchestration. | 17:20 |
| QuentinM | it just templates manifests with variables and use `kubectl create` to send them to kubernetes | 17:20 |
| sdake | kfox1111 sorry had to drop out of the convo for awhile - had phone call | 17:20 |
| sdake | kfox1111 anywhere the diagram can be improved? | 17:20 |
| *** TxGirlGeek has quit IRC | 17:20 | |
| inc0 | kfox1111, it's just launching all the stuff at same time tho right? | 17:20 |
| kfox1111 | sdake: I think the compute operator takes in a 3rd party resource, and pushes the keystone third party resource. | 17:20 |
| kfox1111 | inc0: yeah. but using entrypoint to block things to get an ordering. | 17:21 |
| QuentinM | kfox1111: inc0: yeah kpm just templates the manifests and send everything to kubernetes | 17:21 |
| sdake | kfox1111 probably something no tclear on that diagram is i think everything has a third party resource at the service level | 17:21 |
| QuentinM | the entrypoint is not kpm/helm related | 17:21 |
| inc0 | so that's not really orchiestration | 17:21 |
| pprokop | Hi Quentin :D | 17:21 |
| kfox1111 | sdake: right. | 17:21 |
| inc0 | that's just pushing all the things | 17:21 |
| QuentinM | kpm in our case is just meant to template and send all the resources to kubernetes and that's it. | 17:21 |
| kfox1111 | inc0: no, it is, as its orcestrating a deployment. | 17:21 |
| inc0 | for i in resources; kubectl create i | 17:21 |
| QuentinM | hi pprokop, whatup | 17:21 |
| QuentinM | inc0: exactly | 17:22 |
| kfox1111 | its an implicit workflow rather hten an explicit one. | 17:22 |
| *** DTadrzak has quit IRC | 17:22 | |
| QuentinM | it pulls a package from a repository in the given version / release channel, template and kubectl create | 17:22 |
| kfox1111 | implicit ones make me nervious as an operator. | 17:22 |
| inc0 | orchiestration...or choreography...is done with entrypoint | 17:22 |
| inc0 | really | 17:22 |
| QuentinM | yes | 17:22 |
| kfox1111 | you never know when the implicitness decides to do something bad. seen it far to often. | 17:22 |
| QuentinM | kpm is just a loop over templates and send them to the k8s API | 17:23 |
| kfox1111 | we had a cloud afflicted by chef deciding to recreate a keystone user over and over again. | 17:23 |
| kfox1111 | showed up as periodical rest flakyness. but most of the time thigns were fine. | 17:23 |
| sdake | kfox1111 that sounds like compoe + glance ;) | 17:23 |
| inc0 | so kfox1111 how about having external pod that will consume graph of resources generated by helm | 17:23 |
| inc0 | + have notion of dependencies | 17:23 |
| kfox1111 | thats an orperator I think. :) | 17:23 |
| inc0 | and just create resource when all the deps are met | 17:23 |
| inc0 | well, it will be one mechanism for everything | 17:24 |
| kfox1111 | operator | 17:24 |
| inc0 | not writing keystone operator | 17:24 |
| kfox1111 | you still have to orchestrate across service. | 17:24 |
| kfox1111 | too | 17:24 |
| inc0 | it's dependency operator | 17:24 |
| inc0 | dependency operator can operate sub-dependency operators | 17:24 |
| kfox1111 | I think for now, we shouldn't overoptomize by assuming all the operators look the same. | 17:24 |
| sdake | ya that is what is in the spec inc0 | 17:24 |
| kfox1111 | in the end, we might find out thats true and make the code into just one thing. | 17:25 |
| *** chas_ has quit IRC | 17:25 | |
| sdake | the thing that is delta from that model is that we don't have a generic implementation for it | 17:25 |
| inc0 | so unless we go crazy about it, it's really just evolution of entrypoint mechanism | 17:25 |
| inc0 | QuentinM, thoughts? | 17:26 |
| kfox1111 | we can make a base class they all share to share as much as we can. in the end, if we prove out the base class is all thats ever needed, | 17:26 |
| inc0 | pprokop | 17:26 |
| kfox1111 | we can collaps it down to one operator and an input file or something. | 17:26 |
| sdake | kfox1111 agree, i see that in the future as well | 17:26 |
| QuentinM | kfox1111: about your job being flaky, it can't happen since Kubernetes uses etcd to store the state. | 17:26 |
| QuentinM | kfox1111: the KV store doesn't flake. | 17:26 |
| inc0 | kfox1111, like this?;) https://github.com/inc0/navigator/blob/master/mariadb.yaml | 17:26 |
| kfox1111 | QuentinM: I've seen it personally flake. :) | 17:27 |
| inc0 | every distributed database is flaky | 17:27 |
| pprokop | inc0: ? | 17:27 |
| inc0 | it's like...mathematically proven;) | 17:27 |
| inc0 | pprokop, read couple lines above plz | 17:27 |
| inc0 | I'd be interested in your thoughts | 17:27 |
| kfox1111 | had 3 etcd's on one laptop. no one said it wouldn't work, but I saw multiple kube-controller-managers gain master at once. | 17:27 |
| *** adrian_otto has joined #openstack-kolla | 17:27 | |
| kfox1111 | and upgrading a deployment did some spectacurally wieird things. :) | 17:28 |
| kfox1111 | beign a dev is about imagining whats possible. | 17:28 |
| kfox1111 | being an operator is imaginging what will bite you. :) | 17:28 |
| inc0 | if its distributed and have a state, it's broken | 17:28 |
| inc0 | question is, how broken | 17:28 |
| inc0 | and when does it show | 17:28 |
| sdake | thirdpartyresource (i.e. etcd) contains the state | 17:28 |
| inc0 | yeah, and imho it shouldn't | 17:29 |
| kfox1111 | theres' desaster recovery to handle too. | 17:29 |
| pprokop | If you want to implement a operator for deps see k8s-appcontroller, maybe you do not trust mirantis but they already wrote it | 17:29 |
| inc0 | state == kubectl get pods + kubectl get jobs | 17:29 |
| kfox1111 | double failure in etcd. it happens. | 17:29 |
| *** TxGirlGeek has joined #openstack-kolla | 17:29 | |
| kfox1111 | we go to restore from backups. | 17:29 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Fix fact gathering when using --limit https://review.openstack.org/397851 | 17:29 |
| pbourke | awiddersheim: ^ | 17:30 |
| pbourke | whenever you have a chance :) | 17:30 |
| kfox1111 | how much in k8s is critical state, and how much is held in git, and can be easily just reuploaded? | 17:30 |
| kfox1111 | requireing completed jobs to hang out for orchestration purposes makes that harder. | 17:30 |
| inc0 | kfox1111, you mean like "step deployment"? | 17:31 |
| inc0 | press C to continue? | 17:31 |
| kfox1111 | not exactly sure what I mean. but just gota make sure those cases are handled. | 17:31 |
| inc0 | doable in multiple ways | 17:32 |
| kfox1111 | with low level packages without orchestration tied in, | 17:32 |
| inc0 | 1. don't just start everyting, start things in turns manually | 17:32 |
| kfox1111 | its easier to just relaunch the packages manually. | 17:32 |
| kfox1111 | with orchestration built in, then it assumes things that may not hold true during desaster recovery. | 17:32 |
| inc0 | 2. create virtuall dependency that you'll run manually, like lock-job | 17:32 |
| inc0 | well any orchiestration will need to trust it's single source of truth | 17:33 |
| rhallisey | finally caught up. Phew | 17:33 |
| QuentinM | kfox1111: we have that too | 17:33 |
| inc0 | for ansible these are facts | 17:33 |
| inc0 | for k8s - it's k8s api | 17:33 |
| QuentinM | kfox1111: sometimes I just destroy everything keystone related and run kpm deloy stackanetes/keystone again | 17:33 |
| inc0 | if that's broken, you're in pain anyway | 17:33 |
| QuentinM | or when I edit a configmap, I can just run deploy again to get it there | 17:34 |
| kfox1111 | inc0: its better not to have to deal with extra orchestration pain, while your dealing with the rest. :) | 17:34 |
| *** newmember has quit IRC | 17:34 | |
| rhallisey | kfox1111, agree with what you're saying. We want client side workflow to only run based on user input or k8s input | 17:34 |
| *** Jeffrey4l has quit IRC | 17:34 | |
| kfox1111 | rhallisey: yeah. I just want the workflow out of the way sometimes so it doesn't "help" and actually cause more pain then help. | 17:34 |
| kfox1111 | so packages magically launching create db jobs, register keystone entrypoints, etc | 17:35 |
| rhallisey | yup, optional, layered and flexible client side | 17:35 |
| kfox1111 | I'm starting to run multiregion clouds, | 17:35 |
| pprokop | I would like to make every component not required. Keep Kolla as modular as possible. | 17:35 |
| rhallisey | openstack is so complex, in can be configued to run a million ways | 17:35 |
| inc0 | maybe let's stick to ansible then:) | 17:35 |
| pprokop | And an operator can build deployment from small blocks | 17:35 |
| inc0 | client side is not cloud native at all | 17:36 |
| kfox1111 | and I really don't like giving admin access to create entrypoint jobs. I like to do those manually. less dangerous to the other regions. | 17:36 |
| kfox1111 | pprokop: +1. | 17:36 |
| inc0 | kfox1111, so thing is...any orchiestration can be manual | 17:36 |
| pprokop | One can choose kpm, one helm | 17:36 |
| sdake | pprokop i believe the spec states that ;) | 17:36 |
| kfox1111 | lets make packages do what they are good at. templating and shiping the templates. | 17:36 |
| pprokop | one can take config generator | 17:36 |
| kfox1111 | lets leve orchestration out to a different layer. | 17:36 |
| inc0 | pprokop, too much granularity means we'll stick to support 20 thnings doing same thing | 17:36 |
| rhallisey | agreed | 17:36 |
| inc0 | and that's bad | 17:36 |
| kfox1111 | I'm good having entrypoint manage dependencies passively. | 17:37 |
| *** Serlex has quit IRC | 17:37 | |
| pprokop | I'm good when Kolla will give user a choice what deps managemnet to choose | 17:37 |
| kfox1111 | an init-container should not do workflow though. no bootstrap/upgrade stuff outside of a job thats launched by a k8s operator or a human operator. | 17:37 |
| inc0 | I'm not really, we can't spread ourselves thin | 17:37 |
| inc0 | it will kill our productivity | 17:38 |
| *** Pavo has quit IRC | 17:38 | |
| rhallisey | inc0, what will spread us thin? | 17:38 |
| sdake | inc0 we have 10 people signed up in the spe t od o work | 17:38 |
| rhallisey | specifically | 17:38 |
| inc0 | doing multiple orch tools | 17:38 |
| *** TxGirlGeek has quit IRC | 17:38 | |
| *** narasimha_SV has joined #openstack-kolla | 17:38 | |
| *** fragatina has joined #openstack-kolla | 17:38 | |
| kfox1111 | having a layered/ortoganal stack should make it easy to paralell develop the work. | 17:38 |
| inc0 | at the end | 17:38 |
| rhallisey | this is all layered | 17:38 |
| *** TxGirlGeek has joined #openstack-kolla | 17:38 | |
| kfox1111 | yeah. | 17:38 |
| *** Pavo has joined #openstack-kolla | 17:38 | |
| *** fragatina has quit IRC | 17:38 | |
| inc0 | I'm talking about support later | 17:38 |
| inc0 | more code = bad for support | 17:39 |
| kfox1111 | inc0: yeah. we should implement one thing, but not exclude anyone else wanting to implement some other workflow | 17:39 |
| QuentinM | cloud native would be to have an entrypoint querying something about whether it can start now or not | 17:39 |
| QuentinM | try and retry | 17:39 |
| kfox1111 | QuentinM: I'm curious, | 17:39 |
| *** fragatina has joined #openstack-kolla | 17:39 | |
| QuentinM | it could query an external dep graph or whatever | 17:39 |
| inc0 | kfox1111, yeah, make workflow pluggable and optional but keep it one | 17:39 |
| *** msimonin has quit IRC | 17:39 | |
| kfox1111 | most of openstack services actually do that natively. | 17:39 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Fix fact gathering when using --limit https://review.openstack.org/397851 | 17:39 |
| kfox1111 | how much have you ran into that didnt? | 17:39 |
| pprokop | K8s component itself uses polling k8s-apiserver for state | 17:40 |
| sbezverk | kfox1111: remember neutron? noa does not wait for ovs to be up | 17:40 |
| *** fragatin_ has joined #openstack-kolla | 17:40 | |
| kfox1111 | sbezverk: yeah. thats one case I think entrypoint woudl be great for. | 17:40 |
| narasimha_SV | while deploying ceilometer with gnocchi backend ceilometer-api container is not getting created. getting failed at bootstrap | 17:40 |
| sbezverk | it just fails | 17:40 |
| kfox1111 | how many other places? | 17:40 |
| QuentinM | kfox1111: several unfortunately and we need to cope with boostrapping too | 17:40 |
| QuentinM | like you said | 17:41 |
| QuentinM | for upgrades too | 17:41 |
| narasimha_SV | anyone tried ceilometer with gnocchi ??? | 17:41 |
| kfox1111 | if bootstrapping used the same mechanism, then yeah. | 17:41 |
| inc0 | kfox1111, thing is...we don't want to find out which ones the hard way;) | 17:41 |
| kfox1111 | the other bootstrapping option is to just not deploy the object until its initial bootstrap depenedencies were created. | 17:41 |
| sdake | narasimha_SV jeffrey4l did the implementation work - if its broken fiel a bug i'm sure it will be processed along with our other 300 bugs at some point ;-) | 17:41 |
| QuentinM | I am the new version of Neutron, can I start? Nope, the migration process didn't happen. | 17:41 |
| *** fragatin_ has quit IRC | 17:42 | |
| kfox1111 | QuentinM: I think its bad to even try and start the new neturon until the right phase in the upgrade. | 17:42 |
| v1k0d3n | i man...step away for a little bit, and there's a lot more out here. | 17:42 |
| v1k0d3n | kolla is hard to keep up with lately :) | 17:42 |
| inc0 | v1k0d3n, welcome to our wold | 17:42 |
| v1k0d3n | reading... | 17:42 |
| inc0 | world | 17:42 |
| v1k0d3n | lol | 17:42 |
| kfox1111 | I guess the difference here is, do you start everything and have the upgrade send messages/commuinicate somehow to a running upgraded thing to unblock it, | 17:43 |
| v1k0d3n | well...this is an especially busy week or so. | 17:43 |
| kfox1111 | or start it once its at the right step in the upgrade. | 17:43 |
| QuentinM | kfox1111: well yeah ;) | 17:43 |
| kfox1111 | I'm the latter. for example, k8s has a great thing in deployments. | 17:43 |
| inc0 | kfox1111, if you can reflect step of upgrade in kubectl get jobs | 17:43 |
| QuentinM | kfox1111: depends whether you want your cluster to self-manage or have a client-sided client that does it | 17:43 |
| inc0 | it's dependency-resolvable | 17:43 |
| kfox1111 | I don't want to launch a new deployment for the n+1 ver and block it. | 17:43 |
| QuentinM | k8s is all about containers self managing themselves and resolving deps, doing discovery | 17:43 |
| kfox1111 | I want to run some upgrade jobs, | 17:44 |
| *** haplo37 has joined #openstack-kolla | 17:44 | |
| kfox1111 | then do a kubectl replace nova-api-deployment | 17:44 |
| *** g3ek has joined #openstack-kolla | 17:44 | |
| *** fragatina has quit IRC | 17:44 | |
| kfox1111 | and let k8s do the rolling upgrade. | 17:44 |
| QuentinM | ofc | 17:44 |
| *** fragatina has joined #openstack-kolla | 17:44 | |
| *** haplo37_ has joined #openstack-kolla | 17:44 | |
| kfox1111 | k8s has really good orchestratin there. better then the pods will do themselves. | 17:44 |
| kfox1111 | I don't think entyrpoint would be a good fit there. | 17:45 |
| v1k0d3n | oh adrian_otto is here too | 17:45 |
| v1k0d3n | nice to have you. brandon from the summit (talked towards the end of the second day) | 17:45 |
| kfox1111 | QuentinM: not quite client sided. operator. extend k8s to know how to deal with opensatck upgrades. | 17:46 |
| narasimha_SV | sdake: it is working when I am using mongodb as its backend | 17:46 |
| sdake | narasimha_SV sounds like a bug | 17:46 |
| sdake | we need lots o gates | 17:46 |
| sdake | not sure who is responsible for that | 17:46 |
| narasimha_SV | ok | 17:46 |
| sdake | narasimha_SV also I am taking a break from bug triage :) | 17:46 |
| adrian_otto | hi sdake and v1k0d3n | 17:46 |
| sdake | narasimha_SV pretty sure I've earned it :) | 17:47 |
| narasimha_SV | sdake: ok let me file a bug then | 17:47 |
| kfox1111 | so I think you can still do k8s native stuff with an operator for managing some of the workflow, and really still use the real power of k8s. k8s deployments for rolling upgrades / canarie upgrades, etc | 17:47 |
| v1k0d3n | tbf i'm catching up since 11:43. like i said...kolla is super busy these days. | 17:47 |
| kfox1111 | I think entrypoint is a very cleaver solution to a bunch of orchestration issues, but care should be used with it not to overuse it and have it block using some of the advanced k8s features. | 17:48 |
| sdake | v1k0d3n its been like this going on about a year+ :) | 17:48 |
| rhallisey | kfox1111, yes, I agree. That's exactly what we're after. We can't make this entirely about making this app kubernetes like. Openstack is an exception because of it's complexity | 17:48 |
| *** mgoddard_ has quit IRC | 17:48 | |
| rhallisey | we want to deploy and hand off to kubernetes to manage | 17:48 |
| *** mgoddard has joined #openstack-kolla | 17:49 | |
| rhallisey | and any complex operations a user needs we handle as a workflow | 17:49 |
| rhallisey | openstack has a *method* that makes sense to deploy it as just like k8s has a *method* to how apps are expected to behave | 17:50 |
| kfox1111 | yeah. | 17:50 |
| rhallisey | we need to take the line that is: kubernetes <--------> openstack and bend it to be more like a circle | 17:51 |
| rhallisey | which means we can't go to heavily on way or the other | 17:52 |
| openstackgerrit | Merged openstack/kolla: Deprecate scheduler_max_attempts option in nova https://review.openstack.org/397547 | 17:52 |
| kfox1111 | I think helm/kubernetes-entrypoint/operators/containers/k8s all together for appropriate tasks = win. :) | 17:52 |
| sdake | time to watch ptich black | 17:53 |
| *** unicell has quit IRC | 17:54 | |
| kfox1111 | so are we all on the same page? concerns? | 17:57 |
| *** eaguilar has joined #openstack-kolla | 17:57 | |
| sdake | kfox1111 i have a question then | 17:59 |
| kfox1111 | sure | 17:59 |
| sdake | kfox1111 what is the delta between that conversation and the current spec, and can someone record it there for ryan to address | 17:59 |
| kfox1111 | I think its mostly unchanged? | 17:59 |
| kfox1111 | the spec doesn't mention entrypoint though, which I was kind of thinkging implicitly we'd use, | 18:00 |
| kfox1111 | but maybe we want to put that in explicitly? | 18:00 |
| pprokop | Pleas put everything explicitly | 18:00 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Fix fact gathering when using --limit https://review.openstack.org/397851 | 18:00 |
| inc0 | well that was what whole argument last 2 days was about | 18:00 |
| narasimha_SV | sdake: when a container bootstrap is getting failed, what can be the reason | 18:00 |
| sdake | inc0 no idea | 18:00 |
| sdake | say guys i've got a staff meeting now bacak in an hour | 18:01 |
| kfox1111 | inc0: I think the last 2 days were about operators and if we wanted them at all. | 18:01 |
| sdake | can someone sort out what needs to change ;) | 18:01 |
| sdake | in the spec | 18:01 |
| kfox1111 | or can something like entrypoint make them not needed. | 18:01 |
| sdake | something clearly must | 18:01 |
| *** athomas has quit IRC | 18:01 | |
| sdake | oh nm | 18:01 |
| kfox1111 | I think they are needed, but entrypoint is too. | 18:01 |
| sdake | my staff meeting is at 12 | 18:01 |
| sdake | dst ftl | 18:01 |
| sdake | were we in agreement hat entrypoint wasn't going in every container? | 18:02 |
| *** TxGirlGeek has quit IRC | 18:02 | |
| sbezverk | QuentinM: quick question, do you know if containers in the same pod can have access to the shared memory? | 18:02 |
| sdake | kfox1111 sorry phone calls today couldn't keep up with chat | 18:03 |
| inc0 | kfox1111, we don't really need operators for deployment, that was the point I was trying to make | 18:03 |
| sdake | inc0 your right we dont | 18:03 |
| kfox1111 | inc0: I think they are very important. | 18:03 |
| sdake | inc0 if your using only manual workflows | 18:03 |
| inc0 | yes, just not right away | 18:03 |
| inc0 | or entrypoint.. | 18:04 |
| sdake | we are at the point of manual workflows work | 18:04 |
| sdake | next step is automatic workflows | 18:04 |
| kfox1111 | sdake: right. yeah. | 18:04 |
| inc0 | entrypoint will automate it | 18:04 |
| inc0 | and it works | 18:04 |
| inc0 | without need of an operator | 18:04 |
| sdake | comon guys lets sort out as a team how to get a solid outcome | 18:05 |
| sdake | I hear "we are all in the same page?" followed by mostly yes followed by back on no more operators | 18:05 |
| adrian_otto | Has there been any discussion of potential security drawbacks to using automated discovery in the context of the data plane? In other words, using entrypoint tools to facilitate orchestration. | 18:06 |
| inc0 | or rather "not an requirement for deployment" | 18:06 |
| adrian_otto | rather than handling all application orchestration in a separate context that the application itself cannot access? | 18:06 |
| kfox1111 | entrypoint wont handle 'k8s deployment' ojbect rolling upgrades well. | 18:06 |
| sdake | adrian_otto we haven't tackled security yet | 18:06 |
| QuentinM | why not | 18:06 |
| sdake | adrian_otto we don't bolt that on | 18:06 |
| sdake | adrian_otto they are built mostly in the containers | 18:06 |
| kfox1111 | operators can drive k8s deployments. | 18:06 |
| QuentinM | it will still check wether mariadb is healthy | 18:06 |
| sdake | adrian_otto but we are working on more base level problems at present | 18:07 |
| QuentinM | and whether keystone-api is healthy | 18:07 |
| QuentinM | before starting the service | 18:07 |
| kfox1111 | QuentinM: the package everying, start in parallel together thing. | 18:07 |
| adrian_otto | there is an argument to consider that perhaps the container should not have access to information beyond the scope of that container's individual role in an application. | 18:07 |
| kfox1111 | I don't want to start a new rc while the old rc is running. | 18:07 |
| kfox1111 | I want to replace the old deployment with the new deployment. | 18:07 |
| kfox1111 | and I not block access while the rolling upgrade happens. | 18:08 |
| kfox1111 | then k8s does the orchestration of the rolling upgrade natively. | 18:08 |
| QuentinM | sbezverk: I don't believe so | 18:08 |
| kfox1111 | I think entrypoint solves the dependencyh issue well. | 18:09 |
| kfox1111 | what I'm less confinced of is can dependencies be used for orcestrating a worfklow. | 18:09 |
| kfox1111 | it can, but not as well as using more native k8s means. | 18:09 |
| *** pbourke has quit IRC | 18:10 | |
| v1k0d3n | alright guys. let's approach this possibly another way; with a goal of nailing down some general concepts and direction either today or by tomorrow. | 18:11 |
| *** pbourke has joined #openstack-kolla | 18:11 | |
| v1k0d3n | we're talking about many concepts; many of which are not specific to openstack services. also many of which are a little new for us to get a full grasp on the goals, so we want to keep an open mind. | 18:12 |
| kfox1111 | I just added my thoughts to an entrypoint section in the spec. | 18:12 |
| kfox1111 | can everyone have a look? | 18:12 |
| v1k0d3n | can we just talk about only pros and cons of entrypoint? | 18:12 |
| adrian_otto | kfox1111: sorry, I missed the link to that spec, can you shoot it to me please? | 18:13 |
| v1k0d3n | because i think adrian_otto brought up a great point about limiting application discovery, and scoping correctly as well (something for consideration). | 18:13 |
| kfox1111 | adrian_otto: sure. :) | 18:13 |
| kfox1111 | https://review.openstack.org/#/c/392257 | 18:13 |
| inc0 | https://review.openstack.org/#/c/392257/13/specs/kolla-kubernetes-arch.rst adrian_otto | 18:13 |
| adrian_otto | thanks | 18:13 |
| kfox1111 | adrian_otto: yeah, I totally agree with you on the security aspect. | 18:13 |
| kfox1111 | we shuld minimize service accounts to the things that really need them. | 18:14 |
| v1k0d3n | ^^ me as well. this is something QuentinM we may want to discuss also. | 18:14 |
| inc0 | kfox1111, well, with init-container approach to dependencies we don't really have this issue | 18:14 |
| *** sdake_ has joined #openstack-kolla | 18:14 | |
| kfox1111 | so for example, in the nova-compute -> openvswitch case, | 18:14 |
| kfox1111 | it could ask k8s what the state of pods on the same node is, | 18:14 |
| kfox1111 | or it could just check to see if openvswitch's socket is there. | 18:14 |
| v1k0d3n | well, before we dig deeper...(sorry guys)... | 18:14 |
| kfox1111 | entrypoint can do either. | 18:14 |
| *** TxGirlGeek has joined #openstack-kolla | 18:15 | |
| kfox1111 | which is awesome. :) | 18:15 |
| v1k0d3n | can we all weigh in on certain aspects of the spec...hitting each one, and moving forward once there is direction? | 18:15 |
| adrian_otto | so my point in brining this up is not to sidetrack any progress, but to help us think about a range of pros and cons before zeroing in on a particular approach | 18:15 |
| kfox1111 | inc0: we do, its just not as much as if it was in every pod. | 18:15 |
| kfox1111 | adrian_otto: +1. | 18:15 |
| QuentinM | kfox1111: yeah we do both, check socket and pod on the same host | 18:16 |
| kfox1111 | QuentinM: but can security be enhanced by not needing a service token by not checking for the pod? | 18:17 |
| *** sdake has quit IRC | 18:17 | |
| kfox1111 | (though thats not quite an option yet. but that feature is progressing) | 18:17 |
| v1k0d3n | do you guys want to start a general etherpad for pros and cons of each? | 18:17 |
| v1k0d3n | adrian_otto sdake_ rhallisey inc0 kfox1111 thoughts? | 18:17 |
| kfox1111 | v1k0d3n: or do we want to start with, what problems we need to solve? | 18:18 |
| kfox1111 | we ahave a lot of tools, and a lot of problems. :) | 18:18 |
| v1k0d3n | yes, general concepts only. what we need to solve. | 18:18 |
| inc0 | we had some etherpad right? rhallisey | 18:18 |
| rhallisey | I'm reading back lol.. I was commenting in the spec | 18:19 |
| rhallisey | one sec | 18:19 |
| v1k0d3n | well, to be fair, remove any mention of operator/entrypoint. to us they are just names at this points. it;s the concepts that we need to nail down. | 18:19 |
| adrian_otto | yes, using an etherpad to source a pro/con list make sense from my perspective | 18:19 |
| QuentinM | kfox1111: the default service token is automatically added to each running container on k8s | 18:19 |
| adrian_otto | v1k0d3n: +1 | 18:19 |
| QuentinM | y | 18:20 |
| kfox1111 | QuentinM: there is a thing in development to remove that. | 18:20 |
| kfox1111 | trying to find the issue. sec | 18:20 |
| v1k0d3n | https://etherpad.openstack.org/p/161115-kolla-kubernetes-cn-discussion | 18:20 |
| sdake_ | v1k0d3n pretty sure that is the completely wrong approach ;) | 18:20 |
| adrian_otto | Best to describe the characteristics of the approach rather than naming it. By naming an approach, you are using a shorthand that means something to those who understand it, and allows for misunderstanding about what that solution actually does or how it works/behaves. | 18:20 |
| sdake_ | we define the charactericits with the name | 18:21 |
| sdake_ | everything has to have a name | 18:21 |
| *** tonanhngo has joined #openstack-kolla | 18:21 | |
| adrian_otto | not if it's a name of a tool that is not well understood by all stakeholders | 18:22 |
| sdake_ | its not the name of a tool | 18:22 |
| sdake_ | its the name of a concept.. | 18:22 |
| adrian_otto | yes, name concepts, not tools. | 18:22 |
| v1k0d3n | ok, so we need to agree before we need to agree here sdake_ | 18:22 |
| v1k0d3n | anyone for concepts over name? +1 | 18:22 |
| sdake_ | adrian_otto that is being done | 18:22 |
| adrian_otto | terrific | 18:23 |
| sdake_ | is done already | 18:23 |
| sdake_ | was odne a week ago ;) | 18:23 |
| rhallisey | inc0, https://etherpad.openstack.org/p/operator-base-class | 18:23 |
| rhallisey | inc0, that one? | 18:23 |
| adrian_otto | line 35 of the spec makes reference to a tool | 18:23 |
| *** newmember has joined #openstack-kolla | 18:23 | |
| inc0 | I guess so, but let's use etherpad Brandon created | 18:23 |
| adrian_otto | as part of "proposed change" | 18:24 |
| inc0 | to start fresh | 18:24 |
| sdake_ | whats in that etherpad is old and not fleshed out | 18:24 |
| rhallisey | k | 18:24 |
| sdake_ | what is in the spec is fleshed out | 18:24 |
| sdake_ | i'd like to see people focus on the spec ;) | 18:24 |
| sdake_ | its gone through 12 iterations with over 120 comments | 18:24 |
| *** tonanhngo has quit IRC | 18:26 | |
| v1k0d3n | we have a hangout going on to discuss. | 18:26 |
| v1k0d3n | https://hangouts.google.com/hangouts/_/fbdm6ejjp5dlladngn2ahwo7lye | 18:26 |
| sdake_ | unfortunately i have a meeting | 18:26 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Fix fact gathering when using --limit https://review.openstack.org/397851 | 18:26 |
| sdake_ | this is why we dont use hangouts typically in openstack | 18:26 |
| sdake_ | because if its not recorded, it takes another 3 days to get eveyrone back up to speed | 18:26 |
| *** tonanhngo has joined #openstack-kolla | 18:26 | |
| pbourke | sdake_: inc0: the .gitreview file needs to be updated on kolla-ansible | 18:27 |
| rhallisey | I'm trying to capture everything in the spec to save the 10000 lines of reading | 18:27 |
| *** unicell has joined #openstack-kolla | 18:27 | |
| sdake_ | the education happens via reading.. | 18:27 |
| pbourke | sdake_: inc0: I can do but have to run right now, so if someone gets a chance before then | 18:28 |
| *** newmember has quit IRC | 18:28 | |
| *** fragatina has quit IRC | 18:28 | |
| *** fragatina has joined #openstack-kolla | 18:29 | |
| v1k0d3n | sdake_: we're just talking over hangouts, but recording discussion over etherpad | 18:30 |
| rhallisey | v1k0d3n, kk sec | 18:30 |
| rhallisey | joining | 18:30 |
| v1k0d3n | sure | 18:30 |
| *** TxGirlGeek has quit IRC | 18:33 | |
| *** TxGirlGeek has joined #openstack-kolla | 18:38 | |
| *** harlowja has quit IRC | 18:38 | |
| *** shardy has quit IRC | 18:39 | |
| *** fragatina has quit IRC | 18:44 | |
| *** fragatina has joined #openstack-kolla | 18:45 | |
| *** jheroux has quit IRC | 18:49 | |
| kfox1111 | droppped out. :/ | 18:50 |
| kfox1111 | trying to rejoin... | 18:50 |
| *** egonzalez90 has joined #openstack-kolla | 18:53 | |
| egonzalez90 | inc0 sdake_ ping | 18:54 |
| inc0 | what's up egonzalez90 | 18:54 |
| egonzalez90 | some kind of sync will be made at kolla-ansible repo? | 18:54 |
| egonzalez90 | last change is from friday | 18:54 |
| *** magicboiz has quit IRC | 18:57 | |
| inc0 | yeah, we'll need to republish changes | 18:58 |
| *** narasimha_SV has quit IRC | 18:59 | |
| wirehead_ | Oh, you think OpenStack is your ally. But you merely adopted OpenStack. I was born in it, mouded by it. I didn’t see the light until I was already a man, by then it was nothing to me but BLINDING! | 19:00 |
| portdirect | wirehead: deep | 19:00 |
| *** neilus has left #openstack-kolla | 19:02 | |
| openstackgerrit | Andreas Jaeger proposed openstack/kolla-ansible: Set up .gitreview https://review.openstack.org/397888 | 19:02 |
| rhallisey | wirehead_, lol.. | 19:03 |
| bjolo_ | wirehead_, sup :) | 19:06 |
| *** harlowja has joined #openstack-kolla | 19:06 | |
| sbezverk | wirehead_: hm you do not look like 6 years old ;-) | 19:08 |
| *** TxGirlGeek has quit IRC | 19:13 | |
| *** jheroux has joined #openstack-kolla | 19:14 | |
| *** TxGirlGeek has joined #openstack-kolla | 19:18 | |
| kfox1111 | dependency-in-container dic? :) | 19:21 |
| bjolo_ | inc0, https://review.openstack.org/#/c/397186/ | 19:21 |
| kfox1111 | diic dependency-in-init-container? | 19:21 |
| bjolo_ | this needs backport to newton as well. is that in progress? how can i check? | 19:21 |
| inc0 | dependency-init-container-kolla | 19:21 |
| inc0 | kfox1111, ^ | 19:22 |
| * kfox1111 chuckes | 19:22 | |
| kfox1111 | was trying to not quite go that far. ;) | 19:22 |
| portdirect | entrycheck? | 19:23 |
| kfox1111 | entrydep? | 19:23 |
| kfox1111 | entrycheck. | 19:24 |
| kfox1111 | I like it. | 19:24 |
| inc0 | depcheckentry | 19:24 |
| *** sdake_ has quit IRC | 19:25 | |
| v1k0d3n | YAY!! | 19:25 |
| v1k0d3n | this proved to be extremely helpful. | 19:25 |
| kfox1111 | +1 | 19:25 |
| kfox1111 | so much easier over the phone. :) | 19:25 |
| *** bjolo_ is now known as bjolo | 19:25 | |
| *** TxGirlGeek has quit IRC | 19:26 | |
| portdirect | not quite: https://www.youtube.com/watch?v=ygE01sOhzz0, but a step in the rieght direction :) | 19:27 |
| *** sdake has joined #openstack-kolla | 19:27 | |
| kfox1111 | portdirect: :) | 19:27 |
| *** krtaylor has quit IRC | 19:27 | |
| kfox1111 | actually, I think we're trying to solve one problem harder then the "hurding cats" problem. | 19:28 |
| kfox1111 | self assembling hurd of cats. :) | 19:28 |
| inc0 | -.- | 19:28 |
| kfox1111 | how many devs can we get to self assemble into a great team. very hard problem. :) | 19:29 |
| kfox1111 | comminication like we just did is key. :) | 19:29 |
| rhallisey | I'm trying to think of some names | 19:30 |
| inc0 | yay for impromptu hangouts | 19:30 |
| portdirect | lol :) frogs in a wheelbarrow, though i think the hardest thing has been the conficting meaning attached to words, as half the time i think we are agreement but using different nonclamenture for similar concepts... | 19:30 |
| rhallisey | how about the Meow pod | 19:30 |
| inc0 | lmao | 19:30 |
| rhallisey | instead of operators, how about admirals | 19:31 |
| inc0 | self assembling herd of cats | 19:31 |
| inc0 | high level operator will be admiral | 19:31 |
| inc0 | service level will be capitan | 19:31 |
| rhallisey | nah that's to complex | 19:31 |
| rhallisey | we need name for 1 concept | 19:31 |
| rhallisey | one name for one concept | 19:31 |
| inc0 | and init container would be what...ensign? | 19:32 |
| rhallisey | thoughts on admiral? | 19:32 |
| QuentinM | captain-deps | 19:32 |
| rhallisey | s/operator/admiral/ | 19:32 |
| inc0 | sarge nova, reportin' for duty sir! | 19:32 |
| rhallisey | deck-hand | 19:32 |
| rhallisey | lol | 19:32 |
| inc0 | tell me about fleet management;) | 19:33 |
| rhallisey | how about ishmael | 19:33 |
| rhallisey | call me Ishmael | 19:34 |
| *** TxGirlGeek has joined #openstack-kolla | 19:34 | |
| kfox1111 | right. one of the hardest things about communication is the assumptions of definitions each party uses not matching. :/ | 19:34 |
| rhallisey | kfox1111, what do you think about s/operator/admiral/ | 19:34 |
| kfox1111 | I'm ok with admiral. :) | 19:35 |
| rhallisey | ok cool | 19:35 |
| kfox1111 | has no previous conflicting definition I'm aware of. :) | 19:35 |
| rhallisey | now we need s/entrypoint/?/ | 19:35 |
| kfox1111 | heh. this is kind of like quantim mechanics. | 19:35 |
| QuentinM | quentin or quantum | 19:36 |
| kfox1111 | all the terms are already used, so lets just use random unused stuff "top, bottom, strange, etc" | 19:36 |
| QuentinM | both are equally awesome but I mean ... | 19:36 |
| kfox1111 | :) | 19:36 |
| QuentinM | x) | 19:36 |
| portdirect | what state you in? | 19:36 |
| kfox1111 | portdirect: semi-sane. :) | 19:36 |
| inc0 | let's please avoid using word quantum...brings back memories I'd like to remain buried | 19:36 |
| kfox1111 | inc0: fair enough. :) | 19:37 |
| portdirect | entrypoint -> schrodinger | 19:37 |
| QuentinM | oh boy | 19:37 |
| QuentinM | that escalated quickly | 19:37 |
| *** Pavo has quit IRC | 19:38 | |
| * kfox1111 chuckles | 19:38 | |
| inc0 | aaaand we're back at cats | 19:38 |
| sbezverk | portdirect: something easier to pronounce please | 19:38 |
| kfox1111 | or just make up words all together... a froob, a traws, a ... | 19:38 |
| kfox1111 | :) | 19:38 |
| kfox1111 | inc0: exactly. :) | 19:38 |
| inc0 | or we can call it Erwin | 19:39 |
| kfox1111 | so, this is the point in a meeting where you get so far off in a tangent, its just time to call the meeting. :) | 19:39 |
| portdirect | hate to say it - it think 'entrycheck' may be the most appropriate, though boring name :( | 19:39 |
| inc0 | yes, time when brains are fried and you just drool | 19:39 |
| inc0 | dependency-init-container-kolla | 19:40 |
| inc0 | that's not boring when you think about it | 19:40 |
| *** matrohon has joined #openstack-kolla | 19:40 | |
| kfox1111 | I like entrycheck. | 19:41 |
| portdirect | or checkpoint? | 19:41 |
| kfox1111 | its pretty clear what it does, and it doesn't conflict with other things. | 19:41 |
| kfox1111 | checkpoint is an hpc term for snapshotting a running code to let you recover from failure. | 19:41 |
| inc0 | checkpoint has a nice ring to it | 19:41 |
| inc0 | checkpoint container | 19:42 |
| sbezverk | portdirect: trademark violation | 19:42 |
| inc0 | actually, I lioke that | 19:42 |
| kfox1111 | no, please no checkpoint. | 19:42 |
| sbezverk | for people who is not into networking | 19:42 |
| inc0 | chokepoint | 19:42 |
| sbezverk | checkpopint is taken by firewall product | 19:42 |
| portdirect | k thats pretty clear, not that. | 19:42 |
| *** Pavo has joined #openstack-kolla | 19:43 | |
| kfox1111 | entrycheck or entrydep? or something in that vain? | 19:43 |
| inc0 | initdep | 19:43 |
| kfox1111 | initdep works. | 19:43 |
| inc0 | I'll just call it dependency init container for a while | 19:43 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture https://review.openstack.org/392257 | 19:47 |
| v1k0d3n | how about galini-kolla | 19:47 |
| portdirect | sanityinit - as that essentially what its doing, but I'm out - it's late here, my mind is fried, and I'm worried that it's affecting all of you, via some sort of weird IRC quantum entanglement. | 19:47 |
| v1k0d3n | it's greek for....(wait for it).... | 19:47 |
| v1k0d3n | tranquility | 19:47 |
| v1k0d3n | :D :D :D nuk nuk | 19:47 |
| inc0 | tranquil glue | 19:48 |
| inc0 | has a nice flair to it | 19:48 |
| *** portdirect is now known as portdirect_away | 19:48 | |
| inc0 | or rather...tranquility glue | 19:48 |
| inc0 | I think that would be good product name, just not in IT | 19:48 |
| kfox1111 | btw. if you ever need a perfect image for the k8s way: https://s-media-cache-ak0.pinimg.com/736x/f5/08/13/f5081398649c7619b35fcd5458870f18.jpg sidecars & cattle. :) | 19:54 |
| kfox1111 | makes me smile every time. :) | 19:54 |
| inc0 | and guy on bike is...operator? | 19:55 |
| *** gfidente has quit IRC | 19:58 | |
| kfox1111 | hehe. | 19:59 |
| kfox1111 | I think thats a pod. | 19:59 |
| kfox1111 | deployment is a biker gang of those. :) | 20:00 |
| kfox1111 | now theres an image.... | 20:00 |
| kfox1111 | teacher, can I go home? my brain's full... | 20:00 |
| sdake | egonzalez90 the answer is each dev will have to sync again unforutnately | 20:03 |
| *** TxGirlGeek has quit IRC | 20:04 | |
| *** TxGirlGeek has joined #openstack-kolla | 20:04 | |
| sdake | portdirect_away lol | 20:05 |
| sdake | QuentinM lol | 20:05 |
| *** krtaylor has joined #openstack-kolla | 20:05 | |
| *** krtaylor has quit IRC | 20:10 | |
| inc0 | I'm off for 30 or so min, drooling, and after that doing kolla-ansible repo split cleanups | 20:10 |
| inc0 | please don't break this nice agreemend we just had | 20:10 |
| *** inc0 has quit IRC | 20:10 | |
| *** matrohon has quit IRC | 20:11 | |
| *** matrohon has joined #openstack-kolla | 20:12 | |
| *** krtaylor has joined #openstack-kolla | 20:14 | |
| egonzalez90 | sdake i mean the repo dont have the changes made since friday. | 20:14 |
| *** krtaylor has joined #openstack-kolla | 20:15 | |
| egonzalez90 | Should someone push all the changes present in kolla to kolla-ansible? | 20:15 |
| *** krtaylor has quit IRC | 20:17 | |
| rhallisey | sdake, so we are re naming operators to admirals | 20:17 |
| *** msimonin has joined #openstack-kolla | 20:17 | |
| sdake | how about overlords ;-) | 20:17 |
| jascott1 | vikings were nautical | 20:17 |
| sbezverk | "cheeftan" would be a better choice if you want to go into vikings ;-) | 20:18 |
| *** gagehugo has left #openstack-kolla | 20:20 | |
| sbezverk | sorry for wrong spelling | 20:21 |
| kfox1111 | overloards... :) | 20:23 |
| kfox1111 | then we can use the concept of minions too. :) | 20:23 |
| *** krtaylor has joined #openstack-kolla | 20:24 | |
| *** newmember has joined #openstack-kolla | 20:25 | |
| *** krtaylor has quit IRC | 20:26 | |
| jascott1 | should we (at some point) develop Chaos Monkey also? | 20:28 |
| *** papacz has quit IRC | 20:29 | |
| jascott1 | could be "Viking Pillage Party" | 20:29 |
| *** chas_ has joined #openstack-kolla | 20:31 | |
| v1k0d3n | quote of the day gentlemen: "please don't break this nice agreemend we just had" - inc0 | 20:31 |
| v1k0d3n | btw, i am very glad that we worked through that. so note for next time...it may be challenging to do, but etherpad and a hangout really helps a lot. | 20:32 |
| *** krtaylor has joined #openstack-kolla | 20:35 | |
| jascott1 | why would config files be TPRs? are we converting them to fields in TPR or just dumping text? | 20:37 |
| jascott1 | the value would be to have fields which means we only care about a select number of fields or we are converting entire config file (lots of work/maintenance) | 20:38 |
| jascott1 | https://review.openstack.org/#/c/392257/14/specs/kolla-kubernetes-arch.rst,unified Line 80 | 20:41 |
| jascott1 | ^ this doesnt make sense to me | 20:44 |
| sdake | v1k0d3n could someone clarify the nice agreement for me :) | 20:44 |
| sdake | v1k0d3n i've heard conflicting reports of the meeting re operators | 20:44 |
| sdake | rhallisey you will be proud of me | 20:45 |
| sdake | check this out | 20:45 |
| v1k0d3n | sdake: https://etherpad.openstack.org/p/161115-kolla-kubernetes-cn-discussion | 20:45 |
| sdake | I worked dovetail into a sentence in one of my meetings | 20:45 |
| rhallisey | good job! | 20:45 |
| sdake | now your turn! | 20:45 |
| rhallisey | way to lead from the front! | 20:46 |
| sdake | dude hwen i said dovetail | 20:46 |
| sdake | i felt dirty | 20:46 |
| sdake | like zebra | 20:46 |
| rhallisey | haha | 20:46 |
| sdake | why on earth did you seed my mind with that term | 20:46 |
| *** berendt has quit IRC | 20:50 | |
| sdake | https://bugzilla.redhat.com/show_bug.cgi?id=473368 | 20:51 |
| openstack | bugzilla.redhat.com bug 473368 in xorg-x11-drv-ati "Dirty Like Zebra when switching workspaces" [Medium,Closed: currentrelease] - Assigned to airlied | 20:51 |
| *** eaguilar has quit IRC | 20:56 | |
| *** eaguilar has joined #openstack-kolla | 21:00 | |
| *** sdake_ has joined #openstack-kolla | 21:00 | |
| *** krtaylor has quit IRC | 21:02 | |
| *** TxGirlGeek has quit IRC | 21:04 | |
| *** sdake has quit IRC | 21:04 | |
| *** eaguilar has quit IRC | 21:04 | |
| *** matrohon has quit IRC | 21:05 | |
| *** matrohon has joined #openstack-kolla | 21:06 | |
| *** ccesario has joined #openstack-kolla | 21:09 | |
| *** eaguilar has joined #openstack-kolla | 21:11 | |
| *** adrian_otto has quit IRC | 21:13 | |
| *** inc0 has joined #openstack-kolla | 21:13 | |
| inc0 | back | 21:14 |
| *** adrian_otto has joined #openstack-kolla | 21:14 | |
| inc0 | soo...how many times have we changed arch approach since I left? | 21:14 |
| *** krtaylor has joined #openstack-kolla | 21:14 | |
| rhallisey | sdake_, what was your opinon on changing the operator name | 21:19 |
| rhallisey | going to make it admiral | 21:19 |
| inc0 | rhallisey, personal request, can we shelve name discussion for now and do poc of init container based deployment? | 21:19 |
| *** TxGirlGeek has joined #openstack-kolla | 21:20 | |
| rhallisey | thought we said we are implementing all the layers | 21:20 |
| inc0 | so we can achieve full deployment using just init containers | 21:20 |
| rhallisey | operators are still in the spec | 21:20 |
| inc0 | operators will deal with upgrades and other harder tasks | 21:21 |
| inc0 | but we can achieve full deployment without ever touching operators | 21:21 |
| rhallisey | that's not how it's reflected in the spec | 21:21 |
| inc0 | but that's what we talked about on hangouts | 21:21 |
| inc0 | kfox1111, v1k0d3n ^ | 21:21 |
| rhallisey | we said there are multiple layers here :) | 21:22 |
| rhallisey | sec, another spec coming soon | 21:22 |
| inc0 | still, full deployment can be done by init container | 21:22 |
| rhallisey | I defined at layer in the spec | 21:22 |
| rhallisey | that should clear what I mean | 21:23 |
| v1k0d3n | inc0: yes...we are on the same page. | 21:23 |
| v1k0d3n | it was really clear at the end of that call. | 21:23 |
| rhallisey | every layer can do deployment | 21:23 |
| *** lrensing has quit IRC | 21:23 | |
| v1k0d3n | it would be nice to get through 24hrs of clarity. | 21:23 |
| v1k0d3n | i think the group needs that. :) | 21:23 |
| *** egonzalez90 has quit IRC | 21:24 | |
| inc0 | yes, please | 21:24 |
| v1k0d3n | esp the PTL. inc0 will start with long hair...end up with none. | 21:25 |
| inc0 | or all white | 21:25 |
| v1k0d3n | welcome to my world buddy :) | 21:25 |
| v1k0d3n | if you're lucky. | 21:25 |
| inc0 | so rhallisey I'll say what's in my brain at this very moment and you'll tell me if that's corresponds with your takeaways from our todays meeting | 21:27 |
| rhallisey | k | 21:27 |
| rhallisey | mine will be reflected in the spec | 21:28 |
| inc0 | 1. We agree that init-container can deal with whole deploy | 21:28 |
| inc0 | 2. We agree that init container can be optional | 21:28 |
| inc0 | 3. Given 1 and 2 we will go ahead and implement deployment with init container (optional) | 21:28 |
| inc0 | after that is done, we will examine alternatives if we'll be unhappy with result | 21:28 |
| *** sdake_ has quit IRC | 21:28 | |
| *** krtaylor has quit IRC | 21:29 | |
| inc0 | but we'll have working deploymetn soon | 21:29 |
| rhallisey | agree with all 3 not 4 | 21:29 |
| rhallisey | well not that I don't agree with 4 | 21:29 |
| rhallisey | I think the spec will be more clear | 21:29 |
| rhallisey | I just don't like the 4th sentence wording | 21:30 |
| inc0 | ok...but you agree with sense right? | 21:30 |
| inc0 | what I mean is, we can just go ahead and start implementing init container now | 21:31 |
| rhallisey | yes | 21:31 |
| rhallisey | that's in the spec | 21:31 |
| inc0 | and keep operator discussion going | 21:31 |
| rhallisey | agree | 21:31 |
| inc0 | ok, fine | 21:31 |
| rhallisey | not with the operation disccuoin | 21:31 |
| rhallisey | that's also in the spec | 21:31 |
| rhallisey | and will be implemented | 21:31 |
| inc0 | rhallisey, fine, we're unblocked then | 21:31 |
| rhallisey | yes | 21:31 |
| rhallisey | we are | 21:31 |
| rhallisey | everything's a layer | 21:31 |
| kfox1111 | back. | 21:31 |
| inc0 | but with that in mind, let's keep discussion about operators ongoing as we just lowered the scope of them | 21:32 |
| kfox1111 | arian has some great feedback. | 21:32 |
| rhallisey | k let me finish spec. One sec | 21:32 |
| inc0 | we need to be crystal clear what problems are we trying to solve with them | 21:32 |
| inc0 | agree, but this is operator discussion...or admiral as Ryan called it, kinda like the ring of it | 21:34 |
| inc0 | so rhallisey an idea | 21:34 |
| inc0 | let's separate inti container from spec | 21:34 |
| inc0 | and just agree on it | 21:34 |
| *** jtriley has quit IRC | 21:34 | |
| rhallisey | crap this isn't going to be done today | 21:34 |
| inc0 | follow up with writing the damn thing | 21:34 |
| inc0 | that will unblock us from spec | 21:35 |
| inc0 | because role of operators in grand scheme of things seems to be fuzzy still | 21:35 |
| inc0 | is this course of action acceptable? I can propose bp about init containers right away | 21:36 |
| inc0 | and we just approve it and start implementation | 21:36 |
| rhallisey | there's nothing to stop us from wiritng code | 21:37 |
| inc0 | lack of clarity is really stopping | 21:37 |
| rhallisey | the reason I'm trying to use the spec so much is so everyone understand the direction and why | 21:37 |
| inc0 | people don't want to put work into somethign we don't know we throw away | 21:37 |
| rhallisey | our direction now is in much better shape | 21:37 |
| rhallisey | init containers aren't being thrown away | 21:38 |
| *** Pavo has quit IRC | 21:38 | |
| inc0 | yes, hence bp-> approval-> implementation | 21:38 |
| inc0 | quick stream | 21:38 |
| kfox1111 | I think we're in a good place to do the work items in: | 21:38 |
| rhallisey | ya I said that's fine | 21:38 |
| kfox1111 | https://etherpad.openstack.org/p/161115-kolla-kubernetes-cn-discussion | 21:38 |
| kfox1111 | I've got a prototype for helm stuff here: | 21:39 |
| kfox1111 | https://review.openstack.org/#/c/396296/ | 21:39 |
| kfox1111 | I've got some work in progress to make it a bit more generic. | 21:39 |
| kfox1111 | and then we gota try and get entrypoint woven into it. | 21:39 |
| kfox1111 | that that requires some of the other work items. | 21:40 |
| kfox1111 | so anyone can start working on those right away. | 21:40 |
| inc0 | cool | 21:40 |
| inc0 | v1k0d3n, ^ we're good to go. | 21:40 |
| rhallisey | helm, operators, init container all good to go | 21:41 |
| rhallisey | there is agreement around those topics | 21:41 |
| rhallisey | the spec we will have to delay ~1 more day | 21:41 |
| inc0 | https://blueprints.launchpad.net/kolla-kubernetes/+spec/dependency-init-container | 21:42 |
| kfox1111 | yeah. I think we have them well defined enough people can start working on them. there might be some roughness around the edges, but work can start I think. | 21:42 |
| kfox1111 | rhallisey: one more day shouldn't hurt? (famious last words... :) | 21:42 |
| rhallisey | kfox1111, indeed | 21:42 |
| kfox1111 | but we can start anyway. | 21:42 |
| inc0 | https://blueprints.launchpad.net/kolla-kubernetes/+spec/helm-orchestration | 21:42 |
| kfox1111 | (or more accurately, we can continue with the path we've been on.) | 21:42 |
| rhallisey | specs are a lot of work guys :) | 21:42 |
| rhallisey | started with 80 line | 21:43 |
| inc0 | that's why I'm not big fan of it rhallisey ;) | 21:43 |
| rhallisey | may hit 400 by the end | 21:43 |
| *** Pavo has joined #openstack-kolla | 21:43 | |
| rhallisey | inc0, nah this is perfect | 21:43 |
| kfox1111 | rhallisey: yeah. totally get that. tahnks for keeping it alive. | 21:43 |
| rhallisey | fantastic for educating the community | 21:43 |
| inc0 | anyway, I'm glad, tomorrow on meeting we'll organize work around it | 21:43 |
| rhallisey | inc0, yes | 21:43 |
| kfox1111 | yeah. its a complicated enough thing, it has to dbe documented. | 21:43 |
| rhallisey | I'll send a ML note | 21:43 |
| kfox1111 | this will go a long way. | 21:43 |
| inc0 | thanks | 21:43 |
| rhallisey | ya putting an idea into words is hard | 21:44 |
| *** sdake has joined #openstack-kolla | 21:44 | |
| kfox1111 | rhallisey: I'd call that blueprint helm packaging | 21:44 |
| kfox1111 | less controversial then. | 21:44 |
| rhallisey | what is it called | 21:44 |
| rhallisey | oh | 21:44 |
| rhallisey | yes | 21:44 |
| kfox1111 | :) | 21:44 |
| v1k0d3n | yeah this is really good stuff | 21:44 |
| v1k0d3n | what time is meeting? i am starting to get more dialed in. | 21:45 |
| *** jtriley has joined #openstack-kolla | 21:45 | |
| inc0 | 16 utc | 21:45 |
| sdake | inc0 pretty sure 1-3 isn't what i agreed to ;) | 21:45 |
| kfox1111 | which meeting? | 21:45 |
| v1k0d3n | very happy to see all this working out. today's call was just what we needed. | 21:45 |
| inc0 | kfox1111, weekly | 21:45 |
| inc0 | sdake, sorry | 21:45 |
| kfox1111 | k | 21:45 |
| rhallisey | sdake, I'll expalin in the spec | 21:45 |
| sdake | you can't have a meeeting 4 people and call it a day | 21:46 |
| sdake | thats why this needs to happen on irc | 21:46 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture https://review.openstack.org/392257 | 21:46 |
| inc0 | we had meeting with most of interested parties | 21:46 |
| *** eaguilar has quit IRC | 21:46 | |
| rhallisey | adrian_otto I'll address your comments in the next review | 21:46 |
| inc0 | and come to the conclusion at last | 21:46 |
| inc0 | let's not ruin it please | 21:47 |
| rhallisey | had already a bunch of pending changes | 21:47 |
| sdake | i'm pretty sure i'm an interested party and was unrepresented | 21:47 |
| inc0 | most of | 21:47 |
| sdake | i told you had a conflict | 21:47 |
| rhallisey | inc0, check out the definitions of layers | 21:47 |
| sdake | had the meeting anyway | 21:47 |
| inc0 | https://etherpad.openstack.org/p/161115-kolla-kubernetes-cn-discussion <-sdake | 21:47 |
| *** eaguilar has joined #openstack-kolla | 21:47 | |
| kfox1111 | inc0: a lot of what was discussed isn't in the etherpad unfortunatly. not sure I could reconstruct it though. | 21:49 |
| inc0 | rhallisey, what I'm saying is with current model operators are orthogonal to deployment itself, which is exactly fine | 21:50 |
| inc0 | we can implement them wherever we need | 21:50 |
| kfox1111 | basically I think it came down though to two groups. those taht want to orchestrate with packagemanager/entrypoint, and those that wanted it outside. | 21:50 |
| kfox1111 | we decided to allow both to co'exist, as I understood the outcome. | 21:50 |
| inc0 | yeah, but during ocata at least we focus on one | 21:50 |
| inc0 | make it optional, but if somebody wants to niclude second, possible | 21:51 |
| kfox1111 | my focus is on the underlayers as thats really waht I need and am being paid for. | 21:51 |
| kfox1111 | Ican't say where other's focus is on. | 21:51 |
| inc0 | but we need "an orchiestration" | 21:51 |
| inc0 | kfox1111, underlayers as deployment of services itself? | 21:51 |
| kfox1111 | yes. kolla really needs orchestration for the 90% use case. | 21:51 |
| inc0 | that's fine, we can handle 90% thing with init-container | 21:52 |
| kfox1111 | inc0: in the layer cake, layer 4 through 1, excluding the layer 2 thing that just poped up. | 21:52 |
| *** eaguilar has quit IRC | 21:52 | |
| inc0 | fine, so piece of code to create configmaps and shoot helm package into the k8s | 21:53 |
| inc0 | right? | 21:53 |
| inc0 | while we'll deal with making this thing stand in correct order | 21:53 |
| kfox1111 | yeah. thats a simple operator. | 21:53 |
| kfox1111 | there's one thing I 'm still working on prototyping... | 21:54 |
| jascott1 | are we starting with one db per service? | 21:54 |
| kfox1111 | you can nest packages. | 21:54 |
| inc0 | jascott1, potentially in future | 21:54 |
| inc0 | not now | 21:54 |
| kfox1111 | so if we make fine grained packages, | 21:54 |
| jascott1 | cool | 21:54 |
| kfox1111 | then we can make an overarching package that includes them and does the orchestration jobs. | 21:54 |
| kfox1111 | so that coveres the stackanetes use case. | 21:55 |
| *** harlowja has quit IRC | 21:55 | |
| kfox1111 | and those that don't want helm to orchestrate can use the lower level packages and a an operator or manually deploy them. | 21:55 |
| kfox1111 | I think the idea wiill fly, but need to test. | 21:55 |
| inc0 | yeah, that'd be perfect | 21:55 |
| inc0 | openstack chart that's composed from nova charts, neutron charts | 21:55 |
| inc0 | and so on | 21:55 |
| rhallisey | inc0, did you read the layering section? | 21:57 |
| kfox1111 | right. | 21:57 |
| inc0 | yes | 21:57 |
| jascott1 | at a helm level, if user doesnt want some service will we have a way to exclude from installing resources into k8s? | 21:57 |
| rhallisey | kk | 21:57 |
| kfox1111 | jascott1: yeah. | 21:57 |
| inc0 | jascott1, yes, helm is supposed to handle that | 21:57 |
| kfox1111 | jascott1: my use case requires multiple mariadbs/rabbitmq's. | 21:57 |
| kfox1111 | its actually one of the next things I need to implement. | 21:57 |
| kfox1111 | It comes much more cheeply from having a package manager, and thats one reason I was focused so much on helm the last couple weeks. | 21:58 |
| jascott1 | I think we will have to do something, helm installs everything in a charts directory recursively | 21:59 |
| kfox1111 | actually no. | 22:00 |
| inc0 | jascott1, one way to deal with this is to have chart for mariadb | 22:00 |
| kfox1111 | I foudn a workaround. :) | 22:00 |
| inc0 | and chart for keystone | 22:00 |
| kfox1111 | the trick is, | 22:00 |
| kfox1111 | helm does that for all packages not starting with _. | 22:00 |
| inc0 | and init-container in keystone with dependency on mariadb | 22:00 |
| inc0 | oO | 22:00 |
| kfox1111 | if you put a file in starting with _, and wrap it in a macro, | 22:00 |
| adrian_otto | rhallisey: thanks for looking through my comments. | 22:01 |
| kfox1111 | you can put it in a conditional in a template not starting with _. | 22:01 |
| inc0 | ahh....naming-convention-based-logic | 22:01 |
| inc0 | how fun | 22:01 |
| kfox1111 | so a value can make turning on/off a template work. :) | 22:01 |
| jascott1 | i have seen that used for partials but hadnt thought of using it for package control | 22:02 |
| kfox1111 | jascott1: https://review.openstack.org/#/c/396296/ for the current prototype. | 22:02 |
| kfox1111 | :) | 22:02 |
| kfox1111 | yeah. :) | 22:02 |
| kfox1111 | it hit me all of a sudden. | 22:02 |
| kfox1111 | adrian_otto: thanks for submiting them. :) | 22:02 |
| jascott1 | inc0: so they are separate packages without deps used in helm? | 22:02 |
| inc0 | that's one way of doing it | 22:03 |
| rhallisey | brb | 22:03 |
| jascott1 | i.e. we throw out the helm dep stuff in that case | 22:03 |
| inc0 | so one thing tho, bootstrap jobs | 22:03 |
| rhallisey | I'll get in maybe 1 or 2 more review tonight | 22:03 |
| inc0 | jobs doesn't have init containers right? | 22:03 |
| kfox1111 | bootstrap jobs. | 22:03 |
| kfox1111 | they might. | 22:03 |
| kfox1111 | or, they could I should say. | 22:03 |
| inc0 | ok, nvm then | 22:03 |
| jascott1 | stackanetes's init-container has mechanism to wait for jobs | 22:03 |
| rhallisey | inc0, kfox1111, any discussion I miss here please dump it in the spec | 22:04 |
| kfox1111 | rhallisey: +1 | 22:04 |
| rhallisey | so I can have the spec reflect it | 22:04 |
| inc0 | jascott1, other way around, jobs waiting for something else | 22:04 |
| *** rhallisey has quit IRC | 22:04 | |
| jascott1 | gotcha | 22:04 |
| inc0 | like nova bootstrap job waiting for mariadb | 22:04 |
| kfox1111 | inc0: emagine a mariadb inner db creation job waiting for mariadb pod to be launched. | 22:04 |
| jascott1 | can a job kind have init container | 22:04 |
| kfox1111 | yes. | 22:05 |
| kfox1111 | jobs are really just pods that dont autorestart if the main container exists 0. | 22:05 |
| kfox1111 | all the regular pod features work. | 22:05 |
| sbezverk | jascott1: job translates into the same POD as everything else, it is just has an end, other types usually do not. | 22:05 |
| inc0 | then fine | 22:05 |
| kfox1111 | I can totally see orchestrating an entire deployment with helm/entrypoint. It will work for a lot of people. | 22:07 |
| inc0 | yeah | 22:07 |
| inc0 | that's my point | 22:07 |
| kfox1111 | I'm just not 100% sure it will work for 100% of the people. | 22:07 |
| inc0 | nothing ever will | 22:07 |
| kfox1111 | and I'm not sure I'm in that set. :/ | 22:07 |
| inc0 | but there always will be option to turn them off | 22:07 |
| inc0 | and do it manually | 22:07 |
| kfox1111 | yup. then I'm good. :) | 22:07 |
| inc0 | all you need to do is...nto include init container at all | 22:08 |
| inc0 | and do it all manually | 22:08 |
| sdake | or use an operator | 22:08 |
| kfox1111 | launch helm install --set no_entrypoint foopackage | 22:08 |
| sdake | that is the design compromise kfox1111 proposed to us early on | 22:08 |
| inc0 | kfox1111, exactly | 22:09 |
| inc0 | just keep in mind that with taht helm will just throw everything at k8s | 22:09 |
| inc0 | and hope for the best | 22:09 |
| sdake | if people want to use operators they can, if people want ot use init containers they can | 22:09 |
| kfox1111 | right. and if we make the packages configurable enough, that can be fine too. | 22:09 |
| sdake | if they want to use both god helm them, they can | 22:09 |
| kfox1111 | the one thing I havent figured out yet, | 22:10 |
| jascott1 | heheh | 22:10 |
| kfox1111 | is can we do one package per microservce, and service packages to wrap them all up for those that want that. | 22:10 |
| inc0 | ultimate granularitah! | 22:10 |
| kfox1111 | or can we do the trick I'm doing in the helm prototype, where I just make one package for the service, and load the microservice out of it conditionally. | 22:10 |
| *** matrohon has quit IRC | 22:11 | |
| jascott1 | thats sounds better | 22:11 |
| kfox1111 | gota experiment a bit more to decide. | 22:11 |
| *** matrohon has joined #openstack-kolla | 22:11 | |
| sbezverk | kfox1111: microservice package is actually a kolla's image for a specific service, right? | 22:11 |
| kfox1111 | see https://review.openstack.org/#/c/396296/16/tests/bin/ceph_workflow.sh | 22:11 |
| kfox1111 | for how thats launched, | 22:11 |
| *** matrohon has quit IRC | 22:11 | |
| inc0 | chart for nova-api which will be included in chart nova which will be included in chart openstack | 22:11 |
| inc0 | right> | 22:11 |
| kfox1111 | and this crazy trick: https://review.openstack.org/#/c/396296/16/helm/openstack-neutron/templates/main.yaml :) | 22:11 |
| kfox1111 | inc0: right. | 22:12 |
| jascott1 | inc0 thats too many charts | 22:12 |
| kfox1111 | jascott1: they are pretty trivial to nest. | 22:12 |
| inc0 | yo dawg I heard you like charts so I put chart into your chart | 22:12 |
| kfox1111 | hehe | 22:12 |
| jascott1 | lol | 22:12 |
| sdake | is it clear to everyone that we are doing 1-10 | 22:13 |
| jascott1 | if chart ~= image and all those are on one image | 22:13 |
| sdake | lines 80-90 | 22:13 |
| jascott1 | why have a chart per | 22:13 |
| kfox1111 | cart != one image. but one microservice. | 22:13 |
| *** harlowja has joined #openstack-kolla | 22:13 | |
| *** matrohon has joined #openstack-kolla | 22:13 | |
| kfox1111 | Ie, a pod is made up of the microservice + maybe fluentd + maybe entrypoint, etc. | 22:13 |
| jascott1 | glance api and glance registry will run off different images? | 22:13 |
| *** jtriley has quit IRC | 22:13 | |
| sdake | jascott1 they do currently yes | 22:13 |
| kfox1111 | so a microservice chart is likely involing more then one image. | 22:14 |
| jascott1 | that seems like where an extra chart would come in maybe | 22:14 |
| jascott1 | i just dont see the value in a chart per 'daemon' | 22:14 |
| kfox1111 | for those of us that want to manually build their clouds. | 22:14 |
| kfox1111 | they can reuse just the templates/packages without any orchestration what so ever. | 22:15 |
| kfox1111 | if that same workflow can be replicated in a combined package though, like in the helm prototype review, then that would work too. | 22:15 |
| sdake | ok well since apparently i'm on ignore - i'll go do something else | 22:15 |
| sdake | enjoy | 22:15 |
| kfox1111 | sdake: sorry. was I ignoring you? | 22:16 |
| sdake | nobody answered my question from this mystical meeting | 22:16 |
| sdake | about lines 80-90 | 22:16 |
| kfox1111 | in the currentest spec? | 22:16 |
| sdake | yes | 22:16 |
| kfox1111 | looking. | 22:17 |
| jascott1 | oh i think I understand line 80 now. the TPR is just a list of files | 22:17 |
| jascott1 | well more than that but it contains list of files | 22:18 |
| sdake | jascott1 you got it | 22:18 |
| jascott1 | it read to me like load the config files into the TPR | 22:18 |
| kfox1111 | sdake: ok. the way I interpite that, is for the operator, thats what it will do. so, yes. | 22:18 |
| kfox1111 | for those not using operators, they will have something else do that. such as helm. | 22:18 |
| sdake | kfox1111 this conflicts with what inc0 said earlier 1,2,3 nothing else | 22:18 |
| inc0 | sdake, feel free to write this operator if you want, we just write init-container | 22:19 |
| inc0 | it will be optional, default | 22:19 |
| sdake | what i agreed to was init container optional off | 22:19 |
| kfox1111 | inc0: I think the kolla team is split on "we" there. | 22:19 |
| inc0 | kfox1111, I can get people to work on it right now | 22:20 |
| kfox1111 | I think some want to work on operators, some on helm/entrypoint workflow. | 22:20 |
| sdake | kfox1111 thats great - good there is so much energy :) | 22:20 |
| kfox1111 | Its open source. I think each can work on what they need to. :) | 22:20 |
| inc0 | yup, let's decide what is default when we get at least one merged | 22:20 |
| kfox1111 | I think both are liekly to be done in parallel. | 22:21 |
| inc0 | fine, whatever as long as one works | 22:21 |
| inc0 | in ocata | 22:21 |
| kfox1111 | +1. :) | 22:21 |
| jascott1 | what is our init container going todo that the stackanetes one isnt? | 22:21 |
| inc0 | and fittest will survive | 22:21 |
| kfox1111 | jascott1: they mix entrypoint into every container. | 22:21 |
| inc0 | jascott1, nothing, this is really variant of stackanetes approach | 22:21 |
| kfox1111 | jascott1: the idea we had was to put it in an init container instead, following the sidecar pattern. | 22:22 |
| kfox1111 | otherwise, same as the stackanetes aproach. | 22:22 |
| inc0 | variant as in entrypoint won't be in image itself, it will have dedicated init container image | 22:22 |
| sdake | not fittest will survive | 22:22 |
| jascott1 | ok thanks | 22:22 |
| *** adrian_otto has quit IRC | 22:28 | |
| *** khamtamtun has joined #openstack-kolla | 22:29 | |
| jascott1 | can we change line 80 to "Read Keystone configuration file **names** from the Kubernetes ThirdPartyResource and" | 22:30 |
| *** adrian_otto has joined #openstack-kolla | 22:31 | |
| *** jheroux has quit IRC | 22:32 | |
| *** schwicht has quit IRC | 22:35 | |
| *** matrohon has quit IRC | 22:38 | |
| sdake | jascott1 if you want a change submit in spec | 22:39 |
| sdake | pretty much what i've been saying since ryan has been working on it :) | 22:39 |
| *** adrian_otto has quit IRC | 22:45 | |
| jascott1 | should I do comment or patch? dont want to be lazy ;) | 22:45 |
| jascott1 | well... | 22:45 |
| sdake | jascott1 yes comment in patch use the c button to leave a comment | 22:47 |
| sdake | then click "post" after done | 22:47 |
| *** lamt has quit IRC | 22:48 | |
| jascott1 | done. thanks | 22:49 |
| *** fguillot has quit IRC | 22:52 | |
| *** msimonin has quit IRC | 22:53 | |
| *** mgiles has quit IRC | 22:57 | |
| sdake | thanks jascott1 | 22:58 |
| *** khamtamtun has quit IRC | 22:59 | |
| sdake | sweet all my expense reports were approved | 22:59 |
| sdake | now to get rbergeron to do hers ;) | 22:59 |
| *** harbor has joined #openstack-kolla | 23:09 | |
| *** harbor is now known as portdirect_ | 23:09 | |
| Pavo | good evening | 23:10 |
| portdirect_ | evening pavo | 23:11 |
| Pavo | evening portdirect_ | 23:11 |
| Pavo | so I have found a issue with the current kolla-ansible unless I am doing something wrong | 23:12 |
| Pavo | once you enable Ceph and you do parted $DISK -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1 and then deploy | 23:12 |
| *** dave-mccowan has quit IRC | 23:12 | |
| Pavo | if you restart the storage node or lose power you have to do the parted $DISK -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1 again before your ceph containers stay in a loop restart | 23:13 |
| Pavo | which might want to be documented in the docs at http://docs.openstack.org/developer/kolla/ceph-guide.html | 23:14 |
| *** msimonin has joined #openstack-kolla | 23:15 | |
| *** eaguilar has joined #openstack-kolla | 23:19 | |
| *** sdake has quit IRC | 23:22 | |
| *** lamt has joined #openstack-kolla | 23:23 | |
| *** msimonin has quit IRC | 23:25 | |
| *** inc0 has quit IRC | 23:25 | |
| jascott1 | Pavo no experience but that doesnt sound good/right | 23:27 |
| portdirect_ | pavo: don't use kolla-ansible I'm afraid so can't help, but that doesn't sound nice/right - :( | 23:27 |
| jascott1 | haha | 23:27 |
| Pavo | I agree | 23:27 |
| portdirect_ | :) | 23:27 |
| jascott1 | isnt that labeling the partition? | 23:28 |
| Pavo | I would think that it would just reuse the same partition thats been labled | 23:28 |
| jascott1 | i know there is issue around ceph and locking not being released. is this the same or related? | 23:28 |
| Pavo | I thought is was but apparently its not labeling persistently | 23:28 |
| Pavo | haven't ran into that issue yet jascott1 | 23:29 |
| portdirect_ | yup - which looks weird - but one of the kolla-ansible peeps should be able to offer assistance | 23:29 |
| Pavo | but this is also my first time using Ceph also, so there is that | 23:29 |
| Pavo | like I said I could be doing it wrong just giving feedback on what docs say | 23:30 |
| *** chas_ has quit IRC | 23:33 | |
| *** Pavo has quit IRC | 23:38 | |
| *** Pavo has joined #openstack-kolla | 23:38 | |
| *** TxGirlGeek has quit IRC | 23:39 | |
| *** lamt has quit IRC | 23:41 | |
| *** TxGirlGeek has joined #openstack-kolla | 23:42 | |
| *** lamt has joined #openstack-kolla | 23:55 | |
| *** chas_ has joined #openstack-kolla | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
