| sbezverk | sdake: done with cleanup | 00:00 |
|---|---|---|
| sbezverk | please address my comment here: https://review.openstack.org/#/c/423555/ | 00:00 |
| sdake | working on it | 00:00 |
| * sdake n ot superman ;) | 00:00 | |
| sdake | gerrit is sluggish at present - not accepting my change via git review | 00:04 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Use environment variables to specify versions https://review.openstack.org/423555 | 00:04 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Use environment variables to specify versions https://review.openstack.org/423555 | 00:05 |
| sdake | the reason gerrit is sluggish at present is daily backups at 00:00 | 00:06 |
| sdake | sbezverk can you revisit https://review.openstack.org/#/c/423046/ | 00:07 |
| sdake | kfox1111 - need all 5 of those patches reviewed now that i've made it through the sbezverk documentation guantlet :) | 00:07 |
| sbezverk | sdake: done | 00:08 |
| sdake | sbezverk thx | 00:08 |
| sdake | sbezverk gerrit is showing me 423046 as -1 | 00:08 |
| sdake | wonder if that is thedaily backups or if you retained the -1 vote on it | 00:08 |
| sdake | oh there it goes | 00:08 |
| sdake | wow dailybackups make gerrit pokey | 00:09 |
| portdirect | sdake: not sure about mysql test chart | 00:10 |
| sdake | portdirect not sure abou twhat? | 00:10 |
| portdirect | it requires pvc support, so actually should really fail | 00:10 |
| sdake | portdirect - you added it in there not me :) | 00:10 |
| portdirect | lol | 00:11 |
| sdake | is pvc support not implemented in the dev env? | 00:11 |
| sdake | i thought that was fixed up | 00:11 |
| portdirect | not untill you install kolla... | 00:11 |
| portdirect | so its not there at that point | 00:11 |
| portdirect | we add it | 00:11 |
| sdake | ok - well i can address that in yet another followon patch | 00:11 |
| sdake | what would you recommend installing that doesn't require pvc? | 00:11 |
| sdake | fwiw, I ran the stable/mysql chart and it worked for me | 00:12 |
| sdake | although I didn't test mysql, just that helm placed it in the deploy state | 00:12 |
| portdirect | yeah - helm deploys it fine | 00:12 |
| portdirect | just the pods fail | 00:12 |
| portdirect | anyway +2ed | 00:12 |
| sdake | ok - well we shortly remove it after deploying it | 00:12 |
| sdake | thanks :) | 00:12 |
| sdake | I can change it to somehting else if someone has suggestions | 00:12 |
| portdirect | memcached | 00:14 |
| portdirect | tiny image too :) | 00:14 |
| sdake | portdirect cool | 00:16 |
| sdake | portdirect what was your fix for vagrant | 00:16 |
| sdake | I did a vagrant destroy -and that cratered tiller | 00:16 |
| *** sayantani01 has joined #openstack-kolla | 00:16 | |
| sdake | portdirect and you were typing quickly so I missed the uri you used for tiller and what precisely ou did to workaround the problem | 00:16 |
| portdirect | np | 00:16 |
| portdirect | 2 mins | 00:16 |
| sdake | I thought you did a docker pull port/tiller:2.1.3 | 00:16 |
| sdake | to load tiller in the cache | 00:16 |
| portdirect | i did but thats not needed | 00:17 |
| sdake | k - i'll be h ere :) | 00:17 |
| portdirect | 1:) kubectl delete --namespace=kube-system deploy tiller-deploy | 00:17 |
| portdirect | 2:) helm init --tiller-image docker.io/port/tiller:v2.1.3 | 00:18 |
| portdirect | that image is exactly the same as the gcr.io one | 00:19 |
| portdirect | https://github.com/kubernetes/helm/issues/1838 | 00:20 |
| portdirect | looks like it may be spreading :( | 00:20 |
| sdake | portdirect what is your github id? | 00:21 |
| portdirect | intlabs | 00:22 |
| sdake | portdirect i couldnt' credit you properly - intlabs wasn't foudn by github | 00:24 |
| sdake | but anywya i did dad your irc nick in there ;) | 00:24 |
| portdirect | lol | 00:24 |
| sdake | brb, going to actually verify that works | 00:24 |
| portdirect | https://github.com/intlabs | 00:24 |
| *** sdake has quit IRC | 00:25 | |
| *** sdake has joined #openstack-kolla | 00:26 | |
| sdake | portdirect duno what to tell ya -when i type @intlabs in github issue tracker, it didn't pop up your name in the list | 00:26 |
| portdirect | yeah - I'm actually pretty hard to track down. I tend to work via https://github.com/portdirect, which is an organisation in github - and that really confuses the UI | 00:28 |
| portdirect | kinda nice, cause its virtually impossible to track where I am. KInda rubbish for exactly the same reason - I don't get credited for about 70% of my commits. | 00:29 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Change mysql to memcached in docs https://review.openstack.org/423563 | 00:29 |
| sdake | [sdake@minime-03 halcyon-vagrant-kubernetes]$ helm ls | 00:31 |
| sdake | Error: could not find a ready tiller pod | 00:31 |
| sdake | any ideas? | 00:31 |
| sdake | oh its workign now | 00:31 |
| sdake | just a little slow to respond | 00:31 |
| sbezverk | sdake: helm is not initialized | 00:31 |
| sbezverk | helm init --client-only | 00:31 |
| sdake | sbezverk helm is indeed initialized | 00:31 |
| *** goldyfruit has joined #openstack-kolla | 00:32 | |
| sdake | sbezverk portdirect 's workaround works | 00:32 |
| portdirect | sounds like the image was still pulling | 00:32 |
| sdake | right | 00:32 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Break apart helm init and helm ls oeprations https://review.openstack.org/423566 | 00:37 |
| *** caowei has joined #openstack-kolla | 00:38 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Break apart helm init and helm ls oeprations https://review.openstack.org/423566 | 00:38 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Change mysql to memcached in docs https://review.openstack.org/423563 | 00:38 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Add a workaround for docker-engine 1.13.0 https://review.openstack.org/423568 | 00:46 |
| sdake | portdirect sbezverk srwilkers_ kfox1111 can one of you fellas ack: https://review.openstack.org/#/c/423067/ | 00:47 |
| sdake | actually two of youths | 00:48 |
| sdake | thanks jascott1 :) | 00:49 |
| *** adrian_otto has quit IRC | 00:51 | |
| *** sdake has quit IRC | 00:52 | |
| *** adrian_otto has joined #openstack-kolla | 00:52 | |
| *** sdake_ has joined #openstack-kolla | 00:52 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Add a workaround for docker-engine 1.13.0 https://review.openstack.org/423568 | 00:55 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Break apart helm init and helm ls operations https://review.openstack.org/423566 | 00:55 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Add a simple test to see if horizon is running https://review.openstack.org/422932 | 00:55 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible: [WIP] Split Openvswitch into own role https://review.openstack.org/416793 | 00:57 |
| sdake_ | portdirect quick q about storage providers | 01:00 |
| sdake_ | (for kubernetes) | 01:00 |
| portdirect | shoot | 01:00 |
| sdake_ | portdirect if kolla-kubernetes had a desire to ue a different storage provider | 01:00 |
| sdake_ | what would that take in terms of implementation | 01:01 |
| sdake_ | ue/use | 01:01 |
| sdake_ | for example iscsi or nfs | 01:01 |
| portdirect | or gce/ec2 | 01:01 |
| sdake_ | right | 01:01 |
| sdake_ | there are a whole bunch | 01:01 |
| portdirect | 2 mins | 01:01 |
| sdake_ | sure | 01:02 |
| portdirect | you would need to somehow feed the appropraite values into these: https://github.com/openstack/kolla-kubernetes/blob/master/helm/kolla-common/templates/_common_pv.yaml | 01:03 |
| portdirect | https://github.com/openstack/kolla-kubernetes/blob/master/helm/kolla-common/templates/_common_pvc.yaml | 01:03 |
| portdirect | though i dont know how to do that | 01:03 |
| portdirect | as I've not managed to keep ontop of the globals/values work that kfox/sbezverk have been doing | 01:04 |
| portdirect | I dont think that they have been tested since the jinja2 days | 01:04 |
| sdake_ | looks like there is some gce work in there: https://github.com/openstack/kolla-kubernetes/blob/master/helm/kolla-common/templates/_common_pv.yaml#L25 | 01:05 |
| portdirect | yeah - though those are both straight conversions of the jinja2 templates | 01:05 |
| sdake_ | right - so hasn't been tested lately :( | 01:05 |
| sdake_ | i wonder if it was ever tested | 01:06 |
| portdirect | the prefered way to do it now is to provide a storage class | 01:06 |
| sdake_ | how is that different than what kolla-kubernetes has n ow? | 01:06 |
| portdirect | as those maifests taget the method used in k8s 1.2 | 01:06 |
| portdirect | https://github.com/openstack/kolla-kubernetes/blob/master/helm/kolla-common/templates/_common_pvc.yaml#L8 | 01:06 |
| portdirect | if we just used that, then users could brign whatever storage they wanted | 01:07 |
| portdirect | but I'm not sure how hard it would be unwind kolla-k8s from the way it uses storage currently | 01:07 |
| sdake_ | ok - so storage as used today is base dupon a kubernetes 1.2 construct | 01:08 |
| sdake_ | and in 1.5, there is a better way to do the job | 01:08 |
| portdirect | yes | 01:08 |
| sdake_ | are there docs on the better way to do the job? | 01:08 |
| sbezverk | portdirect: we cannot use storage class, as kube controller has an not yet resolved issue | 01:09 |
| portdirect | though this only beacme possible for bare metal with 1.5 | 01:09 |
| portdirect | sbezverk: what issue? | 01:09 |
| sbezverk | otherwise we would need to lock on att's kube controller | 01:09 |
| sbezverk | missing ceph binaries | 01:09 |
| portdirect | we have kube-controlelr in kolla | 01:09 |
| portdirect | i added it just for this | 01:09 |
| portdirect | sdake_: for context | 01:10 |
| *** sdake_ is now known as sdake | 01:10 | |
| sbezverk | the actual controller kubernetes-controller-manager is missing ceph binaries | 01:10 |
| portdirect | ceph is so uncommon in k8s, that google do not provide rbd binary in the upstream image | 01:10 |
| portdirect | you have to add your own | 01:10 |
| *** sacharya has quit IRC | 01:10 | |
| sdake | what delivable is kube-controller in? | 01:11 |
| portdirect | sbezverk: we can use kolla image? | 01:11 |
| *** sacharya has joined #openstack-kolla | 01:11 | |
| sdake | and what is kube-controller in a little more detail? | 01:11 |
| portdirect | sbezverk: https://github.com/openstack/kolla/tree/master/docker/kube | 01:11 |
| * sdake groans at more curls in the codebase | 01:13 | |
| * portdirect thinks its the only way :( | 01:14 | |
| sdake | portdirect there is always a better way :) | 01:14 |
| sdake | portdirect although a short term curl hack works for now | 01:14 |
| sdake | portdirect the idea then is you deploy this container in the baremetal os? | 01:14 |
| portdirect | what? I'd like to use it | 01:14 |
| sdake | portdirect package it into rpm and deb | 01:14 |
| portdirect | lol | 01:14 |
| sdake | portdirect yup - thats the better way | 01:14 |
| sdake | portdirect then you get signed binaries from a trusted source | 01:15 |
| sdake | vs random curls from who knows where | 01:15 |
| portdirect | yeah mostly id agree with you | 01:15 |
| sdake | anyway back on topic :) | 01:15 |
| portdirect | ok - so sbezverk is right | 01:16 |
| portdirect | in that with a fesh vainila install of k8s via kubeadm you cannot use ceph stoage class | 01:16 |
| portdirect | but you could if you installed from rpms | 01:16 |
| portdirect | aor any method that had the binarys for k8s running on the hosts, not in containers | 01:17 |
| sdake | is there any way to get the righ tcontianer we need into kubeadm? | 01:17 |
| portdirect | yes | 01:17 |
| sdake | tell me more plz :) | 01:17 |
| portdirect | we cannot get ceph rbd into those images upstream | 01:17 |
| sdake | i understan | 01:17 |
| sdake | can kubeadm be overridden as is? | 01:18 |
| portdirect | there are two ways of doing it | 01:18 |
| sdake | to use different images? | 01:18 |
| portdirect | china has this problem in a BIG way | 01:18 |
| portdirect | the recommened advice there - its to prepull images to hosts, and then tag them a gcr.io images | 01:19 |
| portdirect | ^^ WOW :/ | 01:19 |
| portdirect | so kubeadm is being extended to allow you to specify images | 01:19 |
| sdake | nice | 01:19 |
| sdake | when is that landing? | 01:19 |
| portdirect | like we did to unblock you with tiller | 01:19 |
| portdirect | in master now | 01:19 |
| sdake | nice | 01:19 |
| portdirect | will need to check | 01:20 |
| *** unicell has quit IRC | 01:20 | |
| portdirect | but we will still have issue | 01:20 |
| portdirect | as the images kolla makes will not be tagged correctly | 01:20 |
| portdirect | 2 sec | 01:20 |
| kfox1111 | nice. | 01:20 |
| portdirect | and using as a bad example: | 01:21 |
| portdirect | gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.0 | 01:21 |
| portdirect | this is the naming format that kubadm will expect | 01:21 |
| portdirect | '-amd64:v1.5.0' is key here | 01:21 |
| kfox1111 | ah. yeah. :/ | 01:21 |
| portdirect | we need those images in kolla to have a similar suffix | 01:22 |
| kfox1111 | we should try and patch that in trunk asap? | 01:22 |
| portdirect | with platform arch:k8s version | 01:22 |
| portdirect | kfox1111: we should ideally byt Jeffrey4l was very unkeen on the idea | 01:22 |
| sdake | portdirect i can fix that | 01:23 |
| sdake | portdirect i also -1'ed that review | 01:23 |
| sdake | i recall it | 01:23 |
| portdirect | they made me remove the '-amd64' beofre accepting the ps | 01:23 |
| * portdirect crys uncontrollably again | 01:23 | |
| sdake | portdirect you didn't really provide a compelling reason as to why -amd64 was needed :) | 01:23 |
| portdirect | I rememeber sdake, i rememeber | 01:24 |
| sdake | portdirect iirc the review said "this is what kubernetes does" | 01:24 |
| kfox1111 | sdake: yeah, but I did, but was shot down. ;) | 01:24 |
| portdirect | lets move forawrd though | 01:24 |
| kfox1111 | +1 | 01:24 |
| kfox1111 | past is the past. | 01:24 |
| portdirect | that only gets us half way | 01:24 |
| portdirect | as we still need to sort the actuall tag at the end | 01:25 |
| portdirect | kolla-build would need to be mofified to do that | 01:25 |
| sdake | portdirect ther eis a --tag override | 01:25 |
| portdirect | or we would need docs covering it | 01:25 |
| sdake | docs would work | 01:25 |
| portdirect | sdake: docs the :) | 01:25 |
| portdirect | *then | 01:25 |
| sdake | plus the hub can provide the images | 01:25 |
| sdake | its too late to make massive changes to kolla-build.py at this time | 01:25 |
| sdake | have to wait for pike to open up | 01:26 |
| sdake | i'll cover the docs, if someone else can ocver the deliverable rename of the packages | 01:26 |
| sdake | the kolla deliverable that is | 01:26 |
| portdirect | are those images gonna be in ocata? | 01:26 |
| sdake | whatever is in master now is going to ocata yes | 01:26 |
| portdirect | cool | 01:26 |
| portdirect | they are on hub atm | 01:26 |
| sdake | and ocata is cut in about 10 days | 01:26 |
| portdirect | tagged master | 01:26 |
| sdake | portdirect can you submit a review to change the image names to what kubeadm expects | 01:27 |
| sdake | and link this chat log here: | 01:27 |
| sdake | http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2017-01-21.log.html#t2017-01-21T01:00:37 | 01:28 |
| sdake | in the commit msg | 01:28 |
| sdake | portdirect if I submit the change, I can't +2 it or argue really with anyone about why it needs to be done :) | 01:28 |
| kfox1111 | sdake: I have a ps up for revew to add two more packages. think it can still make it in for ocata? | 01:29 |
| *** sdake has quit IRC | 01:29 | |
| portdirect | roger 30 mins - just finishing off something else | 01:29 |
| *** sdake has joined #openstack-kolla | 01:29 | |
| kfox1111 | sdake: https://review.openstack.org/#/c/422950/ | 01:29 |
| sdake | portdirect i got dced | 01:30 |
| portdirect | roger 30 mins - just finishing off something else | 01:30 |
| sdake | portdirect my last msg was 18:27:48] <sdake>portdirect can you submit a review to change the image names to what kubeadm expects | 01:30 |
| sdake | [18:27:56] <sdake>and link this chat log here: | 01:30 |
| sdake | [18:28:10] <sdake>http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2017-01-21.log.html#t2017-01-21T01:00:37 | 01:30 |
| sdake | [18:28:24] <sdake>in the commit msg | 01:30 |
| sdake | [18:28:44] <sdake>portdirect if I submit the change, I can't +2 it or argue really with anyone about why it needs to be done :) | 01:30 |
| sdake | portdirect cool thanks :) | 01:30 |
| kfox1111 | 20:29 < kfox1111> sdake: I have a ps up for revew to add two more packages. think it can still make it in for ocata? | 01:30 |
| sdake | kfox1111 you mean two more containers? | 01:30 |
| *** jascott1_ has quit IRC | 01:30 | |
| sdake | kfox1111 I think so | 01:30 |
| kfox1111 | sdake: yeah. | 01:31 |
| sdake | kfox1111 although the gate is imploding with your change | 01:31 |
| kfox1111 | I think its just failing due to docker 1.13 issue? | 01:31 |
| kfox1111 | it was submitted about that time the other day. | 01:31 |
| kfox1111 | I just kicked off a recheck. | 01:31 |
| kfox1111 | I've never submitted a container before, so I took a wild guess. | 01:32 |
| sdake | you need a recheck as that patch has docker-engine 1.13.0 | 01:32 |
| kfox1111 | please let me know if I did somethign wrong. :) | 01:32 |
| sdake | it looks ok to me | 01:32 |
| kfox1111 | oh... | 01:32 |
| kfox1111 | do I need a rebase? | 01:32 |
| sdake | http://logs.openstack.org/50/422950/1/check/gate-kolla-dsvm-build-centos-binary-centos-7-nv/e89613a/console.html#_2017-01-20_01_15_30_788162 | 01:32 |
| sdake | kfox1111 moment | 01:32 |
| portdirect | kfox1111: you have logs of gate expolosion | 01:33 |
| sdake | kfox1111 your recheck will fix it up i think | 01:33 |
| portdirect | ^^ that it? | 01:33 |
| sdake | portdirect yup thats the log | 01:33 |
| openstackgerrit | Kevin Fox proposed openstack/kolla: Add ceph fs containers https://review.openstack.org/422950 | 01:33 |
| kfox1111 | well, I just rebased just in case. | 01:33 |
| sdake | kfox1111 a recheck will do it - no need to rebase ;) | 01:33 |
| sdake | kfox1111 recheck checks against latest master | 01:33 |
| kfox1111 | if the docker 1.13 fix came after that ps, it would probablly need a rebase. | 01:33 |
| sdake | no that is not the case | 01:34 |
| kfox1111 | really? I'ave always had to rebase for kolla-kubernetes I think. | 01:34 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Load murano dashboard dynamic https://review.openstack.org/402156 | 01:34 |
| sdake | i asked this same question of infra over a year ago and they corredcted me | 01:34 |
| *** adrian_otto has quit IRC | 01:34 | |
| sdake | feel free to ask anyone in openstack-infra :) | 01:34 |
| kfox1111 | hmm... does the 'recheck' do an automatic rebase then? | 01:34 |
| sdake | SamYaple and I both individually chekced their statements as well | 01:34 |
| kfox1111 | weird. ok. | 01:35 |
| sdake | and they were correct | 01:35 |
| sdake | ya its not intuitive | 01:35 |
| *** v1k0d3n has joined #openstack-kolla | 01:35 | |
| sdake | a recheck will do it no need to rebase | 01:35 |
| sdake | Jeffrey4l__ was going that too as of a month or so ago | 01:35 |
| sdake | Jeffrey4l__ and I think he has stopped now that I explained i to to him | 01:36 |
| sdake | Jeffrey4l__ might check with him - he has alot of code output and runs into those typeso fproblems often | 01:36 |
| *** v1k0d3n has quit IRC | 01:36 | |
| sdake | the only time a rebase is needed is if there is a merge conflict or gerrit is "stuck" | 01:36 |
| Jeffrey4l__ | hrm what are u talking? | 01:36 |
| kfox1111 | gotta head out. l8r all. | 01:36 |
| *** Jeffrey4l__ is now known as Jeffrey4l | 01:36 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: test rebooting in the gate https://review.openstack.org/413193 | 01:37 |
| sdake | later kfox1111 | 01:38 |
| *** unicell has joined #openstack-kolla | 01:42 | |
| *** salv-orlando has joined #openstack-kolla | 01:43 | |
| *** unicell has quit IRC | 01:45 | |
| *** zhangkun_ has joined #openstack-kolla | 01:47 | |
| *** salv-orlando has quit IRC | 01:47 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible: [WIP] Split Openvswitch into own role https://review.openstack.org/416793 | 01:49 |
| *** hfu has joined #openstack-kolla | 02:07 | |
| *** hfu has quit IRC | 02:08 | |
| *** vcn has joined #openstack-kolla | 02:09 | |
| *** sacharya has quit IRC | 02:21 | |
| *** sacharya has joined #openstack-kolla | 02:22 | |
| *** hfu has joined #openstack-kolla | 02:24 | |
| openstackgerrit | sayantani proposed openstack/kolla: Update Bifrost documentation https://review.openstack.org/417114 | 02:26 |
| openstackgerrit | sayantani proposed openstack/kolla: [WIP]Update Bifrost documentation https://review.openstack.org/417114 | 02:27 |
| *** pramodrj07 has quit IRC | 02:33 | |
| *** MasterOfBugs has quit IRC | 02:34 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the gate failure https://review.openstack.org/423574 | 02:39 |
| openstackgerrit | Wei Cao proposed openstack/kolla: Install karbor dashboard into horizon image https://review.openstack.org/423575 | 02:42 |
| *** tonanhngo has joined #openstack-kolla | 02:46 | |
| *** fguillot has quit IRC | 02:46 | |
| *** pramodrj07 has joined #openstack-kolla | 02:52 | |
| *** MasterOfBugs has joined #openstack-kolla | 02:52 | |
| openstackgerrit | Wei Cao proposed openstack/kolla-ansible: Load karbor dashboard https://review.openstack.org/423576 | 02:52 |
| *** aj_power has joined #openstack-kolla | 03:08 | |
| *** rhallisey has quit IRC | 03:30 | |
| *** zhubingbing has joined #openstack-kolla | 03:32 | |
| *** vcn has quit IRC | 03:55 | |
| *** adrian_otto has joined #openstack-kolla | 04:07 | |
| *** kristian__ has joined #openstack-kolla | 04:10 | |
| *** adrian_otto has quit IRC | 04:14 | |
| *** kristian__ has quit IRC | 04:15 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible: [WIP] Split Openvswitch into own role https://review.openstack.org/416793 | 04:16 |
| *** sdake_ has joined #openstack-kolla | 04:22 | |
| *** sdake has quit IRC | 04:22 | |
| sdake_ | Jeffrey4l are you about | 04:28 |
| sdake_ | Jeffrey4l earlier we were talking about how you don't need to rebase a patch prior to running recheck | 04:30 |
| sdake_ | Jeffrey4l and also that neeed to rename some of the kolla images related to kubernetes to amd64 (to be loaded properly by kubeadm) | 04:31 |
| *** masuberu has joined #openstack-kolla | 04:40 | |
| * sdake_ -> bed | 04:41 | |
| sdake_ | Jeffrey4l if you come about - could you ack this doc change: https://review.openstack.org/#/c/423067/ | 04:43 |
| *** adrian_otto has joined #openstack-kolla | 04:53 | |
| *** sdake_ has quit IRC | 05:01 | |
| *** caowei has quit IRC | 05:02 | |
| *** esmiurium has quit IRC | 05:02 | |
| *** adrian_otto has quit IRC | 05:18 | |
| *** aj_power has quit IRC | 05:23 | |
| *** goldyfruit has quit IRC | 05:38 | |
| *** tonanhngo_ has joined #openstack-kolla | 05:41 | |
| *** tonanhngo has quit IRC | 05:43 | |
| *** unicell has joined #openstack-kolla | 05:44 | |
| *** sdake has joined #openstack-kolla | 05:53 | |
| *** hfu has quit IRC | 05:59 | |
| *** kristian__ has joined #openstack-kolla | 06:00 | |
| *** kristian__ has quit IRC | 06:01 | |
| *** kristian__ has joined #openstack-kolla | 06:01 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add vitrage images https://review.openstack.org/417593 | 06:01 |
| *** zhubingbing has joined #openstack-kolla | 06:01 | |
| zhubingbing | hello guys | 06:01 |
| *** kristian__ has quit IRC | 06:01 | |
| *** hfu has joined #openstack-kolla | 06:02 | |
| *** hfu has quit IRC | 06:02 | |
| *** hfu has joined #openstack-kolla | 06:03 | |
| *** tonanhngo_ has quit IRC | 06:05 | |
| *** tonanhngo has joined #openstack-kolla | 06:05 | |
| *** hfu has quit IRC | 06:08 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible: Remove precheck.yml in baremetal https://review.openstack.org/423585 | 06:08 |
| *** lamt has joined #openstack-kolla | 06:09 | |
| *** caowei has joined #openstack-kolla | 06:19 | |
| *** sdake_ has joined #openstack-kolla | 06:21 | |
| *** sdake has quit IRC | 06:24 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for senlin https://review.openstack.org/423336 | 06:26 |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible: Add bool enable_elasticsearch in searchlight.conf.j2 https://review.openstack.org/423586 | 06:30 |
| *** zhubingbing has quit IRC | 06:40 | |
| *** tonanhngo has joined #openstack-kolla | 06:42 | |
| *** tonanhngo has quit IRC | 06:47 | |
| *** sacharya has quit IRC | 06:56 | |
| *** sacharya has joined #openstack-kolla | 06:57 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for searchlight https://review.openstack.org/423590 | 06:59 |
| *** kristian__ has joined #openstack-kolla | 07:00 | |
| *** unicell has quit IRC | 07:05 | |
| *** sayantani01 has quit IRC | 07:05 | |
| *** kristian__ has quit IRC | 07:05 | |
| *** kristian__ has joined #openstack-kolla | 07:26 | |
| *** zhurong has joined #openstack-kolla | 07:27 | |
| *** kristian__ has quit IRC | 07:30 | |
| *** bmace_ has quit IRC | 07:46 | |
| *** bmace_ has joined #openstack-kolla | 07:47 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for searchlight https://review.openstack.org/423590 | 07:48 |
| *** salv-orlando has joined #openstack-kolla | 07:48 | |
| *** pramodrj07 has quit IRC | 07:58 | |
| *** MasterOfBugs has quit IRC | 08:00 | |
| *** zhurong has quit IRC | 08:09 | |
| *** sdake_ has quit IRC | 08:19 | |
| *** prameswar has joined #openstack-kolla | 08:22 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for senlin https://review.openstack.org/423336 | 08:27 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Add fluentd role https://review.openstack.org/407392 | 08:33 |
| *** prameswar has quit IRC | 08:39 | |
| *** kristian__ has joined #openstack-kolla | 08:55 | |
| *** msimonin has joined #openstack-kolla | 08:58 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Install all python packages in global-requirements.txt file https://review.openstack.org/394150 | 08:59 |
| *** salv-orlando has quit IRC | 09:12 | |
| openstackgerrit | jianyi proposed openstack/kolla-ansible: Add zun ansible role https://review.openstack.org/416179 | 09:17 |
| *** tonanhngo has joined #openstack-kolla | 09:19 | |
| *** tonanhngo has quit IRC | 09:23 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for ceilometer https://review.openstack.org/423606 | 09:23 |
| *** kristian__ has quit IRC | 09:25 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible: Rename task in destroy role that removes images https://review.openstack.org/423609 | 09:34 |
| openstackgerrit | Merged openstack/kolla: Fix ironic inspector https://review.openstack.org/380774 | 10:05 |
| *** harish23r has joined #openstack-kolla | 10:08 | |
| *** vcn has joined #openstack-kolla | 10:12 | |
| *** harish23r has quit IRC | 10:14 | |
| *** vcn has quit IRC | 10:24 | |
| *** Masber_080 has joined #openstack-kolla | 10:36 | |
| *** zhangkun_ has quit IRC | 10:37 | |
| *** masuberu has quit IRC | 10:38 | |
| openstackgerrit | jianyi proposed openstack/kolla-ansible: Add zun ansible role https://review.openstack.org/416179 | 10:41 |
| *** salv-orlando has joined #openstack-kolla | 10:43 | |
| *** salv-orlando has quit IRC | 10:47 | |
| *** harish23r has joined #openstack-kolla | 10:51 | |
| *** Syffs has joined #openstack-kolla | 11:09 | |
| Syffs | hello, I have kolla deployed as an all-in-one install, but on my host machine: I have this in loop in my /var/log/messages | 11:10 |
| Syffs | Input 'mariadb_logstreamer_input' error: open /var/log/kolla/mariadb/mariadb.log: permission denied#015 | 11:10 |
| Syffs | Input 'openstack_logstreamer_input' error: open /var/log/kolla/neutron/dnsmasq.log: permission denied#015 | 11:11 |
| *** hfu has joined #openstack-kolla | 11:22 | |
| *** prameswar has joined #openstack-kolla | 11:26 | |
| *** sp___ has joined #openstack-kolla | 11:27 | |
| *** caowei has quit IRC | 11:31 | |
| *** salv-orlando has joined #openstack-kolla | 11:44 | |
| *** salv-orlando has quit IRC | 11:48 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible: Sync tools/cleanup-images with kolla repository https://review.openstack.org/423618 | 11:50 |
| *** hfu has quit IRC | 11:51 | |
| sp___ | Jeffrey4l: ping... | 11:55 |
| *** tonanhngo has joined #openstack-kolla | 12:00 | |
| *** Masber_080 has quit IRC | 12:04 | |
| *** tonanhngo has quit IRC | 12:05 | |
| harish23r | hey guys. need some help here in Kolla deployment. | 12:06 |
| harish23r | I'm trying to deploy a single node kolla setup. using the Mitaka release | 12:07 |
| harish23r | I get the following error while "using kolla-deploy" | 12:07 |
| harish23r | ameswar (~prameswar@60.243.144.5) has joined | 12:07 |
| harish23r | * sp___ (~confisury@103.209.223.125) has joined | 12:07 |
| harish23r | * caowei has quit (Ping timeout: 245 seconds) | 12:07 |
| harish23r | * salv-orlando (~salv_orla@5.87.196.101) has joined | 12:07 |
| harish23r | * salv-orlando has quit (Ping timeout: 248 seconds) | 12:07 |
| harish23r | <openstackgerrit> Christian Berendt proposed openstack/kolla-ansible: Sync tools/cleanup-images with kolla repository https://review.openstack.org/423618 | 12:07 |
| harish23r | * hfu has quit (Quit: My MacBook has gone to sleep. ZZZzzz…) | 12:07 |
| harish23r | <sp___> Jeffrey4l: ping... | 12:07 |
| harish23r | * tonanhngo (~tonanhngo@ip-64-134-216-234.public.wayport.net) has joined | 12:07 |
| harish23r | sry. | 12:07 |
| harish23r | This error: | 12:07 |
| harish23r | TASK: [common | starting heka container] ************************************* | 12:07 |
| harish23r | failed: [localhost] => {"changed": true, "failed": true} | 12:07 |
| harish23r | msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.25/conteiners/create?name=heka',),) | 12:07 |
| harish23r | FATAL: all hosts have failed -- aborting | 12:07 |
| harish23r | localhost: 0k=28 changed=0 unreachable=0 failed=1 | 12:07 |
| *** prameswar has quit IRC | 12:08 | |
| Jeffrey4l | sup sp___ | 12:09 |
| sp___ | Jeffrey4l: i have improved the coverage by upto 80% for kolla BP | 12:10 |
| sp___ | https://review.openstack.org/#/c/422444/ | 12:10 |
| sp___ | Jeffrey4l: please review this | 12:10 |
| Jeffrey4l | cool. | 12:10 |
| Jeffrey4l | ok. will review it. | 12:10 |
| sp___ | Jeffrey4l: thanks | 12:10 |
| harish23r | can anyone help me out! | 12:19 |
| harish23r | does it deal with a docker error? or I should configure a setting? | 12:20 |
| *** h16mara_ has quit IRC | 12:23 | |
| *** saneax-_-|AFK is now known as saneax | 12:25 | |
| *** msimonin has quit IRC | 12:32 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible: Set docker_restart_policy_retry to 0 by default https://review.openstack.org/423623 | 12:32 |
| *** salv-orlando has joined #openstack-kolla | 12:45 | |
| *** salv-orlando has quit IRC | 12:49 | |
| *** Masber_080 has joined #openstack-kolla | 12:49 | |
| *** dave-mccowan has joined #openstack-kolla | 13:06 | |
| *** harishr23 has joined #openstack-kolla | 13:23 | |
| *** harish_23r has joined #openstack-kolla | 13:25 | |
| *** harish23r has quit IRC | 13:26 | |
| *** harishr23 has quit IRC | 13:28 | |
| *** esmiurium has joined #openstack-kolla | 13:29 | |
| *** harish_23r has quit IRC | 13:29 | |
| *** dave-mccowan has quit IRC | 13:30 | |
| *** dave-mccowan has joined #openstack-kolla | 13:31 | |
| *** saneax is now known as saneax-_-|AFK | 13:33 | |
| *** sdake has joined #openstack-kolla | 13:45 | |
| *** salv-orlando has joined #openstack-kolla | 13:46 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for cloudkitty https://review.openstack.org/422516 | 13:49 |
| *** salv-orlando has quit IRC | 13:49 | |
| sdake | morning | 13:52 |
| sdake | Jeffrey4l you about? | 13:52 |
| Jeffrey4l | sdake, yep. | 13:52 |
| sdake | would you mind acking a doc patch | 13:52 |
| sdake | in kolla-ubernetes | 13:52 |
| Jeffrey4l | np. link? | 13:52 |
| sdake | https://review.openstack.org/#/c/423067/ | 13:53 |
| sdake | a core reviewer has already been first to review | 13:53 |
| sdake | that is familiar with kolla-kubernetes | 13:54 |
| sdake | i try not to sk you to review code in detail because you probably don't know how it works | 13:54 |
| sdake | although docs are sort of like code, this is simpel enough | 13:54 |
| Jeffrey4l | hrm. kolla-k8s better run on physical machine? but we run gate in vms .. | 13:56 |
| sdake | Jeffrey4l the dev env runs multipel vms | 13:56 |
| sdake | 4 to be precise | 13:56 |
| sdake | the gate runs 1 vm | 13:56 |
| sdake | also this change if you wouldn't midn: https://review.openstack.org/#/c/423563/ | 13:57 |
| sdake | running 4 vms in a vm is feasible, however, then your runnning virt (provided by vagrant) on vagrant | 13:57 |
| sdake | rather virt on virt | 13:57 |
| sdake | running virt (provided by vagrant) on virt (provided by your underlay virtual machine provider) | 13:58 |
| portdirect | Or you can just run the ansible that vagrant drives direct on some hardware. | 13:58 |
| sdake | morning portdirect | 13:58 |
| portdirect | But that's not documented yet ;) | 13:58 |
| portdirect | Hey guys | 13:58 |
| sdake | Jeffrey4l and finally this change: https://review.openstack.org/#/c/423566/ ;-) | 13:59 |
| sdake | all docs changes | 13:59 |
| Jeffrey4l | sdake, hrm. may other kolla-k8s core is better ;) i have no idea and context for there. | 13:59 |
| sdake | Jeffrey4l ok -well can you atleast ack the overview doc change | 13:59 |
| sdake | Jeffrey4l and ill find someone else to ack the other two reviews | 14:00 |
| Jeffrey4l | i am think why memcache rather than mysql ;( | 14:00 |
| sdake | i've like 6 patches lined up | 14:00 |
| sdake | backed up that is in the queue | 14:00 |
| *** schwicht has joined #openstack-kolla | 14:00 | |
| sdake | Jeffrey4l roger on the other patches | 14:00 |
| portdirect | Jeffrey, memcache has no storage requirements | 14:00 |
| portdirect | Mysql does not install ootb | 14:01 |
| sdake | Jeffrey4l if you could just ack https://review.openstack.org/423067 - this chane is innocous enough and reviewed by several people failiar with kolla-kubernetes | 14:01 |
| sdake | I took jascott1 's text verbatum :) | 14:01 |
| portdirect | Oh dear :) | 14:02 |
| Jeffrey4l | ok. done. | 14:02 |
| sdake | thanks fine sir | 14:02 |
| sdake | Jeffrey4l i have another request | 14:02 |
| sdake | the kolla-kubernetes deliverable needs -amd64 for container images | 14:02 |
| sdake | both you and I -1'ed the review because it was out of conformity with the rest of hte containers | 14:03 |
| Jeffrey4l | sdake, why? | 14:03 |
| sdake | for just some container images | 14:03 |
| Jeffrey4l | any meaning for the suffix? | 14:03 |
| sdake | kubeadm depends on -amd64 being in the prefix to override image names | 14:03 |
| sdake | its only af ew containers | 14:03 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Add an overview section to the documentation https://review.openstack.org/423067 | 14:03 |
| sdake | Jeffrey4l portdirect mentioend he would have a patch ready today for it | 14:03 |
| openstackgerrit | Merged openstack/kolla-kubernetes: add a cd to halcyon-vagrant-kubernetes https://review.openstack.org/423039 | 14:04 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Install kubectl and helm on the host OS in docs https://review.openstack.org/423046 | 14:04 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Clean up mysql installed during test of helm https://review.openstack.org/423120 | 14:04 |
| sdake | see what you did there Jeffrey4l ;) | 14:04 |
| Jeffrey4l | just curiosity, if there is no -amd64, it will break anything? | 14:04 |
| sdake | yes it wont work | 14:04 |
| Jeffrey4l | ok. | 14:04 |
| sdake | kubeadm --overide image=kube-controller | 14:04 |
| sdake | kubeadm is compiled for adm64 | 14:04 |
| sdake | it automatically adds -amd64 to the suffix internally it expects | 14:05 |
| sdake | so if there is no -amd64 suffix, the image cannot be found | 14:05 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Use environment variables to specify versions https://review.openstack.org/423555 | 14:05 |
| sdake | which requires manual tagging | 14:05 |
| sdake | i didn't catch it either when i -1'ed it | 14:05 |
| sdake | and the submitter didn't push back hard enough on the reviewers :) | 14:05 |
| sdake | Jeffrey4l i am making up the --overide image=kube-controller syntax, I don't know the exact syntax, but I know there is one | 14:06 |
| Jeffrey4l | ok. | 14:06 |
| sdake | Jeffrey4l and I know kubeadm internally determines what platform its compiled for and launches the image it requires | 14:06 |
| sdake | Jeffrey4l did I explain it clearly? | 14:08 |
| Jeffrey4l | yep. fair enough. | 14:08 |
| sdake | Jeffrey4l rading back it looks a little unclear even to me and I wrote it :) | 14:08 |
| Jeffrey4l | for my understanding, kubeadm adds the suffix and we can not remove it.. | 14:09 |
| *** sdake_ has joined #openstack-kolla | 14:10 | |
| *** schwicht has quit IRC | 14:13 | |
| *** sdake has quit IRC | 14:13 | |
| *** sp___ has quit IRC | 14:15 | |
| openstackgerrit | Pete Birley proposed openstack/kolla: Kubernetes: Rename Images to allow use with native deployment tools https://review.openstack.org/423633 | 14:19 |
| *** sp___ has joined #openstack-kolla | 14:27 | |
| *** salv-orlando has joined #openstack-kolla | 14:29 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for gnocchi https://review.openstack.org/423133 | 14:30 |
| *** zhubingbing has joined #openstack-kolla | 14:36 | |
| *** breitz has quit IRC | 14:51 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Add chrony ansible role https://review.openstack.org/399355 | 14:52 |
| *** breitz has joined #openstack-kolla | 14:52 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for cinder https://review.openstack.org/422384 | 14:59 |
| *** pomac has quit IRC | 15:00 | |
| kfox1111 | morning | 15:08 |
| *** v1k0d3n has joined #openstack-kolla | 15:14 | |
| *** zhubingbing has quit IRC | 15:15 | |
| Syffs | Hello, I have lots of permission denied on mariadb and dnsmasq logs in my /var/log/messages: https://ask.openstack.org/en/question/101760/logs-permission-denied-mariadblog-and-dnsmasqlog/ | 15:17 |
| Syffs | any idea? | 15:17 |
| kfox1111 | selinux? | 15:19 |
| *** sdake has joined #openstack-kolla | 15:20 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for cinder https://review.openstack.org/422384 | 15:20 |
| *** tonanhngo has joined #openstack-kolla | 15:21 | |
| sdake | morning kfox1111 | 15:21 |
| kfox1111 | morning. :) | 15:21 |
| *** sdake_ has quit IRC | 15:21 | |
| kfox1111 | how goes the battle? | 15:21 |
| Syffs | Selinux is disabled | 15:23 |
| sdake | just spilled a soda all over my laptop groan | 15:23 |
| sdake | but it appears unharmed | 15:23 |
| sdake | kfox1111 can you hit the review queue and ack the doc changes | 15:24 |
| sdake | kfox1111 i am working really hard to get the docs into workable shape | 15:24 |
| sdake | running shape | 15:24 |
| sdake | whatever the right words are;) | 15:24 |
| sdake | remove the bit rot in essence | 15:24 |
| kfox1111 | yeah. sec. | 15:25 |
| sdake | doesn't have to be now | 15:25 |
| sdake | just at some point - some of thes epatches sit around for a couple of days | 15:25 |
| sdake | which blocks work since its al on one file | 15:25 |
| sdake | i think the dev env is gtg now with the docker change that was -1'ed | 15:25 |
| sdake | haven't looked why it was -1'ed yet | 15:26 |
| sdake | ;) | 15:26 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Add a workaround for docker-engine 1.13.0 https://review.openstack.org/423568 | 15:33 |
| sdake | yak shaving complete with that change and its children I think :) | 15:33 |
| sdake | off to visit my uncle in the hopsital, then have some tech support for my parents which involves staring at a backup screen for hours on end | 15:34 |
| sdake | during that, I'll take a crack at standing up kolla-kubernetes with ceph | 15:35 |
| *** sayantani01 has joined #openstack-kolla | 15:37 | |
| *** sayantani01 has quit IRC | 15:42 | |
| *** sdake_ has joined #openstack-kolla | 15:45 | |
| sdake_ | ok gott aroll | 15:45 |
| *** eaguilar has joined #openstack-kolla | 15:47 | |
| *** sdake has quit IRC | 15:48 | |
| *** Syfffs has joined #openstack-kolla | 15:49 | |
| *** sdake_ has quit IRC | 15:50 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: test rebooting in the gate https://review.openstack.org/413193 | 15:54 |
| *** Syffs has quit IRC | 15:54 | |
| *** eaguilar has quit IRC | 15:55 | |
| *** tonanhngo has quit IRC | 15:58 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Change mysql to memcached in docs https://review.openstack.org/423563 | 15:58 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Break apart helm init and helm ls operations https://review.openstack.org/423566 | 15:59 |
| *** sayantani01 has joined #openstack-kolla | 15:59 | |
| *** tonanhngo has joined #openstack-kolla | 16:00 | |
| *** sdake has joined #openstack-kolla | 16:06 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for aodh https://review.openstack.org/423670 | 16:07 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon https://review.openstack.org/333996 | 16:07 |
| *** sdake has quit IRC | 16:14 | |
| *** sacharya_ has joined #openstack-kolla | 16:17 | |
| *** dave-mccowan has quit IRC | 16:19 | |
| *** sacharya has quit IRC | 16:20 | |
| *** sdake has joined #openstack-kolla | 16:22 | |
| *** Syfffs is now known as Syffs | 16:22 | |
| *** sacharya_ has quit IRC | 16:22 | |
| *** salv-orlando has quit IRC | 16:22 | |
| *** tonanhngo has quit IRC | 16:26 | |
| *** sdake has quit IRC | 16:29 | |
| *** adrian_otto has joined #openstack-kolla | 16:31 | |
| *** Syffs has quit IRC | 16:38 | |
| *** Syffs has joined #openstack-kolla | 16:39 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon https://review.openstack.org/333996 | 16:39 |
| *** vcn has joined #openstack-kolla | 16:56 | |
| *** vcn has quit IRC | 17:01 | |
| *** Masber_080 has quit IRC | 17:07 | |
| *** adrian_otto1 has joined #openstack-kolla | 17:11 | |
| *** adrian_otto has quit IRC | 17:14 | |
| *** salv-orlando has joined #openstack-kolla | 17:23 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: test rebooting in the gate https://review.openstack.org/413193 | 17:23 |
| *** sacharya has joined #openstack-kolla | 17:26 | |
| *** salv-orlando has quit IRC | 17:28 | |
| *** goldyfruit has joined #openstack-kolla | 17:29 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon https://review.openstack.org/333996 | 17:32 |
| *** adrian_otto1 has quit IRC | 17:49 | |
| *** adrian_otto has joined #openstack-kolla | 17:57 | |
| *** masuberu has joined #openstack-kolla | 17:59 | |
| *** sp___ has quit IRC | 18:03 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon https://review.openstack.org/333996 | 18:07 |
| *** adrian_otto has quit IRC | 18:09 | |
| *** adrian_otto has joined #openstack-kolla | 18:10 | |
| kfox1111 | sbezverk: you around? | 18:19 |
| *** salv-orlando has joined #openstack-kolla | 18:25 | |
| *** goldyfruit has quit IRC | 18:27 | |
| *** salv-orlando has quit IRC | 18:29 | |
| openstackgerrit | Merged openstack/kolla-ansible: Fix docker restart policy issue on docker engine 1.13 https://review.openstack.org/423122 | 18:44 |
| *** adrian_otto has quit IRC | 18:50 | |
| SamYaple | kfox1111: does k8s mount a _single_ rbd? or an rbd per volume created? | 18:58 |
| *** n0isyn0i1e has joined #openstack-kolla | 19:13 | |
| *** n0isyn0ise has quit IRC | 19:16 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible: [WIP] Split Openvswitch into own role https://review.openstack.org/416793 | 19:22 |
| *** sacharya has quit IRC | 19:24 | |
| *** sacharya has joined #openstack-kolla | 19:25 | |
| *** salv-orlando has joined #openstack-kolla | 19:25 | |
| Syffs | hello, I'm having permission denied in loops in my system logs: https://ask.openstack.org/en/question/101760/logs-permission-denied-mariadblog-and-dnsmasqlog/ ... any idea? | 19:29 |
| *** salv-orlando has quit IRC | 19:30 | |
| *** sacharya has quit IRC | 19:30 | |
| *** williamcaban has quit IRC | 19:30 | |
| kfox1111 | SamYaple: rbd per volume. | 19:40 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible: [WIP] Split Openvswitch into own role https://review.openstack.org/416793 | 19:48 |
| *** williamcaban has joined #openstack-kolla | 19:51 | |
| *** adrian_otto has joined #openstack-kolla | 19:53 | |
| *** dave-mccowan has joined #openstack-kolla | 20:07 | |
| *** sacharya has joined #openstack-kolla | 20:26 | |
| *** salv-orlando has joined #openstack-kolla | 20:27 | |
| *** salv-orlando has quit IRC | 20:31 | |
| *** adrian_otto has quit IRC | 20:31 | |
| *** sacharya has quit IRC | 20:31 | |
| *** sdake has joined #openstack-kolla | 20:35 | |
| *** sdake has quit IRC | 20:40 | |
| *** sdake has joined #openstack-kolla | 20:42 | |
| sdake | sup peeps | 20:47 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible: [WIP] Split Openvswitch into own role https://review.openstack.org/416793 | 20:52 |
| kfox1111 | hey | 20:53 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Add a workaround for docker-engine 1.13.0 https://review.openstack.org/423568 | 20:57 |
| sdake | sup kfox1111 | 20:57 |
| sdake | kfox1111 if you could ack https://review.openstack.org/#/c/423568/ | 21:10 |
| sdake | i'd like that to merge prior to monday if possible :) | 21:10 |
| sdake | sbezverk ^^ | 21:10 |
| *** adrian_otto has joined #openstack-kolla | 21:15 | |
| kfox1111 | sdake: I can. is that the correct workaround though? | 21:18 |
| sdake | kfox1111 tested and works for me | 21:18 |
| kfox1111 | seems like its really a bad docker version, and we should recommend around it. | 21:18 |
| sdake | portdirect suggested it | 21:18 |
| sdake | docker isn't changing to permit -'s in their organization names | 21:18 |
| kfox1111 | if it really is a problem with docker not supporting '-'s in repo names. | 21:18 |
| kfox1111 | that would be a docker bug. | 21:19 |
| sdake | i woudl bet 1 dollar against 100 | 21:19 |
| kfox1111 | they use to. | 21:19 |
| sdake | its going to be a permanant docker thing | 21:19 |
| kfox1111 | all other versions of docker did. | 21:19 |
| sdake | in any regard, its a short term fix - i suspect helm can sort it out with the docker cats | 21:19 |
| sdake | the problem without it is nobody can get any work done | 21:19 |
| sdake | and if its not documented its not a thing | 21:19 |
| kfox1111 | I expect docker to revert that change. | 21:20 |
| sdake | its part of docker hub | 21:20 |
| kfox1111 | the other way to document it is with a workaround to avoid that version. | 21:20 |
| sdake | its not like we manually install docker in the instructions | 21:20 |
| kfox1111 | there are lots of docker repos. quey, google's, artefacteries, gitlab | 21:20 |
| sdake | that happens via the dev env | 21:20 |
| kfox1111 | if they use to supprot a -, but don't anymore, that will affect a lot of repo's. | 21:20 |
| sdake | possible - not our problem | 21:21 |
| sdake | its none of our business on docker's policies | 21:21 |
| sdake | if we make it our business, i predict badness wil loccur | 21:21 |
| kfox1111 | pointing at a non official container seems more risky then telling users to avoid a buggy version of docker. | 21:21 |
| sdake | lets let helm sort it out | 21:21 |
| sdake | hsort term fix - that is why there is a date in the docs | 21:21 |
| sdake | the docs are updated continously | 21:22 |
| sdake | or shoudl be | 21:22 |
| sdake | they haven't | 21:22 |
| sdake | that is changing starting as of thursday of last week | 21:22 |
| * kfox1111 shrugs. you asked for my optinion. Thats my opinion. | 21:22 | |
| sdake | i asked for an ack, not a dissertation on how docker is evi :) | 21:22 |
| kfox1111 | working docks are preferable to non working docs. | 21:22 |
| kfox1111 | no, just being consistent. | 21:22 |
| kfox1111 | you didn't like how I ponted to a docker contiainer in my own docker hub namespace before. | 21:23 |
| sdake | ya - fwiw i did ask pete to pin docker-engine to an older version in the dev env | 21:23 |
| kfox1111 | and I did agree with you at the time. | 21:23 |
| sdake | if he does that, we can just remove that warning | 21:23 |
| sdake | yup i dont like it | 21:23 |
| kfox1111 | I don't htink this is different. | 21:23 |
| sdake | i expec tthis to be in the docs for max 1 week | 21:23 |
| sdake | a thundering herd of people are trying to get kolla-kubernetes rolling | 21:23 |
| sdake | and running into problem after p roblem | 21:23 |
| sdake | i say trying because they are failing | 21:24 |
| sdake | me included | 21:24 |
| sdake | if my answer is 'we can't figure out how to get the stuff fixed or worked around" they will be like "are u srs?" | 21:24 |
| sdake | in this case, we are doing two things | 21:24 |
| sdake | working around (by changing dev env that portdirect is taking on, and documenting a seocnd workaroudn until pete gets the docker-engine pinned) | 21:25 |
| sdake | the fixing part comes from helm upstream - I'd hope | 21:25 |
| sdake | the reality is we cna't wiat 2 months for a new docker-engine release to fix this problem | 21:25 |
| kfox1111 | I don't think helm can work around how gogle lays out their repo's. | 21:25 |
| sdake | docker is on a 2 month release cadence | 21:25 |
| sdake | not expecint helm to work around it, i'm expecting them to work with docker to fix it as you requested | 21:26 |
| kfox1111 | yeah. that. | 21:26 |
| sdake | this wil lbe a major emergency for them | 21:26 |
| sdake | it isn't a major emergency for us | 21:26 |
| kfox1111 | yeah. I hope they take it seriously. | 21:26 |
| sdake | our major emergency is that the dev env as docuemtned doesn't work properly :) | 21:27 |
| kfox1111 | agreed. | 21:32 |
| kfox1111 | theres a second smaller emergency. | 21:32 |
| kfox1111 | I've been working on the reboot testing gate. | 21:32 |
| kfox1111 | found an issue with rbd support and the way k8s uses it. | 21:32 |
| kfox1111 | so I don't think we can use ceph rbd to back k8s volumes. | 21:33 |
| *** salv-orlando has joined #openstack-kolla | 21:33 | |
| *** l4yerffeJ has joined #openstack-kolla | 21:34 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 21:35 | |
| *** l4yerffeJ_ has quit IRC | 21:35 | |
| *** Jeffrey4l has quit IRC | 21:35 | |
| *** goldyfruit has joined #openstack-kolla | 21:36 | |
| *** Bigkol has joined #openstack-kolla | 21:38 | |
| Bigkol | Where are 3.0.2 release notes? | 21:41 |
| *** Bigkol has quit IRC | 21:45 | |
| sdake | portdirect if you could pin the docker-engine in the dev env, that owuld be helpful :) | 21:46 |
| * sdake groans | 21:46 | |
| *** klindgren__ has joined #openstack-kolla | 21:48 | |
| *** klindgren_ has quit IRC | 21:49 | |
| sdake | Jeffrey4l_ you around? | 21:50 |
| sdake | Jeffrey4l_ something is odd about the kolla.yaml file | 21:50 |
| sdake | also kolla-ansible.yaml is not in the releases repo | 21:50 |
| * sdake groans | 21:51 | |
| sdake | oh i see - i'm in the newton directory | 21:52 |
| sdake | Jeffrey4l_ I think basically the release-notes section needs to be added to the newton directory | 21:52 |
| sdake | Jeffrey4l_ as well as the ocata directory for kolla.yml and kolla-ansible.yaml | 21:53 |
| *** sdake has quit IRC | 22:02 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon https://review.openstack.org/333996 | 22:03 |
| *** imcsk8_ has quit IRC | 22:04 | |
| *** imcsk8 has joined #openstack-kolla | 22:05 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible: [WIP] Split Openvswitch into own role https://review.openstack.org/416793 | 22:17 |
| portdirect | o/ | 22:19 |
| * portdirect reading | 22:19 | |
| portdirect | kfox1111: sup with k8s rbd? | 22:21 |
| kfox1111 | was trying to get k8s to recover. | 22:21 |
| kfox1111 | seeing mariadb not ever recover. | 22:22 |
| portdirect | you have logs? | 22:22 |
| kfox1111 | dug in and saw the rbd cli stuck. | 22:22 |
| *** l4yerffeJ has quit IRC | 22:22 | |
| *** l4yerffeJ has joined #openstack-kolla | 22:23 | |
| kfox1111 | talked to samyample, and he told me the kernel rbd stuff has a lot of issues. | 22:23 |
| kfox1111 | in particular, there are a few scases on startup, and even if ceph goes offliene for a bit, where the kernel mount can lockup until reboot. | 22:23 |
| portdirect | hmmm | 22:23 |
| portdirect | logs? | 22:23 |
| kfox1111 | logs here: http://logs.openstack.org/93/413193/66/experimental/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-reboot-nv/391a2e1/ | 22:24 |
| *** salv-orl_ has joined #openstack-kolla | 22:24 | |
| kfox1111 | for the ceph case, we're only using it to back mariadb/rabbitmq and helm-repo. | 22:24 |
| kfox1111 | mariadb/rabbbit I was contimplating running with local storage anyway in a cluster. so could work around it that way. | 22:25 |
| kfox1111 | helm-repo we could maybe just ditch the pv. | 22:25 |
| kfox1111 | wasn't sold on the need anyway. | 22:25 |
| portdirect | i kinda wonder if we should pin helm-repo to kube master node(s) | 22:25 |
| portdirect | and then we could just use local storage | 22:26 |
| kfox1111 | yeah. should be easy to do by just tweaking the selector_key | 22:26 |
| *** salv-orlando has quit IRC | 22:27 | |
| portdirect | so looking at logs | 22:27 |
| portdirect | could you remind me of how things work atm? | 22:27 |
| portdirect | mariadb seems running: http://logs.openstack.org/93/413193/66/experimental/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-reboot-nv/391a2e1/console.html#_2017-01-21_17_58_41_081954 | 22:27 |
| kfox1111 | seems that way, but not. | 22:28 |
| kfox1111 | so, I pull the plug on the node to simulate a power failure: | 22:28 |
| kfox1111 | https://review.openstack.org/#/c/413193/66/tests/bin/gate_reboot_master.sh line 84. | 22:29 |
| kfox1111 | k8s comes back up, and things everything is still running. | 22:29 |
| kfox1111 | takes it a bit to figure out it s not. | 22:29 |
| kfox1111 | if you look under logs/pods/mariadb | 22:30 |
| kfox1111 | you can see the state of the current mariadb. it complains there that it cant start due to volume not mounting. | 22:30 |
| kfox1111 | http://logs.openstack.org/93/413193/66/experimental/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-reboot-nv/391a2e1/logs/pods/kolla-mariadb-0.txt | 22:30 |
| portdirect | got it | 22:31 |
| kfox1111 | if you look at the ceph logs, it looks like it running and recovered properly by that point. | 22:31 |
| portdirect | :( | 22:31 |
| kfox1111 | I added a ps -eo etime,cmd to the logs, | 22:31 |
| kfox1111 | to see whats running, and how long: | 22:31 |
| kfox1111 | http://logs.openstack.org/93/413193/66/experimental/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-reboot-nv/391a2e1/console.html#_2017-01-21_18_06_37_173757 | 22:32 |
| kfox1111 | you can see rbd commands stuck for 7 minutes, 21 seconds. | 22:32 |
| kfox1111 | looking in logs/dmesg, you can see its not printing out anything ugly a bout ceph either. | 22:32 |
| kfox1111 | so at that point, Iwent to ceph to ask if they've ever seen rbd lock up. | 22:32 |
| kfox1111 | then SamYample said, its a known ugly with the kernel rbd stuff, and its rather broken. | 22:33 |
| portdirect | yeah - thats not looking too great | 22:33 |
| kfox1111 | said the only way to recover is to reboot. :/ | 22:33 |
| portdirect | moving to dynamic pvc would make it a tiny bit better i think | 22:33 |
| kfox1111 | it fails a timeout sometimes and then wont ever try and recover. | 22:34 |
| kfox1111 | no. would still fail in this case. | 22:34 |
| portdirect | as the controller manager would catch that rbd is busted | 22:34 |
| kfox1111 | as the pvc won't be recreated on reboot. | 22:34 |
| portdirect | so the mariadb pod would not come up at all | 22:34 |
| portdirect | but that doesnt give us a working system :( | 22:34 |
| kfox1111 | I asked if I could wrapper the rbd-fuse client. said it would have a big performance hit. | 22:35 |
| kfox1111 | cephfs is an option too, but again, a performance hit. | 22:35 |
| portdirect | urgh | 22:35 |
| kfox1111 | it may be good enough for a small cloud though. would have to benchmark it. | 22:36 |
| portdirect | gluster, lets use gluster | 22:36 |
| portdirect | :trollface: | 22:36 |
| kfox1111 | heh. | 22:36 |
| kfox1111 | I'd expect it to perform just like cephfs. | 22:36 |
| kfox1111 | so, galera pinned to nodes is probably our best bet then. | 22:36 |
| portdirect | similar, running databases on gluster is stupid | 22:36 |
| kfox1111 | which is desirable for ha anyway. | 22:37 |
| portdirect | i think that that may be ther right solution | 22:37 |
| portdirect | but need to think it through | 22:37 |
| kfox1111 | yeah. | 22:37 |
| kfox1111 | we were considering buying some ssd's to back one of our cloud's cluster databases anyway. | 22:37 |
| kfox1111 | so would probably need that functionality anyway. | 22:38 |
| kfox1111 | this may just expidite it. | 22:38 |
| portdirect | running galera on ceph as always seemed a bit overkill to me | 22:38 |
| kfox1111 | it was a simple way not to have to cluster to get some level of ha. | 22:38 |
| portdirect | yup | 22:38 |
| kfox1111 | so, had a bit of benifit. but if its unstable, then its not worht it. | 22:39 |
| portdirect | though, lets think thos through before we assume the sky is falling | 22:39 |
| kfox1111 | sure. | 22:39 |
| portdirect | can we fix this without restarting a node? | 22:40 |
| kfox1111 | I asked if delaying till ceph was stable would fix the issue. | 22:40 |
| kfox1111 | said it would help, but if ceph goes down for any reason for a while, the kerenl could do the same thing. | 22:40 |
| kfox1111 | I personally don't want to take that risk. :/ | 22:40 |
| kfox1111 | we've had ceph's need attention before. things always recovered after fixing so was just fine. | 22:41 |
| portdirect | did they have rommendations? | 22:41 |
| kfox1111 | no. just, don't use kernel rbd. | 22:41 |
| portdirect | was this SamYaple or peeps at ceph? | 22:41 |
| kfox1111 | said the most performant way was rbd in kvm in a vm. though we don't have that luxury. | 22:41 |
| * portdirect trusts both | 22:41 | |
| kfox1111 | SamYaple. | 22:42 |
| portdirect | lol - we actually could on most installs | 22:42 |
| kfox1111 | yeah. | 22:42 |
| portdirect | but cant really go down that path :( | 22:42 |
| kfox1111 | we'd still need a local mariadb at some point I think. | 22:43 |
| kfox1111 | so the simplest solution would be required in either case I think. :/ | 22:43 |
| portdirect | right - so (and plaing devils advocate) | 22:43 |
| portdirect | the issue with local storage is stat an operator needs to look after a node | 22:43 |
| kfox1111 | right. or a set of nodes. | 22:44 |
| portdirect | vs ceph, where we have a nice blug amorphous blob to puth things in | 22:44 |
| kfox1111 | right. | 22:44 |
| portdirect | local storage is a multiple of times faster than ceph | 22:44 |
| kfox1111 | possibly. | 22:45 |
| kfox1111 | in latency sensitive cases, probably. | 22:45 |
| portdirect | my gut stays go local | 22:45 |
| kfox1111 | yeah. mine too. | 22:45 |
| portdirect | but this is too big for us to decide here | 22:45 |
| kfox1111 | its in the relm of building blocks too though. so we can always provide different building block options and let the operators decide the best way for themselves. | 22:46 |
| portdirect | +1 | 22:46 |
| portdirect | if you can work it in as an option simply then I think that that makes sense | 22:46 |
| kfox1111 | yeah. I thinnk we can. just put a conditional around the pvc and add an attribute to pin it to a node. | 22:47 |
| kfox1111 | though, I think there's one other unknown... | 22:47 |
| portdirect | just so you know (and remeber my scale has not been the same as yours) | 22:47 |
| kfox1111 | currenty its implemented as a statefulset. | 22:47 |
| portdirect | i've typically run my dbs with local storage backed by simple raid | 22:47 |
| kfox1111 | we could do it locally, as a deployment, a statefulset, or a daemonset. | 22:47 |
| kfox1111 | yeah. we have too. | 22:48 |
| kfox1111 | only contimplating ssd's on one of my clouds, as we've foudn the db load high enoguh, | 22:48 |
| kfox1111 | that when we pull out/put back a member, it takes about an hour to fully sync. | 22:48 |
| kfox1111 | performing a full rolling upgrade then takes 3. :/ | 22:49 |
| portdirect | do we have a health check on mariadb pod? | 22:49 |
| kfox1111 | doesn't look like it yet. | 22:49 |
| portdirect | we need that | 22:49 |
| kfox1111 | yeah, we need a bp to do it everywhere. | 22:49 |
| portdirect | as it would have sorted the issue in the first logs you showed | 22:50 |
| kfox1111 | no, | 22:50 |
| portdirect | not improved the situation, just the visibility of it | 22:50 |
| kfox1111 | the very first issue was k8s showing data from etc right after reboot, | 22:50 |
| kfox1111 | but before it talked to kubelet. | 22:50 |
| kfox1111 | etcd\ | 22:50 |
| portdirect | ah | 22:50 |
| portdirect | how fast does a node reboot? | 22:50 |
| kfox1111 | pretty quick. 10's of seconds I think? | 22:51 |
| portdirect | if its under 30 then that is a problem (i think) | 22:51 |
| kfox1111 | it took longer for kubelet to recover kube-apiserver. | 22:51 |
| kfox1111 | and the controller manager. | 22:51 |
| portdirect | right - so thinking | 22:53 |
| portdirect | local stoarge = good | 22:53 |
| portdirect | need to work out reboot issues a bit more | 22:54 |
| portdirect | rebooting a AIO is not the same a s a DC power outage | 22:54 |
| portdirect | similar but not the same :/ | 22:54 |
| kfox1111 | yeah. | 22:55 |
| portdirect | soon as we can run outside of the gat this weill become a lot easier to work out | 22:55 |
| kfox1111 | well, the gate is starting to test it now. | 22:55 |
| kfox1111 | so we can test more frequently then is possible with dedicated hardware | 22:55 |
| portdirect | yeah but the gate is not interactive :/ | 22:56 |
| kfox1111 | true that. | 22:56 |
| kfox1111 | anothther advantage of doing it with local, is it does simplify the install instructions. | 22:57 |
| kfox1111 | as k8s no longer needs ceph support at all. | 22:57 |
| portdirect | also while cant run outside of gate - we don't really have a deverable, but a science project :) | 22:57 |
| kfox1111 | true. we have building blocks for assembling a cloud. | 22:57 |
| kfox1111 | but no instructions for how to do it. | 22:58 |
| portdirect | can you take that on? | 22:58 |
| kfox1111 | building one outside, or instructions? | 22:58 |
| portdirect | al teast an set of rough notes that someone could massage | 22:58 |
| portdirect | either/both | 22:59 |
| kfox1111 | next week or the weke after, I'm going to be deploying it on our test hardware. | 22:59 |
| kfox1111 | sdake's working on fresh docs right now. | 22:59 |
| kfox1111 | I offered to help him too, wherever he needs it. | 22:59 |
| kfox1111 | so I think docs will show up soon. | 22:59 |
| portdirect | great :) | 22:59 |
| portdirect | thanks man | 23:00 |
| kfox1111 | np. :) | 23:00 |
| kfox1111 | sbezverk has it running on some of his hardware too. | 23:00 |
| kfox1111 | so it does work outside the gate for sure. | 23:00 |
| kfox1111 | I had it running on my laptop in a vm for a while too. but I let that bitrot as the gate was more important to prevent regressions. | 23:01 |
| portdirect | yeah - not trying to needle you , just its a bit of a black box atm | 23:01 |
| portdirect | and behing honest, I gave up | 23:01 |
| portdirect | and thats not good :/ | 23:01 |
| kfox1111 | yeah, no needling felt. yeah, its a black box currently. :/ | 23:01 |
| kfox1111 | agreed. | 23:01 |
| kfox1111 | if you needed help though, you could have asked? or did I miss it? | 23:02 |
| kfox1111 | happy to help | 23:02 |
| portdirect | cheers man :) I know you are - just had too much else on to devote time to it | 23:02 |
| kfox1111 | k. yeah. I get that. | 23:03 |
| portdirect | will have a shot once sdakes got docs in progress | 23:03 |
| kfox1111 | cool. sounds good. | 23:03 |
| portdirect | :) | 23:03 |
| portdirect | right I better get back to git merge hell | 23:03 |
| kfox1111 | always good to have someone that hasn't tried the docs, or done a deploy test out the new docs. :) | 23:03 |
| kfox1111 | fun. :/ | 23:03 |
| portdirect | I miss gerrit for some of the stuff im working so much at the moment | 23:03 |
| kfox1111 | oh... | 23:03 |
| kfox1111 | so, back to the previous question real quick. | 23:04 |
| *** dave-mccowan has quit IRC | 23:04 | |
| kfox1111 | I can see 3 options for local storage mariadb. | 23:04 |
| portdirect | shoot | 23:04 |
| kfox1111 | statefulset, daemonset, or deployment. all can be pinned to a set of nodes. | 23:04 |
| portdirect | satefulset | 23:04 |
| kfox1111 | statefulset would work, but replica's woudl always have to be set to 1, | 23:04 |
| kfox1111 | and then pinned to the node. | 23:05 |
| *** adrian_otto has quit IRC | 23:05 | |
| kfox1111 | if you wanted a cluster, you'd have to launch multiple of them. | 23:05 |
| kfox1111 | same with deployment. | 23:05 |
| kfox1111 | though deployment is a little ligher wieght for that use case. | 23:05 |
| portdirect | kinda | 23:05 |
| kfox1111 | daemonset's interesting, in that you coudl just label the nodes as mariadb, | 23:05 |
| portdirect | so (and working this though as I type) | 23:05 |
| kfox1111 | and set net=host | 23:05 |
| portdirect | you could do the same with staefulset | 23:05 |
| portdirect | thats what i was going to suggest | 23:06 |
| portdirect | but you would need to have same number of nodes as replicas | 23:06 |
| kfox1111 | and instances too. | 23:06 |
| kfox1111 | where with a daemonset, you just launch one. and it launches one on all labeled nodes. | 23:06 |
| portdirect | replicas=insatces | 23:06 |
| kfox1111 | so a cluster might be easier. | 23:06 |
| portdirect | daemonset would work for sure | 23:06 |
| kfox1111 | though without rolling suppor t yet, | 23:07 |
| portdirect | but k8s gods would not like you | 23:07 |
| kfox1111 | no way to do a rolling cluster upgrade. | 23:07 |
| portdirect | yes | 23:07 |
| kfox1111 | thats in ps review though. | 23:07 |
| portdirect | mirantis have a soltion for that as well | 23:07 |
| portdirect | https://github.com/Mirantis/k8s-daemonupgradecontroller | 23:08 |
| kfox1111 | ah. | 23:08 |
| *** salv-orl_ has quit IRC | 23:08 | |
| kfox1111 | seems like they really like to add stuff outside of k8s. :/ | 23:08 |
| kfox1111 | coreos has a ps going to k8s itself based on the deployment upgrade code. | 23:08 |
| * portdirect wont say the things he wants too | 23:09 | |
| * portdirect but maybe that comment just said it? | 23:09 | |
| kfox1111 | perhaps. | 23:09 |
| portdirect | yeah i like the coreos approach to this | 23:09 |
| kfox1111 | you see their ps for self hosting kubelet? | 23:10 |
| kfox1111 | its fasinating. :) | 23:10 |
| kfox1111 | not sure it will work well, but they have a prototype working I hear. | 23:10 |
| kfox1111 | they can then use the deployment upgrade code to upgrade all the kubelets. | 23:10 |
| kfox1111 | pretty slick. | 23:10 |
| portdirect | yeah - I've been looking into k8s-on-k8s quite seriously | 23:11 |
| kfox1111 | so, the tradeoff between deployments and statefulset, is a deployment is easily upgraded. | 23:11 |
| kfox1111 | we wouldn't be usinjg any of the features of statefulset either. | 23:11 |
| kfox1111 | so in a cluster, a deployment with local storage and net=host might be better? | 23:12 |
| portdirect | yeah - though the really nice thing about statefulsets for us is how they start up | 23:12 |
| portdirect | if you have livelyness checks | 23:12 |
| portdirect | they start in order | 23:12 |
| portdirect | and for galera this pretty sweet | 23:12 |
| kfox1111 | yeah. but since replica's = 1 in this case always, it wouldn't help. :/ | 23:12 |
| kfox1111 | yeah. | 23:12 |
| portdirect | but whay run replicas=1? | 23:13 |
| portdirect | why not replicas=3? | 23:13 |
| kfox1111 | case we need to pin the pod to the host for local storage to always be there. | 23:13 |
| kfox1111 | since there is no automated way to do that yet. :/ | 23:13 |
| portdirect | but is you set replcias=3, and then have only 3 nodes labeled mariadb | 23:13 |
| portdirect | that will do what you want | 23:13 |
| kfox1111 | hmm... | 23:14 |
| kfox1111 | ordered start wouldn't be garanteed then? | 23:14 |
| kfox1111 | well... | 23:14 |
| portdirect | it would be | 23:14 |
| kfox1111 | well, yes/no... | 23:14 |
| portdirect | you just would not be able to controll the order of nodes that the pod was started on | 23:14 |
| kfox1111 | the names would always be 0-2, | 23:14 |
| portdirect | but therw ould always be ordeing in st startup | 23:14 |
| kfox1111 | but which nodes got which pods would be random. | 23:14 |
| portdirect | yes | 23:14 |
| kfox1111 | so the order of galera startup would be random. | 23:15 |
| portdirect | but thats fine | 23:15 |
| portdirect | but the first to start would always be 0 | 23:15 |
| kfox1111 | yeah. | 23:15 |
| kfox1111 | that might be ok. | 23:15 |
| kfox1111 | autorecovery there has always made me a little worried there. | 23:15 |
| kfox1111 | the galera docs left it as an exercise to the operator to determine "the last db down", | 23:16 |
| kfox1111 | and chosen to be the first up. | 23:16 |
| portdirect | "the last db down",none of what we have discussed solves that :/ | 23:16 |
| kfox1111 | yeah. | 23:16 |
| portdirect | and thats a whole other question :) | 23:16 |
| kfox1111 | thats another issue. though, statefulsets may make it worse? | 23:17 |
| portdirect | no | 23:17 |
| kfox1111 | one way I can think to solve it, | 23:17 |
| kfox1111 | is to do a trigger file on the host via hostpath. | 23:17 |
| kfox1111 | the operator touches a file on the host saying, this is the master to start first. | 23:17 |
| kfox1111 | if all 3 pods came up at the same time, watched for the file to show, and only changed to be the master if they saw it, | 23:18 |
| kfox1111 | it should then come up pretty smooth. | 23:18 |
| portdirect | but how would the operator know where to puth that file? | 23:18 |
| kfox1111 | would have to be in the recovery docs. | 23:18 |
| portdirect | meaning manual? | 23:18 |
| kfox1111 | yeah. | 23:19 |
| portdirect | ok | 23:19 |
| kfox1111 | so that passes the buck the same way gallera passed the buck. | 23:19 |
| portdirect | i mean that works | 23:19 |
| kfox1111 | as the end user, I have no idea most of the time how to tell, other then random. | 23:19 |
| kfox1111 | :/ | 23:19 |
| kfox1111 | one of the problems with galera. :/ | 23:20 |
| portdirect | thats not really true | 23:20 |
| portdirect | but its not easy | 23:20 |
| *** salv-orl_ has joined #openstack-kolla | 23:20 | |
| kfox1111 | yeah. there's some basic sanity checks. | 23:20 |
| mgkwill | Hiya, if anyone is hanging around from kolla-ansible, I'd love further review of 'Split Openvswitch into own role' https://review.openstack.org/#/c/416793 | 23:20 |
| kfox1111 | but if all 3 nodes basically died at the same time, and all seem to have data, its hard to guess which one is newest. | 23:21 |
| portdirect | you whould be able to work out which node went down last by looking at the state files on each node | 23:21 |
| kfox1111 | pretty much every time I've had to recover the cluster it was due to power outage. | 23:21 |
| portdirect | but getting that automated is a proejct in itself :/ | 23:21 |
| kfox1111 | yeah. | 23:21 |
| portdirect | right back to merging hell :/ laters man | 23:22 |
| kfox1111 | ok. l8r. | 23:22 |
| *** jistr has quit IRC | 23:41 | |
| *** sdake has joined #openstack-kolla | 23:43 | |
| *** jistr has joined #openstack-kolla | 23:49 | |
| sdake | kfox1111 aren't ceph rbd volumes used now to back k8s volumes? | 23:50 |
| sdake | kfox1111 sorry missed your msg from earlier - was in the middle of getting rady for naptime :) | 23:50 |
| sdake | kfox1111 or does it have to do with what happens after a reboot? | 23:50 |
| *** dims has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!