Sunday, 2017-01-22

*** dims has joined #openstack-kolla00:00
*** jistr has quit IRC00:01
*** jistr has joined #openstack-kolla00:03
*** sayantani01 has quit IRC00:09
*** salv-orl_ has quit IRC00:15
kfox1111sdake: it looks like k8s's support for backing pv's with ceph rbd relies on kernel support which is flaky. :/00:15
sdakei see - so it works most of the time but not all of the time?00:15
kfox1111there were two cases SamYaple said coudl cause problems.00:16
kfox1111if the volumes try to mount before ceph starts.00:16
kfox1111or if ceph enters a bad state for a while.00:17
kfox1111the former might be able to be worked around.00:17
kfox1111the latter, is a big problem.00:17
kfox1111I wouldn't want to my production db's dependent on it.00:17
kfox1111ceph support in glance/nova/cinder is uneffected.00:17
kfox1111it only really effects rabbit/mariadb.00:17
openstackgerritMerged openstack/kolla-kubernetes: Add a workaround for docker-engine 1.13.0  https://review.openstack.org/42356800:24
sdakeis nfs an option or iscsi?00:25
sdakewhat do we have in use now, ceph?00:26
portdirectnfs would not be great00:26
portdirectiscsi would be a really good option for this00:27
kfox1111nfs is usually a scalability/ha problem.00:27
kfox1111same with iscsi.00:27
portdirectsmae with local...00:27
kfox1111local is, unless you use clustering.00:27
portdirectyeah but we would be wouldnt we?00:27
sdakeok - so i think nfs is pretty terrible myself - wondering what it would take to enable all of the options00:28
sdakeso its just configurable and the operator could choose what they want00:28
kfox1111portdirect: I think we probably should.00:28
kfox1111cephfs is another option.00:28
kfox1111but performance will be a bit worse then nfs. but doesn't have the ha problem.00:28
sdakenfs can be setup in HA mode, but ya, nfs is a pile of steam00:28
portdirectim not sure cephfs is a good for mariadb00:29
sdakeso in kolla-ansible - which is my only experience with ceph00:29
sdakeceph was only used for glance/nova/cinder via the RBD driver00:29
sdakein kolla-kubernetes, ther eare two ceph's running 9from my understanding, which is limited)00:29
kfox1111yeah. so it would be closer to kolla-ansible then. local storage with clustering.00:29
sdakethere is one that provides storage for the kubernetes PVS00:29
sdakeand one that provides storage internally00:29
kfox1111two different ways its used. one ceph.00:30
portdirectsdake: same ceph for both atm00:30
sdakecan someone validate the above from me, or am I misunderstanding how persistent storage is handle?00:30
kfox1111right. one for pvs for k8s,00:30
kfox1111and the rest for nova/glance/cinder.00:30
sdakeportdirect and kfox1111 you guys just siad opposite things :)00:30
kfox1111the problem is only with the pvc's.00:30
portdirectkfox1111: could you clarify00:31
kfox1111k8s is using kernel based rbd for k8s pv's.00:31
kfox1111nova/glance/cinder is using librbd.00:31
kfox1111the instability is with the kernel based rbd support.00:31
portdirectkfox1111: but to help sdake, could you calarifyt that they are both coming from the same ceph cluster atm00:32
portdirectand you are talking now about how that cluster is consumed in two different ways00:32
sdakesounds like same ceph cluster, two different access mechanisms00:32
portdirectyes00:33
kfox1111portdirect: right.00:33
kfox1111once ceph was handling both use cases.00:33
sdakecool thanks for clearing that up00:33
sdakeand that is what we have at present - and it works most of the time :)00:33
sbezverkportdirect: I tried iscsi and hit issue00:33
sdakeand maybe we need to transition to something else, so it works all of the time00:33
portdirect60% of the time it works every time00:33
sbezverkI opened an issue with kube people, so far no answer00:33
portdirecthmm :(00:34
portdirective been using iscsi with a driver w wrote very well00:34
portdirectbut not k8s own driver00:34
*** zhurong has joined #openstack-kolla00:34
kfox1111portdirect: flexdriver?00:35
sdakecan we enable a general approach to storage backing where anything canbe used?00:35
portdirectHDS now reommend it  with some of their arrays :D00:35
sbezverkhttps://github.com/kubernetes/kubernetes/issues/3915900:35
portdirectkfox1111: yes00:35
*** goldyfruit has quit IRC00:35
sdakeor is that not a thing00:35
sdakereading the FM it says there are multiple storage backends00:35
kfox1111sdake: its pretty muc htat way now. its just the underlying options are less then ideal at the moment. :/00:35
portdirectsdake: thats what 1.5 PVC lets us do00:35
sbezverkportdirect: I like iscsi very much, it just does not work :(00:35
kfox1111isci's hard to make ha though.00:36
kfox1111so defeats the purpose for this use case I think.00:36
sdakeok - so ignoring the actual making it ha00:36
portdirectsbezverk: https://youtu.be/h1et3FR6zis?t=28m40s00:36
sdakeand assuming that is someone elses problem  (TM)00:36
kfox1111unless you actually have ha iscsi externally from another vendor.00:36
sdakeit sound slike with k8s 1.5 we can have a general approach to storage00:37
kfox1111yeah. but you need a capable storage backend.00:37
kfox1111ceph rbd seems to be a bit lacking at the moment the way k8s supports it. :/00:37
sdakeright so we could say something like "kolla-kubernetes coudl work with XYZ"00:37
sdakehowever in the gate we test with "Y"00:38
sbezverkkfox1111: not every solution must be software solution, you can build h/w redundant storage cluster and offer iscsi connectivity00:38
kfox1111well, this is a batteries included issue I think.00:38
sdakekfox1111 right - we test with "Y"00:38
sbezverkkfox1111: it is just people are looking for "sheap" things00:38
kfox1111sbezverk: right. thats what I meant. netap for example sells an appliance that would work.00:38
sbezverk"cheap"00:38
sdakesbezverk the word you are looking for is cheap :)00:38
sdakesbezverk sorry i thought you were making a play on words ;)00:38
kfox1111netapp is the opposite of cheep though.00:38
kfox1111next best solution I can think of is a cluster with local storage.00:40
sbezverkkfox1111: I think, and maybe I am totally wrong, when you build mission critical env, cheep, should not be in first 5 reason for technology selections ;)00:40
portdirectsbezverk: the acutal driver is here https://github.com/portdirect/harbor/blob/f82d7ded096e27cda8df8a5e03860a789066e41e/docker/kubernetes/kubernetes-kubelet/assets/usr/libexec/kubernetes/kubelet-plugins/volume/exec/cinder/cinder00:40
kfox1111sbezverk: there's levels of sla and cheep. :)00:40
sbezverkportdirect: I do not think cinder driver and iscsi driver are the same thing?!?00:41
portdirectkfox1111: betteies included are nice, but this is satrting to get a bit out of scope is is not00:41
portdirectsbezverk: that does both :)00:41
portdirectuses cinder to create and manage iscsi00:42
sbezverkportdirect: hm interesting..00:42
kfox1111portdirect: which being out of scope?00:42
portdirecti think that we should try and use the k8s sortage options, so that people deplying it can bring whatever the want to the party00:43
sdakeportdirect ++00:43
kfox1111how would you back mariadb with cinder, which needs a mariadb?00:43
kfox1111portdirect: I don't disagree with that.00:43
kfox1111put local storage is an option too.00:43
portdirectand speding too much time on building out ha storage moves us towards being a distro00:43
portdirectrather than a set of building blocks00:43
kfox1111nothing stopping you from deploying a mariadb on pv's if you have a suitable one.00:43
*** caowei has joined #openstack-kolla00:43
kfox1111not arguing we should remove support for them.00:43
sdakeportdirect - so what i'm after is how to get kolla-kubernetes stood up in general :)00:44
sdakeportdirect so i can document it00:44
portdirectwhat I'm saying though it im not sure it should be a blocker00:44
sdakeportdirect so others can deveop for it00:44
portdirectsdake: exactly00:44
kfox1111I am pretty sure I'll need a more ha db then just a single node can provide.00:44
sdakeportdirect it doesn't have to be pretty or fantastic, it just has to sort of work :)00:44
kfox1111so I'll probably contribute the local storage cluster code.00:44
portdirectkfox1111: we all will, but kolla-k8s doesnt need00:44
portdirectsdake: we are in total agreement00:45
kfox1111if all of us do, why wouldn't others?00:45
kfox1111sounds like it belongs under the kolla-kubernetes umbrella then.00:45
portdirectbeacsue we need it - but ha kubernetes storage is not reeally kolla-k8s's problem in my opinion00:45
kfox1111kolla-ansible is already doing it.00:45
sdakealready doing which? :)00:45
kfox1111I'm not arguing we implement a sotrage system from scratch.00:45
kfox1111just supporting galera with local storage.00:46
kfox1111kolla-ansible already does.00:46
sdakekolla-ansible uses bindmounts for galera00:46
kfox1111maybe we are talking about differen things?00:46
kfox1111sdake: exactly.00:46
sdakeis that what ou mean by local storage?00:46
kfox1111which are prety much the same as k8s host path's.00:46
kfox1111yeah.00:46
portdirectso lets do that - but via pvc annotations00:47
portdirectand not include anything else in the scope of kolla-k8s00:47
portdirectie do all sotage section via the annotation00:47
kfox1111was just going to do it as an if conditional?00:47
portdirectthis lets operators use whatever they want00:47
sdakeportdirect would that allow ops to choose whatever storage backend they want?00:48
portdirectyes00:48
sdakewfm00:48
kfox1111{{if local_storage }} host_path {{else}} pv: name: ... {{end}}00:48
kfox1111?00:48
portdirectno00:48
portdirectthats the old wat00:48
portdirectway00:48
kfox1111there isn't a new way that works with hostpaths I think?00:49
sbezverkportdirect: if you have config example of working iscsi storage option for kube, pleae let me know..00:49
portdirectwhat we should o is delacre a storage class00:49
sdakewow my mothers computer has a bunch of crap on it00:49
kfox1111how do yho udeclare a storage class for local storage?00:49
portdirectand then use the annotation to select it00:49
portdirect2 sec (im sure you can)00:49
portdirectminikube does i think00:49
kfox1111oh. that would be cool if that works.00:50
*** caowei has quit IRC00:50
*** caowei has joined #openstack-kolla00:52
openstackgerritLi Yingjun proposed openstack/kolla-kubernetes: Fix different repo issue for documentation  https://review.openstack.org/40505400:52
sdakeportdirect were you able to get that pin sorted out00:53
portdirectyeah - going to move to upstream docker00:53
portdirectwill be in on monday00:53
sdakeportdirect i looked at the running vagrant, its already running docker-engine dude00:54
sdakeportdirect that is what the problem is - docker-engine 1.13 is busted00:54
sdakedocker-engine = upstream docker00:55
sdakedocker = red hat docker00:55
sdakered hat is not on docker-engine 1.13 :)00:55
sdake(centos that is)00:55
portdirectdamn, thought i'd pulled that out https://github.com/att-comdev/halcyon-kubernetes/blob/master/kube-deploy/roles/deploy-kube/tasks/centos.yml#L3900:55
portdirectwill have it fisec on monday regardless00:55
sdakeportdirect cool - so i think the key is to pin to an older version of docker00:56
sdakelike 0.12.600:56
sdakegiven the complexity of windows, I have no idea how people maintain their own windows computers00:57
sdakeI can barely maintain my parents, and my version of maintenace involves a full restore from a pristine installed copy from a NAS :)00:57
*** hfu has joined #openstack-kolla01:05
kfox1111portdirect: been able to find an example of a local storage class?01:08
kfox1111I'm looking through the code base but not finding much.01:08
portdirectSame here, it seems there is some provisioning stuff in there, but no examples01:09
kfox1111I fired up a new minikube, but kubectl get storageclass has none defined.01:10
kfox1111hhhmm... though. looks like my minikube binary is a bit old... let me download a newer one.01:11
portdirectSame here01:11
portdirectIts the iso that matters not minikube itself really01:11
sdakeiso as in cdrom?01:12
kfox1111yeah. but easier to just download the new minikube and minikube delete; minikube start01:12
portdirectdims would know01:12
portdirectsdake minikube launches a custom iso with a modified version of kyperkube (localkube) for testing, used to be based on boot2docker, but now is a buildroot based distro that google/coreos made01:16
sdakei see01:16
sdakeis it CI/CDed as kubernetes is released?01:17
sdakeperhaps it has an old version of k8s01:17
portdirectnah - it runs any version you want01:17
portdirectfor example01:17
portdirecthttps://www.irccloud.com/pastebin/HB2vBrOD/01:18
sdakeportdirect cool so minikbube is an integration wrapper much like the current dev env we have today, however, the devenv today is based upon kubeadm01:19
portdirectyup01:19
portdirectwe needed kubadm based for multinode01:19
kfox1111sdake: it was made to be the most simple way to deploy a test k8s.01:19
kfox1111downlad the binary and run it.01:19
kfox1111it lacks multinode support though, as most folks don't realy need it.01:19
portdirectbut if we go local stoage and drop the ceph dep, then we could use it01:19
kfox1111it works with ceph.01:20
sdakeportdirect i think using kubeadm is fine01:20
sdakesince kubeadm is multinode01:20
sdakealthough I don't really understand what kubeadm is either :)01:20
sdakewe do want our dev env to virtualize multliple nodes01:20
sdakethe gate may not needi t but devs do01:21
* portdirect wonders if maybe kolla should use ansible01:21
sdakeportdirect which deliverable of kolla :)01:21
portdirectlol01:21
portdirectall of them01:22
portdirect:)01:22
kfox1111the gate is multinode. ;)01:22
sdakeso in the earlly days of k8s - 0.9.7 or so - we tried making kolla use k8s and failed01:22
sdakethen we tried shell scripts and that worked sort of01:22
sdakethen we tried compose and that worked better01:22
sdakethen we went to ansible and that was a winner01:22
*** masuberu has quit IRC01:22
sdakek8s lacked host=pid and net=pid support - which were major blockers01:23
portdirectlol - you should have stuck with it :) (no i know im just being rude)01:23
sdakeit was probably missing a whoel slew of other crap too that would have dragge don and on01:23
portdirecti fork your work back then01:23
kfox1111ok. redeployed minikube. its 1.5.1 now.01:23
kfox1111no storage clsases defined though.01:23
portdirectand got round that by just bind mounting to docker socker01:23
sdakeso k8s lacks a storage class at all?01:24
sdakeor we have to load the min01:24
sdakeperhaps its some kind of plugin01:24
kfox1111sdake: no. just looking to see if minikube provided any storageclasses out of the box  for local storage.01:24
sdakejust throwing darts ata wall here :)01:24
kfox1111it does not seem to.01:24
*** masuberu has joined #openstack-kolla01:24
kfox1111if there is a way to do pvc provisioning from local storage via storage classes, I can't figure out how to.01:25
sdakeok cna someone define the difference between kubeadm and kube-spray while we are on this topic?01:25
portdirectsdake01:25
portdirectnot easily01:25
portdirectthats a bit like the difference between packstack and fuel01:25
sdakekubeadm is packstack in the anaology?01:26
portdirectno they both work :D01:26
sdakei actually quite liked packstack, I think if rht would have stuck with packstack, they woudl have won :)01:26
portdirectjust different ways of deploying a cluster01:26
* portdirect was reffering to fuel01:26
kfox1111sdake: kubeadm is under the k8s umbrella.01:26
kfox1111like helm.01:27
kfox1111its newer, so less advanced.01:27
sdakekfox1111 i know - so is kubespray01:27
sdakekubespray in the incubator01:27
kfox1111kubespray is?01:27
sdakethis is literall ythe first free 8 hour block of time i've had since late nov to take a look at anything01:27
sdakeand I just want to get going :)01:28
kfox1111oh. seems like it is. thats news to me. :)01:28
sdakethe most meaningful contribution i can make to kolla-kubernetes at htis point is documenting how all this stuff works and making it easy for devs to onramp01:28
sdakeif devs canm't onramp, ops wont be able to onramp either01:29
kfox1111+101:29
openstackgerritMarcus Williams proposed openstack/kolla-ansible: Split Openvswitch into own role  https://review.openstack.org/41679301:29
openstackgerritMarcus Williams proposed openstack/kolla-ansible: Add OpenDaylight role  https://review.openstack.org/41636701:29
sdakenot that devs are smarter then ops - just they are typically more motivated and stick with it longer01:29
sdakewe need to have a 30 minute "here is how you go from aero to winning"01:29
kfox1111we paid a high cost for helm. :/01:29
kfox1111sdake: and to dig through code to figure out how.01:29
sdakekfox1111 right - although that was the right move in retrospec01:29
sdakekfox1111 i have dug throug hteh gate code01:30
portdirectkfox1111: yeah but it helps to be +ve01:30
sdakewhere I am jammed up now is setting up ceph01:30
portdirectother projects (ones that im onvoved in) can go from nothing to fully deployed on bare metal in under 8 mins now01:30
sdakehence the questions aroudn storage01:30
portdirect13 lines of bash01:30
sdakeportdirect is this aic-helm or something else?01:30
portdirectboht opensatck-helm, and harbor01:31
kfox1111I used this to deply my ceph: http://docs.ceph.com/docs/jewel/rados/deployment/01:31
portdirectharbor take 45 mins01:31
openstackgerritMarcus Williams proposed openstack/kolla-ansible: Add OpenDaylight role  https://review.openstack.org/41636701:31
kfox1111portdirect: kolla-kubernetes will get there. like I said, we paid a HIGH cost for helm.01:31
kfox1111:/01:32
sdakewhy ahe one snowflake in your datacenter when you can have a blizzard :)01:32
kfox1111I had totally working docs at one point. then it all fell apart due to 3 or 4 code refactors. :/01:32
sdakekfox1111 refactors are a natural part of development01:32
sdakethere are 3 things that make a project great01:32
sdakefour things actualy01:32
sdakecommunity, code, docs, and gating01:32
kfox1111yeah. It was an investment in community to do helm.01:33
sdakeany one of those doesn't materialize, the chair falls over and the operator or dev sitting on the chair cracks his skull open01:33
kfox1111but at the same time, it prevents more devs from starting up, as the docs bitrotted.01:33
portdirectnone of the openstack on k8s projects has all four01:33
sdakefortunately docs are fixable01:33
sdakecommunity is really hard to fix01:33
kfox1111so short term, it hurt the community.01:33
kfox1111yeah.01:33
kfox1111ture.01:33
kfox1111true01:33
sdakeso lets fix the docs01:34
sdakei'm doing that01:34
sdakewhere i need help is storage :)01:34
sdakei think i can put together a 10-20 line shell script that does the job01:34
kfox1111yup.01:34
sdakeit may be longer - who knows01:34
sdakei have managed to stand up kolla-kubernetes on friday in 3 hours wihtout storage and it seemed to all work01:35
sdakeminus the containers failing to start and whatnot :)01:35
portdirectlol - but getting the templates to render is agood 1st step01:36
sdakethe dev env doc is perfect now, just need the quickstart to be so01:36
sdakeportdirect i have been taking alot of notes and have history recorded and formulating a shell script to do the job01:36
portdirectsweet :)01:36
sdakei think our top mission to solve - right now - is how to get me unblocked so I can get eveyrone else unblocked :)01:37
portdirectIf you push a DNM, ps it would be great to go over it01:37
kfox1111sdake: so, where are you stuck?01:37
sdakei relaize not eveyrone is blocked01:37
sdakebut eveyrone using the gate to test is super suboptiaml01:37
sdakekfox1111 stuck on how to setup storage01:37
portdirecti'sd like to be able to run koll-k8s as well01:37
kfox1111no, thats too generic.01:37
kfox1111what step of "deploy ceph" are you stuck on?01:37
kfox1111or is it, step 1, 'pick a deployment tool'?01:37
sdakei haven't deployed ceph - ran out of time on friday01:37
sdakei got through all the gating and got everything written down01:37
kfox1111ah. ok.01:38
kfox1111so, stuck on time. need time to try deploying it?01:38
sdakeit sounds now like we don't want to deploy ceph01:38
sdakebut want to use a local storage thingamagic :)01:38
kfox1111we need to write that support.01:38
kfox1111and I still highly recomment backing nova/glance/cinder with ceph.01:39
openstackgerritMarcus Williams proposed openstack/kolla-ansible: Split Openvswitch into own role  https://review.openstack.org/41679301:39
portdirectkfox, whould you help make this work https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/build_test_ceph.sh ouside of the gate (for devs)01:39
sdakeportdirect cool i hadn't seen that file01:40
sdakeportdirect seen/got to it yet :)01:40
portdirectthats how the gte runs ceph01:40
kfox1111thats the majority of the logic.01:40
portdirecti mageed to make it run outside with a lot of sed01:40
portdirectonce01:40
kfox1111the rest is tests/bin/setup_gate_loopback.sh01:41
sdakeportdirect ready to devconf :)01:41
portdirectnot yet  - gona get the pp done monday01:41
portdirectgot a lot to fit in01:42
sdakei'm sure it will be great01:42
kfox1111tools/setup_simple_ceph_users.sh, and tools/setup-ceph-secrets.sh01:42
sdakebecause thats how you roll :)01:42
kfox1111I tried to seperate out the last two files so they could be used with an external ceph.01:42
*** nihilifer has quit IRC01:42
kfox1111though never tried that yet.01:42
portdirectbasicly gonna give 10 mins to each: why?, apline images & building in dockerhub, deploying k8s/rpm-ostree, freeipa/ipsion & keystone, demo01:43
sdakei've got a 2 hour meetup in tempe in february01:44
sdakeneed to write the abstract01:44
* sdake groans01:44
kfox1111sdake. theres another option.01:44
kfox1111if you only want to do a two node dev env, one controller one compute,01:44
openstackgerritMarcus Williams proposed openstack/kolla-ansible: Split Openvswitch into own role  https://review.openstack.org/41679301:45
openstackgerritMarcus Williams proposed openstack/kolla-ansible: Add OpenDaylight role  https://review.openstack.org/41636701:45
kfox1111you could use the iscsi branch of the gate instead.01:45
kfox1111then you don't need ceph at all.01:45
sdakekfox1111 i am after multinode for devs, as the dev env simulates 4 nodes01:45
kfox1111what do the 4 nodes do?01:45
*** nihilifer has joined #openstack-kolla01:45
sdakeone is controller 3 are assigned whatever roels you like - the current docs recommend compute nad controlelr on all 3 nodes01:46
portdirectkfox1111: currently nothing... 1 controller. 3 nodes01:46
sdakeroles/labels01:46
kfox1111one k8s controller, and 3 k8s compute nodes then?01:46
portdirectwe set it up that way when we imagined that we could get it running outside of gat fater01:46
sdakekfox1111 right01:46
kfox1111k.01:46
kfox1111so the 3 nodes coudl be used, for 1 openstack controller and 2 computes01:47
kfox1111or 3 controllers/computes combined.01:47
portdirectthe idea was the 2nd01:47
kfox1111or 1 controller, one nn, and one compute01:47
sdakethe docs indicate the 2nd model01:47
kfox1111k.01:47
portdirectas this would let us test ha of everything at lowest hardware cost01:47
sdakeya - should run in 16-32gb of ram01:47
sdakeon one machine01:47
kfox1111portdirect: I guess the issue is, the iscsi gate will work on one controller,l but not 3.01:48
kfox1111but doesn't need ceph.01:48
portdirectthats fine01:48
kfox1111so the 1 controller 2 compute would be quicker to document.01:48
portdirectwe can use labels to do that easily01:49
kfox1111depends I guess on just how soon you want to onboard new devs.01:49
sdakekfox1111 got 2 coming right now - just got a linked in email two fulltimer devs coming to kolla-kubernetes01:49
kfox1111the test ceph in the gate will only run on one node too.01:49
sdakethey are looking for how to get engaged01:49
sdakeand I'm like "groan - i can't even get engaged:)"01:49
portdirectkfox1111: i dont want to get political - but others are joining other projects beasue of this01:50
kfox1111ceph-deploy could do a 3 node full ha deployment pretty easily.01:50
kfox1111portdirect: yeah, I get that.01:50
kfox1111like I said, helm was not withouit cost. :/01:50
sdakeok lets forget the past01:50
sdakeand focus on teh future01:50
kfox1111yup.01:50
kfox1111so how quickly do yo uwant a dev env.01:50
kfox1111?01:50
sdakeso the future is - we need docs, and right now :)01:50
kfox1111wha tradeoffs are ok, vs what we shoudl delay for?01:51
sdakei am committed to writing the docs01:51
openstackgerritMarcus Williams proposed openstack/kolla-ansible: Split Openvswitch into own role  https://review.openstack.org/41679301:51
sdakei'd like it to match up with the devenv we have already documented01:51
sdakesinc ethat works well01:51
sdakeso tradeoffs01:51
kfox1111if you want something in the next day or two, I'd recommend doing 1 controoler, 2 compute, and basing it on the iscsi_workflow for now.01:51
kfox1111you can skip ceph alltogether then.01:52
kfox1111it sounds like you are really close to being able to do that.01:52
kfox1111then, once thats done, and the pressure is off, we can work on smoothing out the ceph deployment stuff.01:53
kfox1111(and maybe new folks can help with that)01:53
sdakeok - so thinking down those lines01:55
sdakewhere we recommend 1 controller 2 computes01:55
sdakeis the PV hardcoded in the dev env to use ceph portdirect ^^?01:56
kfox1111its just changing the node lables.01:56
kfox1111sdake: no, its flexible.01:56
kfox1111the iscsi gate uses localpath pv's.01:56
sdakekfox1111 i was spekaing about the halcyon devenv we are using01:59
sdakelacyon dev env may or may not make assumptions about the storage in use01:59
sdakei'm unclear if this is the case01:59
sdakeportdirect could you confirm?01:59
kfox1111oh.02:00
*** sayantani01 has joined #openstack-kolla02:00
openstackgerritMarcus Williams proposed openstack/kolla-ansible: Add OpenDaylight role  https://review.openstack.org/41636702:02
portdirectno halcyon is 100% stock k8s02:02
portdirectwith the expetion of adding things required for ceph support if slected02:03
portdirectbut it does not preculde using any other storage02:03
portdirector alter how you would do it02:03
kfox1111nice.02:03
sdakeportdirect does that mean we could potentially use the iscsi workflow then?02:04
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon  https://review.openstack.org/33399602:04
sdakeexternal to the dev env?02:04
portdirectyes02:04
sdakei recall something about halcyon not supporting PVCs02:04
sdakeis that fixed?02:04
portdirectthat was k8s not supporting pvcs at the time (1.4.6) so yes fixed now02:05
sdakecool02:05
kfox1111portdirect: you talking about dynamic pvc's?02:05
sdakeso i shoudl be able to get iscsi workflow working pretty easily and document that in a quickstart02:05
portdirectthe isse there comes from the use of k8s in public clouds02:05
kfox1111cause pvc's have been supported for a while. using them in 1.3 still on one system.02:05
kfox1111ah.02:05
portdirectwhere is you set up a k8s cluster on aws/gce/azure etcd you have pvc support out of the box02:05
portdirectbut not on bare metal02:06
kfox1111the storageclass stuff?02:06
portdirectand halycon is bare metal (in vm's)02:06
sdakeok - so lets focus on baby steps with a goal of having a quickstart in reviewable state by monday or tuesday02:06
portdirecteither sotage class orf via the cloud provider02:06
sdakeI'll write the docs myself02:06
sdakeI have monday and tuesday mostly available02:06
kfox1111portdirect: ok. that makes sense.02:06
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for barbican  https://review.openstack.org/42372802:07
sdakewould an iscsi backend with 1 controller 2 computes be suitable for dev kfox1111 and portdirect ?02:07
portdirectkfox1111: youve leant to ready my jabberings :) I could ahrdly make that out :)02:07
portdirectsdake if it starts its sutuable for dev :)02:07
portdirectthey can help make it better02:08
portdirectif you can get it to a working keystone02:08
kfox1111+1.02:08
kfox1111it will have all the bits to test things out, fix bugs, and add new features.02:08
kfox1111and worst case, they can contribute and check against ceph in the gate.02:09
kfox1111so yea, it whould help a lot I think.02:09
sdakei like whould :)02:09
sdakei'll have to use that02:09
sdake;)02:09
kfox1111then we will l have more time to get ceph support into it.02:09
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for barbican  https://review.openstack.org/42372802:19
*** Syffs has quit IRC02:32
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for barbican  https://review.openstack.org/42372802:32
*** hfu has quit IRC02:44
*** jrich523 has quit IRC02:45
*** masuberu has quit IRC02:50
*** sp___ has joined #openstack-kolla02:53
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for barbican  https://review.openstack.org/42372802:55
Jeffrey4l_sdake, re release note. yes. i will add the link.02:56
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix add kubernetes image release note format  https://review.openstack.org/42373903:26
*** hfu has joined #openstack-kolla03:28
sdakeJeffrey4l_ thanks :)03:29
sdakeJeffrey4l_ someone asked for it today03:29
sdakeor maybe it was yesterday - for the newotn 3.0.2 release notes03:29
sdakeand i noticed they arn't published03:29
Jeffrey4l_sdake, could u review it? https://review.openstack.org/42373703:29
*** sudipto has joined #openstack-kolla03:32
*** sudipto_ has joined #openstack-kolla03:32
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Add chrony ansible role  https://review.openstack.org/39935503:32
openstackgerritjianyi proposed openstack/kolla-ansible: Add zun ansible role  https://review.openstack.org/41617903:34
*** hfu has quit IRC03:34
*** zhurong has quit IRC03:43
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Optimize reconfigure action for nova  https://review.openstack.org/40741603:47
*** salv-orlando has joined #openstack-kolla03:48
*** salv-orlando has quit IRC03:52
*** hfu has joined #openstack-kolla03:56
*** hfu has quit IRC04:06
sdakeJeffrey4l_ i did review04:07
sdakeit lgtm however i am unclear on the unreleased.html thing04:07
sdakeJeffrey4l_ the release core reviewers will know what is correct04:07
Jeffrey4l_roger. i think, during developing, release-notes should be "unreleased.html" and should move to ocata.html at release.04:08
sdakeJeffrey4l_ cool - the release team can review - they will know for certain04:09
sdakeJeffrey4l_ i did add a note tho to the review04:09
Jeffrey4l_i see. thanks.04:09
*** sudipto has quit IRC04:09
*** sudipto_ has quit IRC04:09
sdakeJeffrey4l_ can you review this plz https://review.openstack.org/#/c/423633/04:12
Jeffrey4l_sdake, done.04:13
sdakeJeffrey4l_ tx04:13
openstackgerritcaoyuan proposed openstack/kolla-ansible: Fix mysql database user of ceilometer mysql  https://review.openstack.org/42374404:16
*** caowei has quit IRC04:24
openstackgerritcaoyuan proposed openstack/kolla-ansible: Fix mysql database user of ceilometer  https://review.openstack.org/42374404:32
*** sp___ has quit IRC04:34
openstackgerritMerged openstack/kolla-ansible: Update the upgrade.yml for trove  https://review.openstack.org/42314204:37
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for gnocchi  https://review.openstack.org/42313304:48
*** salv-orlando has joined #openstack-kolla04:49
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for gnocchi  https://review.openstack.org/42313304:53
*** salv-orlando has quit IRC04:53
openstackgerritMerged openstack/kolla: Install karbor dashboard into horizon image  https://review.openstack.org/42357504:54
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for searchlight  https://review.openstack.org/42359004:57
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for barbican  https://review.openstack.org/42372805:02
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for barbican  https://review.openstack.org/42372805:05
*** saneax-_-|AFK is now known as saneax05:07
openstackgerritMerged openstack/kolla: Kubernetes: Rename Images to allow use with native deployment tools  https://review.openstack.org/42363305:10
*** sacharya has joined #openstack-kolla05:11
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for aodh  https://review.openstack.org/42367005:11
*** yangzb09 has joined #openstack-kolla05:13
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for cloudkitty  https://review.openstack.org/42251605:14
*** mnaser has quit IRC05:15
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for senlin  https://review.openstack.org/42333605:16
*** caowei has joined #openstack-kolla05:18
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for senlin  https://review.openstack.org/42333605:19
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for cloudkitty  https://review.openstack.org/42251605:19
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for aodh  https://review.openstack.org/42367005:20
*** sdake has quit IRC05:21
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for barbican  https://review.openstack.org/42372805:21
*** mnaser has joined #openstack-kolla05:24
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for gnocchi  https://review.openstack.org/42313305:25
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for barbican  https://review.openstack.org/42372805:27
*** goldyfruit has joined #openstack-kolla05:37
*** mnaser has quit IRC05:49
*** salv-orlando has joined #openstack-kolla05:49
*** salv-orlando has quit IRC05:54
*** mnaser has joined #openstack-kolla05:57
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for ceilometer  https://review.openstack.org/42360605:58
*** sdake has joined #openstack-kolla06:05
*** trinaths has joined #openstack-kolla06:09
*** SridharP has joined #openstack-kolla06:10
*** SridharP1 has joined #openstack-kolla06:19
*** SridharP has quit IRC06:21
*** saneax is now known as saneax-_-|AFK06:22
*** goldyfruit has quit IRC06:23
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for ceilometer  https://review.openstack.org/42360606:28
openstackgerritDuong Ha-Quang proposed openstack/kolla-kubernetes: Horizon service package  https://review.openstack.org/42301606:31
openstackgerritYao Lu proposed openstack/kolla: Change docker restart policy  https://review.openstack.org/42302606:34
*** zhurong has joined #openstack-kolla06:37
openstackgerritWei Cao proposed openstack/kolla-ansible: Add freezer ansible role  https://review.openstack.org/40222406:40
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for gnocchi  https://review.openstack.org/42313306:41
*** zhurong has quit IRC06:42
*** zhurong has joined #openstack-kolla06:42
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for barbican  https://review.openstack.org/42372806:46
*** salv-orlando has joined #openstack-kolla06:50
*** zhubingbing has joined #openstack-kolla06:53
*** salv-orlando has quit IRC06:55
*** hachi has joined #openstack-kolla07:01
openstackgerrityuhui proposed openstack/kolla: Added oslo_log replacing logging library,more file  https://review.openstack.org/42119907:02
openstackgerrityuhui proposed openstack/kolla: Added oslo_log replacing logging library,more file  https://review.openstack.org/42119907:03
*** yangzb09_ has joined #openstack-kolla07:13
*** yangzb09 has quit IRC07:16
*** saneax-_-|AFK is now known as saneax07:27
openstackgerrityuhui proposed openstack/kolla: Added oslo_log replacing logging library,more file  https://review.openstack.org/42119907:31
*** sacharya has quit IRC07:33
openstackgerrityuhui proposed openstack/kolla: Added oslo_log replacing logging library,more file  https://review.openstack.org/42119907:33
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: [TEST]  https://review.openstack.org/42376607:33
*** bmace_ has quit IRC07:47
*** bmace_ has joined #openstack-kolla07:47
*** sp___ has joined #openstack-kolla07:50
*** salv-orlando has joined #openstack-kolla07:51
*** salv-orlando has quit IRC07:56
*** SridharP1 has quit IRC07:58
*** SridharP has joined #openstack-kolla08:00
*** sdake has quit IRC08:02
*** harish23r has joined #openstack-kolla08:02
zhubingbinghello guys08:02
zhubingbinghttps://review.openstack.org/#/c/423573/08:02
zhubingbingwho can help me look it ?08:02
harish23rhey guys!08:03
harish23rcan anyone help me out with my all in one deployment error? while using "kolla-ansible deploy" command08:03
harish23rThis is my error:08:04
harish23rTASK: [common | starting heka container] *************************************08:04
harish23rfailed: [localhost] => {"changed": true, "failed": true}08:04
harish23rmsg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.25/conteiners/create?name=heka',),)08:04
harish23rFATAL: all hosts have failed -- aborting08:04
harish23rlocalhost: 0k=28 changed=0 unreachable=0 failed=108:04
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for gnocchi  https://review.openstack.org/42313308:08
Jeffrey4l_harish23r, did it works before?08:13
harish23rnope.08:13
Jeffrey4l_try use "docker pull" to pull the image, does it work?08:13
*** sdake has joined #openstack-kolla08:14
harish23ryes. the registry operation works08:14
Jeffrey4l_harish23r, try downgrade the docker-engine to 1.12.6, kolla do not support docker-engine 1.13 now.08:14
harish23rhow did you identify my version?08:15
harish23rbtw, i tried both versions. 1.12 and 1.13. same result.08:15
Jeffrey4l_harish23r, you are using 1.25 api. which provides by docker 1.13.08:16
Jeffrey4l_hrm. how about docker run, does it work?08:16
harish23rI see. But both gave the same error. while starting heka container.08:16
harish23rdocker run works fine.08:16
harish23ri used docker run to start the registry.08:16
openstackgerritMerged openstack/kolla-ansible: Add Karbor doc link in README.rst  https://review.openstack.org/42299508:17
harish23rWhat does the error actually mean? its a docker error right?08:17
Jeffrey4l_could u get more specific error? check the docker logs.08:17
harish23ron a different machine now.08:17
Jeffrey4l_i do not know. docker engine just return that.08:17
harish23ri ran the command: "kolla-ansible deploy"08:18
harish23rit shows many lines and finally stopped with that error.08:18
openstackgerritWei Cao proposed openstack/kolla: Add dragonflow container  https://review.openstack.org/41819308:18
Jeffrey4l_check the docker.log / journalctl -u docker -f , to check what's going on.08:18
harish23rokay.08:19
harish23ri tried it earlier. but im very new to docker & kolla08:20
harish23rcouldn't understand much08:20
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Optimize reconfigure action for nova  https://review.openstack.org/40741608:21
Jeffrey4l_harish23r, try to get more error message .08:22
Jeffrey4l_from docker engine's log08:22
Jeffrey4l_show me you "docker info" harish23r08:22
harish23rjust on another machine now. i'll get back soon!08:22
harish23rthanks Jeffrey41_08:23
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for barbican  https://review.openstack.org/42372808:23
Jeffrey4l_sdake, could u ack this one https://review.openstack.org/42373908:23
sdakeJeffrey4l_ acked08:24
Jeffrey4l_thanks.08:24
sdakeJeffrey4l_ did release notes not previously render properly?08:25
Jeffrey4l_yep. at least the format is OK.08:25
Jeffrey4l_but i have no idea why 3.0.0.0bx still on the current series releaset note sections.  http://docs.openstack.org/releasenotes/kolla/08:26
Jeffrey4l_sdake,08:26
*** williamcaban has quit IRC08:27
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for ceilometer  https://review.openstack.org/42360608:31
openstackgerrityuhui proposed openstack/kolla: Added oslo_log replacing logging library,more file  https://review.openstack.org/42119908:33
*** williamcaban has joined #openstack-kolla08:36
harish23rbtw, Jeffrey41_ , I'm using a local registry. I've build and pushed the required images to the registry.08:36
harish23rdocker images, lists the images in the registry.08:36
harish23rdo you want me to check if "docker pull" works with the local registry?08:37
*** williamcaban has quit IRC08:40
openstackgerritWei Cao proposed openstack/kolla-ansible: Load karbor dashboard  https://review.openstack.org/42357608:41
Jeffrey4l_harish23r, yep. need check whether docker works without kolla.08:45
*** saneax is now known as saneax-_-|AFK08:46
sp___sdake: Jeffrey4l_   I know you both have crunch of time. but it would be great if you review this for BP completion of kolla coverage.  https://review.openstack.org/#/c/422444/08:49
Jeffrey4l_sorry. i will review it now08:49
sp___Jeffrey4l_: np, tx08:50
*** sayantani01 has quit IRC08:52
*** salv-orlando has joined #openstack-kolla08:52
*** sdake has quit IRC08:53
*** salv-orlando has quit IRC08:57
*** harish23r has quit IRC09:00
Jeffrey4l_sp___, done. some nit.09:00
sp___Jeffrey4l_: tx09:00
Jeffrey4l_np.09:00
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for mistral  https://review.openstack.org/42378709:06
*** lamt has quit IRC09:17
openstackgerritZeyu Zhu proposed openstack/kolla-ansible: Use required_if in kolla_docker.py  https://review.openstack.org/42378809:25
openstackgerritMerged openstack/kolla: Fix add kubernetes image release note format  https://review.openstack.org/42373909:30
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Optimize reconfigure action for nova  https://review.openstack.org/40741609:32
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for mistral  https://review.openstack.org/42378709:40
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for mistral  https://review.openstack.org/42378709:48
*** zhurong has quit IRC09:49
openstackgerritZeyu Zhu proposed openstack/kolla-ansible: Use required_if in kolla_docker.py  https://review.openstack.org/42378809:50
*** salv-orlando has joined #openstack-kolla09:53
openstackgerritSridhar Pothuganti proposed openstack/kolla-kubernetes: Correction of file path in the developer environment page  https://review.openstack.org/42379309:55
*** salv-orlando has quit IRC09:58
openstackgerrityuhui proposed openstack/kolla: Added oslo_log replacing logging library,more file  https://review.openstack.org/42119909:58
*** Syffs has joined #openstack-kolla10:02
*** williamcaban has joined #openstack-kolla10:03
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for mistral  https://review.openstack.org/42378710:05
*** williamcaban has quit IRC10:16
*** yangzb09_ has quit IRC10:22
trinathsHi10:25
trinathsWhile setting up the kolla-kibernetes development evironment, following the guide https://github.com/openstack/kolla-kubernetes/blob/master/doc/source/development-environment.rst#managing-and-interacting-with-the-environment , I got the following error, http://paste.openstack.org/show/595964/ . Please help me resolve the issue. I'm setting up the environement on Ubunut 14.04.5. Internet is connected to eth1 interface of the box.10:28
trinathskube1: Configuring and enabling network interfaces...10:28
trinathsis giving the error.10:28
trinathsthanks in advance for the help10:29
*** haplo37_ has quit IRC10:37
*** haplo37_ has joined #openstack-kolla10:37
*** zhubingbing has quit IRC10:47
*** zhubingbing has joined #openstack-kolla10:48
openstackgerritSurya Prakash Singh proposed openstack/kolla: Code coverage improvement for kolla  https://review.openstack.org/42244410:51
*** pq_ has quit IRC10:51
*** salv-orlando has joined #openstack-kolla10:54
*** salv-orlando has quit IRC10:58
openstackgerritWei Cao proposed openstack/kolla-ansible: Fix solum precheck  https://review.openstack.org/42379811:05
*** caowei has quit IRC11:06
trinathsthe virtualbox version is 4.311:07
*** williamcaban has joined #openstack-kolla11:13
*** williamcaban has quit IRC11:17
trinathsMoving forward, found another issue, with kubectl.11:29
trinathsroot@openstack-pike:/tmp/halcyon-vagrant-kubernetes# kubectl run -i -t $(uuidgen) --image=busybox --restart=Never11:29
trinathserror: Error loading config file "/root/.kube/config": yaml: mapping values are not allowed in this context11:29
trinathsdid I miss something11:29
trinaths?11:29
*** salv-orlando has joined #openstack-kolla11:54
*** masber has quit IRC11:57
*** masber has joined #openstack-kolla11:58
*** yangzb09_ has joined #openstack-kolla12:18
trinathswith CentOS it all went well. But MODULE_ARGS failed at Ansible.12:23
trinathsHere is the error http://paste.openstack.org/show/595971/12:24
trinathsIts an ansbile error. I'm unable to solve it.12:24
trinathsany solution to it ?12:25
zhubingbingok12:29
zhubingbingtrinaths  i look it12:30
zhubingbingyou can upgrade ansible version12:30
*** sp___ has quit IRC12:31
*** mgoddard has joined #openstack-kolla12:33
*** williamcaban has joined #openstack-kolla12:34
*** williamcaban has quit IRC12:39
*** sudipto_ has joined #openstack-kolla12:44
*** sudipto has joined #openstack-kolla12:44
*** salv-orlando has quit IRC12:55
*** tyrola has quit IRC12:57
*** tyrola has joined #openstack-kolla12:59
*** SridharP has quit IRC13:00
*** zhubingbing has quit IRC13:03
*** salv-orlando has joined #openstack-kolla13:04
openstackgerritWei Cao proposed openstack/kolla-ansible: Correct the groups for solum in haproxy.cfg.j2  https://review.openstack.org/42040713:10
*** SridharP has joined #openstack-kolla13:10
*** mgoddard has quit IRC13:10
trinathsansible version is 2.2.013:11
trinathsto which version I need to upgrade13:11
*** SridharP has quit IRC13:11
trinaths?13:11
*** esmiurium has quit IRC13:17
*** hieulq_ has joined #openstack-kolla13:18
*** hieulq_ has quit IRC13:21
*** esmiurium has joined #openstack-kolla13:24
*** zhubingbing has joined #openstack-kolla13:27
trinathszhubingbing: Hi13:29
zhubingbinghi13:29
trinathszhubingbing: the ansible version is 2.2.013:29
zhubingbingdocker version ?13:30
*** kristian__ has joined #openstack-kolla13:30
zhubingbingyou deploy k8s?13:30
zhubingbingsorry, I have not been deployed kolla-k8s13:31
zhubingbing;(13:31
trinathszhubingbing: there is an issue with docker in my system. Please check this http://paste.openstack.org/show/595973/13:31
zhubingbing;)13:32
zhubingbingi know13:32
trinathszhubingbing: so, how solve the docker issue13:32
zhubingbingapt-get remove docker-engine13:32
trinathszhubingbing: there is no such package installed, http://paste.openstack.org/show/595974/13:33
*** kristian__ has quit IRC13:33
zhubingbingapt-get remove docker13:34
trinathszhubingbing: done13:34
zhubingbingapt-get -y install docker-engine13:34
trinathszhubingbing: doing...13:36
zhubingbingok13:36
zhubingbingwait finish13:37
trinathszhubingbing: Docker version 1.13.0, build 49bf47413:39
zhubingbingok13:39
zhubingbingdocker  info13:39
zhubingbingyou can continue deploy kolla-k8s13:40
trinathszhubingbing: http://paste.openstack.org/show/595975/13:40
zhubingbingthere is an issue with docker ,i think it have been solove13:40
trinathsI;m following the documentation at: https://github.com/openstack/kolla-kubernetes/blob/master/doc/source/development-environment.rst#managing-and-interacting-with-the-environment13:40
zhubingbing;)13:40
zhubingbinghmm13:41
trinathszhubingbing: at $> vagrant up , I got these errors13:41
zhubingbingyou can continue working for deploy13:41
trinathszhubingbing: ok :)13:42
zhubingbingsorry, I have not been deployed kolla-k8s13:42
zhubingbing8/500013:43
zhubingbingI will go on to deploy kolla-K8S ;)13:43
trinathszhubingbing: okay. 8/5000 ?13:43
zhubingbingI sent the wrong message with 8/500013:44
zhubingbing;)13:44
trinathszhubingbing: :D13:44
zhubingbing;)13:47
*** sudipto has quit IRC13:49
*** sudipto_ has quit IRC13:49
*** mgoddard has joined #openstack-kolla13:51
*** sp___ has joined #openstack-kolla13:52
*** williamcaban has joined #openstack-kolla13:56
openstackgerritWei Cao proposed openstack/kolla: Fix solum dashboard into horizon image.  https://review.openstack.org/42383713:58
trinathszhubingbing: still the same error with ansible14:00
trinathszhubingbing: fatal: [kube1]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Shared connection to 127.0.0.1 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_QYND1A/ansible_module_command.py\", line 212, in <module>\r\n    main()\r\n  File \"/tmp/ansible_QYND1A/ansible_module_command.py\", line 90, in main\r\n    module = CommandModule(argument_spec=dict())\r\n  File \"/tmp/ansible14:00
*** williamcaban has quit IRC14:00
zhubingbing;) you can install docker-engine 1.2014:01
zhubingbingyou can try it14:01
trinathsoh okay14:01
trinathszhubingbing: there is not version 1.20 for ubuntu14:05
trinathszhubingbing: i'm using Ubuntu 14.04.514:05
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Optimize reconfigure action for nova  https://review.openstack.org/40741614:06
sp___Jeffrey4l_:  incorporated your comments please review it again, Now seems ok to me :(  https://review.openstack.org/#/c/422444/914:07
openstackgerritWei Cao proposed openstack/kolla: Fix searchlight dashboard into horizon image.  https://review.openstack.org/42384514:09
*** zhubingbing has quit IRC14:09
*** caowei has joined #openstack-kolla14:10
*** zhubingbing has joined #openstack-kolla14:10
Jeffrey4l_sp___, done.14:13
sp___Jeffrey4l_:  thanks14:14
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Optimize reconfigure action for nova  https://review.openstack.org/40741614:14
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Add chrony ansible role  https://review.openstack.org/39935514:15
sbezverkJeffrey4l_: ping14:16
Jeffrey4l_sbezverk, pong14:16
sbezverkJeffrey4l_: do you see consistent gate failure on centos? ubuntu's jobs retrun success14:18
Jeffrey4l_sbezverk, no. any link?14:18
Jeffrey4l_or error message?14:18
sbezverkhttp://logs.openstack.org/16/423016/3/check/gate-kolla-kubernetes-deploy-centos-binary-2-iscsi-nv/eda3da3/console.html14:18
sbezverkJeffrey4l_: and it is the same error on all cenots14:19
sbezverklike repo busted14:19
*** caowei has quit IRC14:19
Jeffrey4l_yes. seems infra mirror is busted.14:19
Jeffrey4l_http://mirror.iad.rax.openstack.org/epel14:19
Jeffrey4l_may be a recheck works?14:20
sbezverkeffrey4l_: yah, will try in a bit..14:20
*** zhubingbing has quit IRC14:20
Jeffrey4l_otherwise, should ping openstack-infra team to check what going on.14:20
*** caowei has joined #openstack-kolla14:21
Jeffrey4l_i can download the file manually. seems it is OK now.14:21
sbezverkJeffrey4l_: thanks for confirming.. I will run recheck to see14:21
Jeffrey4l_np.14:22
openstackgerritWei Cao proposed openstack/kolla: Fix solum dashboard into horizon image.  https://review.openstack.org/42383714:25
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for octavia  https://review.openstack.org/42384614:29
*** hieulq_ has joined #openstack-kolla14:29
*** SridharP has joined #openstack-kolla14:31
*** caowei has quit IRC14:32
*** williamcaban has joined #openstack-kolla14:33
*** williamcaban has quit IRC14:38
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for senlin  https://review.openstack.org/42333614:39
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for cloudkitty  https://review.openstack.org/42251614:41
*** l4yerffeJ has quit IRC14:42
*** l4yerffeJ has joined #openstack-kolla14:42
*** mgoddard has quit IRC14:43
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for searchlight  https://review.openstack.org/42359014:44
*** hieulq_ has quit IRC14:46
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for tacker  https://review.openstack.org/42203614:48
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon  https://review.openstack.org/33399614:50
*** breitz has quit IRC14:52
*** breitz has joined #openstack-kolla14:52
*** sdake has joined #openstack-kolla14:53
*** mgoddard has joined #openstack-kolla14:55
*** sdake_ has joined #openstack-kolla14:56
*** Mech422 has quit IRC14:57
*** sdake has quit IRC14:58
*** Jeffrey4l_ has quit IRC15:00
openstackgerritcaoyuan proposed openstack/kolla-ansible: Add freezer ansible role  https://review.openstack.org/40222415:02
*** n0isyn0i1e is now known as n0isyn0ise15:10
*** schwicht has joined #openstack-kolla15:16
openstackgerritSurya Prakash Singh proposed openstack/kolla: Code coverage improvement for kolla  https://review.openstack.org/42244415:16
*** SridharP has quit IRC15:22
*** salv-orlando has quit IRC15:24
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon  https://review.openstack.org/33399615:29
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Optimize reconfiguration for tacker  https://review.openstack.org/42203615:31
openstackgerritSurya Prakash Singh proposed openstack/kolla: Code coverage improvement for kolla  https://review.openstack.org/42244415:34
sdake_morning folks15:37
kfox1111morning.15:38
sdake_kfox1111 i was thinking about something last night - and wanted confirmation from you specifically15:38
sdake_do you only test kolla-kubernetes in the gate, or do you also test outside of the gate?15:38
kfox1111right now, my focus has been on the gate. sbezverk has been testing outside the gate.15:39
sdake_kfox1111 cool - I see why you said  that increasing the gate timers would slow down your productivity15:39
kfox1111I wana make sure the base set of objects is sold and it doesnt' regress.15:39
kfox1111yeah.15:39
sdake_kfox1111 generally developing with a dependence on the gate is slow in general15:39
kfox1111yeah.15:39
kfox1111but, it has lead to a much improved gate too.15:40
sdake_most devs use both tehcniques, test locally then gate15:40
kfox1111so kind of a tradeoff.15:40
kfox1111yeah.15:40
sdake_agree, our gate is totally kick ass15:40
sdake_by forcing yourself to only test against the gate, you have (mostly alone:) done a great job getting the gate into rockin shape15:40
sdake_the kolla-kubernetes gate is superior to other gating in kolla15:40
kfox1111if I hadn't had to debug a bunch of problems from within the gate, the gate logging wouldn't be so helpful now. :)15:40
sdake_not that its a contest, just a reality15:40
kfox1111yeah.15:41
kfox1111an unintended sideaffect of depending on the gate for dev.15:41
kfox1111interesting lesson learned.15:41
sp___sdake_: morning15:43
*** jmontleon has quit IRC15:47
*** jmontleon has joined #openstack-kolla15:49
sdake_sup sp___15:49
sdake_kfox1111 yup - although I think we want to get eveyrone into a position where they can develop locally or inside the gate15:50
*** duonghq has joined #openstack-kolla15:50
duonghqevening guys15:50
sdake_sup duonghq15:50
kfox1111sdake_: totally agree.15:50
kfox1111just hasn't been a focus for anyone that has been contributing code. :/15:51
sdake_kfox1111 ok so hate to rehash the past; our conversation from last night, you believe iscsi is the fastest path to getting there?15:51
sdake_its a focus for me :)15:51
kfox1111yeah. if you want something asap, thats probably the fastest way there.15:51
sdake_kfox1111 after our conversatoin trailed off last night i finished most of the job on tech support for my parents15:52
kfox1111yeah. and I'm really grateful your working on it. :)15:52
sdake_kfox1111 but then PTFOed as didnt' have energy to take a look at deploying kolla-kubernetes with the iscsi gate15:53
sdake_I recall when doing the ceph work, there is a hardcode IP address in the gate jobs15:53
sdake_(or bringing up ceph that is)15:53
sdake_then I found this review piling around:15:53
kfox1111no worries. only so much a single person can do. :)15:53
kfox1111the gate hard codes some of its config in its config files, as they don't changne in the gate.15:54
sdake_https://review.openstack.org/#/c/405720/15:54
sdake_kfox1111 tell me about it15:54
kfox1111but the config files are changable.15:54
sdake_the config files come from globals.yaml?15:54
sdake_or somewhere else15:54
duonghqsbezverk, are you there?15:54
sdake_unwrapping the union here15:54
sdake_I think that review above definately doesn't work but is on the right track15:55
sdake_I was going to use it as a starting point15:55
kfox1111yeha, the configs mostly kolla/globals.yaml and a bit of kolla-kubernetes.yaml15:55
kfox1111the gate's versions are mostly in tests/config/15:55
kfox1111they are just appended to the trun versions.15:55
sdake_kfox1111 can you offer a review above that explains the steps that Pete went through - so we can get the gate operational fucntionality out of your brain into developer documentation?15:56
kfox1111yeah.15:56
sdake_cool - that I can work with :)15:56
*** jmontleon has quit IRC15:56
kfox1111ah. he's working on d usting off the docker dev env idea.15:56
kfox1111nice.15:56
sdake_kfox1111 just looking at pete review last night before i trialed off to lala sleep time, I think pete had assumed the globals.yaml wasn't the config file15:57
sdake_kfox1111 ya - I thnk thats acool idea, although I'd like to simplify it to outside of docker15:57
sdake_kfox1111 and possibly tackle a dev env inside docker later15:57
sdake_as in after the ptg15:57
sdake_kfox1111 I just want to get something going as soon as feasible :)15:58
kfox1111sdake_: when we figure out the right way to do configmap generation via helm,15:59
kfox1111we can drop the need for kolla-genconfig, which will eleminate the need for a python based dev environment.15:59
sdake_i dont know if we will do configmap generation via helm or via a separate tool15:59
kfox1111yeah. still gotta figure that out.15:59
sdake_has that been decided (the configmap generation)15:59
sdake_i don't think helm can really do configmap generation15:59
sdake_configmap generation is #2 in my list of things to sort out16:00
kfox1111I think its one of those thigns we may want to support a batteries included option?16:00
sdake_(for kolla, I have many lists;)16:00
kfox1111external configmap generation shoudl be supported for sure.16:00
kfox1111but maybe we support a betteries included set of helm configmap packages.16:00
sdake_i guess i dont understand enough about how helm works to know if helm could do configmap generation16:00
kfox1111I think I figured out a way to map helm yaml to openstack ini files generically.16:01
sdake_what i've seen of other repos is people place configmap.tpl files in their charts16:01
*** salv-orlando has joined #openstack-kolla16:01
kfox1111so theres the possibility to maybe do them unopinionatedly.16:01
*** v1k0d3n has quit IRC16:01
sdake_and then operators are expected the fork the repo and hand-edit the files16:01
sdake_i personally find this unappealing16:01
kfox1111yeah me too.16:01
sdake_what is the technique you had in mind for helm?16:02
kfox1111its anti helm in my optionion.16:02
kfox1111talking with the helm devs, they prefer the binary/source split, and users use binaries, not source.16:02
sdake_its not just anti-helm, its anti-oeprator16:02
*** jmontleon has joined #openstack-kolla16:02
kfox1111yeah. that forces operators to be dev/ops.16:02
kfox1111disalows pure ops.16:02
sdake_operators have enough on their plates already16:03
kfox1111I'm dev/ops, but I have  ops coworkers, so I totally undnerstand where they are coming from.16:03
sdake_kolla's job should be to make their lives easier not harder ;)16:03
kfox1111+116:03
kfox1111we can figure out a way to have binary packages for configmap generation.16:03
kfox1111that should make ops happy.16:03
sdake_ok well le can chat about that later so i don't have to reload all the state in my brain (and you too)16:03
kfox1111there shoudl be a way to do that I think.16:04
sdake_after the dev env is working16:04
kfox1111ok.16:04
sdake_if you could provide a review of pete's devenv shell scripts in particular16:04
sdake_(and also the docker code)16:04
sdake_with an eye to "whats in the repo today"16:04
sdake_vs "where we want it to be in the future"16:04
sdake_that would help me the most understand how to provide a  quickstart that works and is accurate16:05
sdake_it may not be quick, but it will be a start :)16:05
*** v1k0d3n has joined #openstack-kolla16:05
kfox1111yeah. I'm reviewing it now.16:05
sdake_thanks dude :)16:05
sdake_ok - i've got to go to my parents to finish tehe job on tedch support, however, whilethats finishing i willl ssh into my servers remotely from there and see if I can't get iscsi rolling from your review + code review of iscsi16:06
sdake_of the iscsi gate16:06
duonghqsrwilkers_, ping16:08
*** Jeffrey4l_ has joined #openstack-kolla16:09
*** sdake_ has quit IRC16:15
*** schwicht has quit IRC16:22
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for aodh  https://review.openstack.org/42367016:32
kfox1111portdirect: you awake?16:33
duonghqsrwilkers_, ping16:33
duonghqanybody have idea why the kolla-kubernetes's gate is unstable?16:33
kfox1111duonghq: what are you seeing?16:35
duonghqkfox1111,  in my ps https://review.openstack.org/#/c/423016/316:35
duonghqafter 3 recheck in a round, only 1 gate is red16:36
duonghqprevious check failed due to some component like mariadb cannot start16:36
duonghqboth init-element and svc16:36
kfox1111yeah... flaky docker hub. :/16:36
kfox1111it would be really nice to get a container cache going.16:36
duonghqyup, the last check fail is http://logs.openstack.org/16/423016/3/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/4e97eeb/console.html16:37
duonghqdial tcp: lookup registry-1.docker.io on 69.20.0.164:53: no such host16:37
duonghq16:37
kfox1111as we're increasing the number of jobs, the frequency of hub issues is going up.16:37
duonghqya16:37
duonghqdo you think the docker hub is overload?16:37
kfox1111not sure. they have been having some issues the last week or so with stability.16:38
kfox1111not sure if its a bug, or a ddos attack, or just too muich regular traffic.16:38
duonghqokay16:39
kfox1111yeah. I agree your ps looks like its passing all tests. just a flaky docker hub.16:41
duonghqso I'll recheck 1-2 times tomorrow16:42
kfox1111duonghq: the ps is looking very good. one minir issue I think and its good for merge.16:43
duonghqkfox1111, can you tell me16:43
kfox1111its in the review.16:44
duonghqkfox1111, just see, thanks16:44
kfox1111np. :)16:45
duonghqbut I think that it is just kind of soft dependency?16:45
kfox1111maybe. the user can always turn it back off. just thinking in the general case, the user won't want their users to see horizon before authentication works.16:46
kfox1111otherwise they might get support requests for "horizon is broken".16:46
duonghqso, does it need to put in requirements.yml?16:47
kfox1111I don't htink horizon gives useful error messages if its up and keystone is down.16:47
portdirecto/16:47
kfox1111in the values.yaml file with the other deps.16:47
kfox1111let me see if I can fidn the svc name for you.16:47
kfox1111portdirect: looking through your ps for the dev env. looking very nice. :)16:47
kfox1111wondering about forking the files though.16:47
portdirectwhich one?16:47
kfox1111one of the reasons minikube stuff bitrotted, is because I couldn't figure out how to get it into the gate, to prevent regressions.16:48
kfox1111https://review.openstack.org/#/c/405720/16:48
*** hachi has quit IRC16:48
kfox1111in that ps, you fork a few files that we've still been changing relatively often.16:48
kfox1111could we change it to add conditionals in the other files rather then forking?16:49
sbezverkgood morning16:49
kfox1111That way, its tested inside the gate for regressions, and less code to maintain.16:49
duonghqkfox1111, I think the service name is keystone-api?16:49
duonghqmorning sbezverk16:51
kfox1111duonghq: not sure. keystone is a little different. I'm looking through the code now.16:51
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Remove contrib directory  https://review.openstack.org/42255916:52
*** duonghq__ has joined #openstack-kolla16:56
*** sdake has joined #openstack-kolla16:56
kfox1111finally found it. its talkign to keystone-public dns entry.16:59
sbezverkkfox1111: duonghq: I think if you want to depend on keystone this service name should be used: keystone-admin16:59
kfox1111yeah, so keystone-public is the svc to depend on.16:59
sbezverkkfox1111: strange since horizon runs within a cluster why would it use public svc?17:00
kfox1111sbezverk: I thought so too, but http://logs.openstack.org/96/333996/14/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/1f4e1a1/logs/kolla/horizon/local_settings17:00
kfox1111and search for OPENSTACK_KEYSTONE_URL.17:00
sbezverkin some cases public ip might not be even reachable if it would go through proxy17:00
kfox1111I'd expect public ip's to always be reachable from the cluster?17:01
kfox1111not sure what woudl break if that wasn't the case.17:01
sbezverkkfox1111: nope, I publish "public name" which is not resolved by internal cluster dns17:02
*** yangzb09_ has quit IRC17:02
kfox1111ah.17:02
kfox1111that makes sense.17:02
kfox1111well, genconfig is doing something bad then.17:02
sbezverkthis separate makes sense for extra sec protection17:02
sbezverkkfox1111: very possible17:02
*** hachi has joined #openstack-kolla17:03
kfox1111sbezverk: can you please post a kolla-ansible ps to fix that? I think you know that process better then I do.17:04
duonghqso, keystone-admin is better solution?17:04
kfox1111keystone-internal I think.17:04
duonghqnice17:05
sbezverkkfox1111: internal would work too17:06
duonghqbut does it needs go to requirement yaml together, or just for k8s-entrypoint?17:06
sbezverkkfox1111: for kolla-ansible it might make sense17:06
portdirectkfox1111: sorry got caught up in something17:06
sbezverkbecaseu they do not "kubernetes cluster with internal dns"17:06
portdirectkfox1111: your suggestion makes total sense - and where i would have tried to go if id had the time to finish it17:07
portdirectkfox1111: i think ideally a developer should be able to work locally - and pretty much only push to gerrit when they expect it to pass the gate17:07
kfox1111portdirect: k.17:07
kfox1111agreed.17:07
portdirectkfox1111: if we get there, then every possible workflow is pretty much accounted for17:08
*** salv-orlando has quit IRC17:08
kfox1111so, to review that review, I'm going to have to do a local checkout and do a diff between the forked file and whats in the gate.17:08
portdirecti think sdake said he was interested in taking it over17:08
kfox1111if not much has changed, I could add the conditionals if your really busy?17:08
portdirectthat would be great17:08
kfox1111then we'd co-author it?17:09
portdirectyeah man :)17:09
sdakeportdirect take which over17:09
kfox1111ok. cool. didn't want to step on your toes.17:09
sdakesorry ws doing tech support day 2 operations :)17:09
sdakelet me read scrollback17:09
portdirectkfox1111: taking workload on is deffo not stepping on my toes :)17:09
sdakeportdirect this is less then a military op i've got rollign at my parents :)17:09
sdakekfox1111 - ya we work as a team so its not like anyone owns any particular part of the code base17:09
kfox1111portdirect: glad you feel that way. :) not everyone does. just checking.17:09
portdirectlol - things going south on yah17:10
sdakekfox1111 so - dont worry about toe bashing17:10
portdirectsdake: nuke it from orbit 'tis the only way to be sure17:10
*** sudipto has joined #openstack-kolla17:10
*** sudipto_ has joined #openstack-kolla17:10
sdake[10:08:24]  <kfox1111>so, to review that review, I'm going to have to do a local checkout and do a diff between the forked file and whats in the gate.17:11
kfox1111sdake: some devs would rather a ps be standaline, and just get reviews. just a procedural difference between devs. I get it. so just trying to play nice.17:11
* portdirect if kfox touches my ps... mumble mumble... somthing something..17:11
portdirect:)17:11
*** kbaegis has joined #openstack-kolla17:11
kfox1111yeah. that. :)17:11
sdakei was dc'ed driving  to my parents house so missed the context of the convo17:11
kfox1111sdake: the ps you pointed me at for review.17:11
sdakekfox1111 i didn't understand what you meant is why i pasted it17:11
portdirectsdake: kfox is gonna see how rotted it is17:12
sdakewas going to ask a followup if you could expand on what you meant :)17:12
kfox1111sdake: the ps forks some files that are used for the gate, and makes a few changes I think.17:12
sdakeoh that particular patch17:12
kfox1111its hard to see what changed, when the whole file changed.17:12
kfox1111to really review it, I'll have to checkout the ps, then do a diff on the forked files.17:12
sdakekfox1111 cool if you want to co-author on it i'm sure portdirect wouldn't mind17:12
sdakethe only person who really cares about co-authors is sambetts|afk17:12
sdakesamyaple that is17:12
sdakekfox1111 is that something you could do today - or no  time?17:13
portdirectkfox1111: from that you should be able waht we need to support halycon - but ideally it should be generic17:13
portdirectkfox1111: if you need anything just hit me up17:13
kfox1111portdirect: k. I may hit you up to test it.17:15
kfox1111I don't have a way to do so currently.17:15
*** sudipto has quit IRC17:15
*** sudipto_ has quit IRC17:15
kfox1111so I'll just do the diff, add the if's and repost.17:15
openstackgerritDuong Ha-Quang proposed openstack/kolla-kubernetes: Horizon service package  https://review.openstack.org/42301617:15
duonghqthank kfox111117:16
portdirectkfox1111: that would make my day :)17:16
kfox1111np. :)17:17
*** sayantani01 has joined #openstack-kolla17:17
sdakekfox1111 cool getting something up that doesn't work is better then getting nothing up :)17:17
kfox1111yup.17:19
kfox1111portdirect: with this ps, are we going to switch it to iscsi for now?17:20
portdirectkfox1111: I'm easy on that front17:20
kfox1111k.17:20
portdirectkfox1111: whatever is easiest17:20
kfox1111k.17:21
sbezverkportdirect: have you made iscsi and storageclass working?17:21
portdirectnever tried17:21
portdirectthough I'd look at the openshift docs17:21
kfox1111theres no infra in iscsi to provide iscsi for k8s though.17:22
portdirectas i think that the mostly use iscsi or gluster for on-prem17:22
*** v1k0d3n has quit IRC17:22
sbezverkkfox1111: what do you mean?17:22
kfox1111portdirect: who mostly use?17:22
portdirectopenshift17:22
sbezverkkfox1111: it supposed to work in theory, but I suspect there is a bug17:23
kfox1111sbezverk: the gate has stuff for getting cinder to provision iscsi stuff.17:23
kfox1111getting iscsi volumes provisioned without cinder is a lot more work.17:23
*** v1k0d3n has joined #openstack-kolla17:23
kfox1111more work I think then getting ceph stuff finsihed off.17:23
sbezverkkfox1111: ok please keep me in the loop, I would really like to see it working..17:24
sp___sdake:  it would be gr8 if you review  https://review.openstack.org/#/c/422444/1117:25
kfox1111sbezverk: will do. I don't have any plans on implementing non cinder iscsi provisioning though.17:25
*** goldyfruit has joined #openstack-kolla17:25
sbezverkkfox1111: for cinder to provide this functionaity, does it need other components like keystone and mariadb or it is some sort of a special version?17:27
portdirectsbezverk: i've used cinder for this purpose (as in that think i showd you yesterday) it would need a kesytone and dbs for both17:28
portdirectsbezverk: its possible to run without any auth - but yuck17:28
portdirect(also never tried)17:28
sbezverkportdirect: but then it is really lame solution, you depend on opensatck to bring kube cluster to run opensatck.. brrr I would neven use this crap17:29
portdirect+217:29
portdirectyeah - I need to run the ks and cinder db stroage outside of itself17:29
portdirecti manged to get them to self host - but it was to scary17:30
kfox1111and you would loose all ha support in the process. :/17:30
kfox1111as cinder doesn't support ha with iscsi.17:30
portdirectnot nesisarity - but its too much work - if you want to do that just use tripple-o17:30
sbezverkkfox1111: idk in this case I would just make efforts to get just iscsi with kube work, no cinder and all extra crap..17:31
kfox1111I gave up long ago trying to make iscsi ha without just paying a vendor for an appliance.17:31
kfox1111ceph handles that case usually much better.17:31
portdirectsbezverk: thats the approach that we should take17:32
kfox1111I think the best soltuion is to just use galera clustering and skip the pv's altogether.17:32
sbezverkkfox1111: with ceph you will always have less performance as there is no "ceph" enabled NIC17:32
portdirect+117:32
sbezverkwhich is different that with iscsi optimized nics17:32
*** mgoddard has quit IRC17:32
kfox1111sbezverk: true/false.17:32
kfox1111stroage is usually the bottleneck these days, not network.17:33
kfox1111and ceph can scale out much farther then the traditional iscsi appliance.17:33
* portdirect rabbitholes17:33
kfox1111hehe. yeah.17:33
*** portdirect is now known as portdirect_away17:33
sbezverkkfox1111: never had to do any study on scalability vs specialized hardware fr for storage, but nothing beats specially designed forwarding acics in router's/switches world..17:35
kfox1111sbezverk: true. well, mostly.17:36
kfox1111if you could have one switch with asics,17:36
kfox1111or 100x without, eventually if you have enough parallel load,17:36
kfox1111parallel will outperform single.17:37
sbezverkkfox1111: good luck with electricity bill ;)17:37
kfox1111thats why all hpc systems no longer build snowflake systems, but compute clusters. :)17:37
duonghqnight guys17:37
*** duonghq has quit IRC17:37
kfox1111hpc use to use custom built procesors/mb's/networks.17:38
kfox1111really fast, but bang for buck,17:38
kfox1111the economy of scale of general purpose computing made it much much cheeper per flop to buy a bunch of cheep machines and tie them together.17:38
sbezverkkfox1111: right17:38
kfox1111ceph I think is doing the same for storage.17:38
kfox1111vendors make some really awesome iscsi appliances.17:39
kfox1111but they are designed/priced like bigiron.17:39
sbezverkkfox1111: for computing, I would agree.. there are too too many variations of processing17:39
kfox1111ceph is built like compute clusters. but just for storage.17:39
*** sudipto_ has joined #openstack-kolla17:39
*** sudipto has joined #openstack-kolla17:39
kfox1111ceph doesn't solve all the worlds storage problems. but it seems to be in a good sweetspot for general storage stuff.17:40
*** sudipto has quit IRC17:41
*** sudipto_ has quit IRC17:41
kfox1111it grows cheeper as it expands.17:41
sbezverkkfox1111: I agree growing ceph is easier, but since it is going through linux stack, the performace will always be limiting17:42
kfox1111yeah, but that can be counteracted by suporting sharding of data across servers.17:42
kfox1111100 spiny disks will be faster then 10.17:43
sbezverkkfox1111: what is sharding?17:43
kfox1111say you have a 1tb file.17:43
kfox1111you  can break it into 1g chunks, and put each on a different server set.17:43
kfox1111then when you get the file back, you can stream all the chunks back in parallel.17:44
kfox1111so you can scale your io past any disk/server can do.17:44
kfox1111iscsi doesn't support that.17:44
sbezverkkfox1111: got it.. would it not create extra load when this 1tb file gets changed? curious which process controls sharding?17:45
kfox1111only the chunks that get changed get the io.17:45
kfox1111the sharding process is done by the client itself.17:46
kfox1111and somewhat the mons.17:46
sbezverkkfox1111: right but figuring out where they are located, takes cpu cycles17:46
kfox1111yeah. thats where the crushmap comes in.17:46
kfox1111its a multi level hashing mechanism.17:46
kfox1111very cleaver, and very fast.17:46
kfox1111thats cephs real claim to fame.17:47
sbezverkkfox1111: I see, interesting..17:47
kfox1111so it basically lets you scale out your storage io and disk space the same way hpc scales out computation.17:47
*** kbaegis has quit IRC17:48
*** sudipto_ has joined #openstack-kolla17:52
*** sudipto has joined #openstack-kolla17:52
*** kbaegis has joined #openstack-kolla17:53
sbezverkkfox1111: horizon check failed duonghq PS17:53
sbezverkI do not see much logging generated by that curl17:54
kfox1111sec...17:55
sbezverkkfox1111: but the url looks good to me..17:56
*** kbaegis has quit IRC17:57
kfox1111yeah. if I had to guess, I'd say kubernetes-entrypoint blocked it for a little bit?17:58
kfox1111the horizon check doesn't currently check to see if its ready yet.17:59
kfox1111so the added delay by the dependencies is not accounted for by the horizon test.17:59
kfox1111probably need a check loop.18:00
*** goldyfruit has quit IRC18:00
sbezverkkfox1111: horizon check has been merged?18:04
sbezverknvm my master was not up to date18:06
*** sudipto has quit IRC18:06
*** sudipto_ has quit IRC18:06
kfox1111sbezverk: I just posted a bit of code to the review I think will fix it.18:09
*** sacharya has joined #openstack-kolla18:09
sbezverkkfox1111: looks good18:11
sbezverkmaybe you should do it in a separate PS18:11
sbezverkthen we can try to merge and re-check18:12
kfox1111yeah. either way.18:12
sbezverkotherwise will have to wait at least 12 hours18:12
kfox1111do you have time to do it? I'm trying to help portdirect/sdake with getting the dev env fixed up a bit.18:12
sbezverkkfox1111: actually no rush. let's wait18:13
kfox1111ok.18:13
sdakeyo folks - jsut got done doing the copy phase of the critical files for my mother's computer18:28
sdakegoing to eat - since i had no food yesterday at all18:28
sdakewhen i'm back, i should be ready to rock on testing out the dev env and getting it rolling18:29
*** sp___ has quit IRC18:29
*** kbaegis has joined #openstack-kolla18:31
kfox1111sdake: I've got the tweaked ps just about ready to post.18:31
*** sdake_ has joined #openstack-kolla18:31
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm  https://review.openstack.org/40572018:33
*** sdake has quit IRC18:34
*** williamcaban has joined #openstack-kolla18:35
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm  https://review.openstack.org/40572018:36
*** dave-mccowan has joined #openstack-kolla18:39
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon  https://review.openstack.org/33399618:39
*** williamcaban has quit IRC18:39
*** williamcaban has joined #openstack-kolla18:40
*** trinaths has quit IRC18:47
*** wirehead_ has quit IRC18:51
*** wirehead_ has joined #openstack-kolla18:52
*** sdake_ has quit IRC19:00
*** v1k0d3n has quit IRC19:01
*** sdake has joined #openstack-kolla19:02
*** wirehead_ has quit IRC19:03
*** wirehead_ has joined #openstack-kolla19:04
*** sdake has quit IRC19:10
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon  https://review.openstack.org/33399619:13
*** sdake has joined #openstack-kolla19:18
*** wirehead_ has quit IRC19:19
*** wirehead_ has joined #openstack-kolla19:19
kfox1111gota head out for a few. bbiab.19:20
*** wirehead_ has quit IRC19:22
*** wirehead_ has joined #openstack-kolla19:23
*** v1k0d3n has joined #openstack-kolla19:24
*** wirehead_ has quit IRC19:27
*** wirehead_ has joined #openstack-kolla19:28
*** wirehead_ has quit IRC19:32
*** wirehead_ has joined #openstack-kolla19:33
*** goldyfruit has joined #openstack-kolla19:33
*** wirehead_ has quit IRC19:37
*** wirehead_ has joined #openstack-kolla19:38
*** lamt has joined #openstack-kolla19:39
*** kbaegis has quit IRC19:40
*** v1k0d3n has quit IRC19:42
*** v1k0d3n has joined #openstack-kolla19:43
*** v1k0d3n has quit IRC19:43
*** Syffs has quit IRC19:49
*** lamt has quit IRC19:53
*** duonghq__ has quit IRC19:54
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon  https://review.openstack.org/33399619:59
*** dave-mccowan has quit IRC20:07
*** wirehead_ has quit IRC20:11
*** apuimedo has quit IRC20:12
*** wirehead_ has joined #openstack-kolla20:12
*** Serlex has joined #openstack-kolla20:12
*** apuimedo has joined #openstack-kolla20:13
*** apuimedo has quit IRC20:20
*** apuimedo has joined #openstack-kolla20:20
*** apuimedo has quit IRC20:25
*** v1k0d3n has joined #openstack-kolla20:25
*** v1k0d3n has quit IRC20:25
*** v1k0d3n has joined #openstack-kolla20:26
*** apuimedo has joined #openstack-kolla20:27
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon  https://review.openstack.org/33399620:29
*** apuimedo has quit IRC20:31
*** apuimedo has joined #openstack-kolla20:33
*** kbaegis has joined #openstack-kolla20:34
kfox1111sdake: https://review.openstack.org/#/c/422950/ ?20:34
*** sdake has quit IRC20:36
*** apuimedo has quit IRC20:38
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP - local registry in gates  https://review.openstack.org/41855020:38
*** kbaegis has quit IRC20:38
*** goldyfruit has quit IRC20:39
*** apuimedo has joined #openstack-kolla20:40
*** kbaegis has joined #openstack-kolla20:43
*** apuimedo has quit IRC20:44
*** apuimedo has joined #openstack-kolla20:46
*** schwicht has joined #openstack-kolla20:47
*** lamt has joined #openstack-kolla20:48
*** apuimedo has quit IRC20:51
*** apuimedo has joined #openstack-kolla20:54
*** severion has joined #openstack-kolla20:55
*** severion has quit IRC20:56
*** apuimedo has quit IRC20:58
*** severion has joined #openstack-kolla20:59
*** v1k0d3n has quit IRC20:59
*** severion has quit IRC20:59
*** v1k0d3n has joined #openstack-kolla20:59
*** apuimedo has joined #openstack-kolla21:01
*** apuimedo has quit IRC21:08
*** sacharya has quit IRC21:08
*** apuimedo has joined #openstack-kolla21:09
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon  https://review.openstack.org/33399621:09
*** apuimedo has quit IRC21:13
*** apuimedo has joined #openstack-kolla21:16
*** apuimedo has quit IRC21:21
*** apuimedo has joined #openstack-kolla21:24
*** nihilifer has quit IRC21:27
*** nihilifer has joined #openstack-kolla21:28
*** apuimedo has quit IRC21:31
*** l4yerffeJ_ has joined #openstack-kolla21:34
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon  https://review.openstack.org/33399621:34
*** apuimedo has joined #openstack-kolla21:34
*** Jeffrey4l_ has quit IRC21:34
*** l4yerffeJ has quit IRC21:35
*** Jeffrey4l_ has joined #openstack-kolla21:35
*** sdake has joined #openstack-kolla21:46
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon  https://review.openstack.org/33399622:05
*** mgoddard has joined #openstack-kolla22:07
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP - local registry in gates  https://review.openstack.org/41855022:17
*** kbaegis has quit IRC22:23
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Support rolling upgrades of horizon  https://review.openstack.org/33399622:28
*** mgoddard has quit IRC22:32
*** kbaegis has joined #openstack-kolla22:33
*** Serlex has quit IRC22:37
*** salv-orlando has joined #openstack-kolla22:38
*** sdake has quit IRC22:40
*** salv-orlando has quit IRC22:47
*** kbaegis has quit IRC22:53
*** kbaegis has joined #openstack-kolla22:54
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP - local registry in gates  https://review.openstack.org/41855022:56
*** masber has quit IRC23:03
*** masber has joined #openstack-kolla23:03
*** saneax-_-|AFK is now known as saneax23:04
*** masber has quit IRC23:06
*** williamcaban has quit IRC23:06
*** masber has joined #openstack-kolla23:07
*** williamcaban has joined #openstack-kolla23:11
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Support rolling upgrades of horizon  https://review.openstack.org/33399623:13
*** jrobinson has joined #openstack-kolla23:15
*** williamcaban has quit IRC23:16
*** sdake has joined #openstack-kolla23:22
openstackgerritDan Prince proposed openstack/kolla: Allow mariadb log directory to be configurable  https://review.openstack.org/42386823:22
*** kbaegis has quit IRC23:25
*** williamcaban has joined #openstack-kolla23:29
*** lamt has quit IRC23:31
*** williamcaban has quit IRC23:34
sdakeJeffrey4l_ arey ou about23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!