Tuesday, 2017-04-04

« Monday, 2017-04-03 Index Wednesday, 2017-04-05 »
*** jtriley has joined #openstack-kolla00:05
*** shashank_t_ has quit IRC00:08
*** yangyape_ has quit IRC00:14
*** caoyuan has joined #openstack-kolla00:17
*** caoyuan_ has joined #openstack-kolla00:19
*** japestinho has quit IRC00:20
*** caoyuan has quit IRC00:22
*** jtriley has quit IRC00:29
*** caoyuan has joined #openstack-kolla00:31
*** caoyuan has quit IRC00:31
*** caoyuan_ has quit IRC00:33
*** shashank_t_ has joined #openstack-kolla00:37
*** eaguilar has joined #openstack-kolla00:38
*** adrian_otto has joined #openstack-kolla00:41
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Upgrade gate to kubernetes 1.6.1  https://review.openstack.org/45299300:41
*** tovin07_ has joined #openstack-kolla00:48
*** eaguilar has quit IRC00:52
sbezverkto restore gates, please merge this ps https://review.openstack.org/45299300:57
*** qiliang27 has joined #openstack-kolla00:57
sdakesbezverk what do you make of this: http://logs.openstack.org/50/452850/9/check/gate-kolla-kubernetes-deploy-centos-binary-2-external-ovs-nv/8028cce/console.html#_2017-04-03_23_43_05_19665800:57
sdakesbezverk will ack once gates wrap up00:58
*** yangyapeng has joined #openstack-kolla01:04
*** caoyuan has joined #openstack-kolla01:04
*** adrian_otto has quit IRC01:04
*** eanylin has quit IRC01:05
*** yangyapeng has quit IRC01:06
*** duonghq has joined #openstack-kolla01:13
*** shashank_t_ has quit IRC01:14
duonghqmorning guys01:14
openstackgerritcaoyuan proposed openstack/kolla master: Update min Ansible version to 2.2.0  https://review.openstack.org/45252901:17
*** dave-mccowan has quit IRC01:17
openstackgerritMerged openstack/kolla-ansible master: Fix the check.yml of Zun does not exist  https://review.openstack.org/45203801:18
openstackgerritcaoyuan proposed openstack/kolla master: Update min Ansible version to 2.2.0  https://review.openstack.org/45252901:19
spsuryaduonghq: morning01:25
duonghqhi spsurya01:26
*** eaguilar has joined #openstack-kolla01:27
*** jpstnh has joined #openstack-kolla01:27
*** eanylin has joined #openstack-kolla01:32
*** goldyfruit has joined #openstack-kolla01:34
*** unicell has quit IRC01:40
*** iceyao has joined #openstack-kolla01:50
openstackgerritDuong Ha-Quang proposed openstack/kolla master: Install InfluxDB Plugin in Grafana  https://review.openstack.org/44662401:55
*** harlowja has quit IRC02:06
*** adrian_otto has joined #openstack-kolla02:09
*** qiliang27 has quit IRC02:17
openstackgerritcaoyuan proposed openstack/kolla-kubernetes master: Fix the repository link for README.rst  https://review.openstack.org/45300802:18
*** adrian_otto has quit IRC02:23
duonghqhrw, hi02:23
openstackgerritcaoyuan proposed openstack/kolla master: Fix the repository link for README.rst  https://review.openstack.org/45300902:24
*** adrian_otto has joined #openstack-kolla02:25
*** tvignaud has quit IRC02:26
*** sayantan_ has joined #openstack-kolla02:31
duonghqhrw, ping02:31
*** tvignaud has joined #openstack-kolla02:39
*** adrian_otto has quit IRC02:47
sdakesup duonghq02:49
*** harlowja has joined #openstack-kolla02:50
*** goldyfruit has quit IRC02:52
duonghqhi sdake02:58
duonghqhow about your last weekend02:59
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/mariadb  https://review.openstack.org/45285503:00
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Modify WARNING into RST format  https://review.openstack.org/45286003:00
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/rabbitmq  https://review.openstack.org/45285803:00
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/skydns  https://review.openstack.org/45285903:00
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/memcached  https://review.openstack.org/45285603:00
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/neutron  https://review.openstack.org/45285703:00
sdakeduonghq busy with housework03:01
*** sayanta__ has joined #openstack-kolla03:01
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Upgrade gate to kubernetes 1.6.1  https://review.openstack.org/45299303:02
*** sayantan_ has quit IRC03:04
duonghqsbezverk, so we're moving to k8s 1.6.1?03:10
sbezverkduonghq: yes, 1.6.1 has a major bug fix03:11
sdakelooks like it has major bugs ;-)03:12
sdakejust had a wierd flashback03:13
sdakei guess it was more like deja vu03:14
sdakei see kubeadm is using the same kubelet version #s03:16
sdakewhile kubeadm may have fixed somethign and the rpm packaging may have fixed something - not conviced whatever was broken in kubelet was fixed03:17
sdake# kubectl version03:19
sdakeClient Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}03:19
sdakeServer Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}03:19
*** eaguilar has quit IRC03:24
*** eaguilar has joined #openstack-kolla03:25
spsuryasdake: for https://review.openstack.org/#/c/447356/31/doc/source/deployment-guide.rst  +217         will this file initially be empty  /etc/kubernetes/manifests/kube-apiserver.yaml    before adding the insecure-port and insecure-bind-address03:33
spsurya?03:33
sdakespsurya no clue03:34
sdakespsurya are ou using 1.5 or 1.6?03:34
spsurya1.503:34
spsuryasdake:03:34
sdakestill no clue :)03:34
spsuryasdake: ok03:35
spsuryaeanylin: around ?03:35
eanylinspsurya: Yah, trying out 1.6.103:36
spsuryaeanylin: for https://review.openstack.org/#/c/447356/31/doc/source/deployment-guide.rst  +217         will this file initially be empty  /etc/kubernetes/manifests/kube-apiserver.yaml    before adding the insecure-port and insecure-bind-address03:37
eanylinspsurya: No, this file is populated when you run kubeadm03:37
eanylinthe insecure-port will be 0 initially03:38
eanylinI updated it to 808003:38
eanylinkubeadm init *03:38
*** rstarmer has joined #openstack-kolla03:39
spsuryaeanylin: correct file name should be /etc/kubernetes/manifests/kube-apiserver.json03:40
spsuryanot .yaml03:40
rstarmerhi folks.  Anyone using the LVM cinder backend? I'm not getting the configuration right, it keeps complaining that my backends aren't properly defined.03:40
eanylin[root@kube01 ~]# vi /etc/kubernetes/manifests/03:41
eanylinetcd.yaml                     kube-apiserver.yaml           kube-controller-manager.yaml  kube-scheduler.yaml03:41
eanylin[root@kube01 ~]#03:41
eanylinactually I am on 1.6.1, not sure if its the same as your 1.503:41
sdakeok - so - if 1.6.1 works03:41
sdakelets document 1.6.103:41
eanylinsdake: Yeah, I think sbezverk's commit should work, i.e. https://review.openstack.org/#/c/452993/3/tools/setup_kubernetes.sh03:42
spsuryahttps://www.irccloud.com/pastebin/i7jbOXqV/03:43
spsuryaeanylin: getting json one with 1.503:43
eanylinspsuyra: Ok, I been testing with higher versions03:43
eanylinI think I must have been on 1.5++ when I started few weeks back...03:44
spsuryaeanylin: what if i got this till line no. 172        Every 5.0s: kubectl get pods --all-namespaces                                                                                                               Tue Apr  4 03:48:16 201703:48
spsuryaNAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE03:48
spsuryakube-system   calico-policy-controller-qb62c             1/1   Running   0          1h03:48
spsuryakube-system   canal-etcd-6c2ft                           1/1   Running   0          1h03:48
spsuryakube-system   canal-node-gf9qc                           3/3   Running   0          1h03:48
spsuryakube-system   dummy-526455539-lsnpr                      1/1   Running   0          2h03:48
spsuryakube-system   etcd-ip-172-31-13-146                      1/1   Running   0          2h03:48
spsuryakube-system   kube-apiserver-ip-172-31-13-146            1/1   Running   0          2h03:48
spsuryakube-system   kube-controller-manager-ip-172-31-13-146   1/1   Running   0          2h03:48
spsuryakube-system   kube-discovery-4047383258-p3f23            1/1   Running   0          2h03:48
spsuryakube-system   kube-dns-2099893933-8d6zt                  3/4   Running   81         2h03:48
spsuryakube-system   kube-proxy-wvs9b                           1/1   Running   0          2h03:48
spsuryakube-system   kube-scheduler-ip-172-31-13-146            1/1   Running   0          2h03:48
spsuryaeanylin: seems like i just ran into 1.6 along with 1.5 ;)03:49
eanylinspsurya: Lol03:49
eanylin1.6.1 is running fine for me03:50
eanylin[root@kube01 ~]# kubectl get pods -n kube-system03:50
eanylinNAME                             READY     STATUS    RESTARTS   AGE03:50
eanylincalico-policy-controller-5wg5n   1/1       Running   0          16m03:50
eanylincanal-etcd-s84h1                 1/1       Running   0          16m03:50
eanylincanal-node-2r79r                 3/3       Running   0          16m03:50
eanylinetcd-kube01                      1/1       Running   0          19m03:50
eanylinkube-apiserver-kube01            1/1       Running   0          17m03:50
eanylinkube-controller-manager-kube01   1/1       Running   1          20m03:50
eanylinkube-dns-3913472980-20p42        3/3       Running   0          20m03:50
eanylinkube-proxy-4mqr6                 1/1       Running   0          20m03:50
eanylinkube-scheduler-kube01            1/1       Running   1          20m03:50
eanylin[root@kube01 ~]#03:50
eanylinspsurya: Your dns crashed 81 times, cant be good :D03:50
spsuryaeanylin: i see03:51
*** eaguilar has quit IRC03:52
spsuryaeanylin: with your controller  and scheduler got 1 too ;)03:53
spsuryais that ok ?03:53
eanylinspsurya: That is true :)03:53
eanylinspsurya: I think so.  Its not recurring restart. It restarted during initial startup03:54
spsuryaneed to add as comment too in doc03:54
spsuryaeanylin i think03:54
*** gkadam has joined #openstack-kolla03:55
*** harlowja has quit IRC03:56
*** adrian_otto has joined #openstack-kolla03:56
eanylinspsurya: Ok. We can do that.03:56
*** lamt has joined #openstack-kolla04:02
*** lamt has quit IRC04:07
*** harlowja has joined #openstack-kolla04:07
sdakei think we can just forget 1.5 ever existed if we can get to a stable 1.6 ;)04:09
sdakeonce 1.7 releases upstream, we should be close to releasing kolla-kubernetes 1.0.004:10
*** unicell has joined #openstack-kolla04:23
spsuryasdake: what about uninstalling the kubernetes or deleting the existing pod04:25
spsuryashould not that be documented04:25
spsurya?04:25
sdakespsurya it i documented04:25
spsuryasdake: will you locate in doc to get that04:26
spsuryai need that04:26
sdakeits right at the bottom of the odc04:27
sdakeits a script in the review04:28
sdakesomethign with 15.sh at the end04:28
sdakenote - this totally destroys your kubernetes deployment :)04:28
spsuryasdake: roger04:29
spsuryai see04:29
spsuryaand got that04:29
spsuryai will destroy and redeploy04:29
eanylinsdake: I am with you on that. I think we should use stable 1.6 :)04:29
sdake1.6 is far from stable it appears04:30
spsuryasdake: eanylin yes I am also going for 1.6 :)04:30
eanylinsdake: Yes, 1.6.1 seems stable so far04:31
eanylinspsurya: Cool04:31
sdakeok - well lets use it then04:31
eanylinsdake: Okie04:31
sdakeeanylin if you can document the changes needed in teh deployment document we can move on from 1.504:31
sdakeeanylin i can delete the 1.5 stuff as needed04:32
sdakeI mostly need the 1.6 deploy mechanism04:32
sdakeas I don't think rich's work is complete (I haven't tried it)04:32
spsuryaeanylin: you got the all 0 in restart ?04:32
eanylinsdake: I think most of the comments are already there04:33
sdakeeanylin in the latest ereview?04:33
sdakeeanylin have to admit I have lost track of all of the comments attempting to keep the review up to date04:33
eanylinI know some of those are not required for 1.6.1 due to bug fixes04:33
eanylinsdake: Yeah, too many things changing these past few days04:33
*** harlowja has quit IRC04:33
sdakewell that should settle down now04:33
sdakei guess i chose a poor time to document deployment :)04:33
eanylinspsurya: I think close to 0, not all 004:34
eanylinsdake: Yeah, I am trying to see if I can still deploy to same stage as before on 1.6.104:34
eanylinsdake: And still trying to figure out why I cannot ping from host to Cirros VM. Lol04:35
spsuryasdake: eanylin IMO let it be there till get all 0s04:35
spsuryawe can remove that later too04:35
eanylinspsurya: Ok04:35
duonghqeanylin, you got k8s cluster 1.6.1 worked?04:36
eanylinduonghq: I got 1.6.0 working yesterday, 1 controller + 1 compute, i.e. multi-node04:37
eanylin1.6.1 still testing AIO04:37
duonghqeanylin, thanks04:37
eanylinnp04:37
eanylinduonghq, might not be able to finish testing tonight though, about 20 mins away from midnight and need to work tomorrow :)04:38
duonghqeanylin, roger04:39
duonghqeanylin, did you get master node notready?04:39
eanylinsdake: I will see if I can update the Deployment Guide after I get the cluster up with 1.6.1.  I think the guide is mostly ok, only thing that I am not sure if its correct is04:40
eanylinManually modify ``/etc/kubernetes/manifests/kube-apiserver.yaml`` with your favorite editor.  After this line::04:40
eanylin   - --kubelet-client-key=04:40
eanylinI think I set it to blank and the cluster didnt work properly after that04:40
sdakeeanylin thanks for the tip - i will deploy without that step04:40
eanylinduonghq, my k8s cluster is up, Openstack cluster is still work in progress04:40
duonghqeanylin, hmm, ok04:40
sdakeeanylin wasn't openstack working for you prior?04:40
eanylinsdake: For 1.6.0 yes, 1.6.1 not yet verified04:41
sdakeroger04:41
sdakeeanylin what are your thoughts on kolla-kubernetes thus far?04:41
sdake(minus the networking rpoblem you ahve which is a misconfiguraiton i think)04:41
eanylinsdake: Yeah, I think its good04:42
eanylinI can actually ping Google DNS from my Cirros VM04:42
duonghqhmm, my dns didn't up04:42
sdakethats sweet04:42
eanylinping across 2 Cirros VMs works too04:42
duonghqthe kube-dns04:42
eanylinand my noVNC works too04:42
eanylinso not sure why I cannot reach the Cirros VM from my host = /04:42
eanylinduonghq, you followed steps in https://review.openstack.org/#/c/452993/3/tools/setup_kubernetes.sh ?04:43
eanylinsdake: I was able to create Cinder LVM Volume too btw04:44
eanylinsdake: But volume attach to Cirros VM didnt work, didnt really find time to troubleshoot, will need to look at the nova/cinder logs04:44
duonghqeanylin, I follow sdake guide, /me rechecking04:44
sdakeduonghq my guide i slacking the steps in setup_kubernetes.sh I think04:45
eanylinduonghq: Ok, thats the latest one, I think not merged yet04:45
sdakeas setup_kubernetes.sh has a bunch of workarouns for 1.604:45
eanylinIf you want to use 1.6.1, try that04:45
duonghqsdake, roger04:45
*** bmace has quit IRC04:45
sdakei guess we need to decode setup_kubernetes.sh04:45
eanylinthe 1.6.0 is a bit more involved to do all the workarounds04:45
*** bmace has joined #openstack-kolla04:45
sdakesup bmace04:45
*** rstarmer has quit IRC04:45
*** rstarmer has joined #openstack-kolla04:47
sdakeduonghq - setting up kubernetes 1.6.1 now04:48
duonghqsdake, nice04:49
*** gkadam has quit IRC04:51
*** skramaja has joined #openstack-kolla04:53
*** zhubingbing_ has joined #openstack-kolla04:55
*** jrobinson has quit IRC04:55
*** g3ek has quit IRC04:55
*** jrobinson has joined #openstack-kolla04:58
eanylinsdake duonghq: http://paste.openstack.org/show/605329/05:00
eanylinsdake: Seems like 1.6.1 will work too. I got problems with my VM creation, not sure if I messed up something along the way...05:00
sdakeeanylin if you could capture your history and paste it somewhere that would be helpfu l :)05:00
*** adrian_otto has quit IRC05:00
eanylinok05:01
eanylinbut cirros vm create fail05:01
eanylin| fault                                | {"message": "Build of instance 5c2b8960-f1f1-4686-ad9a-19ae18839227 aborted: Image 0d02887b-d5f8-4897-9efa-73e09df7b477 is unacceptable: Unable to convert image to raw: Image /var/lib/nova/instances/_base/4f726b9f3e20cc62f5f20c753b19237329328dd8.part is unacceptable: ", "code": 500, "details": "  File \"/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py\", line 1780,05:01
eanylinin _do_build_and_run_instance |05:01
eanylin|                                      |     filter_properties)05:01
eanylinwith some strange errors05:01
eanylinmaybe I need to relook again?05:01
eanylinstrange that it says it cannot convert the glance image05:01
sdakeeanylin did you remove /var/lib/kolla/volumes?05:01
spsuryasdake: +105:02
sdakeeanylin between resets?05:02
eanylinsdake: I think i got it up 1 shot05:02
eanylinhmmm05:02
spsuryaeanylin: please save the history05:02
sdakei will convert histor yt o1.6.1 install docs for kubernetes05:02
sdakethat is where most people are blocked atm eanylin05:02
eanylinsdake: Ok05:02
eanylinI will see what I have for now. And see if I can have another go with better results.05:03
eanylinI tried some new config this time round05:03
eanylinmaybe thats why VM cannot be created05:03
*** g3ek has joined #openstack-kolla05:04
*** eanylin_ has joined #openstack-kolla05:05
*** dasTor_ has joined #openstack-kolla05:05
*** lazyPower has joined #openstack-kolla05:06
eanylin_My session logs got over-written from earlier05:06
eanylin_as in, buffer over-run05:07
*** gkadam has joined #openstack-kolla05:07
sdakeeanylin_ next time you give it a run - run in script05:09
sdakescript will capture your input/output05:09
sdakethen type exit when done05:09
sdakeit will save a file called typescript05:09
eanylin_ok05:09
sdakeput that somewhere, anywhere, plz :)05:09
eanylin_yeah, I dont usually save session logs05:09
eanylin_will do05:09
sdakeeanylin_ i recognize its late and you probably dont want to give it another go05:09
bmacehey sdake hope you are doing well :)05:10
sdakeeanylin_ if you can do at your earliest convience and post a link to the mailing list, i'll get the doc updated05:10
sdakebmace halcyon is busted05:10
eanylin_its ok, I think it will come up (fingers crossed)05:10
*** skramaja_ has joined #openstack-kolla05:10
eanylin_ok05:10
sdakebmace i get impression halcyon wont be maintained long term but i'm not sure05:10
sdakebmace nobody is really working to maintain it at present and portdirect has moved to a different tool05:11
sdakebmace and yup doing well thanks - how about yourself ;)05:11
*** harlowja has joined #openstack-kolla05:11
bmacesdake: that is non-optimal.  internally we have done some similar things to the kolla-k8s devenv stuff for kolla-ansible also, it has been nice :)05:12
*** rstarmer has quit IRC05:12
bmacesdake: we can nuke all that code if you want, if we don't feel like it is going to get fixed.  seems like with the dev docs you guys updated people may not need it as much anyway?05:12
sdakebmace well i  think its premature ot remove it now05:12
sdakejust giving you a heads up05:12
sdakewe should put a warning in the docs stating halcyon is busted atm05:13
bmacekk, understood, thanks.05:13
*** skramaja has quit IRC05:13
*** rstarmer has joined #openstack-kolla05:13
*** rstarmer has quit IRC05:13
*** rstarmer has joined #openstack-kolla05:14
*** skramaja_ is now known as skramaja05:14
*** eanylin has quit IRC05:14
*** dasTor has quit IRC05:14
*** retr0h has quit IRC05:14
*** lp|swap has quit IRC05:14
*** jbadiapa has quit IRC05:14
*** zhenguo has quit IRC05:14
*** rstarmer has quit IRC05:14
zhubingbing_around sdake ?05:20
sdakezhubingbing_ indeed05:21
zhubingbing_https://blueprints.launchpad.net/kolla-kubernetes/+spec/services-redux05:21
sdakezhubingbing_ yes?05:21
zhubingbing_we should remove all service/xx ?05:21
*** jbadiapa has joined #openstack-kolla05:21
sdakenotall of them05:21
sdakezhubingbing_ just most of them05:21
sdakezhubingbing_ reviews are already in teh queue for that blueprint05:21
zhubingbing_emm05:21
zhubingbing_ok05:21
*** retr0h has joined #openstack-kolla05:22
sdakezhubingbing_ what really needs love is the deployment doc at this point :)05:22
zhubingbing_yeah05:22
sdakezhubingbing_ good news, ordered a new server for the house to heat up my office05:22
sdake128gb ram - 2 xeon cpus05:22
sdakeprobably will get another soon05:23
zhubingbing_wow05:23
zhubingbing_;)05:23
zhubingbing_good05:23
sdakei will deploy it in about 2 weeks05:23
sdakepossibly this week05:23
sdakeyour free to use it if you like05:23
sdakedepends on whether or not this deployment doc is merged05:23
sdakei'm not deploying any gear until deploy doc is merged05:24
sdakegear arrives wednesday05:24
zhubingbing_in recent days I can not ssh your box machine05:24
zhubingbing_i like05:24
zhubingbing_；）05:24
sdakezhubingbing_ my gear is suffering hardware problems05:24
sdakezhubingbing_ box is up atm05:24
zhubingbing_i‘ll check it again05:26
zhubingbing_sdake i ssh your box success05:29
sdakeyup its up05:29
sdakei was doing housework this weekend - didnt' touch the computers -came back to a kernel oops05:30
sdakenew box has remote power cycle05:30
sdakeremote kvm05:30
sdakeetc05:30
sdakehttp://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/c220m4-lff-spec-sheet.pdf05:31
sdakehttp://www.provantage.com/~7CSC8AUN.htm05:31
sdakeis what i ordere05:31
sdaked05:31
*** eanylin has joined #openstack-kolla05:35
*** eanylin_ has quit IRC05:35
eanylinsdake duonghq spsurya: http://paste.openstack.org/show/605332/05:40
eanylinsorted out the logs, these are the steps to get the k8s cluster up on CentOS 7 base VM05:41
sdakeeanylin thanks :)05:41
eanylinsdake: No problem, hopefully it can help other guys to move forward :)05:41
*** sayanta__ has quit IRC05:42
*** madgoat has joined #openstack-kolla05:42
eanylinsdake: I think the link that Rich put earlier for canal.yaml will not work with the 1.605:43
*** madgoat has left #openstack-kolla05:43
eanylinit was crashing earlier when I used it05:43
*** unicell has quit IRC05:44
*** targon has joined #openstack-kolla05:44
*** unicell has joined #openstack-kolla05:45
sdakeeanylin your log doesn't have a link to canal.yaml05:46
sdakeeanylin could you provide one?05:46
eanylinthe first line I did a wget05:49
eanylinhttps://raw.githubusercontent.com/projectcalico/canal/master/k8s-install/kubeadm/1.6/canal.yaml05:50
sdakeeanylin got it thanks05:50
eanylinnp05:50
eanylinsdake: I left a comment with the link as well, stating that its needed for 1.605:51
sdakethe port 808005:52
sdakewhere does that go?05:52
eanylinsdake: /etc/kubernetes/manifests/kube-apiserver.yaml05:53
sdakeeanylin can you paste that file please?05:55
sdakehaving trouble connecting with kubectl05:56
sdakeport 8080 connection refused05:56
eanylinsdake: try this05:56
eanylinsed -i 's/insecure-port=0/insecure-port=8080/' /etc/kubernetes/manifests/kube-apiserver.yaml05:56
sdakei already replaced the 0 with 808005:56
sdakeapiserver is running with 808005:56
eanylinthen do05:57
eanylinsystemctl daemon-reload05:57
eanylinsystemctl start docker05:57
eanylinsystemctl restart kubelet05:57
eanylinand wait for like 30 seconds05:57
eanylinjust do a watch05:57
eanylinwatch kubectl get pod -n kube-system05:57
eanylinsdake: Did it work or still having problems?05:59
sdakerebooted05:59
sdakemoemnt05:59
sdakeworking now06:00
sdakethanks ;)06:00
eanylincool :)06:00
*** iceyao has quit IRC06:00
*** iceyao_ has joined #openstack-kolla06:00
*** jaosorior_away is now known as jaosorior06:01
*** harlowja has quit IRC06:02
sdakehttp://paste.openstack.org/show/60533706:05
sdakespsurya duonghq zhubingbing_ ^^06:05
duonghqsdake, nice06:05
openstackgerritMerged openstack/kolla-ansible master: Fix ceph log location  https://review.openstack.org/45140706:05
eanylinsdake: My steps worked :D06:06
sdakemore or less ;)06:06
spsuryasdake: nice some of them are restarting06:06
spsuryaalthough good06:06
sdakespsurya check this: http://paste.openstack.org/show/60533806:07
eanylinspsurya: Some of the restarts were due to fact that canal is not fully up06:07
spsuryasdake: eanylin i see looks really good06:08
spsurya:)06:08
sdakebusybox is in pednign state eanylin06:08
sdakeany tips for that problem?06:08
eanylinsdake: Hmm, I never try spinning busy box, I can try that06:09
eanylinbut I was able to get tiller working06:09
eanylin[root@kube03 ~]# kubectl run -i -t $(uuidgen) --image=busybox --restart=Never06:10
eanylinIf you don't see a command prompt, try pressing enter.06:10
eanylinServer:    172.16.128.1006:10
eanylinAddress 1: 172.16.128.10 kube-dns.kube-system.svc.cluster.local06:10
eanylinName:      kubernetes06:10
eanylinAddress 1: 172.16.128.1 kubernetes.default.svc.cluster.local06:10
eanylinhttp://paste.openstack.org/show/605339/06:11
eanylinseems like it came up without issue for me06:11
sdakehttp://paste.openstack.org/show/60534006:11
sdakemaybe i did something wrong :)06:11
sdakelemme destory canal - perhaps missed the sed steps06:11
eanylinyou tainted the master?06:12
eanylinkubectl taint nodes --all node-role.kubernetes.io/master-06:12
sdakesdake@kolla ~]$ kubectl taint nodes --all node-role.kubernetes.io/master-06:12
sdakeerror: taint key="node-role.kubernetes.io/master" and effect="" not found.06:12
eanylinok, you probably did earlier06:13
duonghqsdake, what is your output of kubectl version06:15
sdake1.6.106:16
duonghqboth client and server?06:16
sdakemoemnt06:16
duonghqI got this error: unable to decode "canal.yaml": no kind "ClusterRoleBinding" is registered for version "rbac.authorization.k8s.io/v1beta1"06:18
duonghq06:18
duonghqguess that there is version mismatch in my box06:18
sdakehttp://paste.openstack.org/show/60534106:18
sdakeduonghq are you using kubernetes 1.6.1?06:18
duonghqyes06:19
sdakeduonghq try yum update06:19
sdakeduonghq actually paste your version06:19
duonghqhmm, I'm using Ubuntu, kubelet, kubeadm, kubectl are 1.6.106:19
sdakei see - using centos here06:19
eanylinhttp://paste.openstack.org/show/605343/06:20
sdakeduonghq how did you get your canal.yaml?06:20
eanylinmine looks like this06:20
eanylinsdake: Still having problems or?06:20
duonghqhttp://paste.openstack.org/show/605344/06:20
duonghqsdake, I use canal.yaml from eanylin paste06:20
sdakehttp://paste.openstack.org/show/60534506:21
sdakeeanylin sorting through them06:21
duonghqsdake, weird06:22
duonghqmy system is weird06:22
*** zhubingbing_ has quit IRC06:23
*** zhubingbing__ has joined #openstack-kolla06:24
eanylinsdake: Ok06:24
*** jrobinson has quit IRC06:26
duonghqsdake, I'm not sure my client still 1.506:27
sdakeduonghq run which kubectl06:28
duonghqsdake, seem that one version of old kubectl deployed manually still hang around in my VM06:28
sdakeyup rm it :)06:28
sdakei had that same problem06:28
sdakeduonghq after rming it make sure to log out and log back in to the system06:30
duonghqsdake, just do that, :)06:31
*** iceyao_ has quit IRC06:32
openstackgerritMerged openstack/kolla master: Use separate sudoers for ironic conductor modprobe  https://review.openstack.org/45220706:34
sdakehttps://da.gd/k9KMi -> https://paste.fedoraproject.org/paste/Gs63MtQ1GmMcnJcWuCQ98l5M1UNdIGYhyRLivL9gydE=/06:38
sdakeduonghq - these are the minimal steps i ran through06:38
sdakeduonghq ping working for me06:38
sdakerather nslookup06:38
sdakeone thing missing from that typescript is the taint06:39
sdakeeanylin your typescript was optimized a bit ;)06:39
sdakei shall write docs now06:39
sdakebbi1hour06:39
*** Serlex has joined #openstack-kolla06:40
duonghqsdake, added insecure-port sed, than it's fine06:40
duonghqthanks06:40
eanylinsdake: Hmm, I think I have that one06:41
sdakeeanylin nah i mean from my typescript taint is missing06:41
eanylinline 286 http://paste.openstack.org/show/605332/06:41
eanylinoh06:41
eanylinokie06:41
sdakei ran it in another window06:41
eanylinI see06:41
eanylinso its all good now?06:41
sdakehaven' ttried kolla-kubernetes06:42
sdakegoing to document k8s deploy while  its fresh in the brian06:42
sdakeeanylin thanks for the hlep there ;)06:42
eanylinI think I messed up my VM with all the things that I was doing = /06:42
eanylinsdake: No problem, anytime06:42
eanylinsdake: Going to be 2am here, I better go and sleep06:42
eanylinlet me know if you run into other issues06:42
sdakeeanylin thanks for the help :)06:42
sdakeeanylin i probably wlil - hopefully i can sort em out06:42
spsuryaeanylin: thanks06:42
sdakei know some stuff is needed for rbac06:43
eanylinsdake: No worries, see you in the morning :)06:43
eanylinoh06:43
eanylinthat one06:43
eanylinI can paste it06:43
eanylinyou want?06:43
sdakesure06:43
eanylinI wasnt sure if its needed06:43
sdakeit may not be in 1.606:43
sdakei am not sure06:43
sdakerather 1.6.106:43
sdakealthough its int eh gate06:43
eanylinDo this at the end06:44
eanylinhttp://paste.openstack.org/show/605346/06:44
eanylinyeah, its in the gate06:44
eanylinsbezverk didnt remove in his latest commit06:44
eanylinyou might be right that its still needed06:44
eanylinas in, you are most probably right06:44
eanylinI think we can add that06:45
eanylinits not going to cause any damage I think06:45
*** manheim has joined #openstack-kolla06:47
*** manheim has quit IRC06:48
*** manheim has joined #openstack-kolla06:48
*** rmart04 has joined #openstack-kolla06:52
*** matrohon has joined #openstack-kolla06:53
*** satyar has joined #openstack-kolla06:56
*** rmart04 has quit IRC06:57
*** iceyao has joined #openstack-kolla07:02
hrwduonghq: pong07:04
hrwmorning everyone07:04
duonghqhrw, can you bring it https://review.openstack.org/#/c/434453/ to this week meeting?07:04
*** athomas has joined #openstack-kolla07:05
duonghqI'm not sure if you discussed with others core07:05
hrwduonghq: sure, can07:05
duonghqhrw, nice07:05
*** caoyuan has quit IRC07:06
*** shardy has joined #openstack-kolla07:09
hrwduonghq: iirc from times when I built with jessie-backports there were several external repos required to get things like ceph, libvirt up to date. especially on aarch6407:10
duonghqhrw, hmm, ok,07:11
hrwduonghq: and if jessie stays then we need to make jessie-backports to be default instead of jessie07:12
hrwI had patch for that but unable to find on review.o.o07:12
hrwhttps://review.openstack.org/#/c/432780/07:13
duonghqhrw, ya07:13
duonghqbut the change affects not so few people, so we need consensus07:14
duonghq:)07:14
hrwduonghq: as Debian support in Kolla had status "no one uses, no one works on it, will be removed" I decided to go easier path and do stretch instead of jessie07:14
duonghqhrw, sure07:14
*** zhenguo has joined #openstack-kolla07:15
hrwadded to agenda, bumped date07:16
hrwduonghq: thanks for reminding07:16
duonghqhrw, nice07:16
duonghqsdake, which helm version are you using?07:17
eanylinduonghq: Should be similar to mine07:19
eanylin[root@kube03 opt]# helm version07:19
eanylinClient: &version.Version{SemVer:"v2.2.3", GitCommit:"1402a4d6ec9fb349e17b912e32fe259ca21181e3", GitTreeState:"clean"}07:19
eanylinServer: &version.Version{SemVer:"v2.2.3", GitCommit:"1402a4d6ec9fb349e17b912e32fe259ca21181e3", GitTreeState:"clean"}07:19
eanylin[root@kube03 opt]#07:19
duonghqeanylin, hmm, I'm using 2.2.207:19
eanylinhmm, ok07:19
*** pcaruana has joined #openstack-kolla07:20
duonghqeanylin, I'm getting this error: Error: release mariadb failed: the server does not allow access to the requested resource (get namespaces kolla)07:20
duonghqdid you get it before?07:20
eanylinduonghq: I think I got that earlier when I messed up my VM07:21
eanylinwasnt sure how I landed in that state07:21
eanylinI usually clean up the mariadb volume and it will work07:21
eanylinseems like this is more related to helm07:21
eanylinhelm ls   didnt work for me too07:22
eanylinI shut that VM down, was thinking of restoring to the base image...07:22
duonghqeanylin, yup, nothing about helm is working Error: the server does not allow access to the requested resource (get configmaps07:22
eanylinyeah07:22
eanylinsame error07:22
eanylinI had a full system running before this07:22
eanylinI did a wipe out07:22
eanylinmaybe I didnt do it correctly07:23
eanylinwhen I did that07:23
eanylinso helm complains after that, maybe need to delete and reinstall helm?07:23
eanylinthat might help07:23
duonghqeanylin, will try07:24
duonghqthanks07:24
eanylinsure07:24
*** jascott1_ has quit IRC07:25
*** shardy has quit IRC07:25
eanylinduonghq: Am leaving for the night. Talk to you later.07:26
duonghqeanylin, thank you,07:27
duonghqcya07:27
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735607:27
sdakeduonghq if you get that error you need to rm -rf your /var/lib/kolla/volumes/*07:28
sdakeits in the troubleshooting section07:28
duonghqsdake, okay07:28
duonghqsdake, after restart my box, canal and kube-dns crashed07:29
sdakeduonghq - kubeadm reset07:30
sdakestart over07:30
sdakewith docs just published07:30
sdakethey can be rendered via tox -e docs07:30
duonghqsdake,  you mean volumes can mess up with helm?07:30
sdakevolumes stick around - helm deosn't clean them up07:30
*** Jezekus has joined #openstack-kolla07:40
sdakemissing manifest step in those docs07:40
sdakeneed to add that07:40
sdakeactually have the sed07:41
sdakejsut no file specified07:41
*** egonzalez has joined #openstack-kolla07:44
duonghqsdake, due to k8s 1.6.1 is place config file in ~/admin.conf, so we need cp it to ~/.kube/config07:44
duonghqin my system07:44
duonghqhow about you?07:44
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735607:45
sdakeduonghq no clue - haven't seen that07:45
sdakefirst time I have had 1.6.1 runnign07:45
sdakei am making it more generic so it runs on any network hardware with rfc191807:45
duonghqsdake, this one: http://paste.openstack.org/show/605361/07:45
sdakeyup i saw it i just dont know what it means07:46
sdakethat may allow us to get rid of the insecure-port sed07:46
sdakenot certain07:46
duonghqit's credential for kubectl access kubernetes cluster07:46
*** skramaja is now known as skramaja_afk07:46
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735607:47
sdakeyup07:47
sdakei get that much - not sure what that buys us as it works with the sed 8080 - wonder if it works without the sed with those commands07:48
sdakelets iterate on what we got in the docs :)07:48
sdakeas in lets make sure they work first07:48
duonghqsure07:48
*** nathharp has joined #openstack-kolla07:50
sdakeok - those docs are verified by me for 1.6.1 on a mostly cleaned up system07:52
sdakethe step 1 and step 2 parts :)07:52
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735607:56
*** calbers has joined #openstack-kolla07:57
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible master: Add graceful_timeout argument to kolla_docker  https://review.openstack.org/45305508:09
sdakeok every step works but starting the helm charts for me :)08:10
*** gfidente has joined #openstack-kolla08:13
duonghqsdake, same here08:15
duonghqsdake, still the server does not allow access to the requested resource (get namespaces kolla) error08:15
sdakeduonghq http://paste.openstack.org/show/605346/08:16
sdakerun that08:16
sdakeyou dont have to reset to do so08:16
sdakejustrun that and then paste the helm chart startups08:16
duonghqsdake, it's new feature in k8s 1.6.1?08:16
sdake1.6.008:17
duonghqah, yes08:17
sdakehttp://paste.openstack.org/show/60536408:17
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Fix Telegraf retention policy not found  https://review.openstack.org/45305808:22
sdakewell init-runonce doesnt' work for me08:22
duonghqsdake, I'm bringing up keystone08:23
sdakethe services are no t responding08:23
sdakealthough everything is operational08:23
sdakeworked that time08:23
duonghqsdake, seem that dns issue?08:23
sdakelet me update docs08:23
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735608:26
sdakehttp://paste.openstack.org/show/60536508:30
sdakethats what I get08:31
sdakeafter init-runonce08:31
sdakethose docs work all the way up to server create :)08:31
duonghqsdake, what is error?08:32
*** mgoddard has joined #openstack-kolla08:33
sdakehttp://paste.openstack.org/show/60536608:33
duonghqlegendary one08:34
sdakeya - although it actually returns a backtrace in ocata :)08:35
duonghqI have 10 po is initializing :(08:35
sdakeit takes my gear 8 minutes to spin up the helm charts08:35
duonghqseem that its downloading image from docker hub08:36
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735608:37
spsuryasdake: duonghq what if i want to clean up the kubernetes setup in the middle not at the end using tools/cleanup-k8s-15.sh ?08:38
spsuryawould it be possible08:39
duonghqspsurya, sorry, do not understand08:39
spsuryai uninstalled the rpms08:39
spsuryaduonghq: suppose i installed kubeternes only i.e upto step 1 only08:39
spsuryaand i want to desotroy08:40
spsuryaeverything and want clean the environment i setup08:40
duonghqspsurya, just kubeadm reset08:41
spsuryaduonghq: does kubeadm reset uninstall the rpm ?08:42
duonghqno08:42
duonghqyou must uninstall it by yourself08:42
spsuryaduonghq: i want to clear all files which was generated by [centos@ip-172-31-13-146 release]$ ll /etc/kubernetes/manifests/08:45
spsuryaetcd.json                     kube-apiserver.json           kube-controller-manager.json  kube-scheduler.json08:45
*** sklgromek has quit IRC08:45
spsuryaduonghq: ignore08:45
spsuryaduonghq: kubeadm reset did what all i wanted08:48
spsuryathanks08:48
duonghqspsurya, np08:48
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735608:48
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735608:49
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735608:50
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible master: Add graceful_timeout argument to kolla_docker  https://review.openstack.org/45305508:53
*** eaguilar has joined #openstack-kolla08:53
sdakeplz make sure to rm -rf /var/lib/kolla/volumes/* before redeploy08:55
sdakezhubingbing__ can you verify those docs on a vm or something on my host :)08:55
sdakeok peeps bedtime for me08:57
sdakenight all08:57
*** eaguilar has quit IRC08:58
spsuryasdake: night08:58
duonghqwnight sdake08:58
sdakewow lastpass fail: https://www.cnet.com/how-to/lastpass-update-hackers-cybersecurity-keep-passwords-safe/09:03
sdakeon that note, ptfoing09:03
*** haplo37_ has quit IRC09:04
*** haplo37_ has joined #openstack-kolla09:04
*** athomas has quit IRC09:23
*** shashank_t_ has joined #openstack-kolla09:26
*** jascott1_ has joined #openstack-kolla09:27
spsuryaduonghq: at line no. 90 got this09:28
spsurya[centos@ip-172-31-13-146 kollaK8s]$ sudo sed -i 's/insecure-port=0/insecure-port=8080/' /etc/kubernetes/manifests/kube-apiserver.yaml09:28
spsuryased: can't read /etc/kubernetes/manifests/kube-apiserver.yaml: No such file or directory09:28
*** magicboiz has quit IRC09:29
duonghqspsurya, can you kubectl get po -n kube-system?09:29
spsuryaduonghq: [centos@ip-172-31-13-146 kollaK8s]$ kubectl get po -n kube-system09:30
spsuryaThe connection to the server localhost:8080 was refused - did you specify the right host or port?09:30
*** ccesario has quit IRC09:30
*** shashank_t_ has quit IRC09:30
*** magicboiz has joined #openstack-kolla09:31
spsuryathis file not generated /etc/kubernetes/manifests/kube-apiserver.yaml09:31
*** athomas has joined #openstack-kolla09:31
duonghqseem that your k8s cluster has not started yet09:31
*** jascott1_ has quit IRC09:32
*** skramaja_afk is now known as skramaja09:33
*** bjolo has joined #openstack-kolla09:34
*** magicboiz has quit IRC09:35
spsuryaduonghq:  have you review sdake latest patch09:35
duonghqya, but this time I cannot get this run09:36
duonghqwas success last time09:36
spsuryalast time i checked with 1.509:37
spsuryaduonghq: now i am checking with 1.6.109:37
duonghqthis afternoon I successed with this :(09:37
duonghqwith 1.6.109:37
spsuryaduonghq: i see09:38
*** shardy has joined #openstack-kolla09:39
duonghqspsurya, the file only be populated when k8s is up09:40
*** japestinho has joined #openstack-kolla09:42
spsuryaduonghq: this time facing some09:44
spsuryayes09:44
spsuryacluster is getting up09:44
spsuryais not*09:44
*** Jezekus has quit IRC09:46
*** bjolo has quit IRC09:47
*** manheim has quit IRC09:47
*** Jezekus has joined #openstack-kolla09:47
*** magicboiz has joined #openstack-kolla09:48
*** zhubingbing_ has joined #openstack-kolla09:56
*** zhubingbing__ has quit IRC09:57
*** Jezekus has quit IRC09:58
*** manheim has joined #openstack-kolla10:00
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Split keepalived own role  https://review.openstack.org/45217710:04
*** sambetts|afk is now known as sambetts10:04
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla master: Update oraclelinux tag to new slim base image  https://review.openstack.org/43732910:10
*** iceyao has quit IRC10:11
*** iceyao has joined #openstack-kolla10:13
*** satyar has quit IRC10:14
*** tovin07_ has quit IRC10:18
*** iceyao has quit IRC10:20
*** Jezekus has joined #openstack-kolla10:24
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu  https://review.openstack.org/43278710:35
*** ccesario has joined #openstack-kolla10:36
*** pbourke has quit IRC10:37
*** shardy has quit IRC10:38
*** pbourke has joined #openstack-kolla10:39
*** shardy has joined #openstack-kolla10:40
openstackgerritzhubingbing proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939510:41
*** duonghq has quit IRC10:41
openstackgerritzhubingbing proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939510:42
*** dasTor_ has quit IRC10:43
*** shardy_ has joined #openstack-kolla10:43
*** dasTor has joined #openstack-kolla10:43
*** shardy has quit IRC10:44
*** qiliang27 has joined #openstack-kolla10:46
*** Jezekus has quit IRC10:49
*** bjolo has joined #openstack-kolla10:50
*** rstarmer has joined #openstack-kolla11:01
*** bjolo has quit IRC11:04
*** bjolo has joined #openstack-kolla11:04
*** skramaja_ has joined #openstack-kolla11:04
*** skramaja has quit IRC11:06
*** rwellum has joined #openstack-kolla11:09
openstackgerritPavel Gluschak (scsnow) proposed openstack/kolla-ansible master: Remove unused driver attr in cinder_backends map  https://review.openstack.org/45312911:09
*** skramaja_ has quit IRC11:19
*** skramaja_ has joined #openstack-kolla11:20
*** dave-mccowan has joined #openstack-kolla11:20
*** rhallisey has joined #openstack-kolla11:22
*** rwallner has joined #openstack-kolla11:30
*** rmart04 has joined #openstack-kolla11:34
*** srwilkers has joined #openstack-kolla11:44
openstackgerritMerged openstack/kolla master: Install InfluxDB Plugin in Grafana  https://review.openstack.org/44662411:48
*** deadnull has joined #openstack-kolla11:53
manheimhello ppls11:54
manheimif i'm writing a new role, i can call vars from global.yml directly from roles tasks, right?11:55
pbourkeyes11:56
manheimthanks11:57
mnasersdake and other interested parties.. https://review.openstack.org/#/c/401003/11:58
*** rwallner has quit IRC11:59
*** eaguilar has joined #openstack-kolla11:59
*** Jezekus has joined #openstack-kolla12:00
*** rwallner has joined #openstack-kolla12:01
*** blallau has joined #openstack-kolla12:02
*** signed8bit_Zzz is now known as signed8bit12:04
*** goldyfruit has joined #openstack-kolla12:05
*** goldyfruit has quit IRC12:05
*** qiliang27 has quit IRC12:07
*** shardy_ is now known as shardy_lunch12:08
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Update senlin config  https://review.openstack.org/45314612:09
spsuryaegonzalez: can you please confirm comment  https://review.openstack.org/#/c/452392/112:10
spsuryaseems like varible need to be added in configuration one too12:10
spsuryaBRB12:10
egonzalezspsurya, replied in gerrit12:24
*** ccesario has quit IRC12:26
*** rstarmer has quit IRC12:27
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: [WIP] Add external rabbitmq  https://review.openstack.org/45315212:30
openstackgerritMark Goddard proposed openstack/kolla master: Use ironic-inspector user for ironic-inspector  https://review.openstack.org/44160512:31
*** skramaja has joined #openstack-kolla12:38
*** shardy_lunch is now known as shardy12:40
*** skramaja_ has quit IRC12:41
*** Pavo has joined #openstack-kolla12:42
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla master: Reparent kolla-toolbox from openstack-base  https://review.openstack.org/43502312:42
*** rstarmer has joined #openstack-kolla12:44
*** skramaja_ has joined #openstack-kolla12:44
*** skramaja has quit IRC12:47
*** signed8bit is now known as signed8bit_Zzz12:56
*** signed8bit_Zzz is now known as signed8bit12:57
*** skramaja_ has quit IRC12:57
*** Jezekus has quit IRC13:00
*** goldyfruit1 has joined #openstack-kolla13:01
spsuryaegonzalez: thanks13:04
spsuryastill around ?13:04
egonzalezspsurya, yep13:04
spsuryaegonzalez:  i was talking about changes needed in https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/templates/neutron.conf.j213:04
spsuryais not that required ?13:05
egonzalezspsurya, as far i can see all required changes are already present in neutron.conf13:05
*** manheim has quit IRC13:07
spsuryaegonzalez:  condition check is not present for variable "enable_neutron_qos"  in https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/templates/neutron.conf.j213:09
*** shardy is now known as shardy_mtg13:10
egonzalezspsurya, it is https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/defaults/main.yml#L334 results in a list of service_plugins here: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/templates/neutron.conf.j2#L3913:10
spsuryathanks13:15
spsuryagot this13:15
*** mbruzek has joined #openstack-kolla13:20
*** mbruzek has joined #openstack-kolla13:20
*** eanylin has quit IRC13:21
sdakemorning13:23
sdakehey spsurya13:23
sdakei see you ran into some trouble?13:23
sdakespsurya it is possible i need to rerun throug hthe directions13:23
sdakespsurya i tink that sed for hte manifest needs to hapepn after kubeadm start13:23
sdakeand then afte rthat kubelet mustbe systemctl restart kubelet13:23
spsuryasdake: right13:23
sdakespsurya i needed to go to bed, and may hve been ain a rush :)13:24
*** jtriley has joined #openstack-kolla13:24
spsuryasdake: np13:24
spsuryaMay be later13:24
sdakei'll updaete it now13:24
sdakethanks for finding the bug13:25
spsuryasdake: I am also going cook something for dinner.13:25
*** qiliang27 has joined #openstack-kolla13:26
spsuryasdake: no problem, we have make the doc as good as possible13:27
spsuryahave to*13:27
spsuryaI was running with 1.6.113:27
sdakemorning rwallner13:28
sdakemorning rwellum :)13:28
openstackgerritMerged openstack/kolla-ansible master: Remove unused driver attr in cinder_backends map  https://review.openstack.org/45312913:28
sdakei see eveyrone ran into that problem tis morning or last night13:28
rwallnerhaha forever getting that mixed up sdake :)13:28
sdakerwallner its tabcomplelte13:28
rwallnerno worries :)13:28
sdakerwallner your new to the channel so i am not used to pressing tab twice :)13:28
spsuryaguys BRB13:29
sdakerwallner welcome btw :)13:29
rwallnerthanks!13:29
rwellumMorning sdake13:29
sdakerwellum did you sort ou the error in the doc (the workaround)13:29
sdakerwallner or jammed up currently?13:29
sdakegah13:29
rwellumsdake: from my end, running through the very latest doc. I've added a couple of issues seen so far.13:31
*** qiliang27 has quit IRC13:31
sdakerwellum cool - do you find the new doc to be much improved?13:32
sdakerwellum are they in the view at present?13:32
*** qiliang27 has joined #openstack-kolla13:32
rwellumI do sdake - it's almost copy-n-pasteable :)13:32
sdakerather have you submittedd them in the review (via the reply butoton and a -1) so I can address them13:32
rwellumsdake: yes13:32
*** ccesario has joined #openstack-kolla13:34
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Fix senlin workers and events  https://review.openstack.org/45314613:34
*** eanylin has joined #openstack-kolla13:36
*** jaosorior is now known as jaosorior_away13:36
sdakei dont understand the insecure-port change13:37
sdakeit makes no sense to me why tha twould be disabled - i think we are doingit wrong13:37
sdakethere is somehing during kubeadm about setting up credentails13:38
*** targon has quit IRC13:40
*** jmccarthy has joined #openstack-kolla13:41
eanylinsdake: Could it like what duonghq mentioned last night13:42
*** targon has joined #openstack-kolla13:42
sdakeeanylin i am atemptin th now13:42
eanylinhttp://paste.openstack.org/show/605361/13:42
eanylinok, let me know13:42
eanylinI went to bed already13:42
*** targon has quit IRC13:42
eanylinby the time he posted that13:42
*** manheim has joined #openstack-kolla13:42
sdakeeanylin queestion - i had a  problem where i could not start vms13:43
sdakeeanylin amqp messaging timed out13:43
sdakeeanylin wonder if you ahve seen that13:43
sdakeeanylin there is a paste in the review13:43
eanylinCirros VM?13:45
eanylinhmm, amqp sounds more like network related13:45
eanylinmy rabbit was ok13:46
eanylinI encountered some other strange problems13:46
eanylinwith the glance image last night13:46
*** manheim has quit IRC13:47
*** manheim has joined #openstack-kolla13:47
eanylinsdake: You are on bare metal right? Not VM like me?13:48
sdakebare metal13:48
*** qiliang27 has quit IRC13:48
*** qiliang27 has joined #openstack-kolla13:49
eanylinok13:49
eanylinI think the procedures should be similar13:49
eanylinfooliouno tested on bare metal (with lower version though, I think 1.5.2)13:50
*** cuongnv has joined #openstack-kolla13:56
*** shardy_mtg is now known as shardy13:57
*** lamt has joined #openstack-kolla14:03
pbourkecould someone have a look at https://review.openstack.org/#/c/448654/ please?14:07
*** Pavo has quit IRC14:08
*** Pavo has joined #openstack-kolla14:09
*** Pavo has quit IRC14:09
*** bjolo has quit IRC14:10
*** lamt has quit IRC14:10
*** Pavo has joined #openstack-kolla14:10
*** Pavo has quit IRC14:10
*** Pavo has joined #openstack-kolla14:11
*** Pavo has quit IRC14:11
*** Pavo has joined #openstack-kolla14:11
berendtpbourke: done14:12
*** Pavo has quit IRC14:12
*** satyar has joined #openstack-kolla14:12
*** Pavo has joined #openstack-kolla14:12
*** Pavo has quit IRC14:12
*** Jezekus has joined #openstack-kolla14:13
srwilkersmorning14:14
berendtpbourke: changed to +1 because the missing documentation is not directly related to the change itself14:17
*** gkadam has quit IRC14:19
*** lrensing has joined #openstack-kolla14:19
*** jpstnh has quit IRC14:20
*** srwilkers has quit IRC14:20
*** hrww has joined #openstack-kolla14:21
*** qiliang28 has joined #openstack-kolla14:21
*** srwilkers has joined #openstack-kolla14:22
*** hrw has quit IRC14:23
*** hrww is now known as hrw14:23
*** ccesario has quit IRC14:23
*** qiliang27 has quit IRC14:24
*** qiliang28 is now known as qiliang2714:24
*** manheim has quit IRC14:26
*** rwallner has quit IRC14:27
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Only start one glance_api when using file backend  https://review.openstack.org/44865414:28
*** shashank_t_ has joined #openstack-kolla14:30
spsuryamorning srwilkers14:31
*** manheim has joined #openstack-kolla14:32
*** rwallner has joined #openstack-kolla14:32
*** shashank_t_ has quit IRC14:33
mgoddardI'm seeing intermittent failures in the kolla unit tests in master. Example: http://logs.openstack.org/05/441605/3/check/gate-kolla-python35/d4091ca/console.html14:35
*** srwilkers has quit IRC14:35
mgoddardI see the same on a fresh checkout of master. Is there a bug covering this?14:35
*** eragavi has joined #openstack-kolla14:36
pbourkealso looking for reviews on https://review.openstack.org/#/c/451358/ please :)14:36
*** srwilkers has joined #openstack-kolla14:38
srwilkersmorning spsurya14:40
*** manheim has quit IRC14:44
*** ccesario has joined #openstack-kolla14:45
*** Deys1 has joined #openstack-kolla14:47
*** erhudy has joined #openstack-kolla14:48
japestinhomorning guys14:49
*** lamt has joined #openstack-kolla14:49
japestinhoI'm trying to deploy openstack with kolla-ansible with lates 4.0.1 tags14:49
japestinhowhen installation finished I got exited memcached container14:50
japestinhoI check docker logs memchaed and got this14:50
japestinhohttps://www.irccloud.com/pastebin/pRtWnf90/memchaed%20logs14:51
*** yingjun has joined #openstack-kolla14:53
japestinhoI tried stop and remove memcached container and run kolla-ansible deploy again and still got restarting memcached container14:53
*** yingjun has quit IRC14:53
*** shashank_t_ has joined #openstack-kolla14:55
*** qiliang27 has quit IRC14:55
*** TxGirlGeek has joined #openstack-kolla14:56
egonzalezjapestinho, try adding memcached_connection_limit: "2000" into globals.yml and reconfigure memcached14:58
*** skramaja has joined #openstack-kolla14:58
*** cuongnv has quit IRC15:02
egonzalezjapestinho, better user 1024 instead  of 2000. 1024 is memcached defaults15:03
*** lucasxu has joined #openstack-kolla15:04
*** ccesario has quit IRC15:05
*** srwilkers has quit IRC15:05
*** rwallner has quit IRC15:08
*** srwilkers has joined #openstack-kolla15:10
*** lrensing has quit IRC15:11
*** lrensing has joined #openstack-kolla15:11
sdakeegonzalez you good with the sudo's in the kolla kubernetes deployment uide15:12
egonzalezsdake, yep not a problem just was saying15:12
*** Guest52040 has joined #openstack-kolla15:12
sdakeegonzalez thanks :)15:13
sdakeegonzalez fwiw I htink kolla-ansible needs this same sort of change15:13
sdakeegonzalez although I am not the person to change it15:13
sdakei dont understand why that change was made to the documentation guidelines15:13
sdakeif we understood that perhaps we would have better understanding of how to proceed15:14
*** fooliouno has joined #openstack-kolla15:14
*** rwallner has joined #openstack-kolla15:17
sdakesbezverk thanks for confirming that we don't need the insecure-port15:18
*** rwallner has quit IRC15:18
*** rwallner has joined #openstack-kolla15:18
japestinhothanks egonzalez it work well after I reconfigured it. I used 1024 instead 2000 like you told. cheers :)15:21
egonzalezjapestinho, probably you have a low ulimit in your system and will likely need to increase it if don't want to have a bottleneck in memcached15:22
*** shashank_t_ has quit IRC15:24
*** shashank_t_ has joined #openstack-kolla15:24
*** sayantan_ has joined #openstack-kolla15:30
*** srwilkers_ has joined #openstack-kolla15:30
*** srwilkers has quit IRC15:31
kfox1111morning.15:31
*** tovin07 has joined #openstack-kolla15:32
*** Jezekus has quit IRC15:34
*** mbruzek has quit IRC15:37
*** mbruzek has joined #openstack-kolla15:38
*** mbruzek has joined #openstack-kolla15:38
sdakemorning folks15:39
sdakekfox1111 morning due15:39
sdakekfox1111 the deploy docs  are almost done15:39
sdakekfox1111 after they merge, I am considering moving all the o ther docs out of the repo permanently15:40
sdake(as in git rm)15:40
*** iceyao has joined #openstack-kolla15:40
sdakekfox1111 to avoid confusion15:40
sdakeone doc to rule them all for the short term15:40
sdakei may keep the deployment doc for the short term as well - although that depends on the maintenance status of halycon15:40
sdakeor if we need to reinvent a new dev env we maintain15:41
kfox1111ok with me.15:41
kfox1111its hard enough to keep one doc up and going when its not gated and contains lots of bits.15:41
kfox1111let alone several.15:42
kfox1111and we're drasticly changing some of the bits.15:42
kfox1111(config stufff)15:42
kfox1111though I think we might be able to seperate the docs up just a bit so some can stay?15:42
*** unicell1 has joined #openstack-kolla15:43
kfox1111like, seperate the dev docs to just be, setup k8s with halycon, then use the deployment docs from there?15:43
sdakeit isn't clear to me halcyon will be maintained from speaking  with portdirect15:44
sdakei contributed to it and got yelled because of legal stuff15:44
sdakeyelled at15:44
*** lucasxu has quit IRC15:44
sdakeso - i can't fix it personally15:44
*** unicell has quit IRC15:44
sdakei really want to document how to setup kubernetes as well15:45
kfox1111ok. I don't really have a horse in that race.15:45
sdakeas the upstream docs are poor15:45
sdakei dont have a horse in that race either15:45
portdirectsdake: I'm not in any position to comment re halycon, simply that I dont use it - v1k0d3n may have some insight though15:45
sdakei just want a dev env that works15:45
kfox1111my point though was, we should maybe split the docs into "setup k8s", "setup kolla-kubernetes"15:45
sdakeportdirect fair enough15:45
sdakekfox1111 agreed15:45
v1k0d3nwhat seems to be the trouble?15:45
sdakekfox1111 i'd like to do that after the deployment guide is merged15:45
sdakev1k0d3n kubernetes 1.6.1 came out15:46
v1k0d3nyup15:46
sdakepeople complaining halcyon doesn't work for them anymore15:46
sdakeit may be a temporary thing15:46
sbezverkkfox1111: I got 1.6.1 gate update working15:46
sdakesbezverk nice :)15:46
sbezverkuntil centos repos went kaput15:46
kfox1111sbezverk: nice. :)15:46
sbezverkkfox1111: so now all centos jobs are bloody red15:47
v1k0d3nis it something that your cores can fix sdake ?15:47
kfox1111sbezverk: fun. :/15:47
kfox1111sbezverk: we really need to finish that repo cache. :/15:47
v1k0d3nmeaning...is it fixable?15:47
kfox1111sbezverk: I'll try and prioritize getting the cache going.15:47
*** Jezekus has joined #openstack-kolla15:48
sbezverkv1k0d3n: opensatck infra is working on it..15:48
v1k0d3nsounds good sbezverk!15:49
v1k0d3nlet me know if i can help at all.15:49
sdakev1k0d3n well they aren't my cores - but I dont know if they can fix it - everyone spread pretty thin15:50
*** shashank_t_ has quit IRC15:50
sdakev1k0d3n yes halcyo is probably fixable - its changing some configs around and stuff15:50
sdakesbezverk openstack infra is working on what?15:51
sdakesbezverk halcyon?15:51
*** shashank_t_ has joined #openstack-kolla15:51
v1k0d3nwhen i say "your", that's as in kolla ;)15:51
sdakev1k0d3n ok :)15:51
sbezverksdake: no fixing centos repos15:51
sdakesbezverk right - i think you may ahv econfused v1k0d3n there with halcyon being fixed (or atleast i was confused by it)15:51
v1k0d3nif something is wrong with halcyon, i would encourage anyone to contribute to it.15:52
v1k0d3nthat's why it's completely open source. :)15:52
sbezverkmy mistake, I must have missed the point of conversation.. I was talking about my pain ;)15:52
sdakesbezverk your pain is real ;)15:52
*** rmart04 has quit IRC15:52
v1k0d3ni don't run that daily. and as you said, 1.6.1 just came out. in fact, i just updated our kube-controller image this morning, and _still_ have yet to test it (although that's my intention soon).15:53
kfox1111v1k0d3n: still doing a custom kube-controller image for rbd support? or for a different reason?15:54
*** tonanhngo has joined #openstack-kolla15:54
*** tonanhngo has quit IRC15:54
kfox1111v1k0d3n: cause if its to slide in rbd into the image, I think tweaking the cephfs external plugin here: https://github.com/kubernetes-incubator/external-storage to support rbd as well would be awesome. :)15:55
*** shashank_t_ has quit IRC15:55
kfox1111would benifit everyone trying to use ceph and k8s.15:56
*** tonanhngo has joined #openstack-kolla15:56
v1k0d3n+115:56
*** hieulq_ has joined #openstack-kolla15:57
v1k0d3nbut to be honest...it's not like it was really causing us issue to begin with.15:57
v1k0d3ndefinitely not anymore painful than having an automated build :)15:57
kfox1111true. true. just thinking its one less thing to maintain.15:58
kfox1111right.15:58
*** shashank_t_ has joined #openstack-kolla15:58
v1k0d3nesp after portdirect 's latest container changes15:58
*** hieulq__ has joined #openstack-kolla15:58
* v1k0d3n nodes in the direction of portdirect15:58
sdakeportdirect  did you see lastpass was hacked15:59
sdakeportdirect or atleast an exploit was found15:59
sdakeportdirect did require clicking on stupid links15:59
v1k0d3nagreed. but that's litterally the _only_ debt we've really maintained. it's all good, and appriciate the link!15:59
portdirectsdake: I'm telling you, Password12345, no one has ever cracked that puppy15:59
sdakeportdirect the recommendation from lastpass is to upgrade the plugin15:59
sdakeportdirect :)15:59
portdirectsdake: not looked into it, but fallout from cloudflare?16:00
kfox1111portdirect: "thats amazing... I have the same combination on my luggage" :)16:00
v1k0d3ngood ol' keypass still working just fine for me :)16:00
Serlexditto16:00
sdakeportdirect no - its a completeliy new security vulnerability16:00
portdirectkfox1111: :)16:00
kfox1111https://www.youtube.com/watch?v=a6iW-8xPw3k16:01
*** pcaruana has quit IRC16:01
*** egonzalez has quit IRC16:02
sdakenew server on the way - arrives wednesday16:02
*** hieulq_ has quit IRC16:02
sdakeneed to order ram for it16:02
srwilkers_sdake, awesome16:02
sdake32 hyperthread cores, will have 128gb ram16:02
sdakeit shipped with 32gb16:03
sdake2.1 ghz cpu turbobosot up to 3.0ghz16:03
sdakeonly 220016:03
sdakehas all kinds of rockin management features16:03
sdakesbezverk has one at his house16:04
sdakewith possiblly a better cpu16:04
*** blallau has quit IRC16:05
*** asmaa has quit IRC16:11
sbezverk32 hyperthreads for dual cpu system is not too great.. most of gear I have use 28 hyperthreads per cpu16:11
sbezverkI noticed I run out of cores faster than out of memory16:11
kfox1111heh. I still mostly turn it off. :)16:11
kfox1111really? weird.16:11
*** tovin07 has left #openstack-kolla16:12
portdirecthyperthreads16:12
portdirectlike threads but hyper16:12
kfox1111its almost always the other way around for us. memory's scarce. cpu's extremely cheep.16:12
rwellumsdake: so following the latest doc, with one exception - and that being the timing of setting the insecure port, and k8s comes up great but nslookup kubernetes is failing.16:12
*** TxGirlGeek has quit IRC16:12
sbezverkkfox1111: cause I use for NFV which are very cpu hungry since they talk directly to pci passthrough devices for higher performance16:13
kfox1111sbezverk: ah... ok.16:13
kfox1111sbezverk: you should check out these: https://exxactcorp.com/Intel-Xeon-Phi/KNL-Solution.php?gclid=Cj0KEQjwiI3HBRDv0q_qhqXZ-N4BEiQAOTiCHiHZAq-7hOqiwkEs7clBCBISPEPSCF3k2aovmNptp1IaAkXL8P8HAQ :)16:14
kfox1111a lot of cores. :)16:14
sbezverkkfox1111: wow 72 cores per CPU I love the sound of that16:14
kfox1111they started with something that looked kind of like a hybrid between a thinned out xeon, and a gpgpu.16:15
kfox1111in the latest iteration they moved it form a board, to a socket like a normal processor.16:15
kfox1111runs a linux kernel as you'd expect. so more closer to a bigiron box then an accellerator now.16:16
*** matrohon has quit IRC16:19
sbezverkyeah, but repos keep crashing ;)16:19
kfox1111infra's aware of the issue?16:19
sbezverkyep, I reported it this morning 2 times, since they are not early risers16:20
kfox1111k16:20
*** hieulq__ has quit IRC16:21
kfox1111so, both containerd and rkt made it into the cncf... interesting.16:22
*** hieulq__ has joined #openstack-kolla16:22
*** ipsecguy_ has joined #openstack-kolla16:24
*** hieulq__ has quit IRC16:25
*** jascott1_ has joined #openstack-kolla16:25
*** ipsecguy has quit IRC16:26
*** TxGirlGeek has joined #openstack-kolla16:26
sdakerwellum gah16:26
sdakekfox1111 a cpu has 10 cores, hyperthreading turns it into 20 cores16:26
sdakekfox1111 (e.g)16:26
sdakekfox1111 the cores share some of the local cpu hardware, but it is essentially a real core16:26
sdakeits a little slower and more cache stalls16:26
kfox1111sdake: I know what it is. ;)16:26
sdakehowever worth the cpu16:27
sdakeok16:27
kfox1111I come from an hpc background.16:27
sdakesorry16:27
kfox1111:)16:27
sdakeyou sounded like you ddin't :16:27
kfox1111the fact that they are almost real, but not quite,16:27
kfox1111actually causes a lot of codes more pain then benifit.16:27
sdakeyour correct16:27
kfox1111and most codes tend to be memory constrained, not cpu constrained, so turning them off tends to balleance an hpc system better.16:27
kfox1111see the link about knights landing. a lot of cores. real and hyperthreads. :)16:28
kfox1111so many cores they can't use a traditional switched smp fabric.16:29
sdakeya - having 128 cores with 64gb ram is not all that helpful :)16:29
kfox1111thats why they became much more interesting when they moved them to a socket. :)16:30
kfox1111no longer constrained by the ram soddered onto the board.16:30
*** jascott1_ has quit IRC16:31
kfox1111here's a better link to them: http://www.intel.com/content/www/us/en/products/processors/xeon-phi/xeon-phi-processors.html#promotions16:32
*** Manheim has joined #openstack-kolla16:34
kfox1111and a picure of the arangement of the processors: https://www.google.com/search?q=phi+x200+architecture&safe=active&tbm=isch&imgil=A_gBQkg4z9OEUM%253A%253BAZaxZyo8Y3P4vM%253Bhttps%25253A%25252F%25252Fwww.servethehome.com%25252Fintel-xeon-phi-knights-landing-generation-now-available%25252F&source=iu&pf=m&fir=A_gBQkg4z9OEUM%253A%252CAZaxZyo8Y3P4vM%252C_&usg=__dE_NGDX7g2iL4feRAp8-aIRdevo%3D&biw=114216:35
kfox1111&bih=814#imgrc=A_gBQkg4z9OEUM:16:35
*** jascott1_ has joined #openstack-kolla16:35
*** nathharp has quit IRC16:37
sdakethe memory controllers in modern chipsets are the next great frontier to conquer for performance  and scalability16:37
*** lucasxu has joined #openstack-kolla16:39
*** iceyao has quit IRC16:39
kfox1111that and disk controllers/software I think.16:39
kfox1111flash is outpacing the iosubsystem.16:39
kfox1111bbiab16:39
*** Manheim has quit IRC16:39
*** Manheim has joined #openstack-kolla16:40
rwellumsdake: fwiw I proceeded and memcache, mariadb and rabbitmq came up successfully, keystone which was the next chart is just continuously restarting.16:41
sdakerwellum can you paste kubectl get pods -n kolla16:41
rwellumhttps://www.irccloud.com/pastebin/ahjmeahE/16:42
rwellumand describe pod...16:44
rwellumhttps://www.irccloud.com/pastebin/0AZBZXpt/16:44
rwellumBut maybe without nslookup kubernetes working this is a non starter16:45
*** unicell1 has quit IRC16:48
*** sayantan_ has quit IRC16:51
*** gfidente is now known as gfidente|afk16:53
*** jascott1_ has quit IRC16:53
*** shashank_t_ has quit IRC16:54
*** sayantan_ has joined #openstack-kolla16:55
*** eaguilar_ has joined #openstack-kolla16:55
*** eaguilar has quit IRC16:56
inc0good morning my dear community16:58
inc0or should I say, dearest of communities16:58
*** athomas has quit IRC16:59
*** dciabrin has quit IRC17:03
*** jascott1_ has joined #openstack-kolla17:05
*** mgoddard has quit IRC17:06
*** mbruzek has quit IRC17:08
Mech422inc0: Morning17:08
*** rwallner has quit IRC17:09
sbezverkinc0: kfox1111: have you seen email from infra about disabling epel? anyu thoughts about this??17:13
sbezverkdo we need to change our gate scripts??17:14
kfox1111looking...17:17
*** unicell has joined #openstack-kolla17:18
kfox1111sbezverk: I think we use jq on the host? I think thats an epel package?17:19
kfox1111so we may need to tweak our scripts to enable it.17:19
kolla-slack<srwilkers> Howdy @inc017:20
sdakerwellum i am a bit slammed atm with work related things - I should free up around 3pm MST17:20
sdakerwellum i know thats not the answerou want to hear17:21
sdakerwellum i just got ambushed ;)17:21
kfox1111sbezverk: I'l do a ps real quick.17:21
sbezverkkfox1111: crudini17:21
kfox1111ah. that too.17:21
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Reenable EPEL  https://review.openstack.org/45327117:22
sdakesup srwilkers_17:24
sbezverkkfox1111: we will not be able merge it though :( until mirrors get fixed and for that there is no eta :(17:25
kfox1111sbezverk: no worries. its there so we can pull the trigger as soon as we're ready.17:26
*** sambetts is now known as sambetts|afk17:27
*** Guest52040 has quit IRC17:28
sdakesbezverk current gateseems flakey and i'm not sure if its my cinder patch or the gate17:28
*** zhubingbing_ has quit IRC17:28
sdakesbezverk i think ll my patchstream is almost ready to go17:28
sdakesbezverk would you mind just  taking  look at th cinder gate17:29
sdakeand tell me if your patchset fixes that17:29
sdakesbezverk its the first one i submitted17:29
*** tonanhngo has quit IRC17:29
sbezverksdake: until the gate is back on track, no reason to bother with deleting things..17:29
sdakesbezverk kfox1111 https://review.openstack.org/#/c/452850/17:29
sbezverksdake: it will not be merged anyway17:30
sdakesbezverk well i'm not merging anything until the gate is working17:30
sdakesbezverk i just want to understand if the gate fialures are "known things" or if i need to make further changes17:30
sbezverksdake: if centos gates are failing on 20th second of run, what do you think can be checked in the log?17:30
sdakesbezverk 1hr timeout and 36min timeout17:31
sdakei dont see a 20 second of the run failing?17:31
sdakerather 36 min failure17:31
sdakehopefuly folks will be ok with merging docs as they dont affect the gate directly17:32
sdakeand they need to be published on docs.oo17:32
kfox1111sdake: looks good to me.17:32
mnaseris gate borked?17:32
* mnaser takes out wrench17:32
kfox1111sdake: yeah, I'm fine with docs only changes.17:32
sdakemnaser kolla-kubernetesgat ei17:32
mnaseroh i dont know much about that :-(17:32
sbezverksdake: cinder failures seems to be not related17:32
sbezverkto your ps17:32
sdakesbezverk kfox1111 thanks - i'll wiat for you to do your good work then :)17:33
kfox1111https://googleprojectzero.blogspot.com/2017/04/over-air-exploiting-broadcoms-wi-fi_4.html very interesting. :)17:33
kfox1111and scary.17:33
sdakesbezverk the plus side of this deletion patch is i have learned a whole lot more context about how kolla-kubernetes worked and how the gate works in general17:33
*** lucasxu has quit IRC17:33
sdake(worked in   the past)17:33
kfox1111sdake: :)17:33
kfox1111a bit sad that its almost dead now.17:34
kfox1111but, for the better.17:34
*** lucasxu has joined #openstack-kolla17:34
sdakekfox1111 which is almost dead?17:37
kfox1111kollakube17:37
sdakeright17:37
sdakei didn't break it :)17:37
*** rwallner has joined #openstack-kolla17:37
sdakei'll leave that to inc017:37
kfox1111yeah.17:37
sdakei may help a bit inc0 - if you need it as I understand what the code is doing now after 8 hrs of studying it17:37
sdakeas in a advisory role17:38
*** deadnull has quit IRC17:38
sdakeinc0 although you can probably pick it up pretty fast on your own17:38
sdakeinc0 justshout if you need a speed walkthrough later17:38
inc0give me a second, juggling few things17:38
sdakebtw folks - April 15th = deadline for 0.6.017:38
sdakeinc0 i can't do it right now17:39
sdakeinc0 this week is firehose wek and next week i am on travel mon-thur17:39
sdakeinc0 may be able to walk you thru it saturday - depends on my backlog17:39
kfox1111I think if inc0 focuses on finishging the genconfig merge,17:39
kfox1111and we work on getting a few of the remaining k8s objects over to helm, we'll be in good shape.17:39
sdakeinc0 also zhubingbing is helping with your genconfig work17:39
sdakekfox1111 agreed17:39
kfox1111we should be pretty close to killing off kollakube at that point.17:40
sdakewe have secret-generator.py which for the moment seems like a suitable solution short term17:40
kfox1111yeah. it works ok.17:40
sdakei will submit a kolla-genpwd to kolla-kubernetes repo when time permits17:40
*** lucasxu has quit IRC17:40
kfox1111I think phase 1 & phase 2 should be doable by 0.6.0.17:40
sdakewe will need that to eject kolla-ansible as a dep17:40
sdakekfox1111 which is phase1 and phase 2:)17:41
kfox1111oh. true.17:41
*** lucasxu has joined #openstack-kolla17:41
kfox1111phase 1, get rid of genconfig. phase 2, get rid of kollakube. phase 3, get rid of the need of a git checkout.17:41
sdakesam wrote part of that code i wrote the other half17:41
kfox1111ie, helmize all the things.17:41
sdakecool phase 3 = 0..7.0 :)17:41
kfox1111yeah. that would be nice.17:42
sdakeas i am pretty much out until the 15th i wont be able to contribute much to phase 1 and phase 217:42
sdakei do intent to finish the job on the deployment guide17:42
sdakei'd like it merged and working before i travel17:42
sdakeits really close17:42
sdakekfox1111 can you look at htis output: http://paste.openstack.org/show/60537117:43
sdakekfox1111 and advise what i'm doing wrong in the deploy guide17:43
sdakeor sbezverk ^^17:43
kfox1111sdake: a whole lot of reasons for that.17:43
*** ccesario has joined #openstack-kolla17:43
sdakekfox1111 look at the backtrace17:44
kfox1111check the nova api logs first.17:44
sdakeamqp timing out17:44
sdakethe backtrace is a new feature in ocata for nova17:44
sdakei'm wondering if tis networking related17:44
kfox1111a whole lot of reasons for that....17:45
*** ccesario has quit IRC17:45
kfox1111dig into the logs.17:45
kfox1111openstack error reporting is horible. :/17:45
sdakegah disabling epel?17:45
sbezverksdake: you know that log is useless ;) you need conductor and scheduler logs17:46
*** ccesario has joined #openstack-kolla17:46
kfox1111sdake: you trying to deploy 4.x?17:46
kfox1111if so, you probably need some more stuff thats in the ironic gate.17:46
sdakerwellum looking at that crashloopbackoff - I can't help butwonder if you started with a dirty environment and fialed to delete /var/lib/kolla/volumes/*17:46
sdakehttps://www.irccloud.com/pastebin/ahjmeahE/17:46
sbezverksdake: keystone does not have a persistent volume17:47
sbezverkrwellum: do not delete volume, you screw up your whole setup17:47
sbezverkrwellum: I suggest to run keystone-cleanup service17:48
sbezverkit will gracefully delete keystone DB from mariadb17:48
*** mgoddard has joined #openstack-kolla17:53
*** skramaja has quit IRC17:53
*** mbruzek has joined #openstack-kolla17:54
sdakesbezverk right - but it uses mariadb17:54
sdakesbezverk when  Imentioned the deletion of that volume - I meant for him to runcleanup-k8s.sh17:55
sbezverksdake: do not see logic, he has running mariadb/rabbitmq17:56
sbezverkwhy runclean?17:56
sdakesbezverk right - was just a gues17:57
sbezverkall he needs, or at least first step would be just delete keystone db if it is there and that is it17:57
sdakesbezverk i'm learning how kolla-kbuernetes works17:57
sdakesbezverk i dont have the same set of context you do17:57
sdakealthoug hcatching up raipdly :)17:57
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Disable placement for v3 gate.  https://review.openstack.org/45327717:57
sdakealso we have 20 peopel on that doc review17:57
sbezverksdake: man care should be taking before giving such recommendations ;)17:57
sdakeso hopefully they will be helping write code in the future17:57
sdakesbezverk the documentation has a big ass red warning box around that section :)17:57
*** sayantan_ has quit IRC17:58
sdakesbezverk its in trobuleshooting (If all else fails -> do this)17:58
sdakesbezverk it uninstalls the rpms, cleans up everything without a reboot, etc17:58
sdakekbueadm reset is farimproved in 1.6.117:59
*** ccesario has quit IRC18:02
*** dciabrin has joined #openstack-kolla18:04
sbezverksdake: kubeadm reset is maintained by people who develop kubernetes, so if there is a change it will be intgrated into kubeadm reset way before you can add it into your script and what is most important, it is minimally sufficient, maybe you clean up more, but who said it really needed??18:05
*** Jezekus has quit IRC18:05
*** ccesario has joined #openstack-kolla18:06
rwellumsdake: I always start from a clean VM18:06
*** eaguilar_ has quit IRC18:08
*** sayantan_ has joined #openstack-kolla18:08
*** jmccarthy has quit IRC18:08
rwellumSo that when working through your deploy guide - it's a fair attempt - not cluttered with previous.18:10
bswartzhey guys I've been trying to get kolla-ansible working for a while now and I'm following the quickstart guide -- I got most of the way though the guide but I end up getting stuck in a container reboot loop (the kolla-toolbox container) when I run kolla-ansible deploy and I can't debug it and I can't fix it18:12
bswartzis there a better set of docs than the quickstart guide?18:12
sdakerwellum right18:13
sdakesbezverk kubeadm reset does not clean up some things like /opt/cni if you switch cni drivers18:13
sdakesbezverk in 1.5.1 it didn't clean up anything18:13
sdakein 1.6.1 it cleans up almost everything18:13
sdakemy script needs tidying as a reuslt of this iimprovement in behavior18:14
sdakeI wan't remove-k8s to return a system to pristine status18:14
sdakeits not ther eyet18:14
sdakesbezverk i do agree with you - kubeadm dudes will do abetter job of maitaning kubeadm then my hacky cleanup script18:14
sdakesbezverk although they aren't quite there yet18:14
sdakesbezverk that script was written for 1.5 - it needs tuning for 1.618:15
sdakealot of tuning18:15
Mech422sdake: how's the new servers working out ?18:15
sdakeit can probably be 3 or 4 commands instead of 2018:15
sdakeMech422 arrives wednesday18:15
sdakeMech422 not sure if i can rack it in time for my travel18:15
Mech422sdake: sweet :-)18:15
sdakesbezverk i actually didn't write the script, vhosaot did18:15
sdakevhosakot did rather18:15
*** Pavo has joined #openstack-kolla18:15
sdakerwellum thanks for the compliment - although it was eveyrones contribution not necesairly mine18:16
sdakeI just organized eveyrones braindumping in that review :)18:16
sdakerwellum i htink its not quite perfect yet18:16
sdakerwellum although very very close18:16
sdakerwellum eanylin was MVP in that review imo - he sorted out alot of specifics based upon how the gate works18:17
sdakerwellum eanylin may be able to help you while i'm out18:17
rwellumsdake: at this rate I'm going to have to coordinate my vacation with yours lol.18:18
sdakerwellum its not a vacation its upstream servant leadership training18:18
rwellums/vacation/$job/g18:19
sdake:)18:19
sdake$$JOB$$18:19
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Helm Native Values.  https://review.openstack.org/44982618:21
rwellumsdake: I've never had a stable cluster where everything looks clean, all pods up - but dns fails - before today that is18:21
*** g3ek has quit IRC18:22
*** rwallner has quit IRC18:22
sdakerwellum is your dns failing?18:23
sdakerwellum (step 2 in the doc)18:23
rwellumsdake: yeah mentioned that a couple of times earlier - sorry if you missed.18:23
sdakerwellum ok -in a meeting atm - could you paste your kubectl get pods -n kube-system18:24
sdakerwellum i'm curious by what you mean "dns failed"18:25
rwellum'nslookup kubernetes' step18:25
sdakerwellum its m ybad i missed it - firehose today :)18:25
rwellumnp sdake18:25
sdakeok - so that fails - is the dns service in 3/3 ?18:25
rwellumhttps://www.irccloud.com/pastebin/ahjmeahE/18:26
rwellumsdake: ^^^18:26
rwellumSo yeah 3/3 - which is what I meant by I've never seen dns fail when pods look healthy18:27
*** jtriley has quit IRC18:28
sdakerwellum hrm - well your deployment looks good18:29
sdakerather kubernetes18:29
sdakehowever - kolla-kubernetes will not deploy if step 2 fails18:29
sdakei guess I should add a warning box there or something18:29
*** g3ek has joined #openstack-kolla18:31
rwellumWe had a warning there once18:31
sdakeya - i'll add it back when i get a chance if you lea ea comment and -1 the review18:32
*** shardy has quit IRC18:32
sdakestruggle to keep a record of the changes requested if they aren't in gerrit :)18:32
sdakeok - so on to your problem18:32
sdakeyour environment is bokred18:32
sdakei'm wondering if you can start over18:32
sdakeif kubernete sdoesn't deploy correctly, bd things happen18:32
rwellum"If nslookup kubernetes fails, kolla-kubernetes will not deploy correctly." - you have it covered18:32
*** rwallner has joined #openstack-kolla18:33
rwellumI did it 2x from a fresh VM sdake18:33
rwellumbut I'll do it again - don't mind.18:33
sdakewell18:34
sdakefresh vm18:34
sdakemaybe its a networking thing18:34
sdakecan you run ip a -> paste18:34
*** kolla-slack has quit IRC18:35
*** kolla-slack has joined #openstack-kolla18:35
rwellumsdake: on a repro or on the current bad state VM?18:36
sbezverkrwellum: there are a couple of technic you can use to debug this.. 1 I personally like is to add a sidecar debug container which runs along side of the container you debug and does tail -f on the crashing container log18:36
sbezverkthis methof guarantees you will understand why your cotnainer is crashing..18:36
sdakesbezverk rwellum 's basic kubernetes dns resolution is not working18:37
sdakesbezverk this is step 2 (validate kubernetes deployment)18:37
rwellumsbezverk: I heard folks talking about that (sidecare) but haven't experimented yet. But this seems more basic. i just continued to see how for i could get - but as sdake says the dns seems borked.18:37
sbezverksdake: do you have a log confirming that?18:38
sdakerwellum pleae psate your nslookup after running the busybox image18:38
sdakerwellum it is entirely possible the ips i picked in the doc are colliding with your vm's ip addresses18:39
sbezverkrwellum: easy test, bring up centos container and connect there with docker exec bash, from within if you do ping mariadb, you should see if names gets resolved..18:39
sdakesbezverk there is actually an easier way - you can run busybox and nslookup kubernetes18:40
sdakewhich returns the dns ip address18:40
*** signed8bit is now known as signed8bit_Zzz18:40
rwellumsdake: did you change the ip18:40
kfox1111sdake: we should finish making the test_kubernetes_dns.sh script fully generic.18:40
rwellum(s)18:40
sdakekfox1111 cool - i just document it so people can undertand it :)18:40
kfox1111make it really easy to debug dns issues, rather then ask folks to manually run stuff.18:40
sdakerwellum indeed18:40
sbezverksdake: returning dns ip address is not the same as name resolution18:40
sdakerwellum everything i did is in the doc18:40
rwellumhttps://www.irccloud.com/pastebin/0oXnlmun/18:40
sbezverkrwellum: your resolv.conf does not look good18:41
sbezverkI do not see search key18:42
rwellumsbezverk: there's a step: kolla-kubernetes/tools/setup-resolv-conf.sh kolla - but that's well after this test18:42
*** jtriley has joined #openstack-kolla18:42
sbezverk[root@kube-1 ~]# kubectl exec horizon-97956455-50f8p -n kolla -- cat /etc/resolv.conf18:43
sbezverksearch kolla.svc.sbezverk.cisco.com svc.sbezverk.cisco.com sbezverk.cisco.com sbezverk.cisco.com18:43
sbezverknameserver 10.57.160.1018:43
sbezverkoptions ndots:518:43
sbezverkrun the same command on rabbitmq, see what it shows to you18:43
*** signed8bit_Zzz is now known as signed8bit18:45
sbezverkkfox1111: infra advised not to run recheck until they say it is ok ;)18:45
kfox1111sbezverk: tehy told me its ok. ;)18:45
sbezverkkfox1111: hmm, pabelanger said nope, oh well18:46
*** Pavo has quit IRC18:46
kfox1111sbezverk: as of when?18:46
rwellumsbezverk:  https://www.irccloud.com/pastebin/pdRG5fxR/18:46
*** nathharp has joined #openstack-kolla18:47
kfox111114:19 < clarkb> kfox1111: pabelanger switched epel mirror sources and get it updated from working source so  our mirror should work now.18:47
sbezverkkfox1111: I guess they changed their minds ;-)18:47
kfox111114:33 < pabelanger> ya18:47
kfox111114:33 < pabelanger> needs to be 18:00 +18:47
kfox1111so should be fixed as of 18:00 utc.18:48
sbezverkrwellum: this one looks good18:48
sbezverkrwellum: kubectl get svc kube-dns18:49
sbezverkthen check if backend ip address is the same as assigned to kubedns container18:49
rwellumsbezverk: what am I doing wrong? https://www.irccloud.com/pastebin/OwpjGLg2/18:51
*** satyar has quit IRC18:51
kfox1111rwellum: -n kube-system18:52
sdakesbezverk yup - i didn't know tools/setup-resolv-conf.sh kolla is needed during step1 (deploying kubernetes) - it wasn't needed for my nenvironment18:53
kfox1111yeah, setup-resolv-conf.sh is a kolla-kubernetes thing, not a k8s thing.18:54
rwellumGah - thanks kfox111118:54
rwellumhttps://www.irccloud.com/pastebin/t2D2BZLA/18:54
*** eaguilar has joined #openstack-kolla18:55
sdakerwellum thta is what it should be18:55
sdakealthough your resolv.conf is totally borked18:55
sbezverkrwellum: same command with describe18:55
sdakeoh thats sbezverk 'z18:55
sdakerwellum your missing a sed operation from the documentation18:55
sdakerwellum or a sudo systemctl restart kubectl18:56
sdakerwellum the docs need to be updated to move that manifests change to later18:56
sdakefollowed y a sudo systemctl retart kubectl18:56
sdakerwellum i am certain that si what the problem is18:56
sbezverksdake: that new ;) "sdo systemctl restart kubectl"18:56
sdakesUdo :)18:56
rwellumI saw that had changed in the doc - but wasn't sure what exactly.18:57
sbezverksdake: no sdake, restarting kubectl is new feature ;)18:57
sdakelooks like a kubeadm bug to me :)18:57
*** mkoderer has quit IRC18:57
sdakerwellum - let me pull the current docs nd update it with what i think it should be18:57
sbezverksdake: you probably wanted to say kubelet18:57
sdakesbezverk right - my bad18:58
rwellumsdake: you mention 3 different things, 1. manifest move 2. restart kubelet and 3. a missed sed. :)19:00
*** mgoddard has quit IRC19:02
sdakerwellum ok - dont do anything yet :)19:02
sdakerwellum lets debug whaty you got19:02
sdake    sudo sed -i 's/10.96.0.10/10.3.0.10/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf19:02
sdakecan you paste that 10-kubeadm.conf file plz19:02
inc0uff I'm back19:03
inc0meeting all the way down19:03
sdakeinc0 \o19:03
inc0sdake: I know about Zhubings work, in fact I want to rebase all the patches19:03
inc0since they're in merge config now19:03
sdakeinc0 cool19:03
sdakeinc0 fire away imo :)19:04
inc0what else you guys wanted to talk with me about?19:04
sdakeinc0 nothing yet - will need a review of this deployment document19:04
inc0kk19:04
sdakerwellum can you paste that 10-kubeadm.conf file plz19:04
rwellumsdake: I noticed this sed command had changed https://www.irccloud.com/pastebin/n1V5z8s7/19:04
inc0I'm thinking whether we should include config generation without k-ansible already19:04
inc0since this thing is merging fast19:05
sdakerwellum changed where?19:05
inc0and we can replace whole section of deployment doc with single command19:05
*** matrohon has joined #openstack-kolla19:05
sdakerwellum in the deployment doc?19:05
rwellumIn the deploy doc yes19:05
sdakeinc0 we can merge as is19:05
sdakeand i'll maintain it - you have my word19:05
inc0kk19:05
inc0fine by me19:05
sdakeatleast until i'm dead or fired whichever comes first :)19:05
sdakeinc0 although its not quite ready yet19:06
sdakeits very close19:06
sbezverkkfox1111: I rechecked gate upgrade, it looks to be working. Appreciate if you ack it so the gate can be restored..19:06
sdakesbezverk link19:06
kfox1111k. sec...19:06
sbezverkit is still running, but here is the link: https://review.openstack.org/#/c/452993/319:06
kfox1111ah. ok.19:06
*** eaguilar_ has joined #openstack-kolla19:07
sbezverkmeeting time, will be back in 1.5 hour19:07
*** eaguilar has quit IRC19:07
kfox1111sbezverk: k. ping me when jenkins finishes, if  Idon't see it first.19:07
*** lucasxu has quit IRC19:07
inc0sdake: we can merge is as is soon and fix it in multiple following patches19:07
sdakerwellum - ok well i'm not sure when it changed - however your missing the sed doc and I just cherry-picked the repo and it looks correct19:07
sdakeinc0 yup -gie me couple hours19:08
inc0it's going to be wip anyway but having a base will be easier to handle19:08
sdakeinc 0its almost perfect19:08
inc0kk19:08
inc0glad to hear that:)19:08
*** lucasxu has joined #openstack-kolla19:08
sdakeit is missing how to make a vm actually function19:08
sdakewhich i hope eanylin can answer19:08
kfox1111sbezverk: ps looks nice though. gets rid of most of the hackies from 1.6.0. :)19:08
sdakeonce it gets on docs.oo it will be alot easier to handle19:08
sdakesbezverk ya lgtm19:08
rwellumsdake - this is the change:  https://www.irccloud.com/pastebin/ePbj7MtE/19:08
sdakesbezverk i'll =2 it now and if it passes the gte you can +2/+w if it has a kfox +2 on it19:09
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: Move memcached config to kolla-k8s  https://review.openstack.org/45062019:09
sdakerwellum in paste https://www.irccloud.com/pastebin/n1V5z8s7/ line 6 is wrong19:10
sdakerwellum this means you are missing the sed operation19:10
sdakeas for the --enable-cri=false no idea what that does19:10
sdakesbezverk kfox1111 any idea what CRI?19:10
sdakeis19:11
kfox1111sdake: cri's their container abstraction engine.19:11
kfox1111its beta in 1.6, but its broken with respect to shared pid namespaces. :/19:12
kfox1111the old driver is still there, and fully works. which is what that flag reenables.19:12
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735619:15
sdakekfox1111 cool so we need it19:15
sdakekfox1111 thanks for the explination19:15
rwellumsdake: trying again fresh vm :)19:18
*** rwallner has quit IRC19:18
sdakerwellum glad your persistent :)19:18
sdakerwellum we are almost there19:18
sdakerwellum where I am stuck in these docs is creating a vm fails19:18
sdakerwellum although I just found a bug in the doc (I think)19:18
sdakerelated to canal setup19:18
sdakerwellum which is fixed in that new review19:18
kfox1111hmm....19:19
kfox1111did seleneum change?19:19
kfox1111seeing this alot all of a sudden: http://logs.openstack.org/93/452993/3/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/8aadbfe/console.html19:20
*** sayantan_ has quit IRC19:21
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: Move cinder config to kolla-k8s  https://review.openstack.org/45055619:23
*** harlowja has joined #openstack-kolla19:24
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: Move rabbitmq config to kolla-k8s  https://review.openstack.org/45058119:25
sdakekfox1111 havne't seen that19:25
sdakeas of yesterday19:25
sdakeand i looked at aobut 20-30 gate logs19:25
inc0ok enough for now19:25
*** rwallner has joined #openstack-kolla19:25
kfox1111sdake: its happening a lot now.19:25
sdakegood thing we dont have voting gates ;)19:25
sdakecurls from all over the internet = winning :)19:26
*** shashank_t_ has joined #openstack-kolla19:26
inc0btw did k8s 1.6.1 fixed our stuff?19:26
sdakeinc0 waiting on the gate19:26
sdakebut we think it improved the situation vastly19:26
sdake1.6.0 is nearly doa19:26
inc0sdake: curl | sh is best deployment tool19:26
kfox1111inc0: hence the conversation about solinum breakage....19:26
sdakeinc0 you got it :)19:26
kfox1111bhel19:26
kfox1111bleh19:26
inc0that's why we love docker19:26
inc0docker is just glorified curl | sh19:27
inc0especially if you use dockerhub imgs;)19:27
kfox1111not exactly.19:27
inc0yeah I knwo19:27
kfox1111its still acceptable due to being able to clone.19:27
kfox1111not easy to clone curl.19:27
inc0not if someone changes :latest ....19:27
inc0and remember, :latest is default19:27
kfox1111thats why you save what you care about.19:27
kfox1111and why I don't use :latest. :)19:27
inc0you might not19:27
kfox1111that was the first thing I checked in solineum when debugging. did I use latest? nope.19:28
inc0but trust me, most people do19:28
kfox1111yeah. I know they do. and why I keep trying to educate them. :)19:28
inc0using docker run --privileged container:latest is equivalent to sudo curl | sh19:28
kfox1111right.19:30
openstackgerritOpenStack Proposal Bot proposed openstack/kolla stable/newton: Updated from global requirements  https://review.openstack.org/45330919:30
inc0that's why I generally dislike dockerhub tbh19:30
kfox1111but I don't recommend either of those. ;)19:30
inc0I'll always suggest to build your own containers19:30
kfox1111nothing wrong with the tool. something wrong with the usage.19:30
inc0base distro images are mainained enough to trust them19:30
kfox1111you gotta trust someone.19:31
inc0on top of that, I'll roll with stuff I control19:31
kfox1111I mostly agree, with the following caviot.19:31
kfox1111your trusting the source code from the source, so its not so bad to trust a prebuilt container built from that same source.19:31
*** ccesario has quit IRC19:31
*** signed8bit has quit IRC19:32
inc0kfox1111: well key is who builds it19:32
kfox1111especially if they have infra to test the binaries and you dont.19:32
*** manheim_ has joined #openstack-kolla19:32
*** signed8bit has joined #openstack-kolla19:33
kfox1111right. and my claim is, they may have better facilities to build it then general person off the street.19:33
*** signed8b_ has joined #openstack-kolla19:33
inc0well, I think mission of kolla is to provide facilities too...as much as possible19:33
rwellumsdake - where you have added the restart kubelet command, I actually need to do '    sudo systemctl daemon-reload'19:33
inc0I mean talking from purely kolla perspective, I will always suggest to build containers yourself19:34
sdakerwellum i dont understnad why that owuld be the case19:34
sdake.yaml is not loaded by systemctl19:34
sdakeall of the systemctl files re changed prior to a daemon-reload19:34
rwellumsdake: actually i was informed to do it:19:34
rwellumhttps://www.irccloud.com/pastebin/IB5qTqXw/19:34
rwellumSo i did and then k8s starts up19:35
inc0mnaser: ping19:35
mnasero/ inc019:35
kfox1111inc0: still think, if someone (kolla) is more likely to keep images up to date then a site, then the site should pull from the source and not build.19:35
sdakerwellum that points to a problem somewhere else in the guide19:35
inc0hey, did you do anything with registry thing? anything I can help with?19:35
sdakethere should only be one daemon-reload19:35
mnaseroh19:35
kfox1111its the same thing as password security. You can set the bar up so high, the people just write all passwords down on stickies,19:35
sdakei'll ahve a pass through it when i get back in the office at 319:35
mnaserinc0 https://review.openstack.org/#/c/401003/ waiting for infra reviews19:35
kfox1111completely eliminating the benifits intended.19:35
mnaserim waiting for open discussion in the meeting going on for infra at openstack-meeting19:35
mnaserto bring it up a bit more19:35
sdakerwellum does your dns now work?19:36
mnaseralso i built and published {centos,ubuntu}-{binary,source} for newton19:36
*** Pavo has joined #openstack-kolla19:36
sdakemnaser which version?19:36
mnaseri'll have to do oraclelinux binary/source and then work on mitaka19:36
inc0mnaser: will this be pushable registry?19:36
sdake3.0.3?19:37
mnasersdake yes19:37
mnaser3.0.319:37
sdakecool thanks mnaser19:37
sdakeoh newton - that would be 2.0.319:37
rwellumwell sdake I was looking at the latest sed command on canal.yaml trying to see if it 10.0.1 or 10.1.0 as you are suggesting19:37
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Fix seleneum blocking the gate  https://review.openstack.org/45331519:37
*** signed8bit has quit IRC19:37
mnasernewton would be 3.0.3 or 3.0.4 i cant remember?19:37
mnaseri believe ocata is 4.x19:37
mnaserand inc0 yes, though the details of how things will be pushed to it is another discussion19:37
sdakerwellum i think it should be 10.1.0.10 although i'd like advice from sbezverk or kfox1111 on what they think19:37
mnaserfor me i can dedicate the resources to get a registry running initially (small steps)19:37
sdakerwellum the original docs had 10.0.1.1019:38
inc0easier discussion19:38
kfox1111for what?19:38
sdakekfox1111 moment:19:38
inc0once we have access to hosted registry, pushing will be in our control19:38
rwellumtrying that now sdake19:38
inc0with caveat of secrets19:38
inc0which we can handle with infra somehow19:38
kfox1111inc0: we need secrets either way.19:38
mnaserbuilding oraclelinux for newton to push to dockerhub soon after19:38
mnaserkfox1111 that can be handled using a dedicated trusted pusher which was another thing that was discussed19:38
kfox1111secrets is the hard part. not where we push to.19:38
mnaseractually if you dont mind19:38
sdakekfox1111 https://review.openstack.org/#/c/447356/41/doc/source/deployment-guide.rst line 152-15419:39
mnaseri can link you to a really productive discussion yesterday with infra19:39
mnaserwhich was over irc19:39
kfox1111mnaser: that would be great. :)19:39
inc0yeah kfox1111 we discussed secrets among others things19:39
sdakekfox1111 this is one of the few unanswered queestions remianing in the document19:39
inc0with zuulv3 it's going to be easier, today it's just possible;)19:39
sdakeminus the question of whether cloud.yaml is correct or not :)19:39
sdakezuulv3 has been coming for a year now :)19:39
mnaserkfox1111 there is a bit before that but ill put in where i came in and where things went - http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-04-03.log.html#t2017-04-03T22:54:5419:39
inc0one idea was to copy stuff tarball style to secure machine and puish from it19:40
kfox1111sdake: it needs to match whatever you are giving kubeadm init.19:40
sdakekfox1111 kubeadm init is here:19:40
mnaserkfox1111 and it resulted in this https://review.openstack.org/#/c/401003/19:40
mnaserwhich i hope to work with infra team to get up and going19:40
sdakekfox1111 line 12219:40
sdakekfox1111 i am far from a network expert - struggling to understand this stuff19:41
sdakerwellum does your nslookup on kubernetes work?19:42
rwellumnegative sdake19:42
kfox1111sdake: to many things going on at the moment. they need to match between the init and the canal.19:42
sdakekfox1111 review: https://review.openstack.org/#/c/447356/41/doc/source/deployment-guide.rst line 122 is kubeadm init - line 152-154 are the sed - what is the answer 152, or 154?19:44
sdakekfox1111 i am not a networking genius19:45
*** krtaylor has quit IRC19:45
kfox1111paste here please.19:45
*** kbyrne has quit IRC19:46
sdake  sed -i "s@192.168.0.0/16@10.0.1.0/16@" canal.yaml19:46
sdake   may need to be:19:46
sdake   sed -i "s@192.168.0.0/16@10.1.0.0/16@" canal.yaml19:46
sdake    sudo kubeadm init --pod-network-cidr=10.1.0.0/16 --service-cidr=10.3.0.0/1619:47
kfox1111which one matches?19:47
sdakeright19:47
sdakeno idea what the answer is19:47
*** ccesario has joined #openstack-kolla19:47
sdakei know it seems like trivial networking things19:47
sdakeis it the first sed or second i need in the doc19:48
sdakei think its the second one19:48
kfox1111no, like, string match.19:48
sdakebut i am not sure as i deployed fine with the first variant19:48
kfox1111should be the latter one.19:48
kfox1111should be string identicle.19:48
sdakethanks19:48
sdakeI deployed with the first variant and it worked fine - although openstack didn't work19:49
sdakeopenstack deployed and could create aa vm - went to error state19:49
sdakei'll try that second version19:49
*** kbyrne has joined #openstack-kolla19:50
*** signed8b_ has quit IRC19:51
*** signed8bit has joined #openstack-kolla19:52
*** rstarmer has quit IRC19:57
*** rstarmer has joined #openstack-kolla19:59
*** ediardo has quit IRC20:00
*** ediardo has joined #openstack-kolla20:01
*** jemcevoy has joined #openstack-kolla20:01
*** rwallner has quit IRC20:01
*** dave-mccowan has quit IRC20:03
jemcevoyinc0:  Are you going to the k8n meetup Thursday? https://www.meetup.com/Portland-Kubernetes-Meetup/events/238574831/20:04
kolla-slack<jascott1> that looks pretty cool20:09
*** manheim_ has quit IRC20:12
sdakerwellum I'll be OOT for rougy 1.5 hours20:14
*** Manheim has quit IRC20:14
sdakebbiaf20:14
*** Manheim has joined #openstack-kolla20:14
*** rstarmer has quit IRC20:17
kolla-slack<jascott1> @kfox1111 that PR was just merged :slightly_smiling_face:20:19
kolla-slack<jascott1> (helm)20:19
*** Pavo has quit IRC20:21
inc0jemcevoy: KElsey Hightower presenting....I think I will20:23
inc0jascott1_, qwang wanna go too?20:23
jemcevoyKelsey is good20:24
jascott1_just hit the button and im waitlisted20:25
inc0hmm https://review.openstack.org/#/c/450620/ this gate output looks strange20:26
inc0yeah waitlist too:(20:27
inc0so are you Jim:P20:27
inc0worst case scenero we'll do Kolla meetup20:28
sbezverkkfox1111: we will not be able to test selenium until gate update gets merged20:32
qwanginc0: I wish it would be hosted at a different time. But if you're going, I'll try.20:33
sbezverksince I have good faith in 1.6.1 update, it was passing last night green20:33
*** matrohon has quit IRC20:33
inc0I'm thinking20:33
inc0I'll probably pass20:33
inc0Denise will be in town20:33
sbezverkI would suggest to merge it and then run other changes20:33
inc0so I'd rather meet by white shed20:34
inc0sbezverk: which gate change btw?20:35
*** Jezekus has joined #openstack-kolla20:37
openstackgerritMerged openstack/kolla-ansible master: Fix Fluentd warn on dnsmasq.log file parsing  https://review.openstack.org/45249820:40
sbezverkinc0: https://review.openstack.org/#/c/452993/320:43
*** nathharp has quit IRC20:43
sbezverkinc0: we have catch 22 situation, kfox1111 selenium got broken by some upstream changes20:46
sbezverkso it prevents gate update to get merged20:46
sbezverkbut we cannot fix selenium without working gate20:46
*** mbruzek has quit IRC20:47
sbezverkthat is why I am suggesting to merge gate update right now, it was ok last night, and then start dealing with other issues20:47
inc0well gates aren't voting20:47
inc0so we can20:47
inc0we shouldn't but we can20:47
sbezverkinc0: unfortunately we do not have a choice here20:48
sbezverkI do not like it either, it is just I do not see an alternative20:48
*** asalkeld has joined #openstack-kolla20:48
inc0yup, I know20:48
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu  https://review.openstack.org/43278720:48
inc0I'm happy that we don't have voting gates right now:P20:48
*** ccesario has quit IRC20:48
sbezverkinc0: right see good in bad20:49
inc0always an optimist eh?;)20:49
*** eaguilar_ has quit IRC20:49
sbezverkinc0: no it is more, there should not be a situation without a solution, even though solution might be not ideal one20:50
*** jascott1_ has quit IRC20:51
*** gfidente|afk is now known as gfidente20:55
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ironic-pxe: handle non-x86 architectures  https://review.openstack.org/43481720:55
*** Jezekus has quit IRC20:58
*** krtaylor has joined #openstack-kolla20:59
*** matrohon has joined #openstack-kolla21:00
*** TxGirlGeek has quit IRC21:01
*** jascott1_ has joined #openstack-kolla21:01
*** ksumit has joined #openstack-kolla21:04
ksumitHello all, I run into the following error while trying the Ansible-Kolla quickstart guide:21:04
ksumit'manifest for kolla/centos-binary-kolla-toolbox:auto not found' ; I'd appreciate if someone could point me in the right direction.21:05
*** dave-mccowan has joined #openstack-kolla21:16
sbezverkksumit: your tag is wrong21:17
sbezverkit should not be auto, it should be something like 4.0.0 or 3.0.321:17
ksumitThanks for the response! Any ideas on how I can change it? Like I said, I was just following the published quickstart guide.21:18
*** matrohon has quit IRC21:18
sbezverkksumit: if I am not mistaken, it should be set in globals.yml in /etc/kolla21:18
*** jascott1_ has quit IRC21:18
*** jtriley has quit IRC21:19
*** Serlex has quit IRC21:19
ksumitUnderstood. I'll take a look at it. Thanks!21:19
ksumitWhere can I see the available tags btw?21:20
*** lrensing has quit IRC21:20
*** eaguilar has joined #openstack-kolla21:21
sbezverkksumit: I do not think you can see them easily, at least I do not know how21:21
ksumitPerhaps on docker hub: https://hub.docker.com/r/kolla/centos-binary-kolla-toolbox/tags/21:22
*** gfidente has quit IRC21:22
sbezverkksumit: here you go :-)21:22
inc0ksumit: openstack_release config21:24
inc0in globals.yaml21:24
ksumitThanks @inc0!21:25
rwellumsbezverk: following the guide yet again, I got dns up and 4 services good. Just going down the list of charts - next on the list glance has not come up all the way.21:25
rwellumhttps://www.irccloud.com/pastebin/jK8kQ4E9/21:25
*** jascott1_ has joined #openstack-kolla21:26
rwellumWith so many pods created and sitting in Init - where do I begin debugging this?21:26
sbezverkrwellum: get logs from glance-create-keystone-service-s5pv521:27
sbezverkrwellum: it is normal that other containers are stuck in init, they wait service create to complete21:28
sbezverkrwellum: it could be that glance job cannot get in touch with keystone to create service end points21:28
rwellumsbezverk:  https://www.irccloud.com/pastebin/PtyHjZk4/21:29
sbezverkrwellum: kubectl log glance-create-keystone-service-s5pv5 -n kolla21:30
*** jrobinson has joined #openstack-kolla21:31
rwellumhttps://www.irccloud.com/pastebin/FTOHYoys/21:32
sbezverkrwellum: kubectl get svc -n kolla | grep 192.168.7.10121:33
rwellumNothing sbezverk21:33
sbezverkrwellum: make sense21:34
rwellumin my cloud.yaml I have21:34
rwellumhttps://www.irccloud.com/pastebin/XCx8tq0L/21:34
sbezverkrwellum: what is this address?21:34
*** david-lyle has quit IRC21:35
rwellumCopy-n-paste from the deploy doc :(21:35
rwellumit's not one of the addresses called out to replace.21:35
sbezverkrwellum: hmmm glance is not supposed to use any public ip address21:39
sbezverkit is supposed to use keystone-admin name21:39
rwellumIs that configurable or a default?21:41
sbezverkrwellum: can you do grep -r keystone_admin_url21:41
sbezverkin kolla-kubernetes repo21:41
ksumitSo I am running into a new issue now, and it looks like centos-binary-manila-data indeed doesn't exist on the Public Docker Hub. Any workarounds? - "kolla/centos-binary-manila-data not found: does not exist or no pull access"21:42
rwellumsbezverk https://www.irccloud.com/pastebin/9EI4d25s/21:42
*** manheim_ has joined #openstack-kolla21:43
*** manheim_ has quit IRC21:43
*** Pavo has joined #openstack-kolla21:43
*** david-lyle has joined #openstack-kolla21:43
sbezverkrwellum: ok in your cloud.yaml, any change you overwrite it?21:43
rwellumYou mean like this:21:44
rwellumhttps://www.irccloud.com/pastebin/M6bb3LtG/21:44
sbezverkrwellum: no this one looks ok21:44
*** Manheim has quit IRC21:45
rwellumConfused, I thought you just said shouldn't be set here?21:47
sbezverkrwellum: dns name will be used for external connectivity to keystone21:47
*** eaguilar has quit IRC21:47
sbezverkwhat I find strange is that internal containers are using external keystone ip address to communicate with it21:48
rwellumok21:48
sbezverkkubectl get svc -n kolla | grep keystone21:48
rwellumhttps://www.irccloud.com/pastebin/l1H3QW37/21:49
sbezverkrwellum: do you deploy glance using service charts?21:49
rwellumYes:21:49
rwellum    helm install --debug kolla-kubernetes/helm/service/glance --namespace kolla --name glance --values ./cloud.yaml21:49
sbezverkrwellum: why do you have different ip 192.168.122.85 and 192.168.7.101???21:50
sbezverkrwellum: helm delete glance --purge21:51
rwellumI think my cloud.yaml is just wrong.21:51
sbezverkrwellum: helm install --dry-run --debug kolla-kubernetes/helm/service/glance --namespace kolla --name glance --values ./cloud.yaml21:51
sbezverkrwellum: hm I thought you have deployed sucessfully kolla-kube once21:52
sbezverkyou should be able just reuse that cloud.yaml21:52
kfox1111back.21:53
sbezverkkfox1111: selenium does not allow to gate update merge, but your selenium fix will not get in without my gate update21:54
sbezverkI want to merge gate update as is, I know it was working then your fix can get merged..21:54
kfox1111sbezverk: one of them will have to merge without testing. :/21:54
kfox1111or we will have to merge them together.21:54
sbezverkkfox1111L last night gate update was green21:55
kfox1111for your 1.6.1 ps?21:55
sbezverkso I am confident21:55
sbezverkyes21:55
kfox1111k. I'll take your word for it. lets merge it.21:55
sbezverksounds good21:55
kfox1111can't get much more broken. :)21:55
*** manheim has joined #openstack-kolla21:55
sbezverkexactly21:55
sbezverkand we are here just in case ;)21:56
kfox1111yup. :)21:56
sbezverkyou will not need to rebase anything for selenium fix right?21:57
*** Pavo has quit IRC21:59
kfox1111I think a recheck will auto rebase after the merge.22:00
openstackgerritMerged openstack/kolla-kubernetes master: Upgrade gate to kubernetes 1.6.1  https://review.openstack.org/45299322:01
*** jascott1_ has quit IRC22:04
sbezverkkfox1111: it is merged, can you re-check22:05
kfox1111kk22:06
kfox1111sbezverk: parent values just merged. :)22:06
kfox1111so, this should be possible very very soon: https://review.openstack.org/#/c/449826/ :)22:06
sbezverkkfox1111: oh you mean in helm22:06
kfox1111right.22:06
kfox1111as soon as they cut a release. :)22:06
sbezverkkfox1111: nice22:07
*** signed8bit is now known as signed8bit_Zzz22:13
*** fooliouno has quit IRC22:14
*** srwilkers_ has quit IRC22:14
sbezverkkfox1111: http://logs.openstack.org/15/453315/1/check/gate-kolla-kubernetes-pep8-ubuntu-xenial/0acd199/console.html#_2017-04-04_22_09_02_12191022:16
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Fix seleneum blocking the gate  https://review.openstack.org/45331522:20
* kfox1111 sighs22:20
kfox1111oh. there was more to it...22:20
kfox1111know off hand the 3.x compatible excepton?22:20
kfox1111I hate they broke 2.x compataibilty with that one. :/22:21
sdakerwellum back in the office22:21
kfox1111oh. its 'as' now.22:21
sbezverkkfox1111: nope sorry22:22
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Fix seleneum blocking the gate  https://review.openstack.org/45331522:22
*** asalkeld has quit IRC22:24
*** lucasxu has quit IRC22:27
sdakehrw where do i buy an arm 64 server22:29
sdakerackmount22:29
sdakehrw reqiurements ipmi + PXE22:29
sdake+32gb ram22:29
*** jascott1_ has joined #openstack-kolla22:31
sdakeinc0 if you have a chance you might send an emial to the maiing list with tag [tc] asking humbly for a review of the stable:follows-policy review22:32
sdakeinc0 otherwise it wont be added to the agenda22:32
sdake192.168.7.101 is my machine's IP22:32
sdakerwellum sbezverk ^^22:32
sdakekfox1111 sbezverk just merge your two patches - that is how the gate is fixed on voting gates22:33
sbezverksdake: what it is doing on rwellum VM ;)22:33
sdakekfox1111 yes a recheck will rebase after a merge22:33
kfox1111sdake: yeah. going to see if the seleneum one actually works though. should now.22:33
kfox1111had a bug in it before. so gating as much as possible is still helpful.22:34
sdakesbezverk probably somethign in the guide needs customization22:34
sbezverksdake: most of gate jobs are past the part which gate update has changed22:34
*** signed8bit_Zzz is now known as signed8bit22:34
sdakesbezverk cool - i was just reading scrollback and adding commentary :)22:34
sdake(was out of office for 1.5 hours)22:34
*** jascott1_ has quit IRC22:35
sdakesbezverk kfox1111 if there is anything i can do to help please advise22:35
sdakesbezverk kfox1111 otherwise i'll keep working on the deployment guide - its almost settled22:35
kfox1111sdake: thanks. will do. I think its in hand though.22:35
sdakekfox1111 sounds good22:36
kfox1111ah... lbaas is kind of fun... but still will be so much nicer when I can get k8s to manage it instead of me manually managing it.22:37
sbezverkkfox1111: still the same selenium error: http://logs.openstack.org/15/453315/3/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-iscsi-nv/3713a0f/console.html#_2017-04-04_22_37_05_01957022:39
sdakemy recommendatio nis to comment selenium out of the gate for now - and then fix it as a followu pto unblock people from commits22:39
kfox1111wow.... thats super, super helpful.. exception='message'22:39
sbezverkkfox1111: it is job security for developers ;)22:40
kfox1111oh..... I see. it still is raising in that case...22:40
sdakesbezverk you know how the russians that worked on jets protected theirfamlieis during the cold war?22:41
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Fix seleneum blocking the gate  https://review.openstack.org/45331522:41
sbezverksdake: no idea22:42
sdakesbezverk each mechanic had 2 jets22:42
sdakethey tailored them specifically to all the tricks they knew22:42
sdakethey were treated well and their families were treated well22:42
sdakeif the jet crashed, they were screwed22:42
sdakethe only epople that could work on the jet was the dude who customized it :)22:42
sbezverksdake: good approach ;)22:43
sdakesbezverk that is job security done right :)22:43
rwellumsdake: short story is i followed the guide again, checking the sed commands did their job, use the correct 10.1.x address and I got dns up. Then i started running charts and blew up on chart #5=glance. Looks like the keystone address in cloud.yaml may be wrong. I moved to a correct address maybe but still stuck.22:43
sdakerwellum that 192.168.7.101 is my eth0 on my gear22:44
sdakerwellum you are doing clean vms on each go right?22:44
rwellumack22:45
rwellumand ack22:45
sdakerwellum note i cut and past eall the helm stuff all at once per block22:45
sdake(e.g. if its in a cut and paste block i cut and pate the whole thing)22:45
sdakei am rendering the existing docs now and trying to reproduce my deploy from 2am this morning22:45
sdakeit will take me about 20 mins to know iff it works22:46
rwellumsdake: we need a note then about another change in the cloud.yaml22:46
sdakerwellum did you modify network_interface to eth0 and neutron_network_interface to eth1?22:46
sdakerwellum agreed22:46
rwellumsdake: yeah22:46
sdakea note needs to go in about that last one too22:46
sdakei am going to make it a sed line I think22:47
sdakei am going to go through the existing docs first and verify they still wfm22:47
sdakeyou said you had five services up22:47
sdakewhic hones?22:47
sdakerwellum cherrypick correctly?22:49
sdakerwellum let me fix that cherrypick22:49
rwellumSorry I grossly exaggerated. The first four came up, ie. mariadb, rabbit, memcached and keystone22:49
rwellumsdake: probably not...22:49
sdakeyou may or may not need cinder for glance22:49
rwellumyeah but I was told you can bring these up in any order.22:50
sdakeok first let me fix the cherrypick22:50
rwellumkk22:50
rwellumdid you fix already then it rebroke?22:50
rwellumI think I'm a combination of a bull in a china shop and those infinite monkeys typing on infinite keyboards in my approach.22:51
sdakerwellum your rockin it dude22:52
sdakerwellum the key is your not giving up22:52
sdakerwellum this as you know is how software is developed :)22:52
sdakerwellum the cherrypick wfm22:53
sdakeI must have pushed that earlier this morning22:53
rwellumit worked, then stopped working - I put it in as a log in my comment22:53
sdakewhat worked then stopped working?22:54
rwellumthe cherry pick22:54
rwellumThis morning broken, then it was fixed, then tried this evening and broken again.22:55
rwellumv. confusing.22:55
rwellumBut I'll try again - 1s.22:55
sdakei just pulled the cherrypick and i twfm22:57
*** eanylin has quit IRC22:57
sdakeclick download->cherrypick in upper right hand corner, click copyt o clipboard22:57
sdakethen paste22:57
sdakeyou know, I may not have updated the docs...22:57
sdakelet me check that22:57
rwellumIt looks lik you have /1722:58
rwellummy notes had /1922:58
rwellumgit fetch git://git.openstack.org/openstack/kolla-kubernetes refs/changes/40/439740/17 && git cherry-pick FETCH_HEAD22:58
*** jtriley has joined #openstack-kolla22:58
sdakegit fetch ssh://sdake@review.openstack.org:29418/openstack/kolla-kubernetes refs/changes/40/439740/21 && git cherry-pick FETCH_HEAD22:58
rwellumAssuming something else.22:58
rwellumaha!22:58
sbezverkkfox1111: yeeaah first green!22:59
kfox1111nice.23:00
sdakerwellum infra team recommends:23:01
sdakegit review -x 43974023:01
sdakechanging docs as such23:02
*** manheim has quit IRC23:03
*** jtriley has quit IRC23:05
*** eragavi has quit IRC23:07
rwellumbrb sdake - dinner time23:07
*** jascott1_ has joined #openstack-kolla23:18
*** eaguilar has joined #openstack-kolla23:19
*** lamt has quit IRC23:20
*** jtriley has joined #openstack-kolla23:20
*** signed8b_ has joined #openstack-kolla23:20
kfox1111sdake: https://review.openstack.org/#/c/453315/ please.23:22
kfox1111almost back to fully working gate.23:22
*** signed8b_ has quit IRC23:23
sdakelgtm23:23
*** signed8bit has quit IRC23:24
*** jascott1_ has quit IRC23:24
sbezverkkfox1111: the multi node issue is so hard to troubleshoot, every time is differnet hick up..23:25
*** signed8bit has joined #openstack-kolla23:25
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap  https://review.openstack.org/43974023:25
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735623:25
kfox1111sbezverk: yeah. we may need to start gatering occurrences on the etherpad. maybe its multiple issues.23:26
*** signed8bit has quit IRC23:26
openstackgerritMerged openstack/kolla-kubernetes master: Fix seleneum blocking the gate  https://review.openstack.org/45331523:27
sdakekfox1111 there is a blueprint for continuous iprovement of the gate23:27
sdakethe etherpad is there23:27
sbezverkkfox1111: I guess23:27
*** goldyfruit1 has quit IRC23:37
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735623:39
sdakeOK23:40
sdakerwellum - rendering that with tox -e docs and then testing23:40
sdakenote that once this hits the repo, it will be published on docs.o.o23:41
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735623:42
sdakemaybe 44 will be the charm:)23:43
spsuryamorning guys23:44
*** qiliang27 has joined #openstack-kolla23:46
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Fernet Token Support  https://review.openstack.org/44627423:49
*** yuanying_ has quit IRC23:53
« Monday, 2017-04-03 Index Wednesday, 2017-04-05 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!