Thursday, 2017-03-30

*** shashank_t_ has joined #openstack-kolla00:00
kfox1111let me rephrase that. kubeadm has awlays configured kubectl on the user to work out of the box.00:00
kfox1111is that not true anymore?00:00
sbezverkkfox1111: it does not set context00:01
*** rwellum has quit IRC00:01
kfox1111kubeadm does not setup ~/.kube/config anymore?00:02
kfox1111that would be a weird regression.00:02
*** hieulq has joined #openstack-kolla00:03
*** yingjun has joined #openstack-kolla00:03
sbezverklooks like we need to do it manually now:00:06
*** schwicht has quit IRC00:06
sbezverk2017-03-29 23:46:00.492016 |   sudo cp /etc/kubernetes/admin.conf $HOME/00:06
sbezverk2017-03-29 23:46:00.492048 |   sudo chown $(id -u):$(id -g) $HOME/admin.conf00:06
sbezverk2017-03-29 23:46:00.492076 |   export KUBECONFIG=$HOME/admin.conf00:06
kfox1111mkdir -p ~/.kube00:07
kfox1111cp /etc/kubernetes/admin.conf ~/.kube/config00:07
kfox1111?00:07
kfox1111wait... how was this ever working...00:07
sbezverkexport KUBECONFIG=/etc/kubernetes/admin.conf00:07
*** rwsu has quit IRC00:08
kfox1111oh... maybe it was using the unauthenticated port? since it was all local.00:08
sbezverkin this case we will just use default config they genereted00:08
kfox1111I doubht its world readable?00:08
sbezverkkfox1111: that is why I enabled 808000:08
kfox1111if it was just 8080 though, it would just work?00:09
sbezverkbut I guess they now force to use secured 6443 port00:09
kfox1111if we copy it to ~/.kube/config,00:09
kfox1111then we don't have to worry about the environment var.00:09
kfox1111which could be dropped in some scripts somehow.00:09
sbezverkkfox1111: ok let's try00:10
kfox1111since 8080 isn't working anyway, maybe we disable it, and just try authenticated the whole way.00:10
*** sayantan_ has quit IRC00:11
*** signed8b_ is now known as signed8bit_Zzz00:11
*** sayantan_ has joined #openstack-kolla00:12
sbezverkkfox1111: one thing for sure, they do not set kube-system as default context00:12
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139100:13
kfox1111sbezverk: guessing you still need the chown?00:14
*** tonanhngo has quit IRC00:14
sbezverkkfox1111: it should inherit, no?00:14
kfox1111I'm guessing its root only.00:15
sbezverkkfox1111: ok will change it next time if it barks back00:16
kfox1111yeah... [kfox@mantis ~]$ ls -l /etc/kubernetes/admin.conf00:16
kfox1111-rw-------. 1 root root 9192 Oct 21 18:29 /etc/kubernetes/admin.conf00:16
kfox1111its going to fail at the cp.00:16
kfox1111the other option is:00:16
kfox1111sudo cat /etc/kubernetes/admin.conf > ~/.kube/config00:16
kfox1111though that might make the perms too loose and kubectl might complain/fail.00:17
kfox1111I think we'll need the sudo cp and the sudo chown for sure.00:17
*** tonanhngo has joined #openstack-kolla00:17
*** tonanhngo has quit IRC00:22
sbezverksudo cp /etc/kubernetes/admin.conf ~/.kube/config00:22
sbezverksudo chown $(id -u):$(id -g) ~/.kube/config00:22
sbezverkkfox1111: should work right?00:23
kfox1111looks ok I think.00:24
kfox1111uh...00:24
kfox1111wait.00:24
kfox1111not sure when the ~ will be interprited.00:24
kfox1111if that goes to the users dir, or root's dir.00:24
kfox1111users dir.00:24
kfox1111ok. I think its good00:25
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139100:25
sbezverkkfox1111: we will see soon enough00:25
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes 1.5.4 and kubeadm  https://review.openstack.org/45155600:26
kfox1111k. :)00:26
sdakeimo should unblock the gate asap and in parallel work on sorting out workarounds for 1.6.000:26
kfox1111sdake: +100:27
kfox1111sdake: that ps looks good to me, if it passes the gates.00:27
kfox1111thanks for working on it.00:27
sdakekfox1111 right - if it passses gate00:27
sdakethe centos bits do00:27
sbezverksdake: I would agree if ubuntu would be working00:27
sdakei am not ceratin on the ubuntu pins00:28
sbezverkotherwise we still have broken gate00:28
sdakesbezverk i submitted a new review which pins ubuntu as well00:28
sdakei was out most of the day - apologies for that00:28
kfox1111sdake: no worries. I've been out too.00:29
kfox1111but wasn't worried about it so much as sbezverk's been doing a great job. :)00:29
sdakesbezverk i'd definately keep working on 1.6.0 tho - we need that asap ;)00:29
sbezverksdake: I will, it is not in my nature to give up00:30
sdakesbezverk of course not :)00:30
sdakesbezverk that is why we get along so well :)00:30
sdakeamong other reasons00:30
sdakepersistence to a fault ;-)00:30
kfox1111yay persistence! :)00:32
*** schwicht has joined #openstack-kolla00:32
kfox1111the openstack mission review's really taking off. :)00:33
kfox1111very happy to see it.00:34
kfox1111just hoping at this point its more then just talk.00:34
sbezverkkfox1111: itdoes not want to communicate to 6443 :(00:35
kfox1111are we missing something in the config maybe?00:36
kfox1111add a cat ~/.kube/config please00:37
kfox1111maybe the certs are not readable.00:37
kfox1111hmm...00:37
kfox1111that failure is before the copy?00:37
sbezverkafter00:38
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139100:38
sbezverkI added netstat -tunlp00:38
kfox1111perhaps we're not giving enough time for kube-apiserver to start?00:38
sbezverkto see if api is listening on this port00:38
sbezverkpossibly00:38
kfox1111in the logs: http://logs.openstack.org/91/451391/23/check/gate-kolla-kubernetes-deploy-centos-binary-3-ceph-multi-nv/75d35f1/console.html00:39
kfox1111well....00:40
kfox1111yeah... should be after...00:40
kfox1111so... yeah. wondering if its more files needed...00:40
sdakegroan: http://logs.openstack.org/56/451556/3/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/799b87c/console.html#_2017-03-30_00_31_34_78042700:41
kfox1111gotta head out for a few. will try and check back soon.00:41
kfox1111heh. did they pull the package?00:41
kfox1111we really need an rpm cache. :/00:41
sbezverksdake: they removed it I guess00:44
kfox1111wrong solution to having named a release rpm -0. :/00:44
sdakehaha kfox111100:44
openstackgerritzhongshengping proposed openstack/kolla-ansible master: Remove min_l3_agents_per_router option  https://review.openstack.org/45115900:45
sdakecan't expect eveyrone to be an expert in rpm :)00:45
kfox1111"lets delete the alpha", not "lets name the release -1 like everyone else does"00:45
*** david-lyle_ has joined #openstack-kolla00:45
openstackgerritzhongshengping proposed openstack/kolla-ansible master: Remove verbose option  https://review.openstack.org/45115500:46
*** david-lyle has quit IRC00:48
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155600:49
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155600:54
sdakekfox1111 sbezverk if your leaving commentary upstream - you might ask them to follow the standard versioning for RPMs specified here: https://fedoraproject.org/wiki/Packaging:Versioning00:56
*** yangyapeng has joined #openstack-kolla00:56
*** cuongnv has joined #openstack-kolla01:02
*** eanylin has quit IRC01:05
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155601:09
kfox1111back.01:11
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139101:14
openstackgerritMarcus Williams proposed openstack/kolla master: Add OpenDaylight Container to kolla  https://review.openstack.org/41636901:14
kfox1111sbezverk: seems like it still isn't 100% credentialed for the jenkins user...01:17
kfox1111weird though, that the setup_kubernetes kubectl get pods seems to work...01:17
sbezverkkfox1111: I discovered another issue :( when AllowAlways is used, in my cluster apiserver after 3-4 minutes dies01:17
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Optimize reconfiguration for haproxy  https://review.openstack.org/42415601:17
kfox1111sbezverk: really?01:18
kfox1111weird.01:18
sbezverkkfox1111: with rbac it stays up01:18
kfox1111any logs?01:18
kfox1111thats unsettling....01:18
kfox1111that may be related.01:18
kfox1111cause it takes abit to build the helm packages.01:18
*** jrobinson has quit IRC01:19
sbezverkkfox1111: yeah, I think they should have not released it right after rc1, it looks too green to me01:21
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Split Openvswitch into own role  https://review.openstack.org/41679301:22
sdakeyup kubernetes alpha long gone01:23
sdakewho has a pool without water?01:23
sdakerather kubeadm01:24
kfox1111hopefully by 1.7, we will have rpm caching done. :/01:24
sdakerpm caching is relativey easy if you can coax infra into making a mirror f or you01:24
kfox1111yeah.01:24
kfox1111or we could just tar it up in the gate job we have already.01:24
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Fix cinder config group in nova.conf  https://review.openstack.org/45161301:24
sdakekfox1111 https://review.openstack.org/#/c/349278/01:25
sbezverkI am off for today, have a good one.. I will check back later if sdake's pin works I will +2 it.. I have a feeling, 1.6.0 uprgade might take a bit of time..01:25
sdakeubuntu passes ;)01:26
sdakekubeadm has some wonky new interfaces01:26
kfox1111sbezverk: mind if I carry on the ps?01:26
sdakettyl sbezverk01:26
sdakea 1.0.0 should have a deprecation policy imo :)01:26
kfox1111sdake: if we copy the k8s rpms the same way we are building the cached containers,01:27
kfox1111we know the rpms actualy work.01:27
kfox1111would be fully tested in the gate.01:27
kfox1111even better I think then just mirroring.01:27
sdakeya thats a good idea - i like it01:27
sbezverkkfox1111: sure , go ahead01:27
kfox1111sbezverk: ok. thanks.01:28
kfox1111sdake: k. I'll take a stab at it eventually if no one else gets to it first.01:29
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role  https://review.openstack.org/41636701:33
*** david-lyle__ has joined #openstack-kolla01:34
sdakehttps://github.com/kubernetes/kubernetes/issues/4381501:35
kfox1111yeah. sbezverk and I have been keeping an eye on that.01:36
sdakekfox1111 do you ahe an old version of kubeadm source rpm?01:36
kfox1111I don't. :/01:36
*** david-lyle_ has quit IRC01:37
kfox1111we may  have to try and get kubeadm 1.6.0 to deploy a 1.5.x01:37
sdakethis is difficult as kubeadm has dependencies on 1.6.0 of kubelet01:37
sdake(on centos)01:37
kfox1111so, combine both of the patches. :/01:37
*** zhubingbing has joined #openstack-kolla01:37
sdakei do believe i can make kubeadm ignore those deps tho01:37
kfox1111oh.... whats unfortunate. :/01:37
kfox1111the rpm has a dep, or kubeadm checks for it?01:38
sdakekubeadm has a dep for kubectl>=1.6.001:38
sdakeand kubelet01:38
sdakeand the cni thing01:38
kfox1111lame.01:38
kfox1111ok.01:38
sdakerpm -i --nodeps gets it done01:38
kfox1111not sure the best path forward then.01:38
sdakestruggling to find the url for the RPMS though01:38
sdakewhy on earth was the older version of kubeadm deleted from the repos (1.5.whtever/)01:39
* sdake shakes fists01:39
sdakerpm -i --nodeps https://URL-TO-RPM is a thing01:40
sdakei can use it if I could determine where the URL-TO-RPM was01:40
kfox1111we may just need to wait a day or two for them to fix things. :/01:42
sdakeok we could block for 2 days01:42
sdakeor we could figure out how to get an uri to the rpm for kubeadm01:43
kfox1111somehow the ps is back to RBAC mode..01:44
kfox1111oh. it was commented out.01:45
*** eanylin has joined #openstack-kolla01:45
kfox1111so.... lets see.. we should dump out the rbac rules...01:47
openstackgerritchenyingnan proposed openstack/kolla-ansible master: Allow nfs, rpcbind and mountd service on server firewall  https://review.openstack.org/45121001:48
*** Pavo has joined #openstack-kolla01:49
sdakehttps://github.com/kubernetes/release/blob/master/rpm/kubelet.spec01:50
*** jrobinson has joined #openstack-kolla01:52
*** PyroMani has quit IRC01:56
*** eaguilar has joined #openstack-kolla01:58
*** dixiaoli has joined #openstack-kolla01:58
*** PyroMani has joined #openstack-kolla02:00
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139102:02
openstackgerritchenyingnan proposed openstack/kolla-ansible master: Allow nfs, rpcbind and mountd service on server firewall  https://review.openstack.org/45121002:05
*** unicell has quit IRC02:06
sdakekfox1111 i've managed to find the 1.5.4 spec file that ws used to build the release02:06
sdakeGetting https://dl.k8s.io/ci-cross/v1.6.0-alpha.0.2074+a092d8e0f95f52/bin/linux/amd64/kubeadm to ./kubeadm02:08
sdake  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current02:08
sdake                                 Dload  Upload   Total   Spent    Left  Speed02:08
sdake100   161  100   161    0     0    271      0 --:--:-- --:--:-- --:--:--   27102:08
sdake  0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     002:08
sdakecurl: (22) The requested URL returned error: 50302:08
* sdake GROANS02:08
*** schwicht has quit IRC02:08
*** rwallner has joined #openstack-kolla02:12
*** Pavo has quit IRC02:14
*** rwallner has quit IRC02:16
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139102:18
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155602:20
kfox1111sdake: most of the k8s stuff is statically linked binaries.02:22
sdakeyes i know02:22
kfox1111maybe we could find a bin, rather then an rpm for kubeadm.02:22
sdakekfox1111 check out that last review02:22
*** esharao has joined #openstack-kolla02:22
*** yuanying has quit IRC02:22
kfox1111sdake: ah. cool. nm then. :)02:24
*** yangyapeng has quit IRC02:24
sdakebundled binaries = evil02:24
sdakeruby anti-pattern02:25
*** schwicht has joined #openstack-kolla02:26
*** shashank_t_ has quit IRC02:28
*** shashank_t_ has joined #openstack-kolla02:28
*** sayanta__ has joined #openstack-kolla02:29
*** sayantan_ has quit IRC02:31
*** ntpttr has quit IRC02:31
*** adrian_otto has joined #openstack-kolla02:31
kfox1111hmm... more progress...02:33
kfox1111ceph starting to deploy....02:33
kfox1111that might have done the trick.02:33
kfox1111sdake: yeah.02:33
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155602:36
*** yangyapeng has joined #openstack-kolla02:40
kfox1111sdake: I think the 1.6.0 ps is working.02:40
kfox1111got a race to fix I think, but otherwise, one of the jobslooks like its going to pass.02:41
kfox1111so its close.02:41
sdakecool02:41
*** ntpttr_ has joined #openstack-kolla02:44
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155602:45
kfox1111still failed.. but much closer.02:46
kfox1111almost all services deployed.02:46
kfox1111http://logs.openstack.org/91/451391/27/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/b42ba89/console.html02:47
sdakekfox1111 are ou using 1.6 version of canal02:50
kfox1111yeah.02:50
sdakeok02:50
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139102:50
sdake1.5->1.6 = major version upgrade02:50
kfox1111yeah.02:51
sdake1.0.0->2.0.0 should be major versoin upgrade02:51
kfox1111hmm.... so, openvswitch said it started.. but neutron-openvswitch-agent didn't find it..02:51
sdakeya pretty much not surprising networking is busted02:51
kfox1111wait.. this feels familior....02:52
sdakegood to hear :)02:52
kfox1111was there a changelog entry on this....02:52
*** adrian_otto has quit IRC02:53
kfox1111damn... yeah, I think so...02:53
kfox1111https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md02:53
kfox1111Docker version 1.12.6 known issues02:53
kfox1111* no shared pid namespace support02:53
*** adrian_otto has joined #openstack-kolla02:53
kfox1111they only say they support s 1.10.3, 1.11.2, 1.12.6 though...02:55
kfox1111we may have to try to downgrade to 1.10.3...02:55
kfox1111really lame they broke this. :/02:55
* sdake groans more02:56
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139102:56
*** fooliouno has joined #openstack-kolla02:59
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere  https://review.openstack.org/45128903:01
*** adrian_otto has quit IRC03:03
*** adrian_otto has joined #openstack-kolla03:03
*** adrian_otto has quit IRC03:04
*** gkadam has joined #openstack-kolla03:06
kfox1111weird.... no joy with 1.10.3....03:18
kfox1111just paused now.03:18
kfox1111really feels like they never tested this stuff. :/03:24
*** unicell has joined #openstack-kolla03:26
*** dave-mccowan has quit IRC03:27
*** yangyapeng has quit IRC03:27
*** yangyapeng has joined #openstack-kolla03:28
*** esharao has quit IRC03:28
*** duonghq has joined #openstack-kolla03:33
*** dixiaoli has quit IRC03:39
*** dixiaoli has joined #openstack-kolla03:41
v1k0d3nhey guys. sdake in particular...03:43
v1k0d3ni know you guys have been keeping up with the kubeadm issues.03:43
kfox1111yeah..... fun times.03:43
v1k0d3ndo you folks have many users using halcyon? that would be impacted.03:43
v1k0d3nlol yeah.03:43
v1k0d3nwell...03:44
kfox1111not sure the count or if they are impacted.03:44
v1k0d3nwell, do you guys wanna switch out to something like we're starting to use?03:44
v1k0d3nhttps://github.com/att-comdev/poc-topsail03:44
v1k0d3nlook in the ansible dir.03:45
kfox1111got the gate 99% working. but they seem to have broken shared pid support in redhats docker. :/03:45
kfox1111let me reprhase, somehow in 1,5, redhats docker was fine. but k8s 1.6 it isnt.03:46
kfox1111not sure how they managed that.03:46
kfox1111v1k0d3n: cool. I'll have a dig through soon. :)03:47
kfox1111looks nice with just a glance.03:47
*** adrian_otto has joined #openstack-kolla03:48
v1k0d3nyeah, still has a little bit to go, but that's the start of our operations portion for kubernetes.03:52
v1k0d3nand we're just planning on using that for development soon.03:53
v1k0d3ncatches issues a bit quicker that way.03:53
fooliounoduonghq: you there? any luck with starting an instance?03:55
duonghqfooliouno, sorry, I'm very busy today, do not have time for testing yet, now I'll start03:56
fooliounonp. was just checking since we seem to be running into the same issues :)03:57
*** tovin07_ has joined #openstack-kolla03:59
zhubingbinghi guys04:00
fooliounoHi zhubingbing04:05
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere  https://review.openstack.org/45128904:06
zhubingbingfooliouno hi04:06
fooliounoduonghq: ok .. I don't know what changed, but I'm now able to create instances. Doesn't give me the warm fuzzy feeling since I cant figure out why it didnt work earlier and why it works now04:07
eanylinsdake: I added my comments based on my testing today04:08
eanylinsdake: I was able to consistently spin up a Cirros VM even after I reboot my VM04:08
eanylinas in, my box04:09
eanylinfooliouno: Thats great :)04:10
fooliounoeanylin: Thanks! Don't ask me how it worked :)04:12
eanylinfooliouno: I scratched my environment and start from scratch, I was able to bring it back up with Cirros VM creation, the procedures should work04:13
eanylinfooliouno: I added some comments based on what I tested today04:13
fooliounoenaylin: I agree. The guide is in a good shape now.04:14
eanylinfooliouno: Yeah, should work if we dont use the latest 1.6 version04:15
fooliounoI'm still on 1.5.204:16
eanylinok04:16
*** yuanying has joined #openstack-kolla04:19
*** adrian_otto has quit IRC04:20
eanylinactually, seems like I tested with 1.604:20
eanylin[root@kube03 ~]# kubectl version04:20
eanylinClient Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-07T23:53:09Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}04:20
eanylinServer Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}04:20
eanylin[root@kube03 ~]#04:20
eanylindidnt realize it until now04:20
*** tovin07_ has quit IRC04:21
*** sayanta__ has quit IRC04:21
*** sayantan_ has joined #openstack-kolla04:22
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139104:24
*** adrian_otto has joined #openstack-kolla04:27
fooliounokfox1111: I'm unable to ping or ssh to the floating IP of the created nova instance. Any pointers on what to look for?04:33
masberhi04:34
masberwas wondering if there is something wrong with ironic? "Unable to find '/etc/kolla/config/ironic/ironic-agent.kernel' in expected paths.04:34
masberI am getting this error during installation04:35
*** dixiaoli has quit IRC04:36
*** adrian_otto has quit IRC04:37
*** bmace has quit IRC04:44
masberfooliouno, can you ping from the router's namespace to your computer?04:44
masbercan you ping the router?04:44
*** bmace has joined #openstack-kolla04:44
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139104:52
fooliounomasber: I'm on a baremetal node. I tried to ping the floating IP of the VM from the node04:56
duonghqfooliouno, I got error when create instance too04:56
duonghqno valid host was found04:56
fooliounoduonghq: I had the same error. I tried creating from Horizon04:56
fooliounocode 50004:56
masberfooliouno, are you using flat network?04:57
duonghqI used client04:57
duonghqopenstack client04:57
fooliounoFor the overlay its Flannel with vxlan04:57
fooliounoduonghq: nova-manage cell_v2 discover_hosts04:58
fooliounoSomeone suggested running that04:58
eanylinduonghq: Are you using baremetal or? You might want to set virt_type to qemu if you are using VM04:58
fooliounoIt gave an error, but I'm not sure if it fixed it somehow04:58
duonghqeanylin, forgot that, again :(04:58
eanylinduonghq: And I had a problem with losing connections on hypervisor04:59
eanylinI think fooliouno didnt see that problem04:59
eanylinI put the bug and workaround in the dg comment sections04:59
duonghqeanylin, can I set virt_type in cloud.yaml?04:59
fooliounoThis was my error: Message No valid host was found. There are not enough hosts available. Code 50004:59
eanylinduonghq: I think you need to set it on the nova.conf05:00
eanylindont think its in cloud.yaml05:00
eanylincrudini --set /etc/kolla/nova-compute/nova.conf libvirt virt_type qemu05:00
eanylinshould be sufficient05:00
duonghqok05:00
eanylinthis way it will be persistent05:01
fooliounomasber: sorry .. its a little late for me. I will try to ping you tomorrow.05:01
*** fooliouno has quit IRC05:02
masberno worries05:03
*** skramaja has joined #openstack-kolla05:05
*** manheim has joined #openstack-kolla05:10
*** manheim has quit IRC05:14
*** stevebaker has quit IRC05:14
*** dmellado has quit IRC05:14
*** jmontleon has quit IRC05:14
*** jmontleon has joined #openstack-kolla05:16
*** dmellado has joined #openstack-kolla05:16
*** stevebaker has joined #openstack-kolla05:18
openstackgerritZeyu Zhu proposed openstack/kolla master: Optimize the code  https://review.openstack.org/44630305:19
openstackgerritZeyu Zhu proposed openstack/kolla-ansible master: Remove the redundant variable  https://review.openstack.org/44843305:22
*** shashank_t_ has quit IRC05:34
*** dixiaoli has joined #openstack-kolla05:34
*** dixiaoli has quit IRC05:36
*** dixiaoli has joined #openstack-kolla05:36
*** jrobinson is now known as jrobinson-afk05:36
*** dixiaoli has quit IRC05:37
*** prameswar has joined #openstack-kolla05:45
*** jrobinson-afk is now known as jrobinson05:47
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla-kubernetes master: README.txt character improvement  https://review.openstack.org/45165405:52
spsuryaduonghq: around ?05:53
*** dixiaoli has joined #openstack-kolla05:55
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere  https://review.openstack.org/45128906:00
*** magicboiz has quit IRC06:02
*** targon has joined #openstack-kolla06:03
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere  https://review.openstack.org/45128906:03
*** mnasiadka has joined #openstack-kolla06:12
*** zhurong has joined #openstack-kolla06:15
spsuryaeanylin: around ?06:26
*** rwsu has joined #openstack-kolla06:27
spsuryaanyone can please let me know the meaning of   ####This work is experimental at this time. A 1.0.0 version or later signals this work is ready for evaluation####   in https://github.com/openstack/kolla-kubernetes/blob/master/README.rst06:28
*** gkadam is now known as gkadam-afk06:31
spsuryaShould not we keep this like "Currently deliverable is in development phase and a 1.0.0 version or later signals this work is ready for evaluation"06:36
*** zhurong has quit IRC06:38
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role  https://review.openstack.org/41636706:38
*** satyar has joined #openstack-kolla06:41
*** haplo37- has quit IRC06:45
*** pcaruana has joined #openstack-kolla06:47
*** jmccarthy1 has joined #openstack-kolla06:53
*** jmccarthy has quit IRC06:53
*** haplo37_ has joined #openstack-kolla06:53
*** jrobinson has quit IRC06:54
*** manheim has joined #openstack-kolla06:55
*** manheim has quit IRC06:57
*** manheim has joined #openstack-kolla06:57
*** matrohon has joined #openstack-kolla07:02
*** manheim has quit IRC07:03
*** manheim has joined #openstack-kolla07:04
*** oanson has joined #openstack-kolla07:06
*** matrohon has quit IRC07:07
*** Deys has joined #openstack-kolla07:09
*** dixiaoli has quit IRC07:10
*** varto has joined #openstack-kolla07:12
*** dixiaoli has joined #openstack-kolla07:13
*** shardy has joined #openstack-kolla07:14
*** bjolo has joined #openstack-kolla07:15
*** oanson has quit IRC07:18
*** oanson has joined #openstack-kolla07:19
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role  https://review.openstack.org/41636707:20
*** Serlex has joined #openstack-kolla07:24
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role  https://review.openstack.org/41636707:28
*** dixiaoli has quit IRC07:30
*** magicboiz has joined #openstack-kolla07:31
*** mnasiadka has quit IRC07:32
*** mnasiadka has joined #openstack-kolla07:33
*** dixiaoli has joined #openstack-kolla07:35
*** zhurong has joined #openstack-kolla07:38
*** zhangqiankun has joined #openstack-kolla07:39
zhangqiankunhello07:39
*** egonzalez has joined #openstack-kolla07:47
egonzalezDaviey, around?07:49
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Fix secure_proxy_ssl_header option  https://review.openstack.org/44985907:50
*** gkadam_ has joined #openstack-kolla07:52
*** gkadam-afk has quit IRC07:52
*** gkadam_ is now known as gkadam07:52
openstackgerritjianyi proposed openstack/kolla-ansible master: Add zun ansible role  https://review.openstack.org/41617907:53
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role  https://review.openstack.org/41636707:58
japestinhohello guys, I'm unable to create instance after enabling HA mode in global.yml. Before that I created an instance succesfully/07:58
*** bjolo_ has joined #openstack-kolla07:58
japestinhoI got this on error message. ERROR nova.virt.libvirt.driver [req-4e5e5dbc-1205-438c-83c9-59bd66c25002 - - - - -] [instance: ae4c9713-5c61-41e4-aec4-0cb23bd3fb3e] Neutron Reported failure on event network-vif-plugged-0c723c25-0c80-480e-8303-989c569f4744 for instance ae4c9713-5c61-41e4-aec4-0cb23bd3fb3e07:58
*** bjolo has quit IRC07:59
japestinhoAny clue why neutron got failure event?07:59
*** openstackgerrit has quit IRC08:03
egonzalezjapestinho, whats the content in globals.yml, any log from nova-compute or neutron services?08:04
*** athomas has joined #openstack-kolla08:05
*** openstackgerrit has joined #openstack-kolla08:05
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla-kubernetes master: README.txt character improvement  https://review.openstack.org/45165408:05
openstackgerritjianyi proposed openstack/kolla-ansible master: Add zun ansible role  https://review.openstack.org/41617908:07
*** mgoddard has joined #openstack-kolla08:08
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role  https://review.openstack.org/41636708:10
openstackgerritjianyi proposed openstack/kolla-ansible master: Add zun ansible role  https://review.openstack.org/41617908:10
admin0i want to enable and test everything kolla can do .. is there an all-in-one documentaiton that enables me to enable every project and check what works and what not08:11
japestinhoegonzales I added enable_haproxy: "yes" and made enable_neutron_dvr, enable_neutron_lbaas, enable_neutron_agent_ha to "yes" condition08:11
openstackgerritjianyi proposed openstack/kolla-ansible master: Add zun ansible role  https://review.openstack.org/41617908:12
openstackgerritZeyu Zhu proposed openstack/kolla-ansible master: Modify the permission of directory  https://review.openstack.org/44893208:12
japestinhoegonzalez I got this from kibana ERROR nova.compute.manager [req-3cbcf1e0-6955-4bee-8a48-8babe34ebc05 - - - - -] [instance: cfdcf8b3-2306-4ca4-8145-3d66cb88d06e] Build of instance cfdcf8b3-2306-4ca4-8145-3d66cb88d06e aborted: Failed to allocate the network(s), not rescheduling. ERROR nova.compute.manager [req-3cbcf1e0-6955-4bee-8a48-8babe34ebc05 - - - - -]08:14
japestinho[instance: cfdcf8b3-2306-4ca4-8145-3d66cb88d06e] Failed to allocate network(s)08:14
japestinhoadmin0 I think this is enough for your need https://docs.openstack.org/developer/kolla-ansible/quickstart.html08:15
openstackgerritjianyi proposed openstack/kolla-ansible master: Add zun ansible role  https://review.openstack.org/41617908:15
openstackgerritjimmygc proposed openstack/kolla-ansible master: Remove useless variable glance_backend_file  https://review.openstack.org/45168808:16
*** blallau has joined #openstack-kolla08:17
egonzalezjapestinho, do you have a VIP allocated in the nodes, if you do a telnet to neutron endpoint connect?08:18
*** yangyapeng has quit IRC08:19
*** yangyapeng has joined #openstack-kolla08:20
*** yangyapeng has quit IRC08:24
hrwhi08:26
*** haplo37_ has quit IRC08:28
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155608:28
*** haplo37_ has joined #openstack-kolla08:31
*** yuanying has quit IRC08:33
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155608:40
Davieyegonzalez: hey man08:44
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere  https://review.openstack.org/45128908:45
*** zhangqiankun has quit IRC08:45
*** zhangqiankun has joined #openstack-kolla08:46
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role  https://review.openstack.org/41636708:47
japestinhoegonzalez did you mean kolla_internal_vip_address? If so yes I have 1 VIP on my lab using 3 controller nodes. I can telnet to neutron endpoint also.08:48
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155608:48
*** gfidente has joined #openstack-kolla08:50
*** manheim has quit IRC08:52
*** manheim has joined #openstack-kolla08:56
*** magicboiz has quit IRC08:57
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155608:58
*** sayantan_ has quit IRC08:59
*** zhurong has quit IRC09:01
*** zhangqiankun has quit IRC09:01
spsuryasdake: hi... just a lame query09:01
sdakeshoot09:02
sdakesup Daviey09:02
spsuryado we have kolla-k8s architecture diagram09:02
spsuryasdake:09:02
sdakespsurya not really09:02
spsuryasdake: is the architecture docs updated specs/kolla-kubernetes-arch.rst ?09:03
spsuryaone more lamy09:03
spsurya:(09:03
sdakespsurya i'm not sure what the diagram would be much beyond kolla-kubernetes is a helm packaged version of openstack09:03
sdakespsurya once specs files are commited htey arenever changed09:03
sdakespsurya so no not updated09:04
spsuryasdake: thanks09:04
*** dixiaoli_ has joined #openstack-kolla09:08
*** magicboiz has joined #openstack-kolla09:09
openstackgerritMerged openstack/kolla-ansible master: Fix secure_proxy_ssl_header option  https://review.openstack.org/44985909:10
*** dixiaoli has quit IRC09:11
*** pbandark has joined #openstack-kolla09:14
Davieysdake: hey, just having some stable conversations with egonzalez..  Got some good food for thought.09:15
sdakeDaviey cool - inc0 may be looking for stable maintainers egonzalez :)09:15
openstackgerritZeyu Zhu proposed openstack/kolla-ansible master: Use yaml.safe_dump() instead of yaml.dump()  https://review.openstack.org/45171309:15
sdakeI know you ahve a hard time coordinating at you are on other sides of the planet09:15
Davieyworks for me.09:16
openstackgerritMerged openstack/kolla-ansible master: Set to Unix format trove-conductor.conf file  https://review.openstack.org/45153909:16
sdakehttps://review.openstack.org/#/c/346455/09:16
sdakeDaviey you already signed up:)09:16
openstackgerritMerged openstack/kolla-ansible master: Trove fix backup restore with Swift  https://review.openstack.org/45153809:16
Davieysdake: I'm going to send a (subjectively) interesting mail to -dev tomorrow about process...09:16
Davieyegonzalez made some good points09:17
sdakeas long as it doesn't crater our application wfm :)09:17
egonzalezsdake, I volunteer for EMEA timezone09:17
sdakeegonzalez add your name to the stable maint list in the review09:17
Davieyegonzalez: you are in EIRE right?09:17
egonzalezDaviey, yep09:17
sdakeDaviey stable:fllows-policy is prety low hanging fruit for us as we are nearly doing the job already09:18
sdakeDaviey if you do send an email - try not to disrupt that tag application :)09:18
Davieysdake: Well... we are probably not as restrictive on changes as we could be... but not outrageously09:19
*** yuanying has joined #openstack-kolla09:19
sdakeDaviey we can be more restrictive - the policy is on a wiki and not particularly dictative09:19
Davieyright.09:19
sdakeDaviey in other words subject to interpretation and change09:19
Davieysdake: What are your views on things like, https://review.openstack.org/#/c/451465/109:19
sdakelooks like a feature to me09:20
sdakei think the only time a bckport shoudl be done is when someheting is completely busted09:21
Davieysdake: Yeah, egonzalez was talking about formalizing it more for kolla.. But generally i've seen examples of appropriate and inappropriate help explain appropriate changes... and standardization through consensus09:21
sdakei.e. critical defect09:21
*** matrohon has joined #openstack-kolla09:21
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role  https://review.openstack.org/41636709:21
sdakeDaviey part of that is a lack of practice i think09:21
sdakeDaviey the core reviewer team doesn't know which patches to bcakport09:22
sdakeDaviey or rather which to approve and which to -209:22
sdakethat is the whole point of the stable maint team - so they acn make that judgement call09:22
Davieyright09:22
sdakethe good news is we have distributeed stable backports throughout the entire community09:22
sdakeunfortunately not every core reviewer knows what to do in every circumstance09:23
sdakethis is why tag application is so critical - so we can shave off that stable maint team09:23
Davieyyeah.. in the early days, there was a good deal of push back from projects wanting to care about stable/*09:23
Daviey(not everywhere, but certainly vocal)09:24
sdakewe care :)09:24
sdakei'd say broadly speaking the core review team cares09:24
sdakeand the community cares deeply09:24
Davieyhell yeah09:24
*** nhlfr has quit IRC09:24
*** bachp has quit IRC09:24
*** vcn[m] has quit IRC09:24
*** retr0h has quit IRC09:24
sdakealot of our backports come from non-core-reviewers09:24
sdakecore reviewers I think feel compelled to rubber stamp these changes as long as they are in master09:25
sdake(don't know for sure)09:25
Davieysdake: mostly operators, you think?09:25
sdakei dont feel compelled in this way - I just ignore the review :)09:25
sdakeya operators09:25
sdakemost everyone understands we dont take feature backports09:26
sdakethe definition of feature is vague09:26
sdakeand differs from person to person09:26
*** vcn[m] has joined #openstack-kolla09:26
sdakei dont think we want to shut down the output of backports09:27
sdakeI htink instead we want to be more selectie in which backports are accepted (+2+w)09:27
*** sambetts|afk is now known as sambetts09:27
sdakethe way to do that is with a smaller backports team09:27
*** zhangqiankun has joined #openstack-kolla09:28
*** zhangqiankun has quit IRC09:29
Davieyyeah.  I need to dash, perhaps i'll leave the email for now.09:29
sdakeDaviey sounds good - hopefully after the application completes you can beat the team into shape :)09:30
Davieybut egonzalez.. if you are signing up to help with stable.. great :)09:30
*** bachp has joined #openstack-kolla09:31
*** nhlfr has joined #openstack-kolla09:31
*** retr0h has joined #openstack-kolla09:31
sdakeegonzalez just add your name to that review and -1 it egonzalez https://review.openstack.org/#/c/346455/ so inc0 knows to change it09:31
egonzalezsdake, done09:34
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Trove services restarting issue on conf update  https://review.openstack.org/45172009:34
*** athomas has quit IRC09:43
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role  https://review.openstack.org/41636709:44
*** shardy has quit IRC09:47
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla-kubernetes master: README.txt character improvement  https://review.openstack.org/45165409:53
*** yingjun has quit IRC09:54
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere  https://review.openstack.org/45128909:55
*** satyar has quit IRC09:57
hrwcan someone tell me what duonghq meant in https://review.openstack.org/#/c/451268/ comment by "ChangeLog should be put in release note, especially adding CLI option" - where and how can I take care of it?09:58
egonzalezjapestinho, check neutron openvswitch agent logs in compute nodes09:58
egonzalezhrw, create a release note: reno new <release-note-title>, add an explanation of what has been implemented, in you case a new arch option for kolla-build10:01
hrwegonzalez: is there a place where I can read about it?10:01
egonzalezhrw, https://docs.openstack.org/developer/reno/10:02
*** yangyapeng has joined #openstack-kolla10:02
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Karbor services restarting issue on conf update  https://review.openstack.org/45173510:02
hrwegonzalez: thx10:02
hrwegonzalez: so I should use reno to create new release note, write there what is needed and then add it into base_arch patch, right?10:03
egonzalezhrw, yep10:05
hrwthanks egonzalez10:05
japestinhothanks egonzalez I will try to inspect the log10:06
*** hachi has quit IRC10:07
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Karbor services restarting issue on conf update  https://review.openstack.org/45173510:10
*** ipsecguy has joined #openstack-kolla10:10
*** hachi has joined #openstack-kolla10:10
admin0https://docs.openstack.org/developer/kolla-ansible/quickstart.html — this does not say the purpose of the 2 network requirement ?10:10
admin0and what kind of ips they need ot be10:10
egonzalezadmin0, network_interface is the interface for APIs services, keystone, glance, nova10:11
*** yangyapeng has quit IRC10:11
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: add base_arch variable for future non-x86 work  https://review.openstack.org/45126810:12
hrwegonzalez: can you take a look is it right?10:12
egonzalezadmin0, neutron_external_interface is the interface for instance's external networking, an unused interface in neutron nodes without IP10:12
admin0egonzalez: thanks .. lookign into https://github.com/openstack/kolla-ansible/blob/master/etc/kolla/globals.yml ..  if I have say  3 controllers, where/how do I specify those ?10:14
admin0i see it also does cloudkitty .. https://github.com/openstack/kolla-ansible .. everything mentioned there works out of the box ?10:14
admin0https://github.com/openstack/kolla-ansible/blob/master/doc/multinode.rst  â€” does someone have a multinode sample they are using10:15
admin0which gives me an idea of how to map services10:16
egonzalezhrw, added a couple of nit comments10:16
*** yangyapeng has joined #openstack-kolla10:17
admin0and is ubuntu or centos preferred for the baseOS10:17
*** dixiaoli_ has quit IRC10:17
*** manheim has quit IRC10:17
*** hachi has quit IRC10:17
egonzalezadmin0, the one you are more comfortable with10:17
*** dixiaoli has joined #openstack-kolla10:22
*** hachi has joined #openstack-kolla10:22
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155610:22
egonzalezadmin0, example http://paste.openstack.org/show/604821/10:23
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155610:23
*** pbourke has quit IRC10:29
*** pbourke has joined #openstack-kolla10:30
*** cuongnv has quit IRC10:31
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139110:34
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139110:35
sdakepbourke about?10:36
pbourkesdake: hey10:36
sdakesup dude how ya been10:36
sdakelong time no talk10:36
sdakerare for me to pull an all nighter10:36
sdakeunfortunately alarm went off at 1am and once i'm awake i'm awake10:36
* sdake angry with wife for not knowing how to operate her phone...10:36
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Optimize reconfiguration for haproxy  https://review.openstack.org/42415610:36
pbourkeha10:36
*** manheim has joined #openstack-kolla10:37
pbourkeim good10:37
sdakegood to hear10:37
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes  https://review.openstack.org/45155610:44
*** matrohon has quit IRC10:45
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139110:46
*** manheim has quit IRC11:01
*** manheim has joined #openstack-kolla11:02
*** powerd has joined #openstack-kolla11:09
powerdhi!11:10
powerdim playing with kolla and trying to deploy newton - but seeing a problem with mariadb during the kolla-build step11:11
powerdINFO:kolla.image.build.mariadb:rm: cannot remove '/var/lib/mysql/mysql': Directory not empty11:11
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: add base_arch variable for future non-x86 work  https://review.openstack.org/45126811:11
hrwegonzalez: updated commit message and release notes11:11
powerdanyone seen this or know how to work around? ta11:11
*** zhangqiankun has joined #openstack-kolla11:14
hrwpowerd: can you pastebin whole log of image build?11:14
powerdthanks hrw, here you go: https://pastebin.com/BbMkZiJR11:16
*** zhangqiankun has quit IRC11:18
egonzalezpowerd, please paste docker info11:24
*** rmart04 has joined #openstack-kolla11:25
hrwpowerd: can you add 'systemctl stop mariadb &&' right after RUN?11:28
powerddocker info here: https://pastebin.com/qwXg9yB711:29
hrwpowerd: it looks like mariadb is running so mysql database cannot be removed11:29
*** rwallner has joined #openstack-kolla11:29
egonzalezpowerd, you may hit a bug in linux kernel https://github.com/docker/docker/issues/2721411:30
powerdyea id stopped mariadb running on the host be subsequent runs still fail11:31
egonzaleza quick google search for cannot remove directory not empty redirect to docker bugs related to kernel 3.10 on centos 7.211:32
powerdhmmm im on centos 7.3(.1611)11:33
*** yangyapeng has quit IRC11:33
powerdi got everything to work ok on mitaka branch recently on a similar setup...11:33
powerdill retry on another system and with the xfs flag mentioned in the link above and report back - thanks11:34
manheimhi, in a multinode setup, where can I control on which node the haproxy kolla_internal_vip_address IP is set?11:37
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Karbor services restarting issue on conf update  https://review.openstack.org/45173511:40
egonzalezmanheim, by default master is the first haproxy node in the inventory unless it fails and backup becomes master11:40
manheimby master you mean the nodes in the [control] group, right?11:41
manheimi have 2 there11:42
*** rwallner has quit IRC11:43
*** yangyapeng has joined #openstack-kolla11:43
egonzalezmanheim, haproxy runs on network nodes11:44
*** matrohon has joined #openstack-kolla11:44
manheimi see11:44
manheimone more question11:44
manheimso i am looking at what services are proxied, for example nova-conductor is not11:45
manheimnvm11:45
*** dixiaoli has quit IRC11:48
egonzalezmanheim, what do you mean with "proxied"11:49
manheimdo not mind my last bit, i got my answer11:50
*** schwicht has quit IRC11:50
manheimi was refering the the haproxy services11:50
*** rhallisey has joined #openstack-kolla11:54
*** rwallner has joined #openstack-kolla11:56
*** athomas has joined #openstack-kolla12:01
*** dave-mccowan has joined #openstack-kolla12:16
openstackgerritMerged openstack/kolla-ansible master: Updated from global requirements  https://review.openstack.org/45101712:20
*** rwellum has joined #openstack-kolla12:21
powerdjust to close off my query earlier - retried on an ext4 system and it the mariadb build went through fine. this is an XFS bug. Thanks for the pointer!12:22
*** Pavo has joined #openstack-kolla12:23
*** schwicht has joined #openstack-kolla12:30
*** gkadam has quit IRC12:32
*** Pavo has quit IRC12:32
*** schwicht has quit IRC12:35
*** pbandark is now known as pbandark`brb12:39
*** yingjun has joined #openstack-kolla12:40
*** srwilkers has joined #openstack-kolla12:41
pbourkecores, looking for acks on the following please https://review.openstack.org/#/c/425766/ https://review.openstack.org/#/c/426322/ https://review.openstack.org/#/c/435023/ https://review.openstack.org/#/c/435024/ https://review.openstack.org/#/c/437329/12:42
*** yangyapeng has quit IRC12:43
*** Pavo has joined #openstack-kolla12:47
*** schwicht_at_work has joined #openstack-kolla12:49
*** shardy has joined #openstack-kolla12:49
*** shardy is now known as shardy_mtg12:50
*** Pavo has quit IRC12:51
*** jmontleon has quit IRC12:52
*** rhallisey has quit IRC12:53
*** schwicht_at_work has quit IRC12:53
*** rhallisey has joined #openstack-kolla12:54
*** jmontleon has joined #openstack-kolla12:54
*** krtaylor has joined #openstack-kolla12:54
*** prameswar has quit IRC12:54
*** pbandark has joined #openstack-kolla12:55
*** shardy_mtg is now known as shardy12:56
*** pbandark`brb has quit IRC12:56
mliima_morning guys12:57
*** jmontleon has quit IRC12:57
*** manheim has quit IRC12:58
*** jmontleon has joined #openstack-kolla12:59
*** yingjun has quit IRC13:00
sdakemorning mliima_13:03
sdakepbourke this needs a quick rev and then I'll +2 it: https://review.openstack.org/#/c/425766/513:06
*** lamt has joined #openstack-kolla13:06
*** jtriley has quit IRC13:07
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Remove image building related docs from kolla-ansible  https://review.openstack.org/42576613:07
pbourkethanks sdake13:08
*** yangyapeng has joined #openstack-kolla13:10
*** yingjun has joined #openstack-kolla13:10
*** esharao has joined #openstack-kolla13:13
openstackgerritMerged openstack/kolla-ansible master: Optimize reconfiguration for haproxy  https://review.openstack.org/42415613:14
*** goldyfruit has quit IRC13:15
*** goldyfruit has joined #openstack-kolla13:15
sdakepbourke i reviewed the other patches as well except the one where jeffrey had a -113:18
sdakeor where there was already a workflow in place13:18
*** manheim has joined #openstack-kolla13:18
pbourkethanks again13:18
*** schwicht has joined #openstack-kolla13:18
*** zhubingbing has quit IRC13:19
*** signed8bit_Zzz is now known as signed8b_13:19
*** eanylin has quit IRC13:20
openstackgerritMerged openstack/kolla-ansible master: Remove image building related docs from kolla-ansible  https://review.openstack.org/42576613:20
sdakesbezverk http://logs.openstack.org/56/451556/16/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/b57a8a8/console.html#_2017-03-30_10_53_00_80962613:21
sdakesbezverk if you have any idea why this operation is failing would love to know :)13:21
*** lamt has quit IRC13:21
sdakesbezverk happens in both 1.6.0 and 1.5.4 centos13:22
sdakethe rpm release process for kubernetes is not gated13:22
sdakethis is why we have so much trouble with theses new rpms13:22
*** Deys has quit IRC13:22
sdakedeb packaging is the same story - not gated13:22
*** schwicht has quit IRC13:22
sbezverksdake: will look at it later, need to do a few things on my main job..13:23
sdakesbezverk ack13:23
sdakesbezverk i've got other things to do for my day job as well :)13:23
esharaogm guys13:24
sdakehey esharao13:24
esharaonew day - new vm - going to retry the whole installation again :)13:25
mliima_need reviews13:27
mliima_https://review.openstack.org/#/c/450797/13:27
sdakeesharao i'd recommend if your doing AIO to use the 1.5.4 rpms here:13:28
hrw"cp: cannot stat ‘/var/lib/docker/volumes/kolla_logs/_data/haproxy/haproxy_latest.log’: No such file or directory" - hm ;(13:28
sdakehttps://review.openstack.org/#/c/451556/16/tools/setup_kubernetes.sh line 24-2813:29
sdakeesharao this wont work multinode - kubeadm is fubared on kubeadm join - however that step isnt' needed in AIO13:29
sdakeesharao did you give up on ubuntu for the short term?  If not, those deb commands wont work as they dont work in the gate13:31
*** schwicht has joined #openstack-kolla13:32
esharaosdake... yes based on yesterday's discussion i gave up on ubuntu13:32
esharaowill do an AIO on centos13:32
esharaoi tried yesterday with 1.6 but my kube-dns kept on getting ip from the docker0 ip range and busybox nslookup kept on failing13:33
*** eanylin has joined #openstack-kolla13:33
*** Deys has joined #openstack-kolla13:36
*** jaosorior is now known as jaosorior_away13:37
*** varto has quit IRC13:37
*** zhubingbing_ has joined #openstack-kolla13:38
sdakeesharao 1.6 is some serious pain13:38
sdakethe RPM packaging is AFU13:38
sdakesee https://github.com/kubernetes/release/issues/305 for an overview13:39
esharaosdake agree... on 1.6 .. taking a look13:39
esharaomakes sense on gating and tagging.. i thought they would have a better conrolled release process..13:41
openstackgerritMerged openstack/kolla master: Allow the use of previously built parent images  https://review.openstack.org/42632213:42
sdakeesharao one would think13:43
openstackgerritMerged openstack/kolla master: Add the missing project link into README.rst  https://review.openstack.org/44725513:43
sdakeesharao their gating ends at the binary production13:43
sdakebinary being the go binary13:43
*** lucasxu has joined #openstack-kolla13:43
esharaoHopefully inputs from others improves their process13:43
esharaobrb13:44
*** Pavo has joined #openstack-kolla13:46
openstackgerritMerged openstack/kolla-ansible master: Add zun ansible role  https://review.openstack.org/41617913:46
*** Pavo has quit IRC13:50
rwellumsbezverk: trying to follow the steps you went through with esharao last night, k8s is fine, but calico is stuck in Pending. I am seeing: http://paste.openstack.org/show/604859/13:54
*** sayantan_ has joined #openstack-kolla13:56
esharaorwellum can you check the ip of kube-dns?13:57
rwellumhttps://www.irccloud.com/pastebin/G1pvzfbx/13:58
esharaoahh exactly my issue.. the kube-dns gets ip from docker0 and not the cluster ip we assign13:58
sdakerwellum esharao i'd recommend again to use the 1.5.4 kubeadm rpms i built for the moment13:59
sdakesorting out the 1.6 transition is going to take awhile14:00
esharaosdake .. yes.. that issue was from yesterday.. will use the 1.5.4 and retry today14:00
sdakemy rpms dont work multinode14:00
sdakebut they do work single node14:00
esharaoon a single node14:00
*** matrohon has quit IRC14:00
sbezverkrwellum: kubectl get pods -n kube-system14:01
sdakerwellum if you do a rebuild of your system use this:14:01
sdakeyum install -y docker ebtables \14:01
sdake    https://fedorapeople.org/groups/kolla/kubeadm-1.6.0-0.alpha.0.2074.a092d8e0f95f52.x86_64.rpm \14:01
sdake    https://fedorapeople.org/groups/kolla/kubectl-1.5.4-0.x86_64.rpm \14:01
sdake    https://fedorapeople.org/groups/kolla/kubelet-1.5.4-0.x86_64.rpm \14:01
sdake    https://fedorapeople.org/groups/kolla/kubernetes-cni-0.3.0.1-0.07a8a2.x86_64.rpm14:01
*** srwilkers has quit IRC14:01
*** yingjun has quit IRC14:01
*** srwilkers has joined #openstack-kolla14:01
rwellumsdake: tried last night - k8s, cni came up, but genconfig was failing with what looked like bugs fixed a while ago.14:02
rwellumsbezverk: pasted that above: https://www.irccloud.com/pastebin/G1pvzfbx/14:03
sbezverkrwellum: kubectl get ds -n kube-system14:03
sdakerwellum you tried those rpms I pointed out?14:03
rwellumhttps://www.irccloud.com/pastebin/E2nztw7G/14:04
*** papacz has quit IRC14:04
sbezverkrwellum: you are using old version of canal.yaml14:05
sdakerwellum i'd recommend sticking to the guide as much as possible and ignore what is happenign in the 1.6 port patch14:05
sbezverkrwellum: in 1.6.0 labeling changed so your etcd canal container cannot find label14:05
rwellumsbezverk: drat...14:06
rwellumsdake: yes I tried that last night14:06
sdakerwellum can you run rpm -qa | grep kube -> paste14:06
sbezverkrwellum: you can get right canal from this location: url="https://raw.githubusercontent.com/projectcalico/canal/master"14:07
sbezverkurl="$url/k8s-install/kubeadm/1.6/canal.yaml"14:07
rwellumI don't have that 1.5.x system up right now14:07
rwellumThe guide has:     curl -L https://raw.githubusercontent.com/projectcalico/canal/7deb07cda04147ab49115f437151705c747d0374/k8s-install/kubeadm/canal.yaml -o canal.yaml14:07
sdakerwellum i see, your tyring 1.6 then :)14:07
sbezverkrwellum: wrong for 1.6.014:07
sdakerwellum right -t he guide and the instructions sbezverk are providing are mutually incompatible :)14:07
rwellumsdake: the guide doesn't have the 1.5.x rpms though14:08
rwellumsbezverk: ack14:08
rwellumSorry guys was trying to find a green-path - neither worked so this morning I just picked 1.6.0 because esharao had success I thought14:08
sdakerwellum agreed - I'll rev that14:08
sbezverkrwellum: when you get new canal.yaml, do not forget 1: to clean up old one and 2: to change config14:08
*** jtriley has joined #openstack-kolla14:09
esharaorwellum - i used the setup_canal.sh from sbezverk14:09
sdakerwellum my day is capped at 24 hours :)14:09
rwellumesharao: the one they told me was only for gates :)14:09
rwellumappreciate all the help sdake14:09
rwellumsbezverk: what is step '2' - you mean the listen port etc?14:10
sbezverkrwellum: yep, to customize to your env..14:10
rwellumyes sbezverk14:10
esharaorwellum it had support for 1.614:10
rwellumsdake: the other thing I didn't understand in your current guide - you used to have this "    sed -i "s@10.96.232.136@10.3.1.100@" canal.yaml" - then removed it? I still needed it with that version of canal.yaml14:11
sdakerwellum it is possible i made an error in the review - looking14:12
sdakerwellum cool thing about gerrit is it keeps every copy :)14:12
sdakerwellum do you know which review it was present in?14:12
esharaorwellum i thought you needed it based on your cluster config14:13
rwellumDisappeared between 18 and 19 sdake14:13
sdakerwellum gah14:14
sdakerwellum well that is needed14:14
*** satyar has joined #openstack-kolla14:14
rwellumOk I'll add it as a review comment14:14
rwellumI was pretty confused by then14:14
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139114:15
sdakerwellum ya my bad - dd gone wild14:16
*** manheim has quit IRC14:16
*** lamt has joined #openstack-kolla14:19
*** shashank_t_ has joined #openstack-kolla14:22
*** shashank_t_ has quit IRC14:22
*** hrw has quit IRC14:22
*** hrw has joined #openstack-kolla14:22
*** matrohon has joined #openstack-kolla14:22
*** shashank_t_ has joined #openstack-kolla14:24
openstackgerritMerged openstack/kolla master: Update Docker to 1.13.1  https://review.openstack.org/44615814:24
*** Pavo has joined #openstack-kolla14:27
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735614:31
*** schwicht has quit IRC14:33
*** Pavo has quit IRC14:33
sdakeesharao rwellum note i have not updated that document based upon the numerous comments from people - only to solve a couple pain points related to the kubernetes 1.6 change14:35
sdakeI'm now revving it with peoples comments included14:35
esharaosdake: ack14:35
*** yangyapeng has quit IRC14:39
*** mnasiadka has quit IRC14:44
pbourkesdake: replied to comment on https://review.openstack.org/#/c/43502314:45
sdakethanks pbourke - email overflowing14:45
*** Deys has quit IRC14:46
*** schwicht has joined #openstack-kolla14:46
sdakepbourke will +2 if you file a blueprint or bug for sorting out the reparnting problem properly14:46
sdakeprobably need a blueprint for that one :)14:47
pbourkesdake: the parent thing is a huge can of worms :/14:49
sdakepbourke agreed - worth a blueprint no?14:49
*** yangyapeng has joined #openstack-kolla14:49
pbourkewell, a blueprint implies I have a strategy in mind there14:49
*** satyar has quit IRC14:50
pbourkeessentially here Im just reparenting one image to solve a few things, but bringing in a new parent for source or, dropping parents altogether requires a lot of discussion14:51
rwellumsdake: ack to doc reving14:51
pbourkesdake: lets check with inc0 on it14:51
pbourkesdake: he was looking at some pocs in this area14:52
*** skramaja has quit IRC14:52
sdakepbourke ok - wel lthen i'll +2/+w if you agree to speak with inc0 on the topic :)14:52
sdakethat also wfm :)14:52
pbourkesounds like a plan14:52
kolla-slack1<inc0> Give me 30min14:53
kolla-slack1<inc0> Just woken up14:53
pbourkehello over there in slack land14:53
*** rwallner has quit IRC14:54
sdakepbourke gerrit jut locked up :)14:55
*** rwallner has joined #openstack-kolla14:55
sdakethere we go - enjoy14:55
pbourkesdake: appreciate it14:56
*** targon has quit IRC14:58
*** shashank_t_ has quit IRC14:58
kfox1111morning.14:59
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139114:59
sbezverkkfox1111: good morning14:59
sbezverkkfox1111: needed to change ps a bit for multinode scenarion and addressed network instability, appreciate your opinion15:00
kfox1111cool. I'll have a look.15:01
openstackgerritMerged openstack/kolla-ansible master: Fix cinder config group in nova.conf  https://review.openstack.org/45161315:03
*** fooliouno has joined #openstack-kolla15:03
*** klindgren_ has joined #openstack-kolla15:04
kfox1111looking good.15:05
*** manheim has joined #openstack-kolla15:07
*** klindgren__ has quit IRC15:07
rwellumhttps://www.irccloud.com/pastebin/5AdJ76tC/15:08
rwellumsbezverk: sdake ^^^^15:08
sbezverkrwellum: nice !!15:09
sdakerwellum which version is that?15:09
rwellumThat's 1.6 with sbezverk workarounds.15:09
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Magnum: add oslo_messaging_notifications config  https://review.openstack.org/45185715:10
*** Serlex has quit IRC15:10
*** adrian_otto has joined #openstack-kolla15:11
openstackgerritMerged openstack/kolla-ansible master: Remove useless variable glance_backend_file  https://review.openstack.org/45168815:11
*** manheim has quit IRC15:11
*** shashank_t_ has joined #openstack-kolla15:12
rwellumhttp://paste.openstack.org/show/604871/ esharao sbezverk sdake : all steps I took (Centos VM) here.15:12
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735615:13
sbezverkkfox1111: I like your RBAC thing ;) please make sure you add yourself as a co-author, I do not want to carry whole blame for this PS ;)15:14
*** admin0 has left #openstack-kolla15:14
*** adrian_otto has quit IRC15:15
* kfox1111 chuckles15:15
kfox1111sbezverk: k. :)15:16
sdakerwellum can you add that to a new issue against this repo:15:16
sdakehttps://github.com/kubernetes/release/tree/master/rpm15:17
sdakesbezverk blamed you are :)15:17
sdakekfox1111 - say ip was not running in the environment15:17
sdakekfox1111 the gate never ran the fix-iptables script correctly15:18
kfox1111sdake: interesting.15:18
sdakekfox1111 instead it ran - i'm not sure - however ip failed to be found15:18
sdakekfox1111 to resolve it in the fs, i had to run /usr/sbin/ip15:18
kfox1111wonder when that broke. it did run for a while I think.15:18
sdakekfox1111 i'm not sure what it does15:18
sdakekfox1111 infra may ahve changed teh default paths in the images15:18
kfox1111we migh want to do that in a different ps.15:18
*** schwicht has quit IRC15:18
kfox1111it could cause new issues to pop up.15:19
sdakekfox1111 fair enough - i wasn't sure what it does :)15:19
sdakekfox1111 just that it was a problem15:19
*** jtriley has quit IRC15:19
kfox1111hmm..15:19
kfox1111lets see...15:20
kfox1111basically, if that was failing, its running with the infra's iptables stuff disabled.15:21
sbezverkkfox1111: ok returning back kubelet's network config made things worse..15:21
kfox1111normally there is a rule in there that pushes all traffic through infra's chain.15:21
sbezverkwe have two choices then since without network config lots of people reported networking instability15:21
kfox1111for k8s, we need the infra rules to only apply to the public interface.15:21
kfox1111so that tries to rewrite the rules to push it only on the one interface.15:22
sbezverkwe can either live with instability until the fix arrive or just wait for the fix..15:22
*** jtriley has joined #openstack-kolla15:22
kfox1111hmm...15:23
kfox1111have we heard back on the ps? sounded like it might be close.15:23
sbezverkkfox1111: 15 minutes ago I left an agree note asking for 1.6.1 with the fix :)15:24
sbezverkkfox1111: now main kube guy is asking questions15:24
sdakeimo its not really kubeadm that is fubared15:25
sdakewell it may be broken15:25
sdakealhtough - the packaging is busted15:25
sbezverkno it is kubelet bug15:25
kfox1111heh. theres breakage in lots of places.15:25
sbezverkit is uncoditionally waiting for something that not always there15:25
sbezverkit should not always be there15:26
rwellumsbezverk - seems like new (broken) behavior https://www.irccloud.com/pastebin/qXTwpV7K/15:26
sdakei can't get join to work - wtb a solution :)15:26
kfox1111sbezverk: might test your fix without the iptables fix.15:26
kfox1111the firewall might be getting in the way.15:27
sbezverkrwellum: read the scroll back I provided right command or check latest PS15:27
sbezverkkfox1111: do you mean what sdake added?15:27
sbezverkkubectl taint nodes --all=true  node-role.kubernetes.io/master:NoSchedule-15:28
sbezverkrwullum: ^^^15:28
kfox1111sbezverk: yeah.15:28
sbezverksorry rwellum:15:28
sbezverkkfox1111: ok let me give it a shot15:28
rwellumthat worked (taint) - thanks guys.15:30
sbezverkkfox1111: kube people saying that they did not catch that bug because their kubeadm was broken ;) 100% agree not tested --- not working15:31
sbezverkmeant to say: kubeadm testing was broken15:31
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139115:32
*** satyar has joined #openstack-kolla15:33
*** rwallner has quit IRC15:33
*** bjolo_ has quit IRC15:34
kfox1111sbezverk: yeah. can't point to a single thing they did wrong. lots of little mistakes piled up in this release.15:34
kfox1111not uncommon.15:34
kfox1111thats why operators shy away from x.0 releases.15:35
*** hrw has quit IRC15:35
kfox1111and why they throw a fit when x-1.y releases are removed too soon.15:35
*** adrian_otto has joined #openstack-kolla15:35
sbezverkkfox1111: Oh I agree we should have stayed with 1.5.4 if it has not been broken along the way too ;)15:36
kfox1111yeah. :/15:36
*** hrw has joined #openstack-kolla15:36
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735615:37
sdakerwellum with your bleeding edge deployment instructions ^^15:38
sdakerwellum note - kolla-kubernetes requires all other kinds of workarounds to enable to run on 16.015:38
sdakekfox1111 one thing that kubernetes could improve on is to actually gate their RPMS15:40
rwellumsdake: ack15:41
sdakekfox1111 check out: https://github.com/kubernetes/release/issues/30515:41
rwellumsdake: the cherry pick steps are failing for me. That's is nothing to do with k8s at this point.15:41
sdakerwellum which cherrypick - you mean git am?15:41
sdakerwellum or the kollakube cherrypick?15:42
rwellumsdake: I'm looking at it now but wanted to give you a quick heads up https://www.irccloud.com/pastebin/HNyH6GyN/15:43
kfox1111sdake: +1. I think a lot of it is inexperience with package managers. they are use to containers/statically linked go binaries, but not packages.15:44
kfox1111which makes some sence, as their normal modus operandi is to avoid rpms.15:44
*** blallau has quit IRC15:45
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap  https://review.openstack.org/43974015:46
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Fix the "Creating the Zun service and endpoint" failed  https://review.openstack.org/45187615:48
esharaosdake.. checking out the latest guide.. it says that repository not found when trying to clone the kubernetes repo15:48
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139115:50
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735615:50
sdakeesharao which command areou typing in for reference15:50
sdakerwellum that new guide should have the correct fetch instructions ^^15:50
sdakerwellum free of problems relating to merge conflicts15:51
sdakezhubingbing_ - wtb kollakbue resolution :)15:51
*** yangyapeng has quit IRC15:52
esharaosdake i see now that was for building packages15:53
*** rmart04 has quit IRC15:53
sdakeesharao hrm - i haven't tested all the docs yet - busy writing them ;)15:53
sdakeesharao if something is broken please leave a comment in the review and i'll fix it :)15:54
esharaono issues.. will put out all the comments15:54
zhubingbing_hear15:54
inc0ok I'm here15:54
inc0shoot pbourke15:54
sdakezhubingbing_ that kollakube bug - i think we still need that fixed when you have time ;)15:54
zhubingbing_roger15:54
pbourkeinc0: did you manage to get anywhere with your poc for removing base?15:54
zhubingbing_https://review.openstack.org/#/c/439740/15:55
zhubingbing_this patch15:55
zhubingbing_sdake15:55
sdakezhubingbing_ thats the one - i just rebased it as it was in mere conflict for some reasosn15:55
sdakezhubingbing_ atm we are referencing that in our deployment documentatino15:56
sdakethat it is necessary to "git fetch" the patch15:56
sdakethat is not really optimal :(15:56
sdakeas the patch rev changes each time15:56
zhubingbing_i know15:56
zhubingbing_I have prepare a new environment to re-verify that deployment document15:57
inc0pbourke: https://review.openstack.org/#/c/449426/15:57
pbourkethere's still a question of what benefits it brings15:58
pbourkeJeffrey4l seemed concerned about it15:58
inc0it's really niche use case15:58
pbourkeinc0: also are you familiar with this 'new source parent' idea sdake has mentioned?15:58
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Fix the "Creating the Zun service and endpoint" failed  https://review.openstack.org/45187615:58
inc0when you want to build your images in CI, you want them self-contained15:59
zhubingbing_sdake we use kubernetes 1.6?16:00
*** egonzalez has quit IRC16:00
*** yangyapeng has joined #openstack-kolla16:00
inc0as in, I'm testing this keystone container, so I want to deploy it exactly the way it was16:01
inc0tested16:01
*** pbandark has quit IRC16:02
inc0and if I'm rebuilding for example nova, it's going to change base image16:03
inc0so technically your keystone is no longer correct16:04
*** caoyuan has joined #openstack-kolla16:04
inc0but that's important for CI/CD based deployment that rebuilds their containers per service all the time16:04
kfox1111inc0: are you saing you rebuild all the things and redeploy all the things?16:05
sbezverkgents, anybody tried 1.6.0 workaround on Ubuntu?16:05
inc0kfox1111: what I'm saying is when you rebuild base, next time you deploy keystone it will be different16:05
inc0it won't change running containers in any way16:06
inc0so don't worry16:06
kfox1111inc0: thats why you should pin your versions.16:06
kfox1111ie, every new build should get its own tag, and every deployment shoudl specify the version number.16:06
kfox1111 /tag16:06
rwellumsdake: I used the exact fetch command in the current doc and it fails.16:06
kfox1111then its atomic.16:06
inc0we use uppper constrains and all that, but we don't want too much constrains there too for security updates16:07
kfox1111theres more then one way to skin this cat.16:07
inc0yeah16:07
kfox1111for k8s, you enver want a rs to be split.16:07
kfox1111so you must specify tag.16:07
*** matrohon has quit IRC16:07
inc0which works today16:07
kfox1111yeah.16:07
inc0I'm just playing around;)16:07
kfox1111so, not sure I agree with the assertion.16:07
kfox1111but I do think we need a smooth way of tagging.16:08
kfox1111which ties in to needing that container fingerprinting thing.16:08
inc0maybe good idea would be to have per-service tagging16:08
inc0yeah...good idea too16:08
kfox1111ideally per container I think.16:08
kfox1111some containers are easier to upgrade then others.16:08
inc0well you can do it today16:08
kfox1111being able to know that an upgrade is needed to easily upgradable piece A, but not needed for hard to upgrade piece B is important.16:09
inc0kfox1111: let's have this discussion next week ok?16:11
inc0how to rework our tagging16:11
*** zhubingbing_ has quit IRC16:11
inc0I think that's a good thing to do16:11
kfox1111inc0: ok. sounds good. :)16:11
inc0or think about at least16:11
kfox1111sbezverk: did you remove the rpm install too? I think that was the thing that actualy made the iptables stuff start working, not the /usr/sbin thing?16:13
kfox1111yeah. iproute is stil in the rpm list. try pulling that?16:14
sbezverkkfox1111: ok missed that16:14
inc0http://forumtopics.openstack.org/cfp/details/6916:17
sdakerwellum - pretty sure i ttested it befoer i submitted i t- i'll tst again16:18
*** jaosorior_away is now known as jaosorior16:19
esharaosdake quick question.. im getting the flannel ip for the kube-dns instead of cluster dns16:19
esharaowhat am i missing?16:19
*** shardy has quit IRC16:20
sdakeesharao not sure - although looking at rwellum 's problem at the moment16:20
powerdhi, has anyone any tips on forcing kolla-build to ignore a previous build and start from fresh? (i may or may not have mucked up some docker overlay files...)16:20
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139116:21
esharaooh ok16:21
sbezverkbb in 1 hour, meeting time16:21
*** pcaruana has quit IRC16:23
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla master: Reduce size of images built from git repos  https://review.openstack.org/42125216:25
sdakerwellum i cloned a fresh kolla-kubernetes - fetched the latest deployment doc patch - then did this:16:27
sdakehttp://paste.openstack.org/show/60488016:27
sdakethat git fetch works for me16:27
sdakerwellum you may need to remove what you have fetched - try git reset --hard16:28
sdakenote git reset --hard will change any local file changes16:28
sdakeesharao can you link me the review you are currently testing?16:28
esharaohttps://review.openstack.org/#/c/447356/2316:29
rwellumsdake: I tried in a completely different VM with a fresh setup and did not help. I'll go through your log and try it again16:30
sdakerwellum what fetch command are you using?16:30
sdakeesharao rwellum you may already know this - however gerrit is a little wonky about how it works16:32
sdakehttps://review.openstack.org/#/c/447356/16:32
sdakeis different then https://review.openstack.org/#/c/447356/2316:32
sdakeif you click to comment - it takes you to the specific review16:33
sdakethen your browser will stay there and not go back to the main review16:33
sdakesomeone upstream reported 1.5.6 is broken in a rollback for ubuntu16:38
* sdake GROANS16:38
*** mgoddard has quit IRC16:38
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Fix the zun api-paset.ini file not found  https://review.openstack.org/45190316:40
*** sambetts is now known as sambetts|afk16:41
*** athomas has quit IRC16:43
*** caoyuan has quit IRC16:44
*** rwallner has joined #openstack-kolla16:48
*** jaosorior has quit IRC16:50
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla master: Reduce size of images built from git repos  https://review.openstack.org/42125216:51
*** rwallner has quit IRC16:53
*** lucasxu has quit IRC16:53
*** lucasxu has joined #openstack-kolla16:56
*** gfidente is now known as gfidente|afk17:01
*** jtriley has quit IRC17:02
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted briefly to relieve performance issues, and should return to service again momentarily.17:03
kfox1111sbezverk: ps is looking close now. :)17:05
kfox1111sbezverk: http://logs.openstack.org/91/451391/39/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/9d8abdf/console.html#_2017-03-30_16_44_20_02755917:06
rwellumsdake: that was it I was getting '17' regardless17:06
kfox1111I think they said they changed the labeling scheme in 1.6?17:06
sdakerwellum you mean the review/17 problem with gerrit?17:07
sdakeitsn ot really a problem - more of a feature I guess or just the way the tech works17:07
*** jtriley has joined #openstack-kolla17:08
rwellumYes sdake17:10
*** MasterOfBugs has joined #openstack-kolla17:10
*** signed8b_ is now known as signed8bit_Zzz17:12
rwellumsdake: now the next patch failed and that one is quite simple (for the API placement fix)17:17
rwellumsdake:  https://www.irccloud.com/pastebin/C18bUkGb/17:17
sdakerwellum if you want to submit a patch to fix that - i can help you with the workflow17:21
sdakerwellum that way I can eanble you :)17:21
sdakerwellum if you don't have time let me know, and i'll fix it17:22
sdakerwellum should take roughly 20 mins on webex17:22
*** powerd has quit IRC17:23
rwellumI don't mind - either way tbh - was just going through the depl. guide 1 step at a time and this part failed - could have been operator error like the last one... :)17:23
sdakerwellum lets do it in webex so you can learn how to do this work :)17:24
sdakerwellum http://cisco.webex.com/join/stdake17:24
sdakeanyone welcome to join in to watch17:24
rwellumSorry I have a meeting in 5m, and finish at 3pm est :(17:25
sdakeok - well i'll get it done then as I have a conflict at 3pm est17:25
rwellumCan we do post 3pm?17:25
rwellumAh - sorry.17:26
sdakelet me check calendar - moment17:26
sdake3pm est = 12pm MST17:26
sdakethat is actually open17:27
sdakelets do it then17:27
rwellumok sounds good17:27
rwellumBTW I do have a review open, https://review.openstack.org/#/c/449790/ - no-one signed up yet17:27
sdakerwellum looking at that review, where does $CONFIG come from?17:29
*** david-lyle__ is now known as david-lyle17:30
sdakerwellum nm - +2 ftw :)17:31
sdakeJeffrey4l any chance you can sort out how to make kolla-kubernetes doc specific changes not run the entire gate as is done in kolla?17:32
rwellumsorry - was walking to meeting17:32
rwellumty17:32
sdakeJeffrey4l or point me to the appropriate point in pc where its done so I can copy it17:32
*** unicell1 has joined #openstack-kolla17:32
*** unicell has quit IRC17:33
*** financeMann has quit IRC17:33
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: handle rtslib(-fb) package names and dependencies  https://review.openstack.org/45080517:35
*** signed8bit_Zzz is now known as signed8b_17:36
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu  https://review.openstack.org/43278717:38
*** rhallisey has quit IRC17:41
*** schwicht has joined #openstack-kolla17:41
*** rhallisey has joined #openstack-kolla17:41
sdakerwellum when you get out of your meeting - inc0 mentioned the gates are red which we know - so i placed your patch in workflow -1 (this is standard openstack workflow)17:44
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Split Openvswitch into own role  https://review.openstack.org/41679317:50
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role  https://review.openstack.org/41636717:50
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: debian: move to stretch  https://review.openstack.org/43445317:51
*** rwallner has joined #openstack-kolla17:51
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu  https://review.openstack.org/43278717:52
*** karlamrhein_ has joined #openstack-kolla17:54
*** rwallner has quit IRC17:56
*** SamYaple_ has joined #openstack-kolla17:57
*** hogepodge_ has joined #openstack-kolla17:57
*** dciabrin has quit IRC17:58
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: handle ppc64le  https://review.openstack.org/43481017:58
kfox1111hmm.... so dns resolution on second node is not working....17:59
*** hogepodge has quit IRC17:59
*** SamYaple has quit IRC17:59
*** karlamrhein has quit IRC17:59
*** karlamrhein_ is now known as karlamrhein18:00
*** hogepodge_ is now known as hogepodge18:00
rwellumsdake: thanks - the status says 'Not Workflow' though?18:02
*** SamYaple_ has quit IRC18:03
*** SamYaple has joined #openstack-kolla18:03
sdakerwellum ya - I -1'ed the workflow to stop it from being merged - even though it can have 2 +2's on it18:04
sdakerwellum that way reviewers can review it - you can recheck it and any core can workflow it after a recheck18:05
sdakerwellum this should permit inc0 to change his vote to +2 since the review looks good18:05
*** rwallner has joined #openstack-kolla18:05
sbezverkkfox1111: there is another thing which I do not get, how it could have get impacted. openvswitch-vswitchd does not exist and the job fails.18:05
sbezverkhttp://logs.openstack.org/91/451391/39/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/9d8abdf/console.html#_2017-03-30_16_44_20_35487518:05
sdakerwellum and hold off on the +w decision until the gate is operational and a recheck is done18:05
*** dciabrin has joined #openstack-kolla18:06
inc0rwellum: also that's reason why you didn't get reviews18:06
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139118:06
inc0when I see red gates, I just assume WIP and move to different review18:06
kfox1111sbezverk: yeah, I'm a bit weirded out by that....18:07
rwellumack sdake18:07
sbezverkkfox1111: I added docker ps -a to see what is going on18:08
rwelluminc0: well it's 5-days+ old, and gates were fine back then I think. Just my fault for not asking I think.18:08
sdakerwellum nah its not your fault - reviewers are slow to review in kolla-kubernetes deliverable18:08
sdakerwellum asking shouldn't be needed - reviewers should be reviewing daily18:09
sdakerwellum although reviewing daily falls down during emergencies18:09
inc0or heavy dev ususally;)18:09
sdakeinc0++18:09
kfox1111sbezverk: k.18:09
sdakei have a review in gerrit that is 1 year old :)18:09
inc0it gets better once project is out of "omg omg let's get it up asap" state18:09
rwellumsdake: lol18:09
rwellumDefinition of 'merge conflict'18:10
sdakerwellum https://review.openstack.org/#/c/306756/18:10
*** jtriley has quit IRC18:10
sdakemerge conflict means git cherrypick wouldn't work18:10
sdakeor git am wouldn't work18:10
sdakeor any applicatoin of the patch on top of the existing master would fail18:10
sdakeso it requires a rebase - manual or otherwise18:11
sdakerwellum i'll teacch you that at 3pm est18:11
sdakerwellum (how to fix a merge conflict)18:11
rwellumCheers!18:11
rwellum(I've actually used git quite a bit - but gerrit is very new to me)18:12
sdakeits a fundamental contributor skill that is required18:12
sbezverkGents, who prefers Ubuntu to Centos, we need your help to test 1.6.0 upgrade, pleasae help!!18:12
sdakerwellum yup - i try not to assume too much about people when i teach em :)18:12
sdakerwellum if I assume you know everything I miss the steps you need18:12
sdakeif I assume you dont know everything, I end up teaching dupicate info18:13
sdakegenerally my approach is to start with a "doesn't know" and go from there :)18:13
rwellumwfm sdake18:13
sdakeinc0 hows the golang thing18:13
inc0kinda cool, I see lots of use cases where golang will excell...just none that we deal with;)18:14
inc0besides maybe cli clients - cli clients in golang are awesome18:14
*** jtriley has joined #openstack-kolla18:18
*** powerd has joined #openstack-kolla18:19
sdakei like golang18:24
sdakepython + C mashup18:24
sdakeI dislike the bundled binaries18:24
*** powerd has quit IRC18:24
inc0thats why I say it's cool for CLI18:25
inc0curl and it works18:25
inc0and disk is cheap18:25
kfox1111one of go's primary features is easy staticlly linked things.18:27
kfox1111but I think containers do a way better job at that.18:27
inc0yeah, agree18:28
*** adrian_otto has quit IRC18:28
inc0not a big deal for different things than clients18:28
inc0you don't want to spawn container for each cli call18:28
inc0(interesting use case tho;))18:28
kfox1111not sure you don't though. :)18:28
kfox1111if a container can run in half a second once cached, then its not too bad really.18:29
kfox1111I think they could tune it a little bit more and get even to that point.18:29
kfox1111as is, I do shell script to docker exec cli wrappers for a few things. makes this soo much easier.18:29
inc0well if you create some hyper-lightweight container engine18:30
inc0that could work18:30
inc0it's cost of seting up docker and stuff that I dislike18:30
sbezverkkfox1111: it is frealing crazy, they changed naming of docker containers18:30
sbezverkdocker.io/kolla/centos-binary-openvswitch-vswitchd@sha256:0c2e6a14f9e39e2bbf4c966832f390da4b2361c98a8104f4f39fb1ade74a7ca818:30
kfox1111sbezverk: yeah. was in the changelog.18:30
*** Mech422 has joined #openstack-kolla18:30
sdakedoes that break the universe?18:30
kfox1111their claim is you should be using kubectl, not docker commands.18:31
Mech422sdake: just getting ready to test your deploy docs...18:31
Mech422sdake: any last minute stuff I should be aware of ?18:31
inc0noise18:31
sdakeMech422 cool - note - they are a work in progress - so you may be frustrated18:31
kfox1111docker naming/labeling is their own internal stuff.18:31
Mech422sdake: yeah - no biggie - thats why we test :-)18:31
sdakei see, so no breaking the universe except that using docker cli is aa pita ;)18:31
sdakeis NOW a pita18:32
kfox1111sdake: its always a pita, but now its differently pita. ;)18:32
sdakeMech422 i deploy on bare metal - however  i have gone through them about 40 times, so i recommend a vm for your testing18:32
sdakeMech422 (IOW I know how to clean up properly after lots of pain)18:33
Mech422sdake: I've gotta run thru bare metal just so I know how...18:33
sdakeMech422 sounds good18:33
Mech422sdake: I can do VM testing after though18:33
sdakeMech422 note also we think there are some incomplete areas of the instruction - not sure if anone but eanylin has got cmpute to spawn a vm18:34
Mech422sdake: btw - dunno if its interesting, but it looks like we're settling on SaltStack/Consul for Prod, and I'm sticking with Ansible/Consul for dev18:34
Mech422sdake: we're gonna be using Consul in what sounds like a pretty similiar manner to the etcd stuff that was discussed on the ML18:35
sdakeMech422 I see - I think inc0 is invovled in that18:35
inc0"involved"18:36
sbezverkkfox1111: once this nightmare is over, will change docker exec for kubectl exec18:36
sbezverkkfox1111: or better yet helm chart job to do the same18:37
*** mgoddard has joined #openstack-kolla18:37
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139118:39
*** satyar has quit IRC18:40
kfox1111sbezverk: +118:41
*** mgoddard has quit IRC18:44
rwellumsdake: meeting over fyi18:44
sdakerwellum lets get to it if your up to it18:46
rwellumsure18:46
sdakehttp://cisco.webex.com/join/stdake18:46
sdakeanyone welcome to attend a "how to resolve merge conflicts" session18:46
mnaseri found a gif18:47
mnaserof how we build kolla18:47
mnaserhttps://i.redd.it/etxk36gkujoy.gif18:47
mnaser:D18:47
sdakemnaser what is missing from that gif is the dude out front blowing the mountains up to make room for the tracks :)18:47
*** powerd has joined #openstack-kolla18:51
*** shashank_t_ has quit IRC18:53
*** shashank_t_ has joined #openstack-kolla18:54
*** zhangqiankun has joined #openstack-kolla19:00
sbezverkkfox1111: it looks almost good ;)19:14
kfox1111yeah. :)19:15
*** MasterOfBugs has quit IRC19:15
kfox1111ubuntu's still failing, as is multinode.19:15
kfox1111consistent behavior with each one. so 2 seperate issues I think.19:15
kfox1111dns is failing to work on slave node....19:17
*** zhangqiankun has quit IRC19:18
kfox1111not sure if thats a calico issue or not...19:19
*** jmccarthy1 has quit IRC19:21
sbezverkkfox1111: I am bringing up ubuntu vm to test workaround19:23
kfox1111k.19:24
kfox1111I'm looking at multinode.19:24
*** shashank_t_ has quit IRC19:28
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role  https://review.openstack.org/41636719:28
*** MasterOfBugs has joined #openstack-kolla19:29
kfox1111the multinode one is weird....19:36
kfox1111kube-proxy looks happy. so does canal...19:36
*** gfidente|afk has quit IRC19:37
*** shashank_t_ has joined #openstack-kolla19:38
openstackgerritRich Wellum proposed openstack/kolla-kubernetes master: Fix broken gate due to kolla-ansible change  https://review.openstack.org/45195719:38
*** powerd has quit IRC19:40
Mech422sdake: I just realized - that doc is for kolla-k*s ...19:42
Mech422sdake: is there a vanilla kolla one that needs review ?19:42
sdakeMech422 no idea19:45
sdakekfox1111 sbezverk rwellum 's review needs to go in for anything to work19:45
sdakekfox1111 see your already on it :)19:46
sdakethanks :)19:46
sdakeok - meeting time for 2 hours of my life bbl :)19:46
kfox1111hmm.... not an iptables issue on the slave...19:46
Mech422sdake: my bad - I thought this was the updated guide for otaco..any way, I'll do kolla-ansible, then kolla-kube19:46
kfox1111sdake: ;)19:46
mnaserJeffrey4l around?19:47
openstackgerritMerged openstack/kolla-kubernetes master: Fix broken gate due to kolla-ansible change  https://review.openstack.org/45195719:53
inc0woho19:55
kfox1111inc0: ?19:55
inc0rkt is full fledged cncf project now19:55
kfox1111interesting.19:55
inc0sayantan_: we should check kolla containers running on rkt soon;)19:56
sayantan_inc0 I agree19:56
inc0best case scenerio we can turn kolla-docker to be runtime-agnostic19:56
inc0and allow almost automatically kolla-ansible deploy on rkt19:56
kfox1111inc0: +1. docker is getting farther and farther away from where k8s wants to go I think.19:57
kfox1111inc0: containerd was tryiong to get in too. what happened with that?19:58
sayantan_This is interesting19:58
kfox1111hmm... looks like containerd also made it.19:58
*** unicell1 has quit IRC20:00
*** unicell has joined #openstack-kolla20:03
Mech422inc0: on the 'etcd config' stuff - what are your feelings about confd ?20:06
inc0I don't know confd enough to develop emotional bond of any kind20:06
Mech422inc0: I was thinking of using consul-templates for some of our local stuff - if this is a 'likely' path for kolla/openstack, I could just as easy do confd20:07
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Enable tempest test in gate  https://review.openstack.org/40212220:07
inc0Mech422: can we discuss it on some other day please? on training20:07
Mech422inc0 sorry, sure - whenever you have time :-)20:08
inc0tomorrow I'm back in action;) sorry about that20:08
sbezverksdake: btw you do not need cleanup script anymore20:13
sbezverk"kubeadm reset" does the job..20:13
kfox1111sbezverk: does it do slave nodes too?20:14
sbezverkkfox1111: do not think so20:15
kfox1111sbezverk: k.20:15
kfox1111might be able to run it on the node anyway.20:15
*** rwallner has quit IRC20:23
fooliounoI tried a multinode environment with kolla_controller on one node and kolla_compute on another. All pods came up but got a neutron port binding error during insatnce creation.20:24
*** adrian_otto has joined #openstack-kolla20:24
*** new2kolla has joined #openstack-kolla20:24
*** unicell1 has joined #openstack-kolla20:25
*** rwallner has joined #openstack-kolla20:26
sbezverkinc0: ping20:26
kfox1111hmm... this in flannel looks maybe relevent...20:27
inc0whats up?20:27
kfox1111I0330 18:50:32.041056       1 main.go:132] Installing signal handlers20:27
kfox1111I0330 18:50:32.041248       1 manager.go:136] Determining IP address of default interface20:27
kfox1111I0330 18:50:32.041738       1 manager.go:149] Using interface with name eth0 and address 10.32.218.15920:27
kfox1111I0330 18:50:32.041754       1 manager.go:166] Defaulting external address to interface address (10.32.218.159)20:27
kfox1111E0330 18:50:32.043903       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [3]20:27
kfox1111E0330 18:50:33.044800       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [3]20:27
kfox1111E0330 18:50:34.045718       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [3]20:27
sbezverkinc0: what is cgroup driver you have on your ubuntu box?20:27
kfox1111E0330 18:50:35.046554       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [3]20:27
kfox1111E0330 18:50:36.047588       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [3]20:27
kfox1111E0330 18:50:37.048429       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [3]20:27
kfox1111E0330 18:50:38.049477       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:27
kfox1111E0330 18:50:39.051093       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:27
kfox1111E0330 18:50:40.052150       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:27
kfox1111E0330 18:50:41.053012       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:27
kfox1111E0330 18:50:42.054094       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:27
kfox1111E0330 18:50:43.055203       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:27
sbezverkkfox1111: looks like etcd issue20:27
kfox1111E0330 18:50:44.056081       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:27
kfox1111E0330 18:50:45.057300       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:27
kfox1111E0330 18:50:46.058161       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:27
kfox1111E0330 18:50:47.059203       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:28
kfox1111E0330 18:50:48.060026       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:28
kfox1111E0330 18:50:49.061130       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:28
kfox1111E0330 18:50:50.062059       1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21]20:28
*** unicell has quit IRC20:28
kfox1111I0330 18:50:51.072600       1 local_manager.go:179] Picking subnet in range 172.16.129.0 ... 172.16.131.020:28
kfox1111I0330 18:50:51.077338       1 ipmasq.go:47] Adding iptables rule: -s 172.16.128.0/22 -d 172.16.128.0/22 -j RETURN20:28
kfox1111I0330 18:50:51.081646       1 ipmasq.go:47] Adding iptables rule: -s 172.16.128.0/22 ! -d 224.0.0.0/4 -j MASQUERADE20:28
kfox1111I0330 18:50:51.084031       1 ipmasq.go:47] Adding iptables rule: ! -s 172.16.128.0/22 -d 172.16.128.0/22 -j MASQUERADE20:28
kfox1111I0330 18:50:51.085977       1 manager.go:250] Lease acquired: 172.16.129.0/2420:28
kfox1111I0330 18:50:51.086362       1 network.go:58] Watching for L3 misses20:28
*** rwallner_ has joined #openstack-kolla20:28
kfox1111I0330 18:50:51.086394       1 network.go:66] Watching for new subnet leases20:28
kfox1111doh.20:28
kfox1111sry.20:28
kfox1111I0330 18:51:02.474876       1 network.go:243] L3 miss but route for 172.16.128.100 not found20:28
kfox1111sbezverk: yeah, but it looks like it already gets a flannel range from etcd before then?20:28
*** unicell has joined #openstack-kolla20:29
sbezverkinc0: docker info what is trhe driver you have for cgroup???20:31
*** rwallner has quit IRC20:31
inc0sbezverk: hold on20:31
inc0cgroupfs20:31
*** unicell1 has quit IRC20:32
sbezverkinco0: thanks20:32
*** rwallner_ has quit IRC20:32
*** manheim has joined #openstack-kolla20:34
*** jmccarthy has joined #openstack-kolla20:35
*** manheim has quit IRC20:38
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Enable tempest test in gate  https://review.openstack.org/40212220:40
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139120:41
sbezverkkfox1111: it looks like ubuntu defaults to cgroupfs driver for docker whereas centos uses systemd. Do we want to use systemd for both ubuntu and centos? It is just I ma not familiar what people mostly used in production20:46
*** jmccarthy has quit IRC20:46
fooliounokfox1111: question about multi-node environment. I have a node where all the "controller" host and a "compute" host. Neutron is on the controller host and is trying to bind a port on the compute host.20:46
fooliounoAnd it fails. says that there is no agent of type Openvswitch agent on the compute host20:46
fooliounoShould there be pods other than nova-compute and nova-libvirt running on the compute host20:47
kfox1111sbezverk: not sure. probably whatever's the distro's way of doing things. it may be ubuntu has an older systemd and only supports cgroupfs.20:47
kfox1111fooliouno: you need openvswitch-agents on the compute nodes too.20:47
sbezverkkfox1111: ok then I will get docker's cgroup driver and specify it in kubelet parameters20:48
sbezverkto be consistent with distro setting20:48
kfox1111should be able to run a second instance of neutron-openvswith-agent helm package configured for compute nodes.20:48
kfox1111sbezverk: k20:48
fooliounokfox1111: ok. So, I'm guessing somewhere in our helm charts, we should look for the label and install the openvswitch agents on compute nodes as well?20:49
kfox1111fooliouno: see tests/bin/ceph_workflow.sh, in the section that looks like:20:50
kfox1111[ "x$1" == "xceph-multi" ] &&20:50
kfox1111there's a neutron-openvswitch-agent-daemonset setup for compute listed.20:50
fooliounoThanks kfox1111, let me look into that20:50
kfox1111np20:50
*** unicell has quit IRC20:53
kfox1111hmm... not sure how this ever worked...20:53
kfox1111maybe canal was not very picky with its subnets overlapping before..20:54
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139120:54
kfox1111or k8s proxy maybe...20:54
kfox1111sbezverk: have a tweak in the review.20:55
kfox1111may fix the multinode issue.20:55
*** unicell has joined #openstack-kolla20:56
sbezverkhave you pushed it already? can you base yours on top of what I just pushed?20:58
kfox1111sbezverk: me?21:00
kfox1111no. just commented in your review. if yo ucould push that.21:00
sbezverkkfox1111: ahh I see21:00
kfox1111I think the subnets were overlapping before.21:01
kfox1111but it was fine before. now, not so sure.21:01
*** rhallisey has quit IRC21:01
fooliounokfox1111: In addition to neutron-openvswitch-agent-daemonset, there are other listed for the compute node as well (openvswitch-ovsdb-daemonset, openvswitch-vswitchd-daemonset). Should those two be deployed on the compute host as well.21:03
kfox1111yeah.21:04
*** srwilkers has quit IRC21:05
fooliounook .. thank you21:05
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139121:07
sbezverkkfox1111: pushed, I also added a tweak for ubuntu21:08
sbezverkit is working fine on my newly built ubuntu vm21:08
kfox1111sbezverk: cool.21:08
*** new2kolla has quit IRC21:13
*** schwicht has quit IRC21:16
*** unicell has quit IRC21:17
*** goldyfruit has quit IRC21:17
mnaserkolla-core: i have built images for ubuntu binary for 3.0.3 - we have failures.  http://paste.openstack.org/show/604927/21:21
mnaserdo we try to fix them and release something with those images working?21:22
mnaseror should i push to dockerhub21:22
esharaosdake.. using centos and the updated dg.. got my nova compute to come up (something that did not work for ubuntu).. will try to get a tenant vm up later..21:23
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139121:25
*** yangyapeng has quit IRC21:25
fooliounokfox1111: Just wanted to report that with those new microservices on compute host, I was able to create an instance in a "true" multinode environment.21:26
kfox1111fooliouno: awesome. :)21:26
*** esharao has quit IRC21:27
fooliounokfox1111: thanks for your help :)21:28
*** yangyapeng has joined #openstack-kolla21:29
kfox1111fooliouno: anytime. :)21:29
*** mcnanci has quit IRC21:30
*** dave-mccowan has quit IRC21:32
mnaserfirst semi-successful run of kolla + tempest - http://logs.openstack.org/22/402122/14/check/gate-kolla-ansible-dsvm-deploy-centos-binary-centos-7-nv/c68aab7/console.html21:32
mnaserbased off the work of Jeffrey4l :)21:32
kfox1111mnaser: nice. :)21:34
sbezverkkfox1111: Ubuntu is running :)21:38
kfox1111nice. :)21:38
kfox1111multinode still is failing. :/21:40
*** Pavo has joined #openstack-kolla21:43
*** fooliouno has quit IRC21:43
*** adrian_otto has quit IRC21:45
*** Pavo has quit IRC21:45
kfox1111ubuntu openvswithc failure...21:45
kfox1111maybe the cri issue.21:46
kfox1111do we know what docker version ubuntu's using?21:46
sdakeesmiurium thats good news - i am also at this state21:46
sdakeesmiurium sorry mismessage21:46
sbezverkkfox1111: I added docker info so it should be in the log21:46
sdakekfox1111 its in the gate logs21:46
sdakejust search for docker21:46
sdakekfox1111 i would - but exhausted21:47
kfox1111sbezverk: try adding --enable-cri=false21:47
*** jtriley has quit IRC21:47
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Enable tempest test in gate  https://review.openstack.org/40212221:48
sbezverkkfox1111: to kubelet? what does it do?21:48
sdakemnaser ++21:48
sdakemnaser note we have tried this in the past - and the gates OOM21:49
mnasersdake given that we're not installing $world, the gate might be dealing in an acceptable fashion21:49
mnaserif not, then we have to adopt something like puppet and do a bunch of scenarios21:49
sdakemnaser the current thinking is we run tempest in a second vm21:49
sdakemnaser and other services in one vm21:50
mnaser"once we get zuul"21:50
mnaser:p21:50
sdakemnaser control plane in one vm - compute + tempest21:50
sdakeya if i hear zuulv3 one moer time21:50
sdakei'm pretty sure i'm gonig to punch my monitor21:50
mnaserim sure we can tweak things to make it run21:51
mnaserif devstack-gate can run all the time without ooming, im sure we can learn a thing or two21:51
mnaserim basing a lot of my work on that21:51
sdakemnaser been there done that although you may be smarter then the others that have come before you :)21:51
mnaserim up for a challenge and we all benefit if it works out :>21:51
sdakemnaser ++21:52
kfox1111sbezverk: yeah. same as the centos one.21:53
*** lucasxu has quit IRC21:54
*** Pavo has joined #openstack-kolla21:55
sbezverkkfox1111: any idea here: 017-03-30T21:44:51Z|00001|daemon_unix|WARN|/var/run/openvswitch/ovs-vswitchd.pid: stale pidfile for pid 821:56
sbezverk being deleted by pid 021:56
sbezverkovs-appctl: cannot read pidfile "/var/run/openvswitch/ovs-vswitchd.pid" (No such process)21:56
*** Pavo has quit IRC21:56
kfox1111sbezverk: yeah. theres a changelog entry that they broke shared pid namespaces in docker 1.12.21:56
kfox1111setting --enable-cri=false disables their beta docker driver,21:57
kfox1111using the same one 1.5 used.21:57
kfox1111which seems to work ok with shared pid namespaces.21:57
kfox1111I have no idea why they defautled the beta to enabled. :/21:57
sbezverkkfox1111: ah, go it so I do it on both ubuntu and centos21:57
kfox1111yeah. try. might fix it.21:58
*** Pavo has joined #openstack-kolla21:59
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139121:59
*** schwicht has joined #openstack-kolla22:00
*** manheim has joined #openstack-kolla22:01
*** harlowja has quit IRC22:02
*** mcnanci has joined #openstack-kolla22:04
*** manheim has quit IRC22:04
*** manheim has joined #openstack-kolla22:05
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Enable tempest test in gate  https://review.openstack.org/40212222:11
*** Pavo has quit IRC22:14
sbezverkkfox1111: looks better22:17
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Enable tempest test in gate  https://review.openstack.org/40212222:18
kfox1111yup.22:23
kfox1111all we need now is the multinode gates.22:24
sbezverkkfox1111: other centos are failing from time to time :(22:27
kfox1111sbezverk: right before the: cat > /tmp/dns-test.py22:28
kfox1111in tools/test_kube_dns.sh22:28
kfox1111can you add:22:28
kfox1111curl -s http://172.16.128.100:6666/version22:28
kfox1111please.22:28
kfox1111sbezverk: not sure thats not an old race we've always had.22:29
sbezverkkfox1111: are you sure etcd will be listening on this ip?22:29
kfox1111yeah. we hard code it.22:29
sbezverkah right in canal.yaml22:30
kfox1111kind of curious if that will work or not.22:32
kfox1111everything seems to be in a pretty good state as far as the logs go.22:33
kfox1111but there seems to be some connectivity problem still.22:33
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139122:33
kfox1111thx.22:33
sdakesup peeps22:33
sbezverksdake: we got ubuntu working ;) with 1.6.022:33
*** g3ek has quit IRC22:34
kfox1111sdake: yeah. now just gotta get multinode working and we're back in buisness. :)22:34
sdakegood news22:34
*** lamt has quit IRC22:35
*** schwicht_at_work has joined #openstack-kolla22:36
kfox1111very very nice.22:36
kfox1111spacex is awesome. :)22:36
*** schwicht has quit IRC22:36
masberhi, I am getting a "503 Service Unavailable" after kolla deployment22:37
kfox1111they just landed the same rocket a second time :)22:37
masberopenstack works but not horizon22:37
masberenable_horizon_sahara: "{{ enable_sahara | bool }}" is this the right entry to enable sahara web interface?22:38
masberor do I have to change bool to true?22:38
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Enable tempest test in gate  https://review.openstack.org/40212222:40
sbezverkkfox1111: so rockets containers are reusable now ;) ??22:41
*** schwicht_at_work has quit IRC22:41
*** manheim has quit IRC22:43
*** caoyuan has joined #openstack-kolla22:43
kfox1111yup. :)22:43
*** g3ek has joined #openstack-kolla22:43
sbezverkkfox1111: we can start two tcpdump instances on the host one capturing traffic from kube_dns_test pod and one on host external interface22:48
sbezverkto see where packets get lost..22:48
kfox1111interesting...22:48
*** jrobinson has joined #openstack-kolla22:48
kfox1111so... etcd did work from the pod network.22:49
kfox1111so, k8s services on the second node are working.22:49
kfox1111but kube-dns isn't accessable from the second node...22:49
kfox1111very weird.22:49
sbezverkkfox1111: we need to check pods's /etc/resolv.conf22:49
kfox1111sbezverk: can you22:49
kfox1111exactly. :)22:49
kfox1111can you change the curl to a cat /etc/resolv.conf?22:49
sbezverkkfox1111: yep, one sec22:50
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139122:51
kfox1111really seems like resolv.conf should be right though...22:54
kfox1111sed -i 's/10.96.0.10/172.16.128.10/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf22:55
kfox1111 should always be running if either master or slave.22:55
sdakerwellum you about?22:57
sbezverkkfox1111: hmm how it was running before then??22:58
*** caoyuan has quit IRC22:59
kfox1111something changed from 1.5 to 1.6. but really not sure what.23:00
kfox1111its very strange that:23:00
*** shashank_t_ has quit IRC23:00
kfox1111curling to canal-etcd svc worked.23:01
kfox1111dns on one node is working.23:01
kfox1111dns on the other node is failing.23:01
*** shashank_t_ has joined #openstack-kolla23:01
kfox1111canal on all nodes report happy.23:01
kfox1111kubeproxy's report happy.23:01
kfox1111kube-dns should be running the same way as canal-etcd...23:02
*** Pavo has joined #openstack-kolla23:03
sbezverkkfox1111: or maybe it is collateral of this workaround23:04
*** shashan__ has joined #openstack-kolla23:04
sbezverksome people were reporting issues with flannel between nodes23:04
sbezverkbut they reported complete traffic outage, not just dns traffic23:04
kfox1111resolv.conf looks good. :/23:05
sbezverkkfox1111: one thing that can be broken is nat23:05
*** shashank_t_ has quit IRC23:05
kfox1111yeah. possible it affects udp but not tcp...23:06
sbezverkfor slave node's container source address23:06
sbezverkthat would explain local connectivity and broken connectivity outside23:06
kfox1111does turning the workaround back off after start break the single node case?23:06
sbezverkwe should try to ping23:06
kfox1111pings harder, as svc's are not pingable.23:07
kfox1111they arn't real ip's. :/23:07
sbezverkkfox1111: yep23:07
*** signed8b_ is now known as signed8bit_Zzz23:07
*** sayantan_ has quit IRC23:07
kfox1111thats unfortunate... :/23:07
sbezverkas soon as networking config is back kubelet goes crazy23:07
kfox1111whats it do?23:08
*** shashan__ has quit IRC23:09
sbezverkkfox1111: we can try to ping from container to 8.8.8.823:09
openstackgerritMonty Taylor proposed openstack/kolla-ansible master: Add clouds.yaml file and use it  https://review.openstack.org/45200523:09
sbezverkthen capture it at different points and see where it gets broken and why23:09
sbezverkI do not see any other way23:09
kfox1111sbezverk: couldn't hurt I guess.23:09
kfox1111ping -c 4 8.8.8.8 in the test_kube_dns container.23:10
sbezverkyeah, but where do we capture on docker0 or on host's external interface?23:11
*** schwicht has joined #openstack-kolla23:11
kfox1111I think it will go out a canalXXXX interface from the container, route through iptables nat table and go out.23:11
kfox1111docker0 will be skipped entirely.23:11
kfox1111go ahead and submit just the ping check and we can think about how to do the tcpdump while it tests.23:12
sbezverkyou mean calicoXXXXX23:12
kfox1111yeah. that. :)23:12
sbezverkok, but if ping works, that would be really fun..23:12
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade  https://review.openstack.org/45139123:13
kfox1111gut feeling, I'm pretty sure it will.23:14
sbezverkkfox1111: it would mean then something very dns specific :(23:19
*** adrian_otto has joined #openstack-kolla23:23
*** harlowja has joined #openstack-kolla23:24
*** shashank_t_ has joined #openstack-kolla23:27
kfox1111yup. pings. :/23:29
*** MasterOfBugs has quit IRC23:30
sbezverkkfox1111: damn23:31
*** shashank_t_ has quit IRC23:31
kfox1111very very strange.23:32
sbezverkkfox1111: we need then enable debugging on all dns requests dns container gets23:33
kfox1111hmm... when do we caputure the slaves iptabhes....23:34
kfox1111before... ok.23:34
*** MasterOfBugs has joined #openstack-kolla23:35
sbezverkkfox1111: I doubt now it is iptables issue since we can ping23:37
kfox1111yeeah, but was hoping to see the calico rules.23:39
kfox1111but we aren't capturing them yet.23:39
sbezverk    - --v=2 I think if we increase this value we will get more verbose logging23:40
spsuryamorning all23:40
kfox1111+            # All pods to speak to services that resolve to the same host.23:40
kfox1111             - name: FELIX_DEFAULTENDPOINTTOHOSTACTION23:40
kfox1111               value: "ACCEPT"23:40
kfox1111interesting comment...23:40
kfox1111do we need calico rules now for cross node communication?23:40
*** openstack has joined #openstack-kolla23:46
*** jtriley has joined #openstack-kolla23:49
sbezverkkfox1111: building, please prepare container definition for tests23:51
*** sayantan_ has joined #openstack-kolla23:51
*** rwellum has quit IRC23:51
kfox1111should be able to run the same test_kube_dns script?23:52
*** Pavo has quit IRC23:52
kfox1111so... if it was enforcing, it should be an annotation on the namespace... lets seee.23:53
sbezverkkfox1111: I do not have any git tools installed23:53
kfox1111scp?23:53
*** sayantan_ has quit IRC23:55
sbezverkkfox1111: no worries will figure out something..23:56
kfox1111so, canal between 1.5 and 1.6 mostly is rbac roles are added.23:58
kfox1111shouldn't affect this...23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!