| *** shashank_t_ has joined #openstack-kolla | 00:00 | |
| kfox1111 | let me rephrase that. kubeadm has awlays configured kubectl on the user to work out of the box. | 00:00 |
|---|---|---|
| kfox1111 | is that not true anymore? | 00:00 |
| sbezverk | kfox1111: it does not set context | 00:01 |
| *** rwellum has quit IRC | 00:01 | |
| kfox1111 | kubeadm does not setup ~/.kube/config anymore? | 00:02 |
| kfox1111 | that would be a weird regression. | 00:02 |
| *** hieulq has joined #openstack-kolla | 00:03 | |
| *** yingjun has joined #openstack-kolla | 00:03 | |
| sbezverk | looks like we need to do it manually now: | 00:06 |
| *** schwicht has quit IRC | 00:06 | |
| sbezverk | 2017-03-29 23:46:00.492016 | sudo cp /etc/kubernetes/admin.conf $HOME/ | 00:06 |
| sbezverk | 2017-03-29 23:46:00.492048 | sudo chown $(id -u):$(id -g) $HOME/admin.conf | 00:06 |
| sbezverk | 2017-03-29 23:46:00.492076 | export KUBECONFIG=$HOME/admin.conf | 00:06 |
| kfox1111 | mkdir -p ~/.kube | 00:07 |
| kfox1111 | cp /etc/kubernetes/admin.conf ~/.kube/config | 00:07 |
| kfox1111 | ? | 00:07 |
| kfox1111 | wait... how was this ever working... | 00:07 |
| sbezverk | export KUBECONFIG=/etc/kubernetes/admin.conf | 00:07 |
| *** rwsu has quit IRC | 00:08 | |
| kfox1111 | oh... maybe it was using the unauthenticated port? since it was all local. | 00:08 |
| sbezverk | in this case we will just use default config they genereted | 00:08 |
| kfox1111 | I doubht its world readable? | 00:08 |
| sbezverk | kfox1111: that is why I enabled 8080 | 00:08 |
| kfox1111 | if it was just 8080 though, it would just work? | 00:09 |
| sbezverk | but I guess they now force to use secured 6443 port | 00:09 |
| kfox1111 | if we copy it to ~/.kube/config, | 00:09 |
| kfox1111 | then we don't have to worry about the environment var. | 00:09 |
| kfox1111 | which could be dropped in some scripts somehow. | 00:09 |
| sbezverk | kfox1111: ok let's try | 00:10 |
| kfox1111 | since 8080 isn't working anyway, maybe we disable it, and just try authenticated the whole way. | 00:10 |
| *** sayantan_ has quit IRC | 00:11 | |
| *** signed8b_ is now known as signed8bit_Zzz | 00:11 | |
| *** sayantan_ has joined #openstack-kolla | 00:12 | |
| sbezverk | kfox1111: one thing for sure, they do not set kube-system as default context | 00:12 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 00:13 |
| kfox1111 | sbezverk: guessing you still need the chown? | 00:14 |
| *** tonanhngo has quit IRC | 00:14 | |
| sbezverk | kfox1111: it should inherit, no? | 00:14 |
| kfox1111 | I'm guessing its root only. | 00:15 |
| sbezverk | kfox1111: ok will change it next time if it barks back | 00:16 |
| kfox1111 | yeah... [kfox@mantis ~]$ ls -l /etc/kubernetes/admin.conf | 00:16 |
| kfox1111 | -rw-------. 1 root root 9192 Oct 21 18:29 /etc/kubernetes/admin.conf | 00:16 |
| kfox1111 | its going to fail at the cp. | 00:16 |
| kfox1111 | the other option is: | 00:16 |
| kfox1111 | sudo cat /etc/kubernetes/admin.conf > ~/.kube/config | 00:16 |
| kfox1111 | though that might make the perms too loose and kubectl might complain/fail. | 00:17 |
| kfox1111 | I think we'll need the sudo cp and the sudo chown for sure. | 00:17 |
| *** tonanhngo has joined #openstack-kolla | 00:17 | |
| *** tonanhngo has quit IRC | 00:22 | |
| sbezverk | sudo cp /etc/kubernetes/admin.conf ~/.kube/config | 00:22 |
| sbezverk | sudo chown $(id -u):$(id -g) ~/.kube/config | 00:22 |
| sbezverk | kfox1111: should work right? | 00:23 |
| kfox1111 | looks ok I think. | 00:24 |
| kfox1111 | uh... | 00:24 |
| kfox1111 | wait. | 00:24 |
| kfox1111 | not sure when the ~ will be interprited. | 00:24 |
| kfox1111 | if that goes to the users dir, or root's dir. | 00:24 |
| kfox1111 | users dir. | 00:24 |
| kfox1111 | ok. I think its good | 00:25 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 00:25 |
| sbezverk | kfox1111: we will see soon enough | 00:25 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes 1.5.4 and kubeadm https://review.openstack.org/451556 | 00:26 |
| kfox1111 | k. :) | 00:26 |
| sdake | imo should unblock the gate asap and in parallel work on sorting out workarounds for 1.6.0 | 00:26 |
| kfox1111 | sdake: +1 | 00:27 |
| kfox1111 | sdake: that ps looks good to me, if it passes the gates. | 00:27 |
| kfox1111 | thanks for working on it. | 00:27 |
| sdake | kfox1111 right - if it passses gate | 00:27 |
| sdake | the centos bits do | 00:27 |
| sbezverk | sdake: I would agree if ubuntu would be working | 00:27 |
| sdake | i am not ceratin on the ubuntu pins | 00:28 |
| sbezverk | otherwise we still have broken gate | 00:28 |
| sdake | sbezverk i submitted a new review which pins ubuntu as well | 00:28 |
| sdake | i was out most of the day - apologies for that | 00:28 |
| kfox1111 | sdake: no worries. I've been out too. | 00:29 |
| kfox1111 | but wasn't worried about it so much as sbezverk's been doing a great job. :) | 00:29 |
| sdake | sbezverk i'd definately keep working on 1.6.0 tho - we need that asap ;) | 00:29 |
| sbezverk | sdake: I will, it is not in my nature to give up | 00:30 |
| sdake | sbezverk of course not :) | 00:30 |
| sdake | sbezverk that is why we get along so well :) | 00:30 |
| sdake | among other reasons | 00:30 |
| sdake | persistence to a fault ;-) | 00:30 |
| kfox1111 | yay persistence! :) | 00:32 |
| *** schwicht has joined #openstack-kolla | 00:32 | |
| kfox1111 | the openstack mission review's really taking off. :) | 00:33 |
| kfox1111 | very happy to see it. | 00:34 |
| kfox1111 | just hoping at this point its more then just talk. | 00:34 |
| sbezverk | kfox1111: itdoes not want to communicate to 6443 :( | 00:35 |
| kfox1111 | are we missing something in the config maybe? | 00:36 |
| kfox1111 | add a cat ~/.kube/config please | 00:37 |
| kfox1111 | maybe the certs are not readable. | 00:37 |
| kfox1111 | hmm... | 00:37 |
| kfox1111 | that failure is before the copy? | 00:37 |
| sbezverk | after | 00:38 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 00:38 |
| sbezverk | I added netstat -tunlp | 00:38 |
| kfox1111 | perhaps we're not giving enough time for kube-apiserver to start? | 00:38 |
| sbezverk | to see if api is listening on this port | 00:38 |
| sbezverk | possibly | 00:38 |
| kfox1111 | in the logs: http://logs.openstack.org/91/451391/23/check/gate-kolla-kubernetes-deploy-centos-binary-3-ceph-multi-nv/75d35f1/console.html | 00:39 |
| kfox1111 | well.... | 00:40 |
| kfox1111 | yeah... should be after... | 00:40 |
| kfox1111 | so... yeah. wondering if its more files needed... | 00:40 |
| sdake | groan: http://logs.openstack.org/56/451556/3/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/799b87c/console.html#_2017-03-30_00_31_34_780427 | 00:41 |
| kfox1111 | gotta head out for a few. will try and check back soon. | 00:41 |
| kfox1111 | heh. did they pull the package? | 00:41 |
| kfox1111 | we really need an rpm cache. :/ | 00:41 |
| sbezverk | sdake: they removed it I guess | 00:44 |
| kfox1111 | wrong solution to having named a release rpm -0. :/ | 00:44 |
| sdake | haha kfox1111 | 00:44 |
| openstackgerrit | zhongshengping proposed openstack/kolla-ansible master: Remove min_l3_agents_per_router option https://review.openstack.org/451159 | 00:45 |
| sdake | can't expect eveyrone to be an expert in rpm :) | 00:45 |
| kfox1111 | "lets delete the alpha", not "lets name the release -1 like everyone else does" | 00:45 |
| *** david-lyle_ has joined #openstack-kolla | 00:45 | |
| openstackgerrit | zhongshengping proposed openstack/kolla-ansible master: Remove verbose option https://review.openstack.org/451155 | 00:46 |
| *** david-lyle has quit IRC | 00:48 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 00:49 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 00:54 |
| sdake | kfox1111 sbezverk if your leaving commentary upstream - you might ask them to follow the standard versioning for RPMs specified here: https://fedoraproject.org/wiki/Packaging:Versioning | 00:56 |
| *** yangyapeng has joined #openstack-kolla | 00:56 | |
| *** cuongnv has joined #openstack-kolla | 01:02 | |
| *** eanylin has quit IRC | 01:05 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 01:09 |
| kfox1111 | back. | 01:11 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 01:14 |
| openstackgerrit | Marcus Williams proposed openstack/kolla master: Add OpenDaylight Container to kolla https://review.openstack.org/416369 | 01:14 |
| kfox1111 | sbezverk: seems like it still isn't 100% credentialed for the jenkins user... | 01:17 |
| kfox1111 | weird though, that the setup_kubernetes kubectl get pods seems to work... | 01:17 |
| sbezverk | kfox1111: I discovered another issue :( when AllowAlways is used, in my cluster apiserver after 3-4 minutes dies | 01:17 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Optimize reconfiguration for haproxy https://review.openstack.org/424156 | 01:17 |
| kfox1111 | sbezverk: really? | 01:18 |
| kfox1111 | weird. | 01:18 |
| sbezverk | kfox1111: with rbac it stays up | 01:18 |
| kfox1111 | any logs? | 01:18 |
| kfox1111 | thats unsettling.... | 01:18 |
| kfox1111 | that may be related. | 01:18 |
| kfox1111 | cause it takes abit to build the helm packages. | 01:18 |
| *** jrobinson has quit IRC | 01:19 | |
| sbezverk | kfox1111: yeah, I think they should have not released it right after rc1, it looks too green to me | 01:21 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Split Openvswitch into own role https://review.openstack.org/416793 | 01:22 |
| sdake | yup kubernetes alpha long gone | 01:23 |
| sdake | who has a pool without water? | 01:23 |
| sdake | rather kubeadm | 01:24 |
| kfox1111 | hopefully by 1.7, we will have rpm caching done. :/ | 01:24 |
| sdake | rpm caching is relativey easy if you can coax infra into making a mirror f or you | 01:24 |
| kfox1111 | yeah. | 01:24 |
| kfox1111 | or we could just tar it up in the gate job we have already. | 01:24 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Fix cinder config group in nova.conf https://review.openstack.org/451613 | 01:24 |
| sdake | kfox1111 https://review.openstack.org/#/c/349278/ | 01:25 |
| sbezverk | I am off for today, have a good one.. I will check back later if sdake's pin works I will +2 it.. I have a feeling, 1.6.0 uprgade might take a bit of time.. | 01:25 |
| sdake | ubuntu passes ;) | 01:26 |
| sdake | kubeadm has some wonky new interfaces | 01:26 |
| kfox1111 | sbezverk: mind if I carry on the ps? | 01:26 |
| sdake | ttyl sbezverk | 01:26 |
| sdake | a 1.0.0 should have a deprecation policy imo :) | 01:26 |
| kfox1111 | sdake: if we copy the k8s rpms the same way we are building the cached containers, | 01:27 |
| kfox1111 | we know the rpms actualy work. | 01:27 |
| kfox1111 | would be fully tested in the gate. | 01:27 |
| kfox1111 | even better I think then just mirroring. | 01:27 |
| sdake | ya thats a good idea - i like it | 01:27 |
| sbezverk | kfox1111: sure , go ahead | 01:27 |
| kfox1111 | sbezverk: ok. thanks. | 01:28 |
| kfox1111 | sdake: k. I'll take a stab at it eventually if no one else gets to it first. | 01:29 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 01:33 |
| *** david-lyle__ has joined #openstack-kolla | 01:34 | |
| sdake | https://github.com/kubernetes/kubernetes/issues/43815 | 01:35 |
| kfox1111 | yeah. sbezverk and I have been keeping an eye on that. | 01:36 |
| sdake | kfox1111 do you ahe an old version of kubeadm source rpm? | 01:36 |
| kfox1111 | I don't. :/ | 01:36 |
| *** david-lyle_ has quit IRC | 01:37 | |
| kfox1111 | we may have to try and get kubeadm 1.6.0 to deploy a 1.5.x | 01:37 |
| sdake | this is difficult as kubeadm has dependencies on 1.6.0 of kubelet | 01:37 |
| sdake | (on centos) | 01:37 |
| kfox1111 | so, combine both of the patches. :/ | 01:37 |
| *** zhubingbing has joined #openstack-kolla | 01:37 | |
| sdake | i do believe i can make kubeadm ignore those deps tho | 01:37 |
| kfox1111 | oh.... whats unfortunate. :/ | 01:37 |
| kfox1111 | the rpm has a dep, or kubeadm checks for it? | 01:38 |
| sdake | kubeadm has a dep for kubectl>=1.6.0 | 01:38 |
| sdake | and kubelet | 01:38 |
| sdake | and the cni thing | 01:38 |
| kfox1111 | lame. | 01:38 |
| kfox1111 | ok. | 01:38 |
| sdake | rpm -i --nodeps gets it done | 01:38 |
| kfox1111 | not sure the best path forward then. | 01:38 |
| sdake | struggling to find the url for the RPMS though | 01:38 |
| sdake | why on earth was the older version of kubeadm deleted from the repos (1.5.whtever/) | 01:39 |
| * sdake shakes fists | 01:39 | |
| sdake | rpm -i --nodeps https://URL-TO-RPM is a thing | 01:40 |
| sdake | i can use it if I could determine where the URL-TO-RPM was | 01:40 |
| kfox1111 | we may just need to wait a day or two for them to fix things. :/ | 01:42 |
| sdake | ok we could block for 2 days | 01:42 |
| sdake | or we could figure out how to get an uri to the rpm for kubeadm | 01:43 |
| kfox1111 | somehow the ps is back to RBAC mode.. | 01:44 |
| kfox1111 | oh. it was commented out. | 01:45 |
| *** eanylin has joined #openstack-kolla | 01:45 | |
| kfox1111 | so.... lets see.. we should dump out the rbac rules... | 01:47 |
| openstackgerrit | chenyingnan proposed openstack/kolla-ansible master: Allow nfs, rpcbind and mountd service on server firewall https://review.openstack.org/451210 | 01:48 |
| *** Pavo has joined #openstack-kolla | 01:49 | |
| sdake | https://github.com/kubernetes/release/blob/master/rpm/kubelet.spec | 01:50 |
| *** jrobinson has joined #openstack-kolla | 01:52 | |
| *** PyroMani has quit IRC | 01:56 | |
| *** eaguilar has joined #openstack-kolla | 01:58 | |
| *** dixiaoli has joined #openstack-kolla | 01:58 | |
| *** PyroMani has joined #openstack-kolla | 02:00 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 02:02 |
| openstackgerrit | chenyingnan proposed openstack/kolla-ansible master: Allow nfs, rpcbind and mountd service on server firewall https://review.openstack.org/451210 | 02:05 |
| *** unicell has quit IRC | 02:06 | |
| sdake | kfox1111 i've managed to find the 1.5.4 spec file that ws used to build the release | 02:06 |
| sdake | Getting https://dl.k8s.io/ci-cross/v1.6.0-alpha.0.2074+a092d8e0f95f52/bin/linux/amd64/kubeadm to ./kubeadm | 02:08 |
| sdake | % Total % Received % Xferd Average Speed Time Time Time Current | 02:08 |
| sdake | Dload Upload Total Spent Left Speed | 02:08 |
| sdake | 100 161 100 161 0 0 271 0 --:--:-- --:--:-- --:--:-- 271 | 02:08 |
| sdake | 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 | 02:08 |
| sdake | curl: (22) The requested URL returned error: 503 | 02:08 |
| * sdake GROANS | 02:08 | |
| *** schwicht has quit IRC | 02:08 | |
| *** rwallner has joined #openstack-kolla | 02:12 | |
| *** Pavo has quit IRC | 02:14 | |
| *** rwallner has quit IRC | 02:16 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 02:18 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 02:20 |
| kfox1111 | sdake: most of the k8s stuff is statically linked binaries. | 02:22 |
| sdake | yes i know | 02:22 |
| kfox1111 | maybe we could find a bin, rather then an rpm for kubeadm. | 02:22 |
| sdake | kfox1111 check out that last review | 02:22 |
| *** esharao has joined #openstack-kolla | 02:22 | |
| *** yuanying has quit IRC | 02:22 | |
| kfox1111 | sdake: ah. cool. nm then. :) | 02:24 |
| *** yangyapeng has quit IRC | 02:24 | |
| sdake | bundled binaries = evil | 02:24 |
| sdake | ruby anti-pattern | 02:25 |
| *** schwicht has joined #openstack-kolla | 02:26 | |
| *** shashank_t_ has quit IRC | 02:28 | |
| *** shashank_t_ has joined #openstack-kolla | 02:28 | |
| *** sayanta__ has joined #openstack-kolla | 02:29 | |
| *** sayantan_ has quit IRC | 02:31 | |
| *** ntpttr has quit IRC | 02:31 | |
| *** adrian_otto has joined #openstack-kolla | 02:31 | |
| kfox1111 | hmm... more progress... | 02:33 |
| kfox1111 | ceph starting to deploy.... | 02:33 |
| kfox1111 | that might have done the trick. | 02:33 |
| kfox1111 | sdake: yeah. | 02:33 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 02:36 |
| *** yangyapeng has joined #openstack-kolla | 02:40 | |
| kfox1111 | sdake: I think the 1.6.0 ps is working. | 02:40 |
| kfox1111 | got a race to fix I think, but otherwise, one of the jobslooks like its going to pass. | 02:41 |
| kfox1111 | so its close. | 02:41 |
| sdake | cool | 02:41 |
| *** ntpttr_ has joined #openstack-kolla | 02:44 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 02:45 |
| kfox1111 | still failed.. but much closer. | 02:46 |
| kfox1111 | almost all services deployed. | 02:46 |
| kfox1111 | http://logs.openstack.org/91/451391/27/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/b42ba89/console.html | 02:47 |
| sdake | kfox1111 are ou using 1.6 version of canal | 02:50 |
| kfox1111 | yeah. | 02:50 |
| sdake | ok | 02:50 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 02:50 |
| sdake | 1.5->1.6 = major version upgrade | 02:50 |
| kfox1111 | yeah. | 02:51 |
| sdake | 1.0.0->2.0.0 should be major versoin upgrade | 02:51 |
| kfox1111 | hmm.... so, openvswitch said it started.. but neutron-openvswitch-agent didn't find it.. | 02:51 |
| sdake | ya pretty much not surprising networking is busted | 02:51 |
| kfox1111 | wait.. this feels familior.... | 02:52 |
| sdake | good to hear :) | 02:52 |
| kfox1111 | was there a changelog entry on this.... | 02:52 |
| *** adrian_otto has quit IRC | 02:53 | |
| kfox1111 | damn... yeah, I think so... | 02:53 |
| kfox1111 | https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md | 02:53 |
| kfox1111 | Docker version 1.12.6 known issues | 02:53 |
| kfox1111 | * no shared pid namespace support | 02:53 |
| *** adrian_otto has joined #openstack-kolla | 02:53 | |
| kfox1111 | they only say they support s 1.10.3, 1.11.2, 1.12.6 though... | 02:55 |
| kfox1111 | we may have to try to downgrade to 1.10.3... | 02:55 |
| kfox1111 | really lame they broke this. :/ | 02:55 |
| * sdake groans more | 02:56 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 02:56 |
| *** fooliouno has joined #openstack-kolla | 02:59 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere https://review.openstack.org/451289 | 03:01 |
| *** adrian_otto has quit IRC | 03:03 | |
| *** adrian_otto has joined #openstack-kolla | 03:03 | |
| *** adrian_otto has quit IRC | 03:04 | |
| *** gkadam has joined #openstack-kolla | 03:06 | |
| kfox1111 | weird.... no joy with 1.10.3.... | 03:18 |
| kfox1111 | just paused now. | 03:18 |
| kfox1111 | really feels like they never tested this stuff. :/ | 03:24 |
| *** unicell has joined #openstack-kolla | 03:26 | |
| *** dave-mccowan has quit IRC | 03:27 | |
| *** yangyapeng has quit IRC | 03:27 | |
| *** yangyapeng has joined #openstack-kolla | 03:28 | |
| *** esharao has quit IRC | 03:28 | |
| *** duonghq has joined #openstack-kolla | 03:33 | |
| *** dixiaoli has quit IRC | 03:39 | |
| *** dixiaoli has joined #openstack-kolla | 03:41 | |
| v1k0d3n | hey guys. sdake in particular... | 03:43 |
| v1k0d3n | i know you guys have been keeping up with the kubeadm issues. | 03:43 |
| kfox1111 | yeah..... fun times. | 03:43 |
| v1k0d3n | do you folks have many users using halcyon? that would be impacted. | 03:43 |
| v1k0d3n | lol yeah. | 03:43 |
| v1k0d3n | well... | 03:44 |
| kfox1111 | not sure the count or if they are impacted. | 03:44 |
| v1k0d3n | well, do you guys wanna switch out to something like we're starting to use? | 03:44 |
| v1k0d3n | https://github.com/att-comdev/poc-topsail | 03:44 |
| v1k0d3n | look in the ansible dir. | 03:45 |
| kfox1111 | got the gate 99% working. but they seem to have broken shared pid support in redhats docker. :/ | 03:45 |
| kfox1111 | let me reprhase, somehow in 1,5, redhats docker was fine. but k8s 1.6 it isnt. | 03:46 |
| kfox1111 | not sure how they managed that. | 03:46 |
| kfox1111 | v1k0d3n: cool. I'll have a dig through soon. :) | 03:47 |
| kfox1111 | looks nice with just a glance. | 03:47 |
| *** adrian_otto has joined #openstack-kolla | 03:48 | |
| v1k0d3n | yeah, still has a little bit to go, but that's the start of our operations portion for kubernetes. | 03:52 |
| v1k0d3n | and we're just planning on using that for development soon. | 03:53 |
| v1k0d3n | catches issues a bit quicker that way. | 03:53 |
| fooliouno | duonghq: you there? any luck with starting an instance? | 03:55 |
| duonghq | fooliouno, sorry, I'm very busy today, do not have time for testing yet, now I'll start | 03:56 |
| fooliouno | np. was just checking since we seem to be running into the same issues :) | 03:57 |
| *** tovin07_ has joined #openstack-kolla | 03:59 | |
| zhubingbing | hi guys | 04:00 |
| fooliouno | Hi zhubingbing | 04:05 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere https://review.openstack.org/451289 | 04:06 |
| zhubingbing | fooliouno hi | 04:06 |
| fooliouno | duonghq: ok .. I don't know what changed, but I'm now able to create instances. Doesn't give me the warm fuzzy feeling since I cant figure out why it didnt work earlier and why it works now | 04:07 |
| eanylin | sdake: I added my comments based on my testing today | 04:08 |
| eanylin | sdake: I was able to consistently spin up a Cirros VM even after I reboot my VM | 04:08 |
| eanylin | as in, my box | 04:09 |
| eanylin | fooliouno: Thats great :) | 04:10 |
| fooliouno | eanylin: Thanks! Don't ask me how it worked :) | 04:12 |
| eanylin | fooliouno: I scratched my environment and start from scratch, I was able to bring it back up with Cirros VM creation, the procedures should work | 04:13 |
| eanylin | fooliouno: I added some comments based on what I tested today | 04:13 |
| fooliouno | enaylin: I agree. The guide is in a good shape now. | 04:14 |
| eanylin | fooliouno: Yeah, should work if we dont use the latest 1.6 version | 04:15 |
| fooliouno | I'm still on 1.5.2 | 04:16 |
| eanylin | ok | 04:16 |
| *** yuanying has joined #openstack-kolla | 04:19 | |
| *** adrian_otto has quit IRC | 04:20 | |
| eanylin | actually, seems like I tested with 1.6 | 04:20 |
| eanylin | [root@kube03 ~]# kubectl version | 04:20 |
| eanylin | Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-07T23:53:09Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} | 04:20 |
| eanylin | Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} | 04:20 |
| eanylin | [root@kube03 ~]# | 04:20 |
| eanylin | didnt realize it until now | 04:20 |
| *** tovin07_ has quit IRC | 04:21 | |
| *** sayanta__ has quit IRC | 04:21 | |
| *** sayantan_ has joined #openstack-kolla | 04:22 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 04:24 |
| *** adrian_otto has joined #openstack-kolla | 04:27 | |
| fooliouno | kfox1111: I'm unable to ping or ssh to the floating IP of the created nova instance. Any pointers on what to look for? | 04:33 |
| masber | hi | 04:34 |
| masber | was wondering if there is something wrong with ironic? "Unable to find '/etc/kolla/config/ironic/ironic-agent.kernel' in expected paths. | 04:34 |
| masber | I am getting this error during installation | 04:35 |
| *** dixiaoli has quit IRC | 04:36 | |
| *** adrian_otto has quit IRC | 04:37 | |
| *** bmace has quit IRC | 04:44 | |
| masber | fooliouno, can you ping from the router's namespace to your computer? | 04:44 |
| masber | can you ping the router? | 04:44 |
| *** bmace has joined #openstack-kolla | 04:44 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 04:52 |
| fooliouno | masber: I'm on a baremetal node. I tried to ping the floating IP of the VM from the node | 04:56 |
| duonghq | fooliouno, I got error when create instance too | 04:56 |
| duonghq | no valid host was found | 04:56 |
| fooliouno | duonghq: I had the same error. I tried creating from Horizon | 04:56 |
| fooliouno | code 500 | 04:56 |
| masber | fooliouno, are you using flat network? | 04:57 |
| duonghq | I used client | 04:57 |
| duonghq | openstack client | 04:57 |
| fooliouno | For the overlay its Flannel with vxlan | 04:57 |
| fooliouno | duonghq: nova-manage cell_v2 discover_hosts | 04:58 |
| fooliouno | Someone suggested running that | 04:58 |
| eanylin | duonghq: Are you using baremetal or? You might want to set virt_type to qemu if you are using VM | 04:58 |
| fooliouno | It gave an error, but I'm not sure if it fixed it somehow | 04:58 |
| duonghq | eanylin, forgot that, again :( | 04:58 |
| eanylin | duonghq: And I had a problem with losing connections on hypervisor | 04:59 |
| eanylin | I think fooliouno didnt see that problem | 04:59 |
| eanylin | I put the bug and workaround in the dg comment sections | 04:59 |
| duonghq | eanylin, can I set virt_type in cloud.yaml? | 04:59 |
| fooliouno | This was my error: Message No valid host was found. There are not enough hosts available. Code 500 | 04:59 |
| eanylin | duonghq: I think you need to set it on the nova.conf | 05:00 |
| eanylin | dont think its in cloud.yaml | 05:00 |
| eanylin | crudini --set /etc/kolla/nova-compute/nova.conf libvirt virt_type qemu | 05:00 |
| eanylin | should be sufficient | 05:00 |
| duonghq | ok | 05:00 |
| eanylin | this way it will be persistent | 05:01 |
| fooliouno | masber: sorry .. its a little late for me. I will try to ping you tomorrow. | 05:01 |
| *** fooliouno has quit IRC | 05:02 | |
| masber | no worries | 05:03 |
| *** skramaja has joined #openstack-kolla | 05:05 | |
| *** manheim has joined #openstack-kolla | 05:10 | |
| *** manheim has quit IRC | 05:14 | |
| *** stevebaker has quit IRC | 05:14 | |
| *** dmellado has quit IRC | 05:14 | |
| *** jmontleon has quit IRC | 05:14 | |
| *** jmontleon has joined #openstack-kolla | 05:16 | |
| *** dmellado has joined #openstack-kolla | 05:16 | |
| *** stevebaker has joined #openstack-kolla | 05:18 | |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla master: Optimize the code https://review.openstack.org/446303 | 05:19 |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-ansible master: Remove the redundant variable https://review.openstack.org/448433 | 05:22 |
| *** shashank_t_ has quit IRC | 05:34 | |
| *** dixiaoli has joined #openstack-kolla | 05:34 | |
| *** dixiaoli has quit IRC | 05:36 | |
| *** dixiaoli has joined #openstack-kolla | 05:36 | |
| *** jrobinson is now known as jrobinson-afk | 05:36 | |
| *** dixiaoli has quit IRC | 05:37 | |
| *** prameswar has joined #openstack-kolla | 05:45 | |
| *** jrobinson-afk is now known as jrobinson | 05:47 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: README.txt character improvement https://review.openstack.org/451654 | 05:52 |
| spsurya | duonghq: around ? | 05:53 |
| *** dixiaoli has joined #openstack-kolla | 05:55 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere https://review.openstack.org/451289 | 06:00 |
| *** magicboiz has quit IRC | 06:02 | |
| *** targon has joined #openstack-kolla | 06:03 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere https://review.openstack.org/451289 | 06:03 |
| *** mnasiadka has joined #openstack-kolla | 06:12 | |
| *** zhurong has joined #openstack-kolla | 06:15 | |
| spsurya | eanylin: around ? | 06:26 |
| *** rwsu has joined #openstack-kolla | 06:27 | |
| spsurya | anyone can please let me know the meaning of ####This work is experimental at this time. A 1.0.0 version or later signals this work is ready for evaluation#### in https://github.com/openstack/kolla-kubernetes/blob/master/README.rst | 06:28 |
| *** gkadam is now known as gkadam-afk | 06:31 | |
| spsurya | Should not we keep this like "Currently deliverable is in development phase and a 1.0.0 version or later signals this work is ready for evaluation" | 06:36 |
| *** zhurong has quit IRC | 06:38 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 06:38 |
| *** satyar has joined #openstack-kolla | 06:41 | |
| *** haplo37- has quit IRC | 06:45 | |
| *** pcaruana has joined #openstack-kolla | 06:47 | |
| *** jmccarthy1 has joined #openstack-kolla | 06:53 | |
| *** jmccarthy has quit IRC | 06:53 | |
| *** haplo37_ has joined #openstack-kolla | 06:53 | |
| *** jrobinson has quit IRC | 06:54 | |
| *** manheim has joined #openstack-kolla | 06:55 | |
| *** manheim has quit IRC | 06:57 | |
| *** manheim has joined #openstack-kolla | 06:57 | |
| *** matrohon has joined #openstack-kolla | 07:02 | |
| *** manheim has quit IRC | 07:03 | |
| *** manheim has joined #openstack-kolla | 07:04 | |
| *** oanson has joined #openstack-kolla | 07:06 | |
| *** matrohon has quit IRC | 07:07 | |
| *** Deys has joined #openstack-kolla | 07:09 | |
| *** dixiaoli has quit IRC | 07:10 | |
| *** varto has joined #openstack-kolla | 07:12 | |
| *** dixiaoli has joined #openstack-kolla | 07:13 | |
| *** shardy has joined #openstack-kolla | 07:14 | |
| *** bjolo has joined #openstack-kolla | 07:15 | |
| *** oanson has quit IRC | 07:18 | |
| *** oanson has joined #openstack-kolla | 07:19 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 07:20 |
| *** Serlex has joined #openstack-kolla | 07:24 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 07:28 |
| *** dixiaoli has quit IRC | 07:30 | |
| *** magicboiz has joined #openstack-kolla | 07:31 | |
| *** mnasiadka has quit IRC | 07:32 | |
| *** mnasiadka has joined #openstack-kolla | 07:33 | |
| *** dixiaoli has joined #openstack-kolla | 07:35 | |
| *** zhurong has joined #openstack-kolla | 07:38 | |
| *** zhangqiankun has joined #openstack-kolla | 07:39 | |
| zhangqiankun | hello | 07:39 |
| *** egonzalez has joined #openstack-kolla | 07:47 | |
| egonzalez | Daviey, around? | 07:49 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Fix secure_proxy_ssl_header option https://review.openstack.org/449859 | 07:50 |
| *** gkadam_ has joined #openstack-kolla | 07:52 | |
| *** gkadam-afk has quit IRC | 07:52 | |
| *** gkadam_ is now known as gkadam | 07:52 | |
| openstackgerrit | jianyi proposed openstack/kolla-ansible master: Add zun ansible role https://review.openstack.org/416179 | 07:53 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 07:58 |
| japestinho | hello guys, I'm unable to create instance after enabling HA mode in global.yml. Before that I created an instance succesfully/ | 07:58 |
| *** bjolo_ has joined #openstack-kolla | 07:58 | |
| japestinho | I got this on error message. ERROR nova.virt.libvirt.driver [req-4e5e5dbc-1205-438c-83c9-59bd66c25002 - - - - -] [instance: ae4c9713-5c61-41e4-aec4-0cb23bd3fb3e] Neutron Reported failure on event network-vif-plugged-0c723c25-0c80-480e-8303-989c569f4744 for instance ae4c9713-5c61-41e4-aec4-0cb23bd3fb3e | 07:58 |
| *** bjolo has quit IRC | 07:59 | |
| japestinho | Any clue why neutron got failure event? | 07:59 |
| *** openstackgerrit has quit IRC | 08:03 | |
| egonzalez | japestinho, whats the content in globals.yml, any log from nova-compute or neutron services? | 08:04 |
| *** athomas has joined #openstack-kolla | 08:05 | |
| *** openstackgerrit has joined #openstack-kolla | 08:05 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: README.txt character improvement https://review.openstack.org/451654 | 08:05 |
| openstackgerrit | jianyi proposed openstack/kolla-ansible master: Add zun ansible role https://review.openstack.org/416179 | 08:07 |
| *** mgoddard has joined #openstack-kolla | 08:08 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 08:10 |
| openstackgerrit | jianyi proposed openstack/kolla-ansible master: Add zun ansible role https://review.openstack.org/416179 | 08:10 |
| admin0 | i want to enable and test everything kolla can do .. is there an all-in-one documentaiton that enables me to enable every project and check what works and what not | 08:11 |
| japestinho | egonzales I added enable_haproxy: "yes" and made enable_neutron_dvr, enable_neutron_lbaas, enable_neutron_agent_ha to "yes" condition | 08:11 |
| openstackgerrit | jianyi proposed openstack/kolla-ansible master: Add zun ansible role https://review.openstack.org/416179 | 08:12 |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-ansible master: Modify the permission of directory https://review.openstack.org/448932 | 08:12 |
| japestinho | egonzalez I got this from kibana ERROR nova.compute.manager [req-3cbcf1e0-6955-4bee-8a48-8babe34ebc05 - - - - -] [instance: cfdcf8b3-2306-4ca4-8145-3d66cb88d06e] Build of instance cfdcf8b3-2306-4ca4-8145-3d66cb88d06e aborted: Failed to allocate the network(s), not rescheduling. ERROR nova.compute.manager [req-3cbcf1e0-6955-4bee-8a48-8babe34ebc05 - - - - -] | 08:14 |
| japestinho | [instance: cfdcf8b3-2306-4ca4-8145-3d66cb88d06e] Failed to allocate network(s) | 08:14 |
| japestinho | admin0 I think this is enough for your need https://docs.openstack.org/developer/kolla-ansible/quickstart.html | 08:15 |
| openstackgerrit | jianyi proposed openstack/kolla-ansible master: Add zun ansible role https://review.openstack.org/416179 | 08:15 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Remove useless variable glance_backend_file https://review.openstack.org/451688 | 08:16 |
| *** blallau has joined #openstack-kolla | 08:17 | |
| egonzalez | japestinho, do you have a VIP allocated in the nodes, if you do a telnet to neutron endpoint connect? | 08:18 |
| *** yangyapeng has quit IRC | 08:19 | |
| *** yangyapeng has joined #openstack-kolla | 08:20 | |
| *** yangyapeng has quit IRC | 08:24 | |
| hrw | hi | 08:26 |
| *** haplo37_ has quit IRC | 08:28 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 08:28 |
| *** haplo37_ has joined #openstack-kolla | 08:31 | |
| *** yuanying has quit IRC | 08:33 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 08:40 |
| Daviey | egonzalez: hey man | 08:44 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere https://review.openstack.org/451289 | 08:45 |
| *** zhangqiankun has quit IRC | 08:45 | |
| *** zhangqiankun has joined #openstack-kolla | 08:46 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 08:47 |
| japestinho | egonzalez did you mean kolla_internal_vip_address? If so yes I have 1 VIP on my lab using 3 controller nodes. I can telnet to neutron endpoint also. | 08:48 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 08:48 |
| *** gfidente has joined #openstack-kolla | 08:50 | |
| *** manheim has quit IRC | 08:52 | |
| *** manheim has joined #openstack-kolla | 08:56 | |
| *** magicboiz has quit IRC | 08:57 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 08:58 |
| *** sayantan_ has quit IRC | 08:59 | |
| *** zhurong has quit IRC | 09:01 | |
| *** zhangqiankun has quit IRC | 09:01 | |
| spsurya | sdake: hi... just a lame query | 09:01 |
| sdake | shoot | 09:02 |
| sdake | sup Daviey | 09:02 |
| spsurya | do we have kolla-k8s architecture diagram | 09:02 |
| spsurya | sdake: | 09:02 |
| sdake | spsurya not really | 09:02 |
| spsurya | sdake: is the architecture docs updated specs/kolla-kubernetes-arch.rst ? | 09:03 |
| spsurya | one more lamy | 09:03 |
| spsurya | :( | 09:03 |
| sdake | spsurya i'm not sure what the diagram would be much beyond kolla-kubernetes is a helm packaged version of openstack | 09:03 |
| sdake | spsurya once specs files are commited htey arenever changed | 09:03 |
| sdake | spsurya so no not updated | 09:04 |
| spsurya | sdake: thanks | 09:04 |
| *** dixiaoli_ has joined #openstack-kolla | 09:08 | |
| *** magicboiz has joined #openstack-kolla | 09:09 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix secure_proxy_ssl_header option https://review.openstack.org/449859 | 09:10 |
| *** dixiaoli has quit IRC | 09:11 | |
| *** pbandark has joined #openstack-kolla | 09:14 | |
| Daviey | sdake: hey, just having some stable conversations with egonzalez.. Got some good food for thought. | 09:15 |
| sdake | Daviey cool - inc0 may be looking for stable maintainers egonzalez :) | 09:15 |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-ansible master: Use yaml.safe_dump() instead of yaml.dump() https://review.openstack.org/451713 | 09:15 |
| sdake | I know you ahve a hard time coordinating at you are on other sides of the planet | 09:15 |
| Daviey | works for me. | 09:16 |
| openstackgerrit | Merged openstack/kolla-ansible master: Set to Unix format trove-conductor.conf file https://review.openstack.org/451539 | 09:16 |
| sdake | https://review.openstack.org/#/c/346455/ | 09:16 |
| sdake | Daviey you already signed up:) | 09:16 |
| openstackgerrit | Merged openstack/kolla-ansible master: Trove fix backup restore with Swift https://review.openstack.org/451538 | 09:16 |
| Daviey | sdake: I'm going to send a (subjectively) interesting mail to -dev tomorrow about process... | 09:16 |
| Daviey | egonzalez made some good points | 09:17 |
| sdake | as long as it doesn't crater our application wfm :) | 09:17 |
| egonzalez | sdake, I volunteer for EMEA timezone | 09:17 |
| sdake | egonzalez add your name to the stable maint list in the review | 09:17 |
| Daviey | egonzalez: you are in EIRE right? | 09:17 |
| egonzalez | Daviey, yep | 09:17 |
| sdake | Daviey stable:fllows-policy is prety low hanging fruit for us as we are nearly doing the job already | 09:18 |
| sdake | Daviey if you do send an email - try not to disrupt that tag application :) | 09:18 |
| Daviey | sdake: Well... we are probably not as restrictive on changes as we could be... but not outrageously | 09:19 |
| *** yuanying has joined #openstack-kolla | 09:19 | |
| sdake | Daviey we can be more restrictive - the policy is on a wiki and not particularly dictative | 09:19 |
| Daviey | right. | 09:19 |
| sdake | Daviey in other words subject to interpretation and change | 09:19 |
| Daviey | sdake: What are your views on things like, https://review.openstack.org/#/c/451465/1 | 09:19 |
| sdake | looks like a feature to me | 09:20 |
| sdake | i think the only time a bckport shoudl be done is when someheting is completely busted | 09:21 |
| Daviey | sdake: Yeah, egonzalez was talking about formalizing it more for kolla.. But generally i've seen examples of appropriate and inappropriate help explain appropriate changes... and standardization through consensus | 09:21 |
| sdake | i.e. critical defect | 09:21 |
| *** matrohon has joined #openstack-kolla | 09:21 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 09:21 |
| sdake | Daviey part of that is a lack of practice i think | 09:21 |
| sdake | Daviey the core reviewer team doesn't know which patches to bcakport | 09:22 |
| sdake | Daviey or rather which to approve and which to -2 | 09:22 |
| sdake | that is the whole point of the stable maint team - so they acn make that judgement call | 09:22 |
| Daviey | right | 09:22 |
| sdake | the good news is we have distributeed stable backports throughout the entire community | 09:22 |
| sdake | unfortunately not every core reviewer knows what to do in every circumstance | 09:23 |
| sdake | this is why tag application is so critical - so we can shave off that stable maint team | 09:23 |
| Daviey | yeah.. in the early days, there was a good deal of push back from projects wanting to care about stable/* | 09:23 |
| Daviey | (not everywhere, but certainly vocal) | 09:24 |
| sdake | we care :) | 09:24 |
| sdake | i'd say broadly speaking the core review team cares | 09:24 |
| sdake | and the community cares deeply | 09:24 |
| Daviey | hell yeah | 09:24 |
| *** nhlfr has quit IRC | 09:24 | |
| *** bachp has quit IRC | 09:24 | |
| *** vcn[m] has quit IRC | 09:24 | |
| *** retr0h has quit IRC | 09:24 | |
| sdake | alot of our backports come from non-core-reviewers | 09:24 |
| sdake | core reviewers I think feel compelled to rubber stamp these changes as long as they are in master | 09:25 |
| sdake | (don't know for sure) | 09:25 |
| Daviey | sdake: mostly operators, you think? | 09:25 |
| sdake | i dont feel compelled in this way - I just ignore the review :) | 09:25 |
| sdake | ya operators | 09:25 |
| sdake | most everyone understands we dont take feature backports | 09:26 |
| sdake | the definition of feature is vague | 09:26 |
| sdake | and differs from person to person | 09:26 |
| *** vcn[m] has joined #openstack-kolla | 09:26 | |
| sdake | i dont think we want to shut down the output of backports | 09:27 |
| sdake | I htink instead we want to be more selectie in which backports are accepted (+2+w) | 09:27 |
| *** sambetts|afk is now known as sambetts | 09:27 | |
| sdake | the way to do that is with a smaller backports team | 09:27 |
| *** zhangqiankun has joined #openstack-kolla | 09:28 | |
| *** zhangqiankun has quit IRC | 09:29 | |
| Daviey | yeah. I need to dash, perhaps i'll leave the email for now. | 09:29 |
| sdake | Daviey sounds good - hopefully after the application completes you can beat the team into shape :) | 09:30 |
| Daviey | but egonzalez.. if you are signing up to help with stable.. great :) | 09:30 |
| *** bachp has joined #openstack-kolla | 09:31 | |
| *** nhlfr has joined #openstack-kolla | 09:31 | |
| *** retr0h has joined #openstack-kolla | 09:31 | |
| sdake | egonzalez just add your name to that review and -1 it egonzalez https://review.openstack.org/#/c/346455/ so inc0 knows to change it | 09:31 |
| egonzalez | sdake, done | 09:34 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Trove services restarting issue on conf update https://review.openstack.org/451720 | 09:34 |
| *** athomas has quit IRC | 09:43 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 09:44 |
| *** shardy has quit IRC | 09:47 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: README.txt character improvement https://review.openstack.org/451654 | 09:53 |
| *** yingjun has quit IRC | 09:54 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere https://review.openstack.org/451289 | 09:55 |
| *** satyar has quit IRC | 09:57 | |
| hrw | can someone tell me what duonghq meant in https://review.openstack.org/#/c/451268/ comment by "ChangeLog should be put in release note, especially adding CLI option" - where and how can I take care of it? | 09:58 |
| egonzalez | japestinho, check neutron openvswitch agent logs in compute nodes | 09:58 |
| egonzalez | hrw, create a release note: reno new <release-note-title>, add an explanation of what has been implemented, in you case a new arch option for kolla-build | 10:01 |
| hrw | egonzalez: is there a place where I can read about it? | 10:01 |
| egonzalez | hrw, https://docs.openstack.org/developer/reno/ | 10:02 |
| *** yangyapeng has joined #openstack-kolla | 10:02 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Karbor services restarting issue on conf update https://review.openstack.org/451735 | 10:02 |
| hrw | egonzalez: thx | 10:02 |
| hrw | egonzalez: so I should use reno to create new release note, write there what is needed and then add it into base_arch patch, right? | 10:03 |
| egonzalez | hrw, yep | 10:05 |
| hrw | thanks egonzalez | 10:05 |
| japestinho | thanks egonzalez I will try to inspect the log | 10:06 |
| *** hachi has quit IRC | 10:07 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Karbor services restarting issue on conf update https://review.openstack.org/451735 | 10:10 |
| *** ipsecguy has joined #openstack-kolla | 10:10 | |
| *** hachi has joined #openstack-kolla | 10:10 | |
| admin0 | https://docs.openstack.org/developer/kolla-ansible/quickstart.html — this does not say the purpose of the 2 network requirement ? | 10:10 |
| admin0 | and what kind of ips they need ot be | 10:10 |
| egonzalez | admin0, network_interface is the interface for APIs services, keystone, glance, nova | 10:11 |
| *** yangyapeng has quit IRC | 10:11 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: add base_arch variable for future non-x86 work https://review.openstack.org/451268 | 10:12 |
| hrw | egonzalez: can you take a look is it right? | 10:12 |
| egonzalez | admin0, neutron_external_interface is the interface for instance's external networking, an unused interface in neutron nodes without IP | 10:12 |
| admin0 | egonzalez: thanks .. lookign into https://github.com/openstack/kolla-ansible/blob/master/etc/kolla/globals.yml .. if I have say 3 controllers, where/how do I specify those ? | 10:14 |
| admin0 | i see it also does cloudkitty .. https://github.com/openstack/kolla-ansible .. everything mentioned there works out of the box ? | 10:14 |
| admin0 | https://github.com/openstack/kolla-ansible/blob/master/doc/multinode.rst — does someone have a multinode sample they are using | 10:15 |
| admin0 | which gives me an idea of how to map services | 10:16 |
| egonzalez | hrw, added a couple of nit comments | 10:16 |
| *** yangyapeng has joined #openstack-kolla | 10:17 | |
| admin0 | and is ubuntu or centos preferred for the baseOS | 10:17 |
| *** dixiaoli_ has quit IRC | 10:17 | |
| *** manheim has quit IRC | 10:17 | |
| *** hachi has quit IRC | 10:17 | |
| egonzalez | admin0, the one you are more comfortable with | 10:17 |
| *** dixiaoli has joined #openstack-kolla | 10:22 | |
| *** hachi has joined #openstack-kolla | 10:22 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 10:22 |
| egonzalez | admin0, example http://paste.openstack.org/show/604821/ | 10:23 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 10:23 |
| *** pbourke has quit IRC | 10:29 | |
| *** pbourke has joined #openstack-kolla | 10:30 | |
| *** cuongnv has quit IRC | 10:31 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 10:34 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 10:35 |
| sdake | pbourke about? | 10:36 |
| pbourke | sdake: hey | 10:36 |
| sdake | sup dude how ya been | 10:36 |
| sdake | long time no talk | 10:36 |
| sdake | rare for me to pull an all nighter | 10:36 |
| sdake | unfortunately alarm went off at 1am and once i'm awake i'm awake | 10:36 |
| * sdake angry with wife for not knowing how to operate her phone... | 10:36 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Optimize reconfiguration for haproxy https://review.openstack.org/424156 | 10:36 |
| pbourke | ha | 10:36 |
| *** manheim has joined #openstack-kolla | 10:37 | |
| pbourke | im good | 10:37 |
| sdake | good to hear | 10:37 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes https://review.openstack.org/451556 | 10:44 |
| *** matrohon has quit IRC | 10:45 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 10:46 |
| *** manheim has quit IRC | 11:01 | |
| *** manheim has joined #openstack-kolla | 11:02 | |
| *** powerd has joined #openstack-kolla | 11:09 | |
| powerd | hi! | 11:10 |
| powerd | im playing with kolla and trying to deploy newton - but seeing a problem with mariadb during the kolla-build step | 11:11 |
| powerd | INFO:kolla.image.build.mariadb:rm: cannot remove '/var/lib/mysql/mysql': Directory not empty | 11:11 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: add base_arch variable for future non-x86 work https://review.openstack.org/451268 | 11:11 |
| hrw | egonzalez: updated commit message and release notes | 11:11 |
| powerd | anyone seen this or know how to work around? ta | 11:11 |
| *** zhangqiankun has joined #openstack-kolla | 11:14 | |
| hrw | powerd: can you pastebin whole log of image build? | 11:14 |
| powerd | thanks hrw, here you go: https://pastebin.com/BbMkZiJR | 11:16 |
| *** zhangqiankun has quit IRC | 11:18 | |
| egonzalez | powerd, please paste docker info | 11:24 |
| *** rmart04 has joined #openstack-kolla | 11:25 | |
| hrw | powerd: can you add 'systemctl stop mariadb &&' right after RUN? | 11:28 |
| powerd | docker info here: https://pastebin.com/qwXg9yB7 | 11:29 |
| hrw | powerd: it looks like mariadb is running so mysql database cannot be removed | 11:29 |
| *** rwallner has joined #openstack-kolla | 11:29 | |
| egonzalez | powerd, you may hit a bug in linux kernel https://github.com/docker/docker/issues/27214 | 11:30 |
| powerd | yea id stopped mariadb running on the host be subsequent runs still fail | 11:31 |
| egonzalez | a quick google search for cannot remove directory not empty redirect to docker bugs related to kernel 3.10 on centos 7.2 | 11:32 |
| powerd | hmmm im on centos 7.3(.1611) | 11:33 |
| *** yangyapeng has quit IRC | 11:33 | |
| powerd | i got everything to work ok on mitaka branch recently on a similar setup... | 11:33 |
| powerd | ill retry on another system and with the xfs flag mentioned in the link above and report back - thanks | 11:34 |
| manheim | hi, in a multinode setup, where can I control on which node the haproxy kolla_internal_vip_address IP is set? | 11:37 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Karbor services restarting issue on conf update https://review.openstack.org/451735 | 11:40 |
| egonzalez | manheim, by default master is the first haproxy node in the inventory unless it fails and backup becomes master | 11:40 |
| manheim | by master you mean the nodes in the [control] group, right? | 11:41 |
| manheim | i have 2 there | 11:42 |
| *** rwallner has quit IRC | 11:43 | |
| *** yangyapeng has joined #openstack-kolla | 11:43 | |
| egonzalez | manheim, haproxy runs on network nodes | 11:44 |
| *** matrohon has joined #openstack-kolla | 11:44 | |
| manheim | i see | 11:44 |
| manheim | one more question | 11:44 |
| manheim | so i am looking at what services are proxied, for example nova-conductor is not | 11:45 |
| manheim | nvm | 11:45 |
| *** dixiaoli has quit IRC | 11:48 | |
| egonzalez | manheim, what do you mean with "proxied" | 11:49 |
| manheim | do not mind my last bit, i got my answer | 11:50 |
| *** schwicht has quit IRC | 11:50 | |
| manheim | i was refering the the haproxy services | 11:50 |
| *** rhallisey has joined #openstack-kolla | 11:54 | |
| *** rwallner has joined #openstack-kolla | 11:56 | |
| *** athomas has joined #openstack-kolla | 12:01 | |
| *** dave-mccowan has joined #openstack-kolla | 12:16 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/451017 | 12:20 |
| *** rwellum has joined #openstack-kolla | 12:21 | |
| powerd | just to close off my query earlier - retried on an ext4 system and it the mariadb build went through fine. this is an XFS bug. Thanks for the pointer! | 12:22 |
| *** Pavo has joined #openstack-kolla | 12:23 | |
| *** schwicht has joined #openstack-kolla | 12:30 | |
| *** gkadam has quit IRC | 12:32 | |
| *** Pavo has quit IRC | 12:32 | |
| *** schwicht has quit IRC | 12:35 | |
| *** pbandark is now known as pbandark`brb | 12:39 | |
| *** yingjun has joined #openstack-kolla | 12:40 | |
| *** srwilkers has joined #openstack-kolla | 12:41 | |
| pbourke | cores, looking for acks on the following please https://review.openstack.org/#/c/425766/ https://review.openstack.org/#/c/426322/ https://review.openstack.org/#/c/435023/ https://review.openstack.org/#/c/435024/ https://review.openstack.org/#/c/437329/ | 12:42 |
| *** yangyapeng has quit IRC | 12:43 | |
| *** Pavo has joined #openstack-kolla | 12:47 | |
| *** schwicht_at_work has joined #openstack-kolla | 12:49 | |
| *** shardy has joined #openstack-kolla | 12:49 | |
| *** shardy is now known as shardy_mtg | 12:50 | |
| *** Pavo has quit IRC | 12:51 | |
| *** jmontleon has quit IRC | 12:52 | |
| *** rhallisey has quit IRC | 12:53 | |
| *** schwicht_at_work has quit IRC | 12:53 | |
| *** rhallisey has joined #openstack-kolla | 12:54 | |
| *** jmontleon has joined #openstack-kolla | 12:54 | |
| *** krtaylor has joined #openstack-kolla | 12:54 | |
| *** prameswar has quit IRC | 12:54 | |
| *** pbandark has joined #openstack-kolla | 12:55 | |
| *** shardy_mtg is now known as shardy | 12:56 | |
| *** pbandark`brb has quit IRC | 12:56 | |
| mliima_ | morning guys | 12:57 |
| *** jmontleon has quit IRC | 12:57 | |
| *** manheim has quit IRC | 12:58 | |
| *** jmontleon has joined #openstack-kolla | 12:59 | |
| *** yingjun has quit IRC | 13:00 | |
| sdake | morning mliima_ | 13:03 |
| sdake | pbourke this needs a quick rev and then I'll +2 it: https://review.openstack.org/#/c/425766/5 | 13:06 |
| *** lamt has joined #openstack-kolla | 13:06 | |
| *** jtriley has quit IRC | 13:07 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Remove image building related docs from kolla-ansible https://review.openstack.org/425766 | 13:07 |
| pbourke | thanks sdake | 13:08 |
| *** yangyapeng has joined #openstack-kolla | 13:10 | |
| *** yingjun has joined #openstack-kolla | 13:10 | |
| *** esharao has joined #openstack-kolla | 13:13 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Optimize reconfiguration for haproxy https://review.openstack.org/424156 | 13:14 |
| *** goldyfruit has quit IRC | 13:15 | |
| *** goldyfruit has joined #openstack-kolla | 13:15 | |
| sdake | pbourke i reviewed the other patches as well except the one where jeffrey had a -1 | 13:18 |
| sdake | or where there was already a workflow in place | 13:18 |
| *** manheim has joined #openstack-kolla | 13:18 | |
| pbourke | thanks again | 13:18 |
| *** schwicht has joined #openstack-kolla | 13:18 | |
| *** zhubingbing has quit IRC | 13:19 | |
| *** signed8bit_Zzz is now known as signed8b_ | 13:19 | |
| *** eanylin has quit IRC | 13:20 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove image building related docs from kolla-ansible https://review.openstack.org/425766 | 13:20 |
| sdake | sbezverk http://logs.openstack.org/56/451556/16/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/b57a8a8/console.html#_2017-03-30_10_53_00_809626 | 13:21 |
| sdake | sbezverk if you have any idea why this operation is failing would love to know :) | 13:21 |
| *** lamt has quit IRC | 13:21 | |
| sdake | sbezverk happens in both 1.6.0 and 1.5.4 centos | 13:22 |
| sdake | the rpm release process for kubernetes is not gated | 13:22 |
| sdake | this is why we have so much trouble with theses new rpms | 13:22 |
| *** Deys has quit IRC | 13:22 | |
| sdake | deb packaging is the same story - not gated | 13:22 |
| *** schwicht has quit IRC | 13:22 | |
| sbezverk | sdake: will look at it later, need to do a few things on my main job.. | 13:23 |
| sdake | sbezverk ack | 13:23 |
| sdake | sbezverk i've got other things to do for my day job as well :) | 13:23 |
| esharao | gm guys | 13:24 |
| sdake | hey esharao | 13:24 |
| esharao | new day - new vm - going to retry the whole installation again :) | 13:25 |
| mliima_ | need reviews | 13:27 |
| mliima_ | https://review.openstack.org/#/c/450797/ | 13:27 |
| sdake | esharao i'd recommend if your doing AIO to use the 1.5.4 rpms here: | 13:28 |
| hrw | "cp: cannot stat ‘/var/lib/docker/volumes/kolla_logs/_data/haproxy/haproxy_latest.log’: No such file or directory" - hm ;( | 13:28 |
| sdake | https://review.openstack.org/#/c/451556/16/tools/setup_kubernetes.sh line 24-28 | 13:29 |
| sdake | esharao this wont work multinode - kubeadm is fubared on kubeadm join - however that step isnt' needed in AIO | 13:29 |
| sdake | esharao did you give up on ubuntu for the short term? If not, those deb commands wont work as they dont work in the gate | 13:31 |
| *** schwicht has joined #openstack-kolla | 13:32 | |
| esharao | sdake... yes based on yesterday's discussion i gave up on ubuntu | 13:32 |
| esharao | will do an AIO on centos | 13:32 |
| esharao | i tried yesterday with 1.6 but my kube-dns kept on getting ip from the docker0 ip range and busybox nslookup kept on failing | 13:33 |
| *** eanylin has joined #openstack-kolla | 13:33 | |
| *** Deys has joined #openstack-kolla | 13:36 | |
| *** jaosorior is now known as jaosorior_away | 13:37 | |
| *** varto has quit IRC | 13:37 | |
| *** zhubingbing_ has joined #openstack-kolla | 13:38 | |
| sdake | esharao 1.6 is some serious pain | 13:38 |
| sdake | the RPM packaging is AFU | 13:38 |
| sdake | see https://github.com/kubernetes/release/issues/305 for an overview | 13:39 |
| esharao | sdake agree... on 1.6 .. taking a look | 13:39 |
| esharao | makes sense on gating and tagging.. i thought they would have a better conrolled release process.. | 13:41 |
| openstackgerrit | Merged openstack/kolla master: Allow the use of previously built parent images https://review.openstack.org/426322 | 13:42 |
| sdake | esharao one would think | 13:43 |
| openstackgerrit | Merged openstack/kolla master: Add the missing project link into README.rst https://review.openstack.org/447255 | 13:43 |
| sdake | esharao their gating ends at the binary production | 13:43 |
| sdake | binary being the go binary | 13:43 |
| *** lucasxu has joined #openstack-kolla | 13:43 | |
| esharao | Hopefully inputs from others improves their process | 13:43 |
| esharao | brb | 13:44 |
| *** Pavo has joined #openstack-kolla | 13:46 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Add zun ansible role https://review.openstack.org/416179 | 13:46 |
| *** Pavo has quit IRC | 13:50 | |
| rwellum | sbezverk: trying to follow the steps you went through with esharao last night, k8s is fine, but calico is stuck in Pending. I am seeing: http://paste.openstack.org/show/604859/ | 13:54 |
| *** sayantan_ has joined #openstack-kolla | 13:56 | |
| esharao | rwellum can you check the ip of kube-dns? | 13:57 |
| rwellum | https://www.irccloud.com/pastebin/G1pvzfbx/ | 13:58 |
| esharao | ahh exactly my issue.. the kube-dns gets ip from docker0 and not the cluster ip we assign | 13:58 |
| sdake | rwellum esharao i'd recommend again to use the 1.5.4 kubeadm rpms i built for the moment | 13:59 |
| sdake | sorting out the 1.6 transition is going to take awhile | 14:00 |
| esharao | sdake .. yes.. that issue was from yesterday.. will use the 1.5.4 and retry today | 14:00 |
| sdake | my rpms dont work multinode | 14:00 |
| sdake | but they do work single node | 14:00 |
| esharao | on a single node | 14:00 |
| *** matrohon has quit IRC | 14:00 | |
| sbezverk | rwellum: kubectl get pods -n kube-system | 14:01 |
| sdake | rwellum if you do a rebuild of your system use this: | 14:01 |
| sdake | yum install -y docker ebtables \ | 14:01 |
| sdake | https://fedorapeople.org/groups/kolla/kubeadm-1.6.0-0.alpha.0.2074.a092d8e0f95f52.x86_64.rpm \ | 14:01 |
| sdake | https://fedorapeople.org/groups/kolla/kubectl-1.5.4-0.x86_64.rpm \ | 14:01 |
| sdake | https://fedorapeople.org/groups/kolla/kubelet-1.5.4-0.x86_64.rpm \ | 14:01 |
| sdake | https://fedorapeople.org/groups/kolla/kubernetes-cni-0.3.0.1-0.07a8a2.x86_64.rpm | 14:01 |
| *** srwilkers has quit IRC | 14:01 | |
| *** yingjun has quit IRC | 14:01 | |
| *** srwilkers has joined #openstack-kolla | 14:01 | |
| rwellum | sdake: tried last night - k8s, cni came up, but genconfig was failing with what looked like bugs fixed a while ago. | 14:02 |
| rwellum | sbezverk: pasted that above: https://www.irccloud.com/pastebin/G1pvzfbx/ | 14:03 |
| sbezverk | rwellum: kubectl get ds -n kube-system | 14:03 |
| sdake | rwellum you tried those rpms I pointed out? | 14:03 |
| rwellum | https://www.irccloud.com/pastebin/E2nztw7G/ | 14:04 |
| *** papacz has quit IRC | 14:04 | |
| sbezverk | rwellum: you are using old version of canal.yaml | 14:05 |
| sdake | rwellum i'd recommend sticking to the guide as much as possible and ignore what is happenign in the 1.6 port patch | 14:05 |
| sbezverk | rwellum: in 1.6.0 labeling changed so your etcd canal container cannot find label | 14:05 |
| rwellum | sbezverk: drat... | 14:06 |
| rwellum | sdake: yes I tried that last night | 14:06 |
| sdake | rwellum can you run rpm -qa | grep kube -> paste | 14:06 |
| sbezverk | rwellum: you can get right canal from this location: url="https://raw.githubusercontent.com/projectcalico/canal/master" | 14:07 |
| sbezverk | url="$url/k8s-install/kubeadm/1.6/canal.yaml" | 14:07 |
| rwellum | I don't have that 1.5.x system up right now | 14:07 |
| rwellum | The guide has: curl -L https://raw.githubusercontent.com/projectcalico/canal/7deb07cda04147ab49115f437151705c747d0374/k8s-install/kubeadm/canal.yaml -o canal.yaml | 14:07 |
| sdake | rwellum i see, your tyring 1.6 then :) | 14:07 |
| sbezverk | rwellum: wrong for 1.6.0 | 14:07 |
| sdake | rwellum right -t he guide and the instructions sbezverk are providing are mutually incompatible :) | 14:07 |
| rwellum | sdake: the guide doesn't have the 1.5.x rpms though | 14:08 |
| rwellum | sbezverk: ack | 14:08 |
| rwellum | Sorry guys was trying to find a green-path - neither worked so this morning I just picked 1.6.0 because esharao had success I thought | 14:08 |
| sdake | rwellum agreed - I'll rev that | 14:08 |
| sbezverk | rwellum: when you get new canal.yaml, do not forget 1: to clean up old one and 2: to change config | 14:08 |
| *** jtriley has joined #openstack-kolla | 14:09 | |
| esharao | rwellum - i used the setup_canal.sh from sbezverk | 14:09 |
| sdake | rwellum my day is capped at 24 hours :) | 14:09 |
| rwellum | esharao: the one they told me was only for gates :) | 14:09 |
| rwellum | appreciate all the help sdake | 14:09 |
| rwellum | sbezverk: what is step '2' - you mean the listen port etc? | 14:10 |
| sbezverk | rwellum: yep, to customize to your env.. | 14:10 |
| rwellum | yes sbezverk | 14:10 |
| esharao | rwellum it had support for 1.6 | 14:10 |
| rwellum | sdake: the other thing I didn't understand in your current guide - you used to have this " sed -i "s@10.96.232.136@10.3.1.100@" canal.yaml" - then removed it? I still needed it with that version of canal.yaml | 14:11 |
| sdake | rwellum it is possible i made an error in the review - looking | 14:12 |
| sdake | rwellum cool thing about gerrit is it keeps every copy :) | 14:12 |
| sdake | rwellum do you know which review it was present in? | 14:12 |
| esharao | rwellum i thought you needed it based on your cluster config | 14:13 |
| rwellum | Disappeared between 18 and 19 sdake | 14:13 |
| sdake | rwellum gah | 14:14 |
| sdake | rwellum well that is needed | 14:14 |
| *** satyar has joined #openstack-kolla | 14:14 | |
| rwellum | Ok I'll add it as a review comment | 14:14 |
| rwellum | I was pretty confused by then | 14:14 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 14:15 |
| sdake | rwellum ya my bad - dd gone wild | 14:16 |
| *** manheim has quit IRC | 14:16 | |
| *** lamt has joined #openstack-kolla | 14:19 | |
| *** shashank_t_ has joined #openstack-kolla | 14:22 | |
| *** shashank_t_ has quit IRC | 14:22 | |
| *** hrw has quit IRC | 14:22 | |
| *** hrw has joined #openstack-kolla | 14:22 | |
| *** matrohon has joined #openstack-kolla | 14:22 | |
| *** shashank_t_ has joined #openstack-kolla | 14:24 | |
| openstackgerrit | Merged openstack/kolla master: Update Docker to 1.13.1 https://review.openstack.org/446158 | 14:24 |
| *** Pavo has joined #openstack-kolla | 14:27 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 14:31 |
| *** schwicht has quit IRC | 14:33 | |
| *** Pavo has quit IRC | 14:33 | |
| sdake | esharao rwellum note i have not updated that document based upon the numerous comments from people - only to solve a couple pain points related to the kubernetes 1.6 change | 14:35 |
| sdake | I'm now revving it with peoples comments included | 14:35 |
| esharao | sdake: ack | 14:35 |
| *** yangyapeng has quit IRC | 14:39 | |
| *** mnasiadka has quit IRC | 14:44 | |
| pbourke | sdake: replied to comment on https://review.openstack.org/#/c/435023 | 14:45 |
| sdake | thanks pbourke - email overflowing | 14:45 |
| *** Deys has quit IRC | 14:46 | |
| *** schwicht has joined #openstack-kolla | 14:46 | |
| sdake | pbourke will +2 if you file a blueprint or bug for sorting out the reparnting problem properly | 14:46 |
| sdake | probably need a blueprint for that one :) | 14:47 |
| pbourke | sdake: the parent thing is a huge can of worms :/ | 14:49 |
| sdake | pbourke agreed - worth a blueprint no? | 14:49 |
| *** yangyapeng has joined #openstack-kolla | 14:49 | |
| pbourke | well, a blueprint implies I have a strategy in mind there | 14:49 |
| *** satyar has quit IRC | 14:50 | |
| pbourke | essentially here Im just reparenting one image to solve a few things, but bringing in a new parent for source or, dropping parents altogether requires a lot of discussion | 14:51 |
| rwellum | sdake: ack to doc reving | 14:51 |
| pbourke | sdake: lets check with inc0 on it | 14:51 |
| pbourke | sdake: he was looking at some pocs in this area | 14:52 |
| *** skramaja has quit IRC | 14:52 | |
| sdake | pbourke ok - wel lthen i'll +2/+w if you agree to speak with inc0 on the topic :) | 14:52 |
| sdake | that also wfm :) | 14:52 |
| pbourke | sounds like a plan | 14:52 |
| kolla-slack1 | <inc0> Give me 30min | 14:53 |
| kolla-slack1 | <inc0> Just woken up | 14:53 |
| pbourke | hello over there in slack land | 14:53 |
| *** rwallner has quit IRC | 14:54 | |
| sdake | pbourke gerrit jut locked up :) | 14:55 |
| *** rwallner has joined #openstack-kolla | 14:55 | |
| sdake | there we go - enjoy | 14:55 |
| pbourke | sdake: appreciate it | 14:56 |
| *** targon has quit IRC | 14:58 | |
| *** shashank_t_ has quit IRC | 14:58 | |
| kfox1111 | morning. | 14:59 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 14:59 |
| sbezverk | kfox1111: good morning | 14:59 |
| sbezverk | kfox1111: needed to change ps a bit for multinode scenarion and addressed network instability, appreciate your opinion | 15:00 |
| kfox1111 | cool. I'll have a look. | 15:01 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix cinder config group in nova.conf https://review.openstack.org/451613 | 15:03 |
| *** fooliouno has joined #openstack-kolla | 15:03 | |
| *** klindgren_ has joined #openstack-kolla | 15:04 | |
| kfox1111 | looking good. | 15:05 |
| *** manheim has joined #openstack-kolla | 15:07 | |
| *** klindgren__ has quit IRC | 15:07 | |
| rwellum | https://www.irccloud.com/pastebin/5AdJ76tC/ | 15:08 |
| rwellum | sbezverk: sdake ^^^^ | 15:08 |
| sbezverk | rwellum: nice !! | 15:09 |
| sdake | rwellum which version is that? | 15:09 |
| rwellum | That's 1.6 with sbezverk workarounds. | 15:09 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Magnum: add oslo_messaging_notifications config https://review.openstack.org/451857 | 15:10 |
| *** Serlex has quit IRC | 15:10 | |
| *** adrian_otto has joined #openstack-kolla | 15:11 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove useless variable glance_backend_file https://review.openstack.org/451688 | 15:11 |
| *** manheim has quit IRC | 15:11 | |
| *** shashank_t_ has joined #openstack-kolla | 15:12 | |
| rwellum | http://paste.openstack.org/show/604871/ esharao sbezverk sdake : all steps I took (Centos VM) here. | 15:12 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 15:13 |
| sbezverk | kfox1111: I like your RBAC thing ;) please make sure you add yourself as a co-author, I do not want to carry whole blame for this PS ;) | 15:14 |
| *** admin0 has left #openstack-kolla | 15:14 | |
| *** adrian_otto has quit IRC | 15:15 | |
| * kfox1111 chuckles | 15:15 | |
| kfox1111 | sbezverk: k. :) | 15:16 |
| sdake | rwellum can you add that to a new issue against this repo: | 15:16 |
| sdake | https://github.com/kubernetes/release/tree/master/rpm | 15:17 |
| sdake | sbezverk blamed you are :) | 15:17 |
| sdake | kfox1111 - say ip was not running in the environment | 15:17 |
| sdake | kfox1111 the gate never ran the fix-iptables script correctly | 15:18 |
| kfox1111 | sdake: interesting. | 15:18 |
| sdake | kfox1111 instead it ran - i'm not sure - however ip failed to be found | 15:18 |
| sdake | kfox1111 to resolve it in the fs, i had to run /usr/sbin/ip | 15:18 |
| kfox1111 | wonder when that broke. it did run for a while I think. | 15:18 |
| sdake | kfox1111 i'm not sure what it does | 15:18 |
| sdake | kfox1111 infra may ahve changed teh default paths in the images | 15:18 |
| kfox1111 | we migh want to do that in a different ps. | 15:18 |
| *** schwicht has quit IRC | 15:18 | |
| kfox1111 | it could cause new issues to pop up. | 15:19 |
| sdake | kfox1111 fair enough - i wasn't sure what it does :) | 15:19 |
| sdake | kfox1111 just that it was a problem | 15:19 |
| *** jtriley has quit IRC | 15:19 | |
| kfox1111 | hmm.. | 15:19 |
| kfox1111 | lets see... | 15:20 |
| kfox1111 | basically, if that was failing, its running with the infra's iptables stuff disabled. | 15:21 |
| sbezverk | kfox1111: ok returning back kubelet's network config made things worse.. | 15:21 |
| kfox1111 | normally there is a rule in there that pushes all traffic through infra's chain. | 15:21 |
| sbezverk | we have two choices then since without network config lots of people reported networking instability | 15:21 |
| kfox1111 | for k8s, we need the infra rules to only apply to the public interface. | 15:21 |
| kfox1111 | so that tries to rewrite the rules to push it only on the one interface. | 15:22 |
| sbezverk | we can either live with instability until the fix arrive or just wait for the fix.. | 15:22 |
| *** jtriley has joined #openstack-kolla | 15:22 | |
| kfox1111 | hmm... | 15:23 |
| kfox1111 | have we heard back on the ps? sounded like it might be close. | 15:23 |
| sbezverk | kfox1111: 15 minutes ago I left an agree note asking for 1.6.1 with the fix :) | 15:24 |
| sbezverk | kfox1111: now main kube guy is asking questions | 15:24 |
| sdake | imo its not really kubeadm that is fubared | 15:25 |
| sdake | well it may be broken | 15:25 |
| sdake | alhtough - the packaging is busted | 15:25 |
| sbezverk | no it is kubelet bug | 15:25 |
| kfox1111 | heh. theres breakage in lots of places. | 15:25 |
| sbezverk | it is uncoditionally waiting for something that not always there | 15:25 |
| sbezverk | it should not always be there | 15:26 |
| rwellum | sbezverk - seems like new (broken) behavior https://www.irccloud.com/pastebin/qXTwpV7K/ | 15:26 |
| sdake | i can't get join to work - wtb a solution :) | 15:26 |
| kfox1111 | sbezverk: might test your fix without the iptables fix. | 15:26 |
| kfox1111 | the firewall might be getting in the way. | 15:27 |
| sbezverk | rwellum: read the scroll back I provided right command or check latest PS | 15:27 |
| sbezverk | kfox1111: do you mean what sdake added? | 15:27 |
| sbezverk | kubectl taint nodes --all=true node-role.kubernetes.io/master:NoSchedule- | 15:28 |
| sbezverk | rwullum: ^^^ | 15:28 |
| kfox1111 | sbezverk: yeah. | 15:28 |
| sbezverk | sorry rwellum: | 15:28 |
| sbezverk | kfox1111: ok let me give it a shot | 15:28 |
| rwellum | that worked (taint) - thanks guys. | 15:30 |
| sbezverk | kfox1111: kube people saying that they did not catch that bug because their kubeadm was broken ;) 100% agree not tested --- not working | 15:31 |
| sbezverk | meant to say: kubeadm testing was broken | 15:31 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 15:32 |
| *** satyar has joined #openstack-kolla | 15:33 | |
| *** rwallner has quit IRC | 15:33 | |
| *** bjolo_ has quit IRC | 15:34 | |
| kfox1111 | sbezverk: yeah. can't point to a single thing they did wrong. lots of little mistakes piled up in this release. | 15:34 |
| kfox1111 | not uncommon. | 15:34 |
| kfox1111 | thats why operators shy away from x.0 releases. | 15:35 |
| *** hrw has quit IRC | 15:35 | |
| kfox1111 | and why they throw a fit when x-1.y releases are removed too soon. | 15:35 |
| *** adrian_otto has joined #openstack-kolla | 15:35 | |
| sbezverk | kfox1111: Oh I agree we should have stayed with 1.5.4 if it has not been broken along the way too ;) | 15:36 |
| kfox1111 | yeah. :/ | 15:36 |
| *** hrw has joined #openstack-kolla | 15:36 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 15:37 |
| sdake | rwellum with your bleeding edge deployment instructions ^^ | 15:38 |
| sdake | rwellum note - kolla-kubernetes requires all other kinds of workarounds to enable to run on 16.0 | 15:38 |
| sdake | kfox1111 one thing that kubernetes could improve on is to actually gate their RPMS | 15:40 |
| rwellum | sdake: ack | 15:41 |
| sdake | kfox1111 check out: https://github.com/kubernetes/release/issues/305 | 15:41 |
| rwellum | sdake: the cherry pick steps are failing for me. That's is nothing to do with k8s at this point. | 15:41 |
| sdake | rwellum which cherrypick - you mean git am? | 15:41 |
| sdake | rwellum or the kollakube cherrypick? | 15:42 |
| rwellum | sdake: I'm looking at it now but wanted to give you a quick heads up https://www.irccloud.com/pastebin/HNyH6GyN/ | 15:43 |
| kfox1111 | sdake: +1. I think a lot of it is inexperience with package managers. they are use to containers/statically linked go binaries, but not packages. | 15:44 |
| kfox1111 | which makes some sence, as their normal modus operandi is to avoid rpms. | 15:44 |
| *** blallau has quit IRC | 15:45 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 15:46 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Fix the "Creating the Zun service and endpoint" failed https://review.openstack.org/451876 | 15:48 |
| esharao | sdake.. checking out the latest guide.. it says that repository not found when trying to clone the kubernetes repo | 15:48 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 15:50 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 15:50 |
| sdake | esharao which command areou typing in for reference | 15:50 |
| sdake | rwellum that new guide should have the correct fetch instructions ^^ | 15:50 |
| sdake | rwellum free of problems relating to merge conflicts | 15:51 |
| sdake | zhubingbing_ - wtb kollakbue resolution :) | 15:51 |
| *** yangyapeng has quit IRC | 15:52 | |
| esharao | sdake i see now that was for building packages | 15:53 |
| *** rmart04 has quit IRC | 15:53 | |
| sdake | esharao hrm - i haven't tested all the docs yet - busy writing them ;) | 15:53 |
| sdake | esharao if something is broken please leave a comment in the review and i'll fix it :) | 15:54 |
| esharao | no issues.. will put out all the comments | 15:54 |
| zhubingbing_ | hear | 15:54 |
| inc0 | ok I'm here | 15:54 |
| inc0 | shoot pbourke | 15:54 |
| sdake | zhubingbing_ that kollakube bug - i think we still need that fixed when you have time ;) | 15:54 |
| zhubingbing_ | roger | 15:54 |
| pbourke | inc0: did you manage to get anywhere with your poc for removing base? | 15:54 |
| zhubingbing_ | https://review.openstack.org/#/c/439740/ | 15:55 |
| zhubingbing_ | this patch | 15:55 |
| zhubingbing_ | sdake | 15:55 |
| sdake | zhubingbing_ thats the one - i just rebased it as it was in mere conflict for some reasosn | 15:55 |
| sdake | zhubingbing_ atm we are referencing that in our deployment documentatino | 15:56 |
| sdake | that it is necessary to "git fetch" the patch | 15:56 |
| sdake | that is not really optimal :( | 15:56 |
| sdake | as the patch rev changes each time | 15:56 |
| zhubingbing_ | i know | 15:56 |
| zhubingbing_ | I have prepare a new environment to re-verify that deployment document | 15:57 |
| inc0 | pbourke: https://review.openstack.org/#/c/449426/ | 15:57 |
| pbourke | there's still a question of what benefits it brings | 15:58 |
| pbourke | Jeffrey4l seemed concerned about it | 15:58 |
| inc0 | it's really niche use case | 15:58 |
| pbourke | inc0: also are you familiar with this 'new source parent' idea sdake has mentioned? | 15:58 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Fix the "Creating the Zun service and endpoint" failed https://review.openstack.org/451876 | 15:58 |
| inc0 | when you want to build your images in CI, you want them self-contained | 15:59 |
| zhubingbing_ | sdake we use kubernetes 1.6? | 16:00 |
| *** egonzalez has quit IRC | 16:00 | |
| *** yangyapeng has joined #openstack-kolla | 16:00 | |
| inc0 | as in, I'm testing this keystone container, so I want to deploy it exactly the way it was | 16:01 |
| inc0 | tested | 16:01 |
| *** pbandark has quit IRC | 16:02 | |
| inc0 | and if I'm rebuilding for example nova, it's going to change base image | 16:03 |
| inc0 | so technically your keystone is no longer correct | 16:04 |
| *** caoyuan has joined #openstack-kolla | 16:04 | |
| inc0 | but that's important for CI/CD based deployment that rebuilds their containers per service all the time | 16:04 |
| kfox1111 | inc0: are you saing you rebuild all the things and redeploy all the things? | 16:05 |
| sbezverk | gents, anybody tried 1.6.0 workaround on Ubuntu? | 16:05 |
| inc0 | kfox1111: what I'm saying is when you rebuild base, next time you deploy keystone it will be different | 16:05 |
| inc0 | it won't change running containers in any way | 16:06 |
| inc0 | so don't worry | 16:06 |
| kfox1111 | inc0: thats why you should pin your versions. | 16:06 |
| kfox1111 | ie, every new build should get its own tag, and every deployment shoudl specify the version number. | 16:06 |
| kfox1111 | /tag | 16:06 |
| rwellum | sdake: I used the exact fetch command in the current doc and it fails. | 16:06 |
| kfox1111 | then its atomic. | 16:06 |
| inc0 | we use uppper constrains and all that, but we don't want too much constrains there too for security updates | 16:07 |
| kfox1111 | theres more then one way to skin this cat. | 16:07 |
| inc0 | yeah | 16:07 |
| kfox1111 | for k8s, you enver want a rs to be split. | 16:07 |
| kfox1111 | so you must specify tag. | 16:07 |
| *** matrohon has quit IRC | 16:07 | |
| inc0 | which works today | 16:07 |
| kfox1111 | yeah. | 16:07 |
| inc0 | I'm just playing around;) | 16:07 |
| kfox1111 | so, not sure I agree with the assertion. | 16:07 |
| kfox1111 | but I do think we need a smooth way of tagging. | 16:08 |
| kfox1111 | which ties in to needing that container fingerprinting thing. | 16:08 |
| inc0 | maybe good idea would be to have per-service tagging | 16:08 |
| inc0 | yeah...good idea too | 16:08 |
| kfox1111 | ideally per container I think. | 16:08 |
| kfox1111 | some containers are easier to upgrade then others. | 16:08 |
| inc0 | well you can do it today | 16:08 |
| kfox1111 | being able to know that an upgrade is needed to easily upgradable piece A, but not needed for hard to upgrade piece B is important. | 16:09 |
| inc0 | kfox1111: let's have this discussion next week ok? | 16:11 |
| inc0 | how to rework our tagging | 16:11 |
| *** zhubingbing_ has quit IRC | 16:11 | |
| inc0 | I think that's a good thing to do | 16:11 |
| kfox1111 | inc0: ok. sounds good. :) | 16:11 |
| inc0 | or think about at least | 16:11 |
| kfox1111 | sbezverk: did you remove the rpm install too? I think that was the thing that actualy made the iptables stuff start working, not the /usr/sbin thing? | 16:13 |
| kfox1111 | yeah. iproute is stil in the rpm list. try pulling that? | 16:14 |
| sbezverk | kfox1111: ok missed that | 16:14 |
| inc0 | http://forumtopics.openstack.org/cfp/details/69 | 16:17 |
| sdake | rwellum - pretty sure i ttested it befoer i submitted i t- i'll tst again | 16:18 |
| *** jaosorior_away is now known as jaosorior | 16:19 | |
| esharao | sdake quick question.. im getting the flannel ip for the kube-dns instead of cluster dns | 16:19 |
| esharao | what am i missing? | 16:19 |
| *** shardy has quit IRC | 16:20 | |
| sdake | esharao not sure - although looking at rwellum 's problem at the moment | 16:20 |
| powerd | hi, has anyone any tips on forcing kolla-build to ignore a previous build and start from fresh? (i may or may not have mucked up some docker overlay files...) | 16:20 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 16:21 |
| esharao | oh ok | 16:21 |
| sbezverk | bb in 1 hour, meeting time | 16:21 |
| *** pcaruana has quit IRC | 16:23 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla master: Reduce size of images built from git repos https://review.openstack.org/421252 | 16:25 |
| sdake | rwellum i cloned a fresh kolla-kubernetes - fetched the latest deployment doc patch - then did this: | 16:27 |
| sdake | http://paste.openstack.org/show/604880 | 16:27 |
| sdake | that git fetch works for me | 16:27 |
| sdake | rwellum you may need to remove what you have fetched - try git reset --hard | 16:28 |
| sdake | note git reset --hard will change any local file changes | 16:28 |
| sdake | esharao can you link me the review you are currently testing? | 16:28 |
| esharao | https://review.openstack.org/#/c/447356/23 | 16:29 |
| rwellum | sdake: I tried in a completely different VM with a fresh setup and did not help. I'll go through your log and try it again | 16:30 |
| sdake | rwellum what fetch command are you using? | 16:30 |
| sdake | esharao rwellum you may already know this - however gerrit is a little wonky about how it works | 16:32 |
| sdake | https://review.openstack.org/#/c/447356/ | 16:32 |
| sdake | is different then https://review.openstack.org/#/c/447356/23 | 16:32 |
| sdake | if you click to comment - it takes you to the specific review | 16:33 |
| sdake | then your browser will stay there and not go back to the main review | 16:33 |
| sdake | someone upstream reported 1.5.6 is broken in a rollback for ubuntu | 16:38 |
| * sdake GROANS | 16:38 | |
| *** mgoddard has quit IRC | 16:38 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Fix the zun api-paset.ini file not found https://review.openstack.org/451903 | 16:40 |
| *** sambetts is now known as sambetts|afk | 16:41 | |
| *** athomas has quit IRC | 16:43 | |
| *** caoyuan has quit IRC | 16:44 | |
| *** rwallner has joined #openstack-kolla | 16:48 | |
| *** jaosorior has quit IRC | 16:50 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla master: Reduce size of images built from git repos https://review.openstack.org/421252 | 16:51 |
| *** rwallner has quit IRC | 16:53 | |
| *** lucasxu has quit IRC | 16:53 | |
| *** lucasxu has joined #openstack-kolla | 16:56 | |
| *** gfidente is now known as gfidente|afk | 17:01 | |
| *** jtriley has quit IRC | 17:02 | |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted briefly to relieve performance issues, and should return to service again momentarily. | 17:03 | |
| kfox1111 | sbezverk: ps is looking close now. :) | 17:05 |
| kfox1111 | sbezverk: http://logs.openstack.org/91/451391/39/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/9d8abdf/console.html#_2017-03-30_16_44_20_027559 | 17:06 |
| rwellum | sdake: that was it I was getting '17' regardless | 17:06 |
| kfox1111 | I think they said they changed the labeling scheme in 1.6? | 17:06 |
| sdake | rwellum you mean the review/17 problem with gerrit? | 17:07 |
| sdake | itsn ot really a problem - more of a feature I guess or just the way the tech works | 17:07 |
| *** jtriley has joined #openstack-kolla | 17:08 | |
| rwellum | Yes sdake | 17:10 |
| *** MasterOfBugs has joined #openstack-kolla | 17:10 | |
| *** signed8b_ is now known as signed8bit_Zzz | 17:12 | |
| rwellum | sdake: now the next patch failed and that one is quite simple (for the API placement fix) | 17:17 |
| rwellum | sdake: https://www.irccloud.com/pastebin/C18bUkGb/ | 17:17 |
| sdake | rwellum if you want to submit a patch to fix that - i can help you with the workflow | 17:21 |
| sdake | rwellum that way I can eanble you :) | 17:21 |
| sdake | rwellum if you don't have time let me know, and i'll fix it | 17:22 |
| sdake | rwellum should take roughly 20 mins on webex | 17:22 |
| *** powerd has quit IRC | 17:23 | |
| rwellum | I don't mind - either way tbh - was just going through the depl. guide 1 step at a time and this part failed - could have been operator error like the last one... :) | 17:23 |
| sdake | rwellum lets do it in webex so you can learn how to do this work :) | 17:24 |
| sdake | rwellum http://cisco.webex.com/join/stdake | 17:24 |
| sdake | anyone welcome to join in to watch | 17:24 |
| rwellum | Sorry I have a meeting in 5m, and finish at 3pm est :( | 17:25 |
| sdake | ok - well i'll get it done then as I have a conflict at 3pm est | 17:25 |
| rwellum | Can we do post 3pm? | 17:25 |
| rwellum | Ah - sorry. | 17:26 |
| sdake | let me check calendar - moment | 17:26 |
| sdake | 3pm est = 12pm MST | 17:26 |
| sdake | that is actually open | 17:27 |
| sdake | lets do it then | 17:27 |
| rwellum | ok sounds good | 17:27 |
| rwellum | BTW I do have a review open, https://review.openstack.org/#/c/449790/ - no-one signed up yet | 17:27 |
| sdake | rwellum looking at that review, where does $CONFIG come from? | 17:29 |
| *** david-lyle__ is now known as david-lyle | 17:30 | |
| sdake | rwellum nm - +2 ftw :) | 17:31 |
| sdake | Jeffrey4l any chance you can sort out how to make kolla-kubernetes doc specific changes not run the entire gate as is done in kolla? | 17:32 |
| rwellum | sorry - was walking to meeting | 17:32 |
| rwellum | ty | 17:32 |
| sdake | Jeffrey4l or point me to the appropriate point in pc where its done so I can copy it | 17:32 |
| *** unicell1 has joined #openstack-kolla | 17:32 | |
| *** unicell has quit IRC | 17:33 | |
| *** financeMann has quit IRC | 17:33 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: handle rtslib(-fb) package names and dependencies https://review.openstack.org/450805 | 17:35 |
| *** signed8bit_Zzz is now known as signed8b_ | 17:36 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu https://review.openstack.org/432787 | 17:38 |
| *** rhallisey has quit IRC | 17:41 | |
| *** schwicht has joined #openstack-kolla | 17:41 | |
| *** rhallisey has joined #openstack-kolla | 17:41 | |
| sdake | rwellum when you get out of your meeting - inc0 mentioned the gates are red which we know - so i placed your patch in workflow -1 (this is standard openstack workflow) | 17:44 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Split Openvswitch into own role https://review.openstack.org/416793 | 17:50 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 17:50 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: debian: move to stretch https://review.openstack.org/434453 | 17:51 |
| *** rwallner has joined #openstack-kolla | 17:51 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu https://review.openstack.org/432787 | 17:52 |
| *** karlamrhein_ has joined #openstack-kolla | 17:54 | |
| *** rwallner has quit IRC | 17:56 | |
| *** SamYaple_ has joined #openstack-kolla | 17:57 | |
| *** hogepodge_ has joined #openstack-kolla | 17:57 | |
| *** dciabrin has quit IRC | 17:58 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: handle ppc64le https://review.openstack.org/434810 | 17:58 |
| kfox1111 | hmm.... so dns resolution on second node is not working.... | 17:59 |
| *** hogepodge has quit IRC | 17:59 | |
| *** SamYaple has quit IRC | 17:59 | |
| *** karlamrhein has quit IRC | 17:59 | |
| *** karlamrhein_ is now known as karlamrhein | 18:00 | |
| *** hogepodge_ is now known as hogepodge | 18:00 | |
| rwellum | sdake: thanks - the status says 'Not Workflow' though? | 18:02 |
| *** SamYaple_ has quit IRC | 18:03 | |
| *** SamYaple has joined #openstack-kolla | 18:03 | |
| sdake | rwellum ya - I -1'ed the workflow to stop it from being merged - even though it can have 2 +2's on it | 18:04 |
| sdake | rwellum that way reviewers can review it - you can recheck it and any core can workflow it after a recheck | 18:05 |
| sdake | rwellum this should permit inc0 to change his vote to +2 since the review looks good | 18:05 |
| *** rwallner has joined #openstack-kolla | 18:05 | |
| sbezverk | kfox1111: there is another thing which I do not get, how it could have get impacted. openvswitch-vswitchd does not exist and the job fails. | 18:05 |
| sbezverk | http://logs.openstack.org/91/451391/39/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/9d8abdf/console.html#_2017-03-30_16_44_20_354875 | 18:05 |
| sdake | rwellum and hold off on the +w decision until the gate is operational and a recheck is done | 18:05 |
| *** dciabrin has joined #openstack-kolla | 18:06 | |
| inc0 | rwellum: also that's reason why you didn't get reviews | 18:06 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 18:06 |
| inc0 | when I see red gates, I just assume WIP and move to different review | 18:06 |
| kfox1111 | sbezverk: yeah, I'm a bit weirded out by that.... | 18:07 |
| rwellum | ack sdake | 18:07 |
| sbezverk | kfox1111: I added docker ps -a to see what is going on | 18:08 |
| rwellum | inc0: well it's 5-days+ old, and gates were fine back then I think. Just my fault for not asking I think. | 18:08 |
| sdake | rwellum nah its not your fault - reviewers are slow to review in kolla-kubernetes deliverable | 18:08 |
| sdake | rwellum asking shouldn't be needed - reviewers should be reviewing daily | 18:09 |
| sdake | rwellum although reviewing daily falls down during emergencies | 18:09 |
| inc0 | or heavy dev ususally;) | 18:09 |
| sdake | inc0++ | 18:09 |
| kfox1111 | sbezverk: k. | 18:09 |
| sdake | i have a review in gerrit that is 1 year old :) | 18:09 |
| inc0 | it gets better once project is out of "omg omg let's get it up asap" state | 18:09 |
| rwellum | sdake: lol | 18:09 |
| rwellum | Definition of 'merge conflict' | 18:10 |
| sdake | rwellum https://review.openstack.org/#/c/306756/ | 18:10 |
| *** jtriley has quit IRC | 18:10 | |
| sdake | merge conflict means git cherrypick wouldn't work | 18:10 |
| sdake | or git am wouldn't work | 18:10 |
| sdake | or any applicatoin of the patch on top of the existing master would fail | 18:10 |
| sdake | so it requires a rebase - manual or otherwise | 18:11 |
| sdake | rwellum i'll teacch you that at 3pm est | 18:11 |
| sdake | rwellum (how to fix a merge conflict) | 18:11 |
| rwellum | Cheers! | 18:11 |
| rwellum | (I've actually used git quite a bit - but gerrit is very new to me) | 18:12 |
| sdake | its a fundamental contributor skill that is required | 18:12 |
| sbezverk | Gents, who prefers Ubuntu to Centos, we need your help to test 1.6.0 upgrade, pleasae help!! | 18:12 |
| sdake | rwellum yup - i try not to assume too much about people when i teach em :) | 18:12 |
| sdake | rwellum if I assume you know everything I miss the steps you need | 18:12 |
| sdake | if I assume you dont know everything, I end up teaching dupicate info | 18:13 |
| sdake | generally my approach is to start with a "doesn't know" and go from there :) | 18:13 |
| rwellum | wfm sdake | 18:13 |
| sdake | inc0 hows the golang thing | 18:13 |
| inc0 | kinda cool, I see lots of use cases where golang will excell...just none that we deal with;) | 18:14 |
| inc0 | besides maybe cli clients - cli clients in golang are awesome | 18:14 |
| *** jtriley has joined #openstack-kolla | 18:18 | |
| *** powerd has joined #openstack-kolla | 18:19 | |
| sdake | i like golang | 18:24 |
| sdake | python + C mashup | 18:24 |
| sdake | I dislike the bundled binaries | 18:24 |
| *** powerd has quit IRC | 18:24 | |
| inc0 | thats why I say it's cool for CLI | 18:25 |
| inc0 | curl and it works | 18:25 |
| inc0 | and disk is cheap | 18:25 |
| kfox1111 | one of go's primary features is easy staticlly linked things. | 18:27 |
| kfox1111 | but I think containers do a way better job at that. | 18:27 |
| inc0 | yeah, agree | 18:28 |
| *** adrian_otto has quit IRC | 18:28 | |
| inc0 | not a big deal for different things than clients | 18:28 |
| inc0 | you don't want to spawn container for each cli call | 18:28 |
| inc0 | (interesting use case tho;)) | 18:28 |
| kfox1111 | not sure you don't though. :) | 18:28 |
| kfox1111 | if a container can run in half a second once cached, then its not too bad really. | 18:29 |
| kfox1111 | I think they could tune it a little bit more and get even to that point. | 18:29 |
| kfox1111 | as is, I do shell script to docker exec cli wrappers for a few things. makes this soo much easier. | 18:29 |
| inc0 | well if you create some hyper-lightweight container engine | 18:30 |
| inc0 | that could work | 18:30 |
| inc0 | it's cost of seting up docker and stuff that I dislike | 18:30 |
| sbezverk | kfox1111: it is frealing crazy, they changed naming of docker containers | 18:30 |
| sbezverk | docker.io/kolla/centos-binary-openvswitch-vswitchd@sha256:0c2e6a14f9e39e2bbf4c966832f390da4b2361c98a8104f4f39fb1ade74a7ca8 | 18:30 |
| kfox1111 | sbezverk: yeah. was in the changelog. | 18:30 |
| *** Mech422 has joined #openstack-kolla | 18:30 | |
| sdake | does that break the universe? | 18:30 |
| kfox1111 | their claim is you should be using kubectl, not docker commands. | 18:31 |
| Mech422 | sdake: just getting ready to test your deploy docs... | 18:31 |
| Mech422 | sdake: any last minute stuff I should be aware of ? | 18:31 |
| inc0 | noise | 18:31 |
| sdake | Mech422 cool - note - they are a work in progress - so you may be frustrated | 18:31 |
| kfox1111 | docker naming/labeling is their own internal stuff. | 18:31 |
| Mech422 | sdake: yeah - no biggie - thats why we test :-) | 18:31 |
| sdake | i see, so no breaking the universe except that using docker cli is aa pita ;) | 18:31 |
| sdake | is NOW a pita | 18:32 |
| kfox1111 | sdake: its always a pita, but now its differently pita. ;) | 18:32 |
| sdake | Mech422 i deploy on bare metal - however i have gone through them about 40 times, so i recommend a vm for your testing | 18:32 |
| sdake | Mech422 (IOW I know how to clean up properly after lots of pain) | 18:33 |
| Mech422 | sdake: I've gotta run thru bare metal just so I know how... | 18:33 |
| sdake | Mech422 sounds good | 18:33 |
| Mech422 | sdake: I can do VM testing after though | 18:33 |
| sdake | Mech422 note also we think there are some incomplete areas of the instruction - not sure if anone but eanylin has got cmpute to spawn a vm | 18:34 |
| Mech422 | sdake: btw - dunno if its interesting, but it looks like we're settling on SaltStack/Consul for Prod, and I'm sticking with Ansible/Consul for dev | 18:34 |
| Mech422 | sdake: we're gonna be using Consul in what sounds like a pretty similiar manner to the etcd stuff that was discussed on the ML | 18:35 |
| sdake | Mech422 I see - I think inc0 is invovled in that | 18:35 |
| inc0 | "involved" | 18:36 |
| sbezverk | kfox1111: once this nightmare is over, will change docker exec for kubectl exec | 18:36 |
| sbezverk | kfox1111: or better yet helm chart job to do the same | 18:37 |
| *** mgoddard has joined #openstack-kolla | 18:37 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 18:39 |
| *** satyar has quit IRC | 18:40 | |
| kfox1111 | sbezverk: +1 | 18:41 |
| *** mgoddard has quit IRC | 18:44 | |
| rwellum | sdake: meeting over fyi | 18:44 |
| sdake | rwellum lets get to it if your up to it | 18:46 |
| rwellum | sure | 18:46 |
| sdake | http://cisco.webex.com/join/stdake | 18:46 |
| sdake | anyone welcome to attend a "how to resolve merge conflicts" session | 18:46 |
| mnaser | i found a gif | 18:47 |
| mnaser | of how we build kolla | 18:47 |
| mnaser | https://i.redd.it/etxk36gkujoy.gif | 18:47 |
| mnaser | :D | 18:47 |
| sdake | mnaser what is missing from that gif is the dude out front blowing the mountains up to make room for the tracks :) | 18:47 |
| *** powerd has joined #openstack-kolla | 18:51 | |
| *** shashank_t_ has quit IRC | 18:53 | |
| *** shashank_t_ has joined #openstack-kolla | 18:54 | |
| *** zhangqiankun has joined #openstack-kolla | 19:00 | |
| sbezverk | kfox1111: it looks almost good ;) | 19:14 |
| kfox1111 | yeah. :) | 19:15 |
| *** MasterOfBugs has quit IRC | 19:15 | |
| kfox1111 | ubuntu's still failing, as is multinode. | 19:15 |
| kfox1111 | consistent behavior with each one. so 2 seperate issues I think. | 19:15 |
| kfox1111 | dns is failing to work on slave node.... | 19:17 |
| *** zhangqiankun has quit IRC | 19:18 | |
| kfox1111 | not sure if thats a calico issue or not... | 19:19 |
| *** jmccarthy1 has quit IRC | 19:21 | |
| sbezverk | kfox1111: I am bringing up ubuntu vm to test workaround | 19:23 |
| kfox1111 | k. | 19:24 |
| kfox1111 | I'm looking at multinode. | 19:24 |
| *** shashank_t_ has quit IRC | 19:28 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 19:28 |
| *** MasterOfBugs has joined #openstack-kolla | 19:29 | |
| kfox1111 | the multinode one is weird.... | 19:36 |
| kfox1111 | kube-proxy looks happy. so does canal... | 19:36 |
| *** gfidente|afk has quit IRC | 19:37 | |
| *** shashank_t_ has joined #openstack-kolla | 19:38 | |
| openstackgerrit | Rich Wellum proposed openstack/kolla-kubernetes master: Fix broken gate due to kolla-ansible change https://review.openstack.org/451957 | 19:38 |
| *** powerd has quit IRC | 19:40 | |
| Mech422 | sdake: I just realized - that doc is for kolla-k*s ... | 19:42 |
| Mech422 | sdake: is there a vanilla kolla one that needs review ? | 19:42 |
| sdake | Mech422 no idea | 19:45 |
| sdake | kfox1111 sbezverk rwellum 's review needs to go in for anything to work | 19:45 |
| sdake | kfox1111 see your already on it :) | 19:46 |
| sdake | thanks :) | 19:46 |
| sdake | ok - meeting time for 2 hours of my life bbl :) | 19:46 |
| kfox1111 | hmm.... not an iptables issue on the slave... | 19:46 |
| Mech422 | sdake: my bad - I thought this was the updated guide for otaco..any way, I'll do kolla-ansible, then kolla-kube | 19:46 |
| kfox1111 | sdake: ;) | 19:46 |
| mnaser | Jeffrey4l around? | 19:47 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Fix broken gate due to kolla-ansible change https://review.openstack.org/451957 | 19:53 |
| inc0 | woho | 19:55 |
| kfox1111 | inc0: ? | 19:55 |
| inc0 | rkt is full fledged cncf project now | 19:55 |
| kfox1111 | interesting. | 19:55 |
| inc0 | sayantan_: we should check kolla containers running on rkt soon;) | 19:56 |
| sayantan_ | inc0 I agree | 19:56 |
| inc0 | best case scenerio we can turn kolla-docker to be runtime-agnostic | 19:56 |
| inc0 | and allow almost automatically kolla-ansible deploy on rkt | 19:56 |
| kfox1111 | inc0: +1. docker is getting farther and farther away from where k8s wants to go I think. | 19:57 |
| kfox1111 | inc0: containerd was tryiong to get in too. what happened with that? | 19:58 |
| sayantan_ | This is interesting | 19:58 |
| kfox1111 | hmm... looks like containerd also made it. | 19:58 |
| *** unicell1 has quit IRC | 20:00 | |
| *** unicell has joined #openstack-kolla | 20:03 | |
| Mech422 | inc0: on the 'etcd config' stuff - what are your feelings about confd ? | 20:06 |
| inc0 | I don't know confd enough to develop emotional bond of any kind | 20:06 |
| Mech422 | inc0: I was thinking of using consul-templates for some of our local stuff - if this is a 'likely' path for kolla/openstack, I could just as easy do confd | 20:07 |
| openstackgerrit | Mohammed Naser proposed openstack/kolla-ansible master: Enable tempest test in gate https://review.openstack.org/402122 | 20:07 |
| inc0 | Mech422: can we discuss it on some other day please? on training | 20:07 |
| Mech422 | inc0 sorry, sure - whenever you have time :-) | 20:08 |
| inc0 | tomorrow I'm back in action;) sorry about that | 20:08 |
| sbezverk | sdake: btw you do not need cleanup script anymore | 20:13 |
| sbezverk | "kubeadm reset" does the job.. | 20:13 |
| kfox1111 | sbezverk: does it do slave nodes too? | 20:14 |
| sbezverk | kfox1111: do not think so | 20:15 |
| kfox1111 | sbezverk: k. | 20:15 |
| kfox1111 | might be able to run it on the node anyway. | 20:15 |
| *** rwallner has quit IRC | 20:23 | |
| fooliouno | I tried a multinode environment with kolla_controller on one node and kolla_compute on another. All pods came up but got a neutron port binding error during insatnce creation. | 20:24 |
| *** adrian_otto has joined #openstack-kolla | 20:24 | |
| *** new2kolla has joined #openstack-kolla | 20:24 | |
| *** unicell1 has joined #openstack-kolla | 20:25 | |
| *** rwallner has joined #openstack-kolla | 20:26 | |
| sbezverk | inc0: ping | 20:26 |
| kfox1111 | hmm... this in flannel looks maybe relevent... | 20:27 |
| inc0 | whats up? | 20:27 |
| kfox1111 | I0330 18:50:32.041056 1 main.go:132] Installing signal handlers | 20:27 |
| kfox1111 | I0330 18:50:32.041248 1 manager.go:136] Determining IP address of default interface | 20:27 |
| kfox1111 | I0330 18:50:32.041738 1 manager.go:149] Using interface with name eth0 and address 10.32.218.159 | 20:27 |
| kfox1111 | I0330 18:50:32.041754 1 manager.go:166] Defaulting external address to interface address (10.32.218.159) | 20:27 |
| kfox1111 | E0330 18:50:32.043903 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [3] | 20:27 |
| kfox1111 | E0330 18:50:33.044800 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [3] | 20:27 |
| kfox1111 | E0330 18:50:34.045718 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [3] | 20:27 |
| sbezverk | inc0: what is cgroup driver you have on your ubuntu box? | 20:27 |
| kfox1111 | E0330 18:50:35.046554 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [3] | 20:27 |
| kfox1111 | E0330 18:50:36.047588 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [3] | 20:27 |
| kfox1111 | E0330 18:50:37.048429 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [3] | 20:27 |
| kfox1111 | E0330 18:50:38.049477 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:27 |
| kfox1111 | E0330 18:50:39.051093 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:27 |
| kfox1111 | E0330 18:50:40.052150 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:27 |
| kfox1111 | E0330 18:50:41.053012 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:27 |
| kfox1111 | E0330 18:50:42.054094 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:27 |
| kfox1111 | E0330 18:50:43.055203 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:27 |
| sbezverk | kfox1111: looks like etcd issue | 20:27 |
| kfox1111 | E0330 18:50:44.056081 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:27 |
| kfox1111 | E0330 18:50:45.057300 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:27 |
| kfox1111 | E0330 18:50:46.058161 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:27 |
| kfox1111 | E0330 18:50:47.059203 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:28 |
| kfox1111 | E0330 18:50:48.060026 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:28 |
| kfox1111 | E0330 18:50:49.061130 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:28 |
| kfox1111 | E0330 18:50:50.062059 1 network.go:102] failed to retrieve network config: 100: Key not found (/coreos.com) [21] | 20:28 |
| *** unicell has quit IRC | 20:28 | |
| kfox1111 | I0330 18:50:51.072600 1 local_manager.go:179] Picking subnet in range 172.16.129.0 ... 172.16.131.0 | 20:28 |
| kfox1111 | I0330 18:50:51.077338 1 ipmasq.go:47] Adding iptables rule: -s 172.16.128.0/22 -d 172.16.128.0/22 -j RETURN | 20:28 |
| kfox1111 | I0330 18:50:51.081646 1 ipmasq.go:47] Adding iptables rule: -s 172.16.128.0/22 ! -d 224.0.0.0/4 -j MASQUERADE | 20:28 |
| kfox1111 | I0330 18:50:51.084031 1 ipmasq.go:47] Adding iptables rule: ! -s 172.16.128.0/22 -d 172.16.128.0/22 -j MASQUERADE | 20:28 |
| kfox1111 | I0330 18:50:51.085977 1 manager.go:250] Lease acquired: 172.16.129.0/24 | 20:28 |
| kfox1111 | I0330 18:50:51.086362 1 network.go:58] Watching for L3 misses | 20:28 |
| *** rwallner_ has joined #openstack-kolla | 20:28 | |
| kfox1111 | I0330 18:50:51.086394 1 network.go:66] Watching for new subnet leases | 20:28 |
| kfox1111 | doh. | 20:28 |
| kfox1111 | sry. | 20:28 |
| kfox1111 | I0330 18:51:02.474876 1 network.go:243] L3 miss but route for 172.16.128.100 not found | 20:28 |
| kfox1111 | sbezverk: yeah, but it looks like it already gets a flannel range from etcd before then? | 20:28 |
| *** unicell has joined #openstack-kolla | 20:29 | |
| sbezverk | inc0: docker info what is trhe driver you have for cgroup??? | 20:31 |
| *** rwallner has quit IRC | 20:31 | |
| inc0 | sbezverk: hold on | 20:31 |
| inc0 | cgroupfs | 20:31 |
| *** unicell1 has quit IRC | 20:32 | |
| sbezverk | inco0: thanks | 20:32 |
| *** rwallner_ has quit IRC | 20:32 | |
| *** manheim has joined #openstack-kolla | 20:34 | |
| *** jmccarthy has joined #openstack-kolla | 20:35 | |
| *** manheim has quit IRC | 20:38 | |
| openstackgerrit | Mohammed Naser proposed openstack/kolla-ansible master: Enable tempest test in gate https://review.openstack.org/402122 | 20:40 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 20:41 |
| sbezverk | kfox1111: it looks like ubuntu defaults to cgroupfs driver for docker whereas centos uses systemd. Do we want to use systemd for both ubuntu and centos? It is just I ma not familiar what people mostly used in production | 20:46 |
| *** jmccarthy has quit IRC | 20:46 | |
| fooliouno | kfox1111: question about multi-node environment. I have a node where all the "controller" host and a "compute" host. Neutron is on the controller host and is trying to bind a port on the compute host. | 20:46 |
| fooliouno | And it fails. says that there is no agent of type Openvswitch agent on the compute host | 20:46 |
| fooliouno | Should there be pods other than nova-compute and nova-libvirt running on the compute host | 20:47 |
| kfox1111 | sbezverk: not sure. probably whatever's the distro's way of doing things. it may be ubuntu has an older systemd and only supports cgroupfs. | 20:47 |
| kfox1111 | fooliouno: you need openvswitch-agents on the compute nodes too. | 20:47 |
| sbezverk | kfox1111: ok then I will get docker's cgroup driver and specify it in kubelet parameters | 20:48 |
| sbezverk | to be consistent with distro setting | 20:48 |
| kfox1111 | should be able to run a second instance of neutron-openvswith-agent helm package configured for compute nodes. | 20:48 |
| kfox1111 | sbezverk: k | 20:48 |
| fooliouno | kfox1111: ok. So, I'm guessing somewhere in our helm charts, we should look for the label and install the openvswitch agents on compute nodes as well? | 20:49 |
| kfox1111 | fooliouno: see tests/bin/ceph_workflow.sh, in the section that looks like: | 20:50 |
| kfox1111 | [ "x$1" == "xceph-multi" ] && | 20:50 |
| kfox1111 | there's a neutron-openvswitch-agent-daemonset setup for compute listed. | 20:50 |
| fooliouno | Thanks kfox1111, let me look into that | 20:50 |
| kfox1111 | np | 20:50 |
| *** unicell has quit IRC | 20:53 | |
| kfox1111 | hmm... not sure how this ever worked... | 20:53 |
| kfox1111 | maybe canal was not very picky with its subnets overlapping before.. | 20:54 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 20:54 |
| kfox1111 | or k8s proxy maybe... | 20:54 |
| kfox1111 | sbezverk: have a tweak in the review. | 20:55 |
| kfox1111 | may fix the multinode issue. | 20:55 |
| *** unicell has joined #openstack-kolla | 20:56 | |
| sbezverk | have you pushed it already? can you base yours on top of what I just pushed? | 20:58 |
| kfox1111 | sbezverk: me? | 21:00 |
| kfox1111 | no. just commented in your review. if yo ucould push that. | 21:00 |
| sbezverk | kfox1111: ahh I see | 21:00 |
| kfox1111 | I think the subnets were overlapping before. | 21:01 |
| kfox1111 | but it was fine before. now, not so sure. | 21:01 |
| *** rhallisey has quit IRC | 21:01 | |
| fooliouno | kfox1111: In addition to neutron-openvswitch-agent-daemonset, there are other listed for the compute node as well (openvswitch-ovsdb-daemonset, openvswitch-vswitchd-daemonset). Should those two be deployed on the compute host as well. | 21:03 |
| kfox1111 | yeah. | 21:04 |
| *** srwilkers has quit IRC | 21:05 | |
| fooliouno | ok .. thank you | 21:05 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 21:07 |
| sbezverk | kfox1111: pushed, I also added a tweak for ubuntu | 21:08 |
| sbezverk | it is working fine on my newly built ubuntu vm | 21:08 |
| kfox1111 | sbezverk: cool. | 21:08 |
| *** new2kolla has quit IRC | 21:13 | |
| *** schwicht has quit IRC | 21:16 | |
| *** unicell has quit IRC | 21:17 | |
| *** goldyfruit has quit IRC | 21:17 | |
| mnaser | kolla-core: i have built images for ubuntu binary for 3.0.3 - we have failures. http://paste.openstack.org/show/604927/ | 21:21 |
| mnaser | do we try to fix them and release something with those images working? | 21:22 |
| mnaser | or should i push to dockerhub | 21:22 |
| esharao | sdake.. using centos and the updated dg.. got my nova compute to come up (something that did not work for ubuntu).. will try to get a tenant vm up later.. | 21:23 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 21:25 |
| *** yangyapeng has quit IRC | 21:25 | |
| fooliouno | kfox1111: Just wanted to report that with those new microservices on compute host, I was able to create an instance in a "true" multinode environment. | 21:26 |
| kfox1111 | fooliouno: awesome. :) | 21:26 |
| *** esharao has quit IRC | 21:27 | |
| fooliouno | kfox1111: thanks for your help :) | 21:28 |
| *** yangyapeng has joined #openstack-kolla | 21:29 | |
| kfox1111 | fooliouno: anytime. :) | 21:29 |
| *** mcnanci has quit IRC | 21:30 | |
| *** dave-mccowan has quit IRC | 21:32 | |
| mnaser | first semi-successful run of kolla + tempest - http://logs.openstack.org/22/402122/14/check/gate-kolla-ansible-dsvm-deploy-centos-binary-centos-7-nv/c68aab7/console.html | 21:32 |
| mnaser | based off the work of Jeffrey4l :) | 21:32 |
| kfox1111 | mnaser: nice. :) | 21:34 |
| sbezverk | kfox1111: Ubuntu is running :) | 21:38 |
| kfox1111 | nice. :) | 21:38 |
| kfox1111 | multinode still is failing. :/ | 21:40 |
| *** Pavo has joined #openstack-kolla | 21:43 | |
| *** fooliouno has quit IRC | 21:43 | |
| *** adrian_otto has quit IRC | 21:45 | |
| *** Pavo has quit IRC | 21:45 | |
| kfox1111 | ubuntu openvswithc failure... | 21:45 |
| kfox1111 | maybe the cri issue. | 21:46 |
| kfox1111 | do we know what docker version ubuntu's using? | 21:46 |
| sdake | esmiurium thats good news - i am also at this state | 21:46 |
| sdake | esmiurium sorry mismessage | 21:46 |
| sbezverk | kfox1111: I added docker info so it should be in the log | 21:46 |
| sdake | kfox1111 its in the gate logs | 21:46 |
| sdake | just search for docker | 21:46 |
| sdake | kfox1111 i would - but exhausted | 21:47 |
| kfox1111 | sbezverk: try adding --enable-cri=false | 21:47 |
| *** jtriley has quit IRC | 21:47 | |
| openstackgerrit | Mohammed Naser proposed openstack/kolla-ansible master: Enable tempest test in gate https://review.openstack.org/402122 | 21:48 |
| sbezverk | kfox1111: to kubelet? what does it do? | 21:48 |
| sdake | mnaser ++ | 21:48 |
| sdake | mnaser note we have tried this in the past - and the gates OOM | 21:49 |
| mnaser | sdake given that we're not installing $world, the gate might be dealing in an acceptable fashion | 21:49 |
| mnaser | if not, then we have to adopt something like puppet and do a bunch of scenarios | 21:49 |
| sdake | mnaser the current thinking is we run tempest in a second vm | 21:49 |
| sdake | mnaser and other services in one vm | 21:50 |
| mnaser | "once we get zuul" | 21:50 |
| mnaser | :p | 21:50 |
| sdake | mnaser control plane in one vm - compute + tempest | 21:50 |
| sdake | ya if i hear zuulv3 one moer time | 21:50 |
| sdake | i'm pretty sure i'm gonig to punch my monitor | 21:50 |
| mnaser | im sure we can tweak things to make it run | 21:51 |
| mnaser | if devstack-gate can run all the time without ooming, im sure we can learn a thing or two | 21:51 |
| mnaser | im basing a lot of my work on that | 21:51 |
| sdake | mnaser been there done that although you may be smarter then the others that have come before you :) | 21:51 |
| mnaser | im up for a challenge and we all benefit if it works out :> | 21:51 |
| sdake | mnaser ++ | 21:52 |
| kfox1111 | sbezverk: yeah. same as the centos one. | 21:53 |
| *** lucasxu has quit IRC | 21:54 | |
| *** Pavo has joined #openstack-kolla | 21:55 | |
| sbezverk | kfox1111: any idea here: 017-03-30T21:44:51Z|00001|daemon_unix|WARN|/var/run/openvswitch/ovs-vswitchd.pid: stale pidfile for pid 8 | 21:56 |
| sbezverk | being deleted by pid 0 | 21:56 |
| sbezverk | ovs-appctl: cannot read pidfile "/var/run/openvswitch/ovs-vswitchd.pid" (No such process) | 21:56 |
| *** Pavo has quit IRC | 21:56 | |
| kfox1111 | sbezverk: yeah. theres a changelog entry that they broke shared pid namespaces in docker 1.12. | 21:56 |
| kfox1111 | setting --enable-cri=false disables their beta docker driver, | 21:57 |
| kfox1111 | using the same one 1.5 used. | 21:57 |
| kfox1111 | which seems to work ok with shared pid namespaces. | 21:57 |
| kfox1111 | I have no idea why they defautled the beta to enabled. :/ | 21:57 |
| sbezverk | kfox1111: ah, go it so I do it on both ubuntu and centos | 21:57 |
| kfox1111 | yeah. try. might fix it. | 21:58 |
| *** Pavo has joined #openstack-kolla | 21:59 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 21:59 |
| *** schwicht has joined #openstack-kolla | 22:00 | |
| *** manheim has joined #openstack-kolla | 22:01 | |
| *** harlowja has quit IRC | 22:02 | |
| *** mcnanci has joined #openstack-kolla | 22:04 | |
| *** manheim has quit IRC | 22:04 | |
| *** manheim has joined #openstack-kolla | 22:05 | |
| openstackgerrit | Mohammed Naser proposed openstack/kolla-ansible master: Enable tempest test in gate https://review.openstack.org/402122 | 22:11 |
| *** Pavo has quit IRC | 22:14 | |
| sbezverk | kfox1111: looks better | 22:17 |
| openstackgerrit | Mohammed Naser proposed openstack/kolla-ansible master: Enable tempest test in gate https://review.openstack.org/402122 | 22:18 |
| kfox1111 | yup. | 22:23 |
| kfox1111 | all we need now is the multinode gates. | 22:24 |
| sbezverk | kfox1111: other centos are failing from time to time :( | 22:27 |
| kfox1111 | sbezverk: right before the: cat > /tmp/dns-test.py | 22:28 |
| kfox1111 | in tools/test_kube_dns.sh | 22:28 |
| kfox1111 | can you add: | 22:28 |
| kfox1111 | curl -s http://172.16.128.100:6666/version | 22:28 |
| kfox1111 | please. | 22:28 |
| kfox1111 | sbezverk: not sure thats not an old race we've always had. | 22:29 |
| sbezverk | kfox1111: are you sure etcd will be listening on this ip? | 22:29 |
| kfox1111 | yeah. we hard code it. | 22:29 |
| sbezverk | ah right in canal.yaml | 22:30 |
| kfox1111 | kind of curious if that will work or not. | 22:32 |
| kfox1111 | everything seems to be in a pretty good state as far as the logs go. | 22:33 |
| kfox1111 | but there seems to be some connectivity problem still. | 22:33 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 22:33 |
| kfox1111 | thx. | 22:33 |
| sdake | sup peeps | 22:33 |
| sbezverk | sdake: we got ubuntu working ;) with 1.6.0 | 22:33 |
| *** g3ek has quit IRC | 22:34 | |
| kfox1111 | sdake: yeah. now just gotta get multinode working and we're back in buisness. :) | 22:34 |
| sdake | good news | 22:34 |
| *** lamt has quit IRC | 22:35 | |
| *** schwicht_at_work has joined #openstack-kolla | 22:36 | |
| kfox1111 | very very nice. | 22:36 |
| kfox1111 | spacex is awesome. :) | 22:36 |
| *** schwicht has quit IRC | 22:36 | |
| masber | hi, I am getting a "503 Service Unavailable" after kolla deployment | 22:37 |
| kfox1111 | they just landed the same rocket a second time :) | 22:37 |
| masber | openstack works but not horizon | 22:37 |
| masber | enable_horizon_sahara: "{{ enable_sahara | bool }}" is this the right entry to enable sahara web interface? | 22:38 |
| masber | or do I have to change bool to true? | 22:38 |
| openstackgerrit | Mohammed Naser proposed openstack/kolla-ansible master: Enable tempest test in gate https://review.openstack.org/402122 | 22:40 |
| sbezverk | kfox1111: so rockets containers are reusable now ;) ?? | 22:41 |
| *** schwicht_at_work has quit IRC | 22:41 | |
| *** manheim has quit IRC | 22:43 | |
| *** caoyuan has joined #openstack-kolla | 22:43 | |
| kfox1111 | yup. :) | 22:43 |
| *** g3ek has joined #openstack-kolla | 22:43 | |
| sbezverk | kfox1111: we can start two tcpdump instances on the host one capturing traffic from kube_dns_test pod and one on host external interface | 22:48 |
| sbezverk | to see where packets get lost.. | 22:48 |
| kfox1111 | interesting... | 22:48 |
| *** jrobinson has joined #openstack-kolla | 22:48 | |
| kfox1111 | so... etcd did work from the pod network. | 22:49 |
| kfox1111 | so, k8s services on the second node are working. | 22:49 |
| kfox1111 | but kube-dns isn't accessable from the second node... | 22:49 |
| kfox1111 | very weird. | 22:49 |
| sbezverk | kfox1111: we need to check pods's /etc/resolv.conf | 22:49 |
| kfox1111 | sbezverk: can you | 22:49 |
| kfox1111 | exactly. :) | 22:49 |
| kfox1111 | can you change the curl to a cat /etc/resolv.conf? | 22:49 |
| sbezverk | kfox1111: yep, one sec | 22:50 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 22:51 |
| kfox1111 | really seems like resolv.conf should be right though... | 22:54 |
| kfox1111 | sed -i 's/10.96.0.10/172.16.128.10/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | 22:55 |
| kfox1111 | should always be running if either master or slave. | 22:55 |
| sdake | rwellum you about? | 22:57 |
| sbezverk | kfox1111: hmm how it was running before then?? | 22:58 |
| *** caoyuan has quit IRC | 22:59 | |
| kfox1111 | something changed from 1.5 to 1.6. but really not sure what. | 23:00 |
| kfox1111 | its very strange that: | 23:00 |
| *** shashank_t_ has quit IRC | 23:00 | |
| kfox1111 | curling to canal-etcd svc worked. | 23:01 |
| kfox1111 | dns on one node is working. | 23:01 |
| kfox1111 | dns on the other node is failing. | 23:01 |
| *** shashank_t_ has joined #openstack-kolla | 23:01 | |
| kfox1111 | canal on all nodes report happy. | 23:01 |
| kfox1111 | kubeproxy's report happy. | 23:01 |
| kfox1111 | kube-dns should be running the same way as canal-etcd... | 23:02 |
| *** Pavo has joined #openstack-kolla | 23:03 | |
| sbezverk | kfox1111: or maybe it is collateral of this workaround | 23:04 |
| *** shashan__ has joined #openstack-kolla | 23:04 | |
| sbezverk | some people were reporting issues with flannel between nodes | 23:04 |
| sbezverk | but they reported complete traffic outage, not just dns traffic | 23:04 |
| kfox1111 | resolv.conf looks good. :/ | 23:05 |
| sbezverk | kfox1111: one thing that can be broken is nat | 23:05 |
| *** shashank_t_ has quit IRC | 23:05 | |
| kfox1111 | yeah. possible it affects udp but not tcp... | 23:06 |
| sbezverk | for slave node's container source address | 23:06 |
| sbezverk | that would explain local connectivity and broken connectivity outside | 23:06 |
| kfox1111 | does turning the workaround back off after start break the single node case? | 23:06 |
| sbezverk | we should try to ping | 23:06 |
| kfox1111 | pings harder, as svc's are not pingable. | 23:07 |
| kfox1111 | they arn't real ip's. :/ | 23:07 |
| sbezverk | kfox1111: yep | 23:07 |
| *** signed8b_ is now known as signed8bit_Zzz | 23:07 | |
| *** sayantan_ has quit IRC | 23:07 | |
| kfox1111 | thats unfortunate... :/ | 23:07 |
| sbezverk | as soon as networking config is back kubelet goes crazy | 23:07 |
| kfox1111 | whats it do? | 23:08 |
| *** shashan__ has quit IRC | 23:09 | |
| sbezverk | kfox1111: we can try to ping from container to 8.8.8.8 | 23:09 |
| openstackgerrit | Monty Taylor proposed openstack/kolla-ansible master: Add clouds.yaml file and use it https://review.openstack.org/452005 | 23:09 |
| sbezverk | then capture it at different points and see where it gets broken and why | 23:09 |
| sbezverk | I do not see any other way | 23:09 |
| kfox1111 | sbezverk: couldn't hurt I guess. | 23:09 |
| kfox1111 | ping -c 4 8.8.8.8 in the test_kube_dns container. | 23:10 |
| sbezverk | yeah, but where do we capture on docker0 or on host's external interface? | 23:11 |
| *** schwicht has joined #openstack-kolla | 23:11 | |
| kfox1111 | I think it will go out a canalXXXX interface from the container, route through iptables nat table and go out. | 23:11 |
| kfox1111 | docker0 will be skipped entirely. | 23:11 |
| kfox1111 | go ahead and submit just the ping check and we can think about how to do the tcpdump while it tests. | 23:12 |
| sbezverk | you mean calicoXXXXX | 23:12 |
| kfox1111 | yeah. that. :) | 23:12 |
| sbezverk | ok, but if ping works, that would be really fun.. | 23:12 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 23:13 |
| kfox1111 | gut feeling, I'm pretty sure it will. | 23:14 |
| sbezverk | kfox1111: it would mean then something very dns specific :( | 23:19 |
| *** adrian_otto has joined #openstack-kolla | 23:23 | |
| *** harlowja has joined #openstack-kolla | 23:24 | |
| *** shashank_t_ has joined #openstack-kolla | 23:27 | |
| kfox1111 | yup. pings. :/ | 23:29 |
| *** MasterOfBugs has quit IRC | 23:30 | |
| sbezverk | kfox1111: damn | 23:31 |
| *** shashank_t_ has quit IRC | 23:31 | |
| kfox1111 | very very strange. | 23:32 |
| sbezverk | kfox1111: we need then enable debugging on all dns requests dns container gets | 23:33 |
| kfox1111 | hmm... when do we caputure the slaves iptabhes.... | 23:34 |
| kfox1111 | before... ok. | 23:34 |
| *** MasterOfBugs has joined #openstack-kolla | 23:35 | |
| sbezverk | kfox1111: I doubt now it is iptables issue since we can ping | 23:37 |
| kfox1111 | yeeah, but was hoping to see the calico rules. | 23:39 |
| kfox1111 | but we aren't capturing them yet. | 23:39 |
| sbezverk | - --v=2 I think if we increase this value we will get more verbose logging | 23:40 |
| spsurya | morning all | 23:40 |
| kfox1111 | + # All pods to speak to services that resolve to the same host. | 23:40 |
| kfox1111 | - name: FELIX_DEFAULTENDPOINTTOHOSTACTION | 23:40 |
| kfox1111 | value: "ACCEPT" | 23:40 |
| kfox1111 | interesting comment... | 23:40 |
| kfox1111 | do we need calico rules now for cross node communication? | 23:40 |
| *** openstack has joined #openstack-kolla | 23:46 | |
| *** jtriley has joined #openstack-kolla | 23:49 | |
| sbezverk | kfox1111: building, please prepare container definition for tests | 23:51 |
| *** sayantan_ has joined #openstack-kolla | 23:51 | |
| *** rwellum has quit IRC | 23:51 | |
| kfox1111 | should be able to run the same test_kube_dns script? | 23:52 |
| *** Pavo has quit IRC | 23:52 | |
| kfox1111 | so... if it was enforcing, it should be an annotation on the namespace... lets seee. | 23:53 |
| sbezverk | kfox1111: I do not have any git tools installed | 23:53 |
| kfox1111 | scp? | 23:53 |
| *** sayantan_ has quit IRC | 23:55 | |
| sbezverk | kfox1111: no worries will figure out something.. | 23:56 |
| kfox1111 | so, canal between 1.5 and 1.6 mostly is rbac roles are added. | 23:58 |
| kfox1111 | shouldn't affect this... | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
