Thursday, 2017-08-03

*** http_GK1wmSU has joined #openstack-containers00:14
*** http_GK1wmSU has left #openstack-containers00:17
*** jberkus has joined #openstack-containers00:30
*** jberkus has quit IRC00:32
*** ricolin has joined #openstack-containers00:44
*** Adri2000 has quit IRC00:56
*** Adri2000 has joined #openstack-containers00:57
*** dixiaoli has joined #openstack-containers00:58
*** hongbin has joined #openstack-containers00:59
*** mdnadeem has joined #openstack-containers01:02
*** EricGonczer_ has joined #openstack-containers01:08
*** DFFlanders has quit IRC01:11
*** yamamoto has quit IRC01:45
*** yamamoto has joined #openstack-containers01:45
*** EricGonczer_ has quit IRC01:51
*** ramishra has joined #openstack-containers01:51
*** EricGonczer_ has joined #openstack-containers01:51
*** EricGonczer_ has quit IRC01:52
*** trinaths has left #openstack-containers02:28
*** yamamoto has quit IRC02:41
*** yamamoto has joined #openstack-containers02:51
*** kbyrne has quit IRC02:54
*** kbyrne has joined #openstack-containers02:58
*** EricGonczer_ has joined #openstack-containers03:07
*** masuberu has joined #openstack-containers03:18
*** masber has quit IRC03:22
*** dixiaoli has quit IRC03:28
*** hongbin has quit IRC03:29
*** yamamoto has quit IRC03:33
*** ykarel has joined #openstack-containers03:39
*** yamamoto has joined #openstack-containers03:42
*** EricGonczer_ has quit IRC03:45
*** ykarel has quit IRC03:46
*** ykarel has joined #openstack-containers03:46
*** yamamoto has quit IRC03:48
*** yamamoto has joined #openstack-containers03:49
*** yamamoto has quit IRC03:54
*** EricGonczer_ has joined #openstack-containers03:56
*** adisky__ has joined #openstack-containers04:03
*** EricGonc_ has joined #openstack-containers04:07
*** EricGonczer_ has quit IRC04:08
*** EricGonc_ has quit IRC04:13
*** ramishra has quit IRC04:22
*** ramishra has joined #openstack-containers04:24
*** janki has joined #openstack-containers04:34
*** mvpnitesh has joined #openstack-containers05:25
*** trinaths has joined #openstack-containers05:26
*** dsariel has joined #openstack-containers05:36
ykarelmnaser, Ok checking05:41
*** robcresswell has quit IRC05:41
*** tonanhngo has joined #openstack-containers05:43
openstackgerritMerged openstack/magnum master: Fix some reST field lists in docstrings in magnum  https://review.openstack.org/48603805:51
*** dims has quit IRC05:52
*** dims has joined #openstack-containers05:54
ykarelmnaser, few nit comments, can you check https://review.openstack.org/#/c/48965705:54
*** livelace has joined #openstack-containers05:58
*** mjura_ has joined #openstack-containers06:00
*** mjura has joined #openstack-containers06:04
*** mjura_ has quit IRC06:05
*** dsariel has quit IRC06:06
*** rcernin has joined #openstack-containers06:19
*** dsariel has joined #openstack-containers06:21
*** tianqing has joined #openstack-containers06:23
*** Oku_OS-away is now known as Oku_OS06:23
*** tonanhngo has quit IRC06:28
tianqinghello...06:32
ykareltianqing, hello06:36
tianqinghello..06:36
ykareltianqing, i created cluster in devstack stable/newton  and it's running: http://paste.openstack.org/show/617273/06:37
tianqingis the dashboard ok?06:38
tianqingfrom browser...06:39
tianqingwhat is u etcd.conf?06:39
ykareletcd.conf: http://paste.openstack.org/show/617368/06:41
ykarelhaven't checked the dashboard06:41
tianqingsame as..06:42
tianqingbut i found this in my kubelet log...06:42
tianqing'client: etcd cluster is unavailable or misconfigured' (will not retry!)06:42
tianqingthe whole sentence is06:42
ykarelthis means your etcd is not running correctly06:43
ykareletcdctl ls06:43
tianqingAug 02 11:28:00 k8-2myfq6xl46-0-6ehjpw74z4ht-kube-master-ocmbtfgauswk.novalocal kubelet[30957]: E0802 11:28:00.458620   30957 event.go:199] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"kube-scheduler-10.0.0.6.14d700522e14244c", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"15516", Generation:0, CreationTimest06:43
ykarelon master node06:43
tianqingsystemctl show it is runing06:43
tianqingand healthz is ok....06:43
tianqingand etcd.conf is same as u..06:43
tianqingETCD_NAME="10.0.0.6" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379" ETCD_LISTEN_PEER_URLS="http://10.0.0.6:2380"  ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.6:2379" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.0.0.6:2380" ETCD_DISCOVERY="https://discovery.etcd.io/5edfa0cd77dc03c1ead91e6a2c83007f"06:44
tianqingi setup one kube cluster without magnum06:44
tianqingi can telnet 127.0.0.1:2379 from master node..06:45
tianqingbut in magnum i can not ...06:45
tianqingand i login into one docker ....06:45
tianqingit also can not nc to 127.0.0.1:2379....06:45
tianqingi do not know why...06:45
tianqingthe crashloopback looks like always try etcd cluster.. .. at last it says not try....06:46
ykarel[fedora@k8-2zs5qzwua7-0-useahxrxlztj-kube-master-fescmb7khklg ~]$ etcdctl ls06:46
tianqing[fedora@k8-2myfq6xl46-0-6ehjpw74z4ht-kube-master-ocmbtfgauswk tianqing]$ etcdctl ls /registry /atomic.io /scheduler /controller06:47
tianqingit is work....06:47
ykarelhmm it's good06:47
ykarellooks like, kubelet gave up when etcd came up06:47
ykarelhow many times you tried creating cluster06:48
ykarelcan you try again06:48
openstackgerritMerged openstack/magnum master: Don't poll heat if no stack exists  https://review.openstack.org/48028606:48
*** pcaruana has joined #openstack-containers06:48
tianqingmany many times...06:49
tianqingkube-controller-manager-10.0.0.6                                   1/1       Running   130        1d06:51
ykarel^^ running06:51
tianqingthe one output of get pod --namespace kube-system06:52
tianqingstare on restarts.....06:52
tianqingcan not find the container on docker ps ...06:52
tianqingAug 03 06:55:25 k8-2myfq6xl46-0-6ehjpw74z4ht-kube-master-ocmbtfgauswk.novaloca│ l kubelet[4166]: E0803 06:55:25.305028    4166 pod_workers.go:184] Error synci│ ng pod 946d09ee3fd8595dd07fe5bf3f0bec65, skipping: failed to "StartContainer" │ for "kube-scheduler" with CrashLoopBackOff: "Back-off 1m20s restarting failed │ container=kube-scheduler pod=kube-scheduler-10.0.0.6_kube-system(946d09ee3fd85│ 95dd07fe5b06:55
tianqingthe kubelet says this...06:56
ykareltianqing, Ok, then it looks rdo package is not good, i will check the source code what's missing06:56
tianqingalso....[root@controller ~]# magnum-api --version 3.3.106:58
tianqingcan you login into the container of scheduler or controller-manager..06:58
tianqingi login and use python06:59
tianqingfind i can not get  http://127.0.0.1:2379/health06:59
tianqingwhat's about u...06:59
*** tonanhngo has joined #openstack-containers07:00
ykarel[fedora@k8-2zs5qzwua7-0-useahxrxlztj-kube-master-fescmb7khklg ~]$ curl http://127.0.0.1:2379/health07:01
ykarel{"health": "true"}[fedora@k8-2zs5qzwua7-0-useahxrxlztj-kube-master-fescmb7khklg ~]$07:01
tianqingoh....07:01
ykarelyou should run  http://10.0.0.6:2379/health from kube-controller/scheduler07:02
*** tonanhngo has quit IRC07:02
tianqinghttp://paste.openstack.org/show/617372/07:04
tianqingthis is magnum generated yaml for kube-scheduler...07:04
*** hishh has joined #openstack-containers07:04
tianqingyou mean the parameter master is wrong?07:04
openstackgerritThomas Bechtold proposed openstack/magnum stable/ocata: Do not require a beta release of kubernetes  https://review.openstack.org/49035907:05
ykarelin my environment also there is same file as you07:06
ykarelsomething else is wrong07:06
ykareli will check and get back to you07:06
tianqingok...07:06
*** hishh1 has joined #openstack-containers07:09
tianqingIn master node, can you nc 10.0.0.6 2379?07:10
*** Dinesh_Bhor has quit IRC07:10
tianqingreplace with your address...07:10
openstackgerritDebo Zhang proposed openstack/magnum master: Update links in README  https://review.openstack.org/49036307:10
*** hishh has quit IRC07:11
*** hishh1 is now known as hishh07:11
*** Dinesh_Bhor has joined #openstack-containers07:16
*** armaan has joined #openstack-containers07:22
*** mgoddard has joined #openstack-containers07:33
*** robcresswell has joined #openstack-containers07:41
tianqing@ykarel... i can run container kube-scheduler by hand....07:45
*** ykarel has quit IRC07:45
tianqingbut very slow....07:45
tianqingwhat's the timeout for kubernetes to run a container?07:45
*** mgoddard has quit IRC07:52
*** mvpnitesh has quit IRC07:59
*** ramishra has quit IRC08:01
*** mvpnitesh has joined #openstack-containers08:05
*** ramishra has joined #openstack-containers08:08
*** mgoddard has joined #openstack-containers08:11
*** mvpnitesh has quit IRC08:12
*** openstackgerrit has quit IRC08:18
*** zenoway has joined #openstack-containers08:32
*** zenoway has quit IRC08:35
*** zenoway has joined #openstack-containers08:36
*** mvpnitesh has joined #openstack-containers08:42
tianqingI0803 08:54:58.220960   22270 docker_manager.go:2069] pod "kube-controller-manager-10.0.0.6_kube-system(1aee4d06c7b713c0fc24afea2b8ebd43)" container "kube-controller-manager" is unhealthy, it will be killed and re-created.08:56
*** ykarel has joined #openstack-containers09:00
*** dims has quit IRC09:04
*** dims has joined #openstack-containers09:08
tianqinghello...09:12
tianqingykarel09:12
*** ykarel has quit IRC09:12
tianqingmay be i found the problem......09:12
*** salmankhan has joined #openstack-containers09:29
*** zenoway has quit IRC09:30
*** zenoway has joined #openstack-containers09:31
*** openstackgerrit has joined #openstack-containers09:31
openstackgerritRicardo Rocha proposed openstack/magnum-specs master: Add Federation API specification  https://review.openstack.org/48960909:31
openstackgerritRicardo Rocha proposed openstack/magnum-specs master: Add Federation API specification  https://review.openstack.org/48960909:33
*** yamamoto has joined #openstack-containers09:41
*** zenoway has quit IRC09:44
*** rochaporto has quit IRC09:49
*** strigazi_OFF has quit IRC09:49
*** rochaporto has joined #openstack-containers09:50
*** strigazi_OFF has joined #openstack-containers09:50
*** ramishra has quit IRC09:50
*** ramishra has joined #openstack-containers09:53
*** ramishra has quit IRC09:57
openstackgerritRicardo Rocha proposed openstack/magnum-specs master: Add Federation API specification  https://review.openstack.org/48960910:00
*** dsariel has quit IRC10:00
*** ramishra has joined #openstack-containers10:00
tianqingare10:03
*** zenoway has joined #openstack-containers10:09
*** dsariel has joined #openstack-containers10:13
*** ramishra has quit IRC10:13
*** ramishra has joined #openstack-containers10:15
*** dsariel has quit IRC10:19
*** dsariel has joined #openstack-containers10:32
*** dsariel has quit IRC10:38
*** tianqing has quit IRC10:41
*** mdnadeem has quit IRC10:47
*** dsariel has joined #openstack-containers10:50
*** dixiaoli has joined #openstack-containers10:59
*** tonanhngo has joined #openstack-containers11:03
*** tonanhngo has quit IRC11:04
*** dixiaoli has quit IRC11:04
*** maestropandy has joined #openstack-containers11:06
maestropandyPlease vote (+3) for our presentation submitted for openstack sydney 2017. Today last day for voting, requesting all to vote. OpenStack cloud storage - Advanced performance tuning & operational best practices with Ceph    https://www.openstack.org/summit/sydney-2017/vote-for-speakers/#/19056  ##### Docker Networking over OpenStack Cloud – Challenges, Solutions & Road-map >>   https://www.openstack.org/summit/sydney-2017/vote-for-speake11:08
*** trinaths has quit IRC11:12
*** maestropandy has left #openstack-containers11:14
*** maestropandy has joined #openstack-containers11:28
*** dsariel has quit IRC11:37
*** maestropandy has quit IRC11:46
*** maestropandy has joined #openstack-containers11:47
*** maestropandy has left #openstack-containers11:47
*** dsariel has joined #openstack-containers11:52
*** jchhatbar has joined #openstack-containers11:59
*** jchhatbar has quit IRC12:00
*** jchhatbar has joined #openstack-containers12:00
*** janki has quit IRC12:02
*** adisky__ has quit IRC12:21
*** armaan has quit IRC12:28
*** catintheroof has joined #openstack-containers12:36
*** maestropandy has joined #openstack-containers12:42
*** maestropandy has left #openstack-containers12:43
*** danil has joined #openstack-containers12:49
*** ykarel has joined #openstack-containers12:58
openstackgerritMohammed Naser proposed openstack/magnum master: Force bs-version workaround only to run for K8S 1.7.X and above  https://review.openstack.org/48965713:03
mnaserykarel addressed comments on both :)13:03
*** jchhatbar has quit IRC13:04
ykarelmnaser, i could only see one13:05
ykarel:)13:05
mnaseroh weird my comments disappeared, ykarel my justification was that if we dropped kube_major -ge 2 and changed the eq, it wouldnt match kubernetes 2.0 to kubernetes 2.713:06
*** yamamoto has quit IRC13:06
mnaserso it would inadvertently switch it back13:06
mnaserwe're probably far away from k8s 2.0 but it would change the behaviour13:06
ykarelmnaser, Ok got it13:08
mnaserykarel cool, id appreciate a +2 on that (yours disappeared after the edit) and if you have a short minute for this -- https://review.openstack.org/#/c/488511/ -- k8s controller manager pod doesnt work at all right now in master against ssl endpoints (so anything production) :>13:08
ykareli voted on cinder driver one, checking the other one13:10
*** armaan has joined #openstack-containers13:11
*** mvpnitesh has quit IRC13:13
*** mvpnitesh has joined #openstack-containers13:13
*** ramishra has quit IRC13:13
*** EricGonczer_ has joined #openstack-containers13:14
*** ramishra has joined #openstack-containers13:15
ykarelmnaser, do you have error logs for the issue you faced13:19
ykarelif yes, it would be good you can file a bug and link to commit message13:19
mnaserykarel i used to have them handy let me check13:19
*** mvpnitesh has quit IRC13:20
mnaserykarel i dont have the exact error handy but i can show how it would would just go nowhere (the /etc/ssl/certs mount has a link in it so its unaccessible)13:20
ykarelmnaser, Ok you can file the bug with the info you have, like for which service you required ssl certs, and the link problem13:22
ykarelalso i have some comments on the commit message13:22
openstackgerritMohammed Naser proposed openstack/magnum master: Remove /etc/ssl/certs in the controller manager pod  https://review.openstack.org/48851113:25
mnaserykarel done and done :)13:25
*** mtanino has joined #openstack-containers13:29
*** EricGonc_ has joined #openstack-containers13:29
ykarelmnaser, Thanks13:31
ykareli voted13:31
*** EricGonczer_ has quit IRC13:31
openstackgerritMohammed Naser proposed openstack/magnum master: Set default iptables FORWARD policy to ACCEPT  https://review.openstack.org/49047813:38
mnaserykarel : thank you :)13:39
mnaser^^ that one drove me crazy, i was up all night trying to figure it out13:39
ykarelmnaser, Ok checking13:42
ykarelmnaser, we are currently on fedora 2513:43
ykarelatomic13:43
ykarelare you using fedora 2513:44
ykarelare you using fedora 2613:44
mnaserykarel i did, then tried 26 and had that issue.  the issue doesnt happen in 25 because it doesnt have docker 1.1313:44
mnaserthe fix is technically harmless because it just redoes the behaviour that happens in 2513:44
mnaserso it would work ok in 25 or 2613:44
ykarelmnaser, I think we can skip this change until we have fedora atomic 26, Also i think setting iptables this way is not persistent across reboot13:59
mnaserykarel that means no newer k8s versions :( .. and it has no harm, also that's fine, because that file runs as a unit service before docker so whenever the docker service is started, it'll run it14:00
*** tonanhngo has joined #openstack-containers14:00
*** tonanhngo has quit IRC14:02
ykarelis this required for new k8s version as well on fedora atomic 2514:02
mnaserykarel i believe new k8s system packages are shipping with f26 but i doubt that new k8s will be shipping in atomic (there was talks of getting docker 1.13 in but that was scratched because they didnt want to introduce changes to a major release)14:03
mnaseri mean the fix itself is harmless and its noop in the scenarios where its not needed14:03
*** hongbin has joined #openstack-containers14:06
*** yamamoto has joined #openstack-containers14:06
ykarelmnaser, Ok then we can take this as for baremetal we would be using f26, and for atomic we can confirm from atomic team what's the plan for docker and k8s version14:09
mnaserykarel yup, for a fact f26 has docker 1.13 (i have it running right now) .. so my plan is to get it fully working/supported while maintaining backwards compatiblity with f2514:09
mnaserso folks can decide to run what they want14:09
ykarelmnaser, Ok14:10
*** yamamoto has quit IRC14:11
*** chhavi has joined #openstack-containers14:18
*** ykarel has quit IRC14:20
*** mjura has quit IRC14:21
*** marst has joined #openstack-containers14:28
*** zenoway has quit IRC14:32
*** zenoway has joined #openstack-containers14:33
*** zenoway has quit IRC14:42
*** irenab has joined #openstack-containers14:42
*** yamamoto has joined #openstack-containers14:47
*** yamamoto has quit IRC14:47
*** slagle has quit IRC14:48
*** slagle has joined #openstack-containers14:50
*** zenoway has joined #openstack-containers14:52
*** armaan has quit IRC14:53
*** trinaths1 has joined #openstack-containers14:58
*** zenoway has quit IRC14:58
*** zenoway has joined #openstack-containers14:58
*** zenoway has quit IRC15:02
*** zenoway has joined #openstack-containers15:02
*** Oku_OS is now known as Oku_OS-away15:05
*** zenoway has quit IRC15:07
*** tonanhngo has joined #openstack-containers15:16
*** tonanhngo has quit IRC15:18
*** hishh has quit IRC15:22
*** yamamoto has joined #openstack-containers15:48
*** penick has joined #openstack-containers15:49
*** yamamoto has quit IRC15:53
*** penick has quit IRC15:53
*** mtanino has quit IRC16:01
*** jchhatbar has joined #openstack-containers16:06
openstackgerritSungjin Kang proposed openstack/magnum-ui master: Fixed typo  https://review.openstack.org/49057916:07
*** rcernin has quit IRC16:12
*** dave-mccowan has joined #openstack-containers16:14
*** jmlowe has quit IRC16:20
*** vishwana_ has joined #openstack-containers16:20
*** vishwanathj has quit IRC16:24
*** dsariel has quit IRC16:34
*** jmlowe has joined #openstack-containers16:41
*** gaowen has quit IRC16:43
*** gaowen has joined #openstack-containers16:43
*** pcaruana has quit IRC16:54
*** mgoddard has quit IRC17:03
*** rcernin has joined #openstack-containers17:03
*** trinaths1 has left #openstack-containers17:04
*** EricGonc_ has quit IRC17:11
*** EricGonczer_ has joined #openstack-containers17:11
*** chhavi has quit IRC17:29
*** chhavi has joined #openstack-containers17:34
*** chhavi has quit IRC17:38
*** jchhatbar has quit IRC17:50
*** danil has quit IRC18:04
*** chhavi has joined #openstack-containers18:07
*** chhavi has quit IRC18:12
*** tonanhngo has joined #openstack-containers18:12
*** salmankhan has quit IRC18:31
*** chhavi has joined #openstack-containers18:34
*** livelace has quit IRC18:49
*** masber has joined #openstack-containers18:57
*** masuberu has quit IRC18:59
*** armaan has joined #openstack-containers19:09
*** EricGonczer_ has quit IRC19:21
*** EricGonczer_ has joined #openstack-containers19:22
*** harlowja has quit IRC19:57
*** salmankhan has joined #openstack-containers20:05
*** salmankhan has quit IRC20:10
*** chhavi has quit IRC20:20
*** armaan has quit IRC20:45
*** armaan has joined #openstack-containers20:45
*** rtjure has joined #openstack-containers21:05
*** harlowja has joined #openstack-containers21:15
*** jmlowe has quit IRC21:18
*** yamamoto has joined #openstack-containers21:21
*** yamamoto has quit IRC21:26
*** shiyer has joined #openstack-containers21:32
*** ricolin_ has joined #openstack-containers21:45
*** ricolin has quit IRC21:47
*** yamamoto has joined #openstack-containers21:58
*** tonanhngo has quit IRC22:11
*** tonanhngo has joined #openstack-containers22:12
*** dave-mccowan has quit IRC22:51
*** EricGonczer_ has quit IRC22:54
*** rcernin has quit IRC22:58
*** tonanhngo has quit IRC23:24
*** EricGonczer_ has joined #openstack-containers23:24
*** jmlowe has joined #openstack-containers23:27
*** tonanhngo has joined #openstack-containers23:30
*** hongbin has quit IRC23:34
*** tonanhngo has quit IRC23:35
*** marst has quit IRC23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!