Thursday, 2018-06-07

*** dardelean_ has quit IRC00:05
*** imdigitaljim has joined #openstack-containers00:16
*** gsimondon has joined #openstack-containers00:17
*** fragatina has quit IRC00:21
*** gsimondon has quit IRC00:22
*** threestrands has quit IRC00:25
*** jmlowe has quit IRC00:26
*** jmlowe has joined #openstack-containers00:27
*** threestrands has joined #openstack-containers00:28
*** Nel1x has joined #openstack-containers01:02
*** dardelean_ has joined #openstack-containers01:05
*** dardelean_ has quit IRC01:09
*** gsimondon has joined #openstack-containers01:11
*** gsimondon has quit IRC01:16
*** hongbin has joined #openstack-containers01:16
*** mdnadeem has joined #openstack-containers01:20
*** threestrands has quit IRC01:35
*** threestrands has joined #openstack-containers01:35
*** threestrands has quit IRC01:36
*** threestrands has joined #openstack-containers01:37
*** threestrands has quit IRC01:38
*** threestrands has joined #openstack-containers01:38
*** threestrands has quit IRC01:38
*** threestrands has joined #openstack-containers01:38
*** threestrands has quit IRC01:39
*** threestrands has joined #openstack-containers01:40
*** threestrands has quit IRC01:40
*** threestrands has joined #openstack-containers01:40
*** threestrands has quit IRC01:41
*** zhubingbing has joined #openstack-containers01:48
*** zhubingbing has quit IRC01:53
*** gsimondon has joined #openstack-containers02:06
*** gsimondon has quit IRC02:10
*** threestrands has joined #openstack-containers02:11
*** threestrands has quit IRC02:11
*** threestrands has joined #openstack-containers02:11
*** georgem1 has joined #openstack-containers02:22
*** gsimondon has joined #openstack-containers03:00
*** gsimondon has quit IRC03:05
*** threestrands has quit IRC03:13
*** fragatina has joined #openstack-containers03:18
*** fragatina has quit IRC03:24
*** Nel1x has quit IRC03:26
*** ramishra has joined #openstack-containers03:32
*** lpetrut has joined #openstack-containers03:42
*** zhubingbing has joined #openstack-containers03:50
*** fragatina has joined #openstack-containers03:53
*** fragatina has quit IRC03:54
*** gsimondon has joined #openstack-containers03:54
*** zhubingbing has quit IRC03:54
*** ricolin has joined #openstack-containers03:57
flwangricolin: ping03:58
flwangricolin: i have some questions about heat in magnum03:58
*** gsimondon has quit IRC03:58
*** fragatina has joined #openstack-containers04:00
*** yamamoto has joined #openstack-containers04:00
*** ricolin has quit IRC04:01
*** fragatina has quit IRC04:01
*** fragatina has joined #openstack-containers04:02
*** udesale has joined #openstack-containers04:02
*** ricolin has joined #openstack-containers04:07
*** lpetrut has quit IRC04:14
*** Nel1x has joined #openstack-containers04:18
*** fragatina has quit IRC04:24
*** Nel1x has quit IRC04:25
*** hongbin has quit IRC04:26
*** georgem1 has quit IRC04:37
*** masayukig_ has quit IRC04:43
*** gsimondon has joined #openstack-containers04:48
*** gsimondon has quit IRC04:53
*** chhagarw has joined #openstack-containers05:06
*** ykarel|training has joined #openstack-containers05:17
*** iranzo has joined #openstack-containers05:40
*** iranzo has joined #openstack-containers05:40
*** mjura has joined #openstack-containers05:42
*** fragatina has joined #openstack-containers05:43
*** zhubingbing has joined #openstack-containers05:51
*** pcichy has quit IRC05:54
*** zhubingbing has quit IRC05:56
*** pcichy has joined #openstack-containers06:00
*** ykarel|training has quit IRC06:02
*** gsimondon has joined #openstack-containers06:04
*** ykarel|training has joined #openstack-containers06:05
*** gsimondo1 has joined #openstack-containers06:13
*** ykarel|training has quit IRC06:14
*** gsimondon has quit IRC06:15
*** lpetrut has joined #openstack-containers06:23
*** ricolin_ has joined #openstack-containers06:33
*** ricolin has quit IRC06:35
*** pcaruana has joined #openstack-containers06:35
openstackgerritzhubingbing proposed openstack/magnum stable/pike: Support local registry  https://review.openstack.org/57310006:46
*** AlexeyAbashkin has joined #openstack-containers07:04
*** armaan has joined #openstack-containers07:10
*** dardelean_ has joined #openstack-containers07:15
*** ktibi has joined #openstack-containers07:26
*** dardelean_ has quit IRC07:32
openstackgerritMerged openstack/magnum master: Strip signed certificate  https://review.openstack.org/57275507:32
*** Alexey_Abashkin has joined #openstack-containers07:34
*** AlexeyAbashkin has quit IRC07:35
*** Alexey_Abashkin is now known as AlexeyAbashkin07:35
*** AlexeyAbashkin has quit IRC07:46
*** mgoddard has joined #openstack-containers07:52
*** AlexeyAbashkin has joined #openstack-containers07:56
*** mago_ has joined #openstack-containers07:59
*** sfilatov has joined #openstack-containers08:08
*** dardelean_ has joined #openstack-containers08:18
*** AlexeyAbashkin has quit IRC08:23
*** ykarel|training has joined #openstack-containers08:26
*** AlexeyAbashkin has joined #openstack-containers08:26
*** AlexeyAbashkin has quit IRC08:33
*** ricolin_ has quit IRC08:34
*** ricolin has joined #openstack-containers08:34
*** ricolin_ has joined #openstack-containers08:35
*** AlexeyAbashkin has joined #openstack-containers08:38
*** ricolin has quit IRC08:39
*** flwang1 has joined #openstack-containers08:40
flwang1strigazi: ping i have questions about multi masters08:40
strigaziflwang1: tell me08:40
flwang1we're test mutil masters with lb(octavia)08:41
flwang1now in devstack, will be soon in our CI08:42
strigaziflwang1: 1st thing you may want to do, remove the etcd LB08:42
flwang1in devstack, IIRC, you told me, the multi masters will be created one by one, right?08:42
flwang1anyway to remove the etcd lb? do we have a config?08:43
strigaziflwang1: no, they are created in parallel08:43
flwang1ok08:43
strigaziflwang1: I don't, we need to change the templates08:43
flwang1I don't think user really need a lb for etcd08:44
flwang1can you just drop it?08:44
flwang1can we just drop it?08:44
strigaziflwang1: you think correct and I agree08:44
flwang1it's an over-engineering, TBH08:44
flwang1sorry for the original author :D08:45
*** AlexeyAbashkin has quit IRC08:45
strigaziit is, I want to do the reverse proxy on minions for some time now08:45
flwang1back to my issue, is the etcd lb the root cause?08:45
strigaziroot cause for what?08:46
flwang1ok, in my env, all 3 api servers are in ready, but the stack still failed08:46
strigaziwhich error?08:47
flwang1wait a sec08:48
flwang1http://paste.openstack.org/show/722860/08:50
strigaziflwang1: something in the calico script, I don't think it is related to the templates08:51
flwang1actually the errors are different each time, but all of them are some like this08:51
flwang1last time it's core dns08:51
strigaziflwang1: a cmd gives non-zero exit code08:51
flwang1this time it's calico08:51
flwang1but it's not a calico issue08:51
strigaziok08:51
flwang1all calico node work fine08:51
flwang1on one master, i can see http://paste.openstack.org/show/722859/ from the cloud-init-output.log08:52
strigazimaybe the fact that we try create the resources form all masters? if you change to apply -f instead of create -f ?08:52
flwang1but i'm not sure if it's related08:52
strigazisee the logs of heat-container-agent in the master nodes08:52
flwang1i did, nothing wrong08:52
flwang1all finished correctly based on my knowledge08:53
strigazionly by checking the heat agent logs we can find out08:53
strigazithe error should be there08:53
flwang1ok, wait a sec08:53
flwang1when we see an error like this, how can we figure out which master failed?08:54
flwang1or we have to check all of them?08:54
strigaziit says resources[2]08:54
strigaziso master 208:55
flwang1cool08:56
*** janki has joined #openstack-containers08:57
flwang1http://paste.openstack.org/show/722863/08:59
flwang1Jun 07 03:32:31 k8sclsuter-fcflbho3xoul-master-2.novalocal runc[2043]: [2018-06-07 03:32:31,075] (heat-config) [DEBUG] Error from server (AlreadyExists): error when creating "/srv/magnum/kubernetes/manifests/calico-deploy.yaml": clusterroles.rbac.authorization.k8s.io "calico-kube-controllers" already exists09:00
flwang1Jun 07 03:32:31 k8sclsuter-fcflbho3xoul-master-2.novalocal runc[2043]: [2018-06-07 03:32:31,075] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-script/78db6430-cef1-4afc-9947-788d90197614. [1]09:00
strigaziAs I guessed :)09:03
flwang1but09:03
flwang1we're using apply -f09:03
flwang1so it could be a race condition issue in k8s?09:03
strigaziI guessed the problem correctly not the solution :)09:03
flwang1so currently, heat will create all master nodes same time?09:04
flwang1and then create all nodes same time?09:04
strigaziCan you try in a working cluster to create with apply -f from different clients the same rolebinding?09:04
strigaziflwang1: yes09:04
flwang1let me try09:05
*** mago_ has quit IRC09:10
*** serlex has joined #openstack-containers09:16
*** mago_ has joined #openstack-containers09:17
*** armaan has quit IRC09:27
flwang1tested09:28
strigaziand?09:28
flwang1on a working cluster, i can rerun the calico script09:28
flwang1and nothing wrong09:28
*** armaan has joined #openstack-containers09:28
*** mago_ has quit IRC09:28
flwang1are you using multi master in cern?09:28
flwang1did you ever see this kind of issue?09:29
flwang1it's really like a race condition issue on kube-api server side09:29
flwang1let me ask another question, will heat-container-agent follow the order like we defined in heat template?09:30
*** sfilatov has quit IRC09:30
*** sfilatov has joined #openstack-containers09:31
*** AlexeyAbashkin has joined #openstack-containers09:31
*** sfilatov has quit IRC09:31
*** sfilatov has joined #openstack-containers09:32
flwang1strigazi: and pls see this error " Error from server (AlreadyExists): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:kube-apiserver-to-kubelet" already exists "09:34
flwang1same issue in https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/kube-apiserver-to-kubelet-role.sh#L1109:35
*** mago_ has joined #openstack-containers09:41
strigaziwe need to rethink multi master then, at cern we don't have multimaster09:42
flwang1OMG09:42
*** AlexeyAbashkin has quit IRC09:50
*** AlexeyAbashkin has joined #openstack-containers10:02
*** pcichy has quit IRC10:06
*** AlexeyAbashkin has quit IRC10:13
*** Alexey_Abashkin has joined #openstack-containers10:13
*** Alexey_Abashkin is now known as AlexeyAbashkin10:16
*** AlexeyAbashkin has quit IRC10:23
*** AlexeyAbashkin has joined #openstack-containers10:25
*** dardelea_ has joined #openstack-containers10:26
*** ykarel|training has quit IRC10:30
*** dardelean_ has quit IRC10:30
*** Alexey_Abashkin has joined #openstack-containers10:38
*** AlexeyAbashkin has quit IRC10:39
*** Alexey_Abashkin is now known as AlexeyAbashkin10:39
*** mdnadeem has quit IRC10:44
*** sfilatov_ has joined #openstack-containers10:47
*** sfilatov has quit IRC10:47
*** sfilatov_ has quit IRC10:59
*** sfilatov has joined #openstack-containers10:59
*** dardelea_ has quit IRC11:00
*** AlexeyAbashkin has quit IRC11:01
*** dardelean_ has joined #openstack-containers11:03
*** sfilatov has quit IRC11:04
*** udesale has quit IRC11:07
*** yamamoto has quit IRC11:14
*** sfilatov has joined #openstack-containers11:20
*** ykarel|training has joined #openstack-containers11:38
*** AlexeyAbashkin has joined #openstack-containers11:51
*** yamamoto has joined #openstack-containers12:10
*** armaan has quit IRC12:12
*** armaan has joined #openstack-containers12:13
*** dave-mccowan has joined #openstack-containers12:14
*** sfilatov has quit IRC12:15
*** dave-mcc_ has joined #openstack-containers12:15
*** armaan has quit IRC12:17
*** vijaykc4 has joined #openstack-containers12:18
*** dave-mccowan has quit IRC12:18
*** yamamoto has quit IRC12:21
*** yamamoto has joined #openstack-containers12:49
*** rcernin has quit IRC12:56
*** zhubingb_ has joined #openstack-containers12:57
*** georgem1 has joined #openstack-containers12:57
*** b_bezak has joined #openstack-containers12:57
*** ricolin_ has quit IRC13:01
*** zhubingb_ has quit IRC13:01
*** ricolin has joined #openstack-containers13:03
*** AlexeyAbashkin has quit IRC13:08
*** Alexey_Abashkin has joined #openstack-containers13:08
*** udesale has joined #openstack-containers13:09
*** Alexey_Abashkin is now known as AlexeyAbashkin13:10
*** vijaykc4 has quit IRC13:10
*** vijaykc4 has joined #openstack-containers13:13
*** janki has quit IRC13:13
*** ykarel|training has quit IRC13:27
*** openstackgerrit has quit IRC13:34
*** ricolin has quit IRC13:39
*** dave-mcc_ has quit IRC13:41
*** dave-mccowan has joined #openstack-containers13:42
*** AlexeyAbashkin has quit IRC13:43
*** dave-mccowan has quit IRC13:46
*** dave-mccowan has joined #openstack-containers13:48
*** b_bezak has quit IRC13:49
*** b_bezak has joined #openstack-containers13:50
*** dave-mcc_ has joined #openstack-containers13:50
*** AlexeyAbashkin has joined #openstack-containers13:50
*** armaan has joined #openstack-containers13:52
*** dave-mccowan has quit IRC13:53
*** b_bezak has quit IRC13:54
*** fragatina has quit IRC13:58
*** fragatina has joined #openstack-containers13:58
*** Alexey_Abashkin has joined #openstack-containers14:02
*** AlexeyAbashkin has quit IRC14:02
*** Alexey_Abashkin is now known as AlexeyAbashkin14:02
*** udesale has quit IRC14:03
*** udesale has joined #openstack-containers14:04
*** mjura has quit IRC14:16
*** fragatina has quit IRC14:26
*** Alexey_Abashkin has joined #openstack-containers14:38
*** AlexeyAbashkin has quit IRC14:41
*** Alexey_Abashkin is now known as AlexeyAbashkin14:41
*** zhubingb_ has joined #openstack-containers14:41
*** openstackgerrit has joined #openstack-containers14:44
*** iranzo has quit IRC14:45
*** chhavi__ has joined #openstack-containers14:48
*** ricolin has joined #openstack-containers14:48
*** chhagarw has quit IRC14:49
*** gsimondo1 has quit IRC14:54
*** markguz_ has joined #openstack-containers15:00
*** Alexey_Abashkin has joined #openstack-containers15:20
*** AlexeyAbashkin has quit IRC15:22
*** Alexey_Abashkin is now known as AlexeyAbashkin15:22
*** AlexeyAbashkin has quit IRC15:22
*** ramishra has quit IRC15:22
*** AlexeyAbashkin has joined #openstack-containers15:23
-openstackstatus- NOTICE: Zuul update for Ansible 2.5 in progress. Scheduler crashed as unexpected side effect of pip upgrade. Will be back and running shortly.15:30
*** ktibi has quit IRC15:30
*** Alexey_Abashkin has joined #openstack-containers15:37
*** AlexeyAbashkin has quit IRC15:37
*** Alexey_Abashkin is now known as AlexeyAbashkin15:37
*** AlexeyAbashkin has quit IRC15:53
*** janki has joined #openstack-containers15:57
*** fragatina has joined #openstack-containers15:58
-openstackstatus- NOTICE: The zuul upgrade to ansible 2.5 is complete and zuul is running again. Changes uploaded or approved between 15:25 and 15:45 will need to be rechecked. Please report any problems in #openstack-infra16:09
*** janki has quit IRC16:11
*** gsimondon has joined #openstack-containers16:13
*** yamamoto has quit IRC16:14
*** serlex has quit IRC16:17
*** georgem11 has joined #openstack-containers16:20
*** yamamoto has joined #openstack-containers16:21
*** georgem1 has quit IRC16:23
*** yamamoto has quit IRC16:26
*** dardelean_ has quit IRC16:33
*** dardelean_ has joined #openstack-containers16:34
*** dardelean_ has quit IRC16:38
*** fragatina has quit IRC16:41
*** zhubingb_ has quit IRC16:42
*** vijaykc4 has quit IRC16:43
*** vijaykc4 has joined #openstack-containers16:44
*** fragatina has joined #openstack-containers16:49
*** lpetrut has quit IRC16:49
*** fragatina has quit IRC16:59
*** gsimondon has quit IRC17:03
*** lpetrut has joined #openstack-containers17:08
*** fragatina has joined #openstack-containers17:18
*** lpetrut has quit IRC17:22
*** yamamoto has joined #openstack-containers17:23
*** udesale has quit IRC17:28
*** yamamoto has quit IRC17:31
*** openstackgerrit has quit IRC17:34
*** lpetrut has joined #openstack-containers17:36
*** ricolin has quit IRC17:44
*** chhavi__ has quit IRC17:45
*** iranzo has joined #openstack-containers17:50
*** armaan_ has joined #openstack-containers17:58
*** armaan has quit IRC17:58
*** lpetrut has quit IRC18:31
*** gsimondon has joined #openstack-containers18:32
*** lpetrut has joined #openstack-containers18:36
*** lpetrut has quit IRC18:40
*** gsimondon has quit IRC18:40
*** lpetrut has joined #openstack-containers18:40
*** zhubingb_ has joined #openstack-containers18:44
*** spotz has quit IRC18:45
*** spotz has joined #openstack-containers18:46
*** armaan_ has quit IRC18:48
*** zhubingb_ has quit IRC18:48
*** jslater has joined #openstack-containers18:49
*** gsimondon has joined #openstack-containers18:55
*** lpetrut has quit IRC19:02
*** gsimondon has quit IRC19:02
*** dardelean_ has joined #openstack-containers19:15
*** mgoddard has quit IRC19:23
*** lpetrut has joined #openstack-containers19:27
*** vijaykc4 has quit IRC19:41
*** flwang1 has quit IRC19:50
*** iranzo has quit IRC20:04
*** lpetrut has quit IRC20:15
*** gsimondon has joined #openstack-containers20:20
*** vijaykc4 has joined #openstack-containers20:26
*** vijaykc4 has quit IRC20:35
*** pcaruana has quit IRC20:39
*** vijaykc4 has joined #openstack-containers20:45
*** zhubingb_ has joined #openstack-containers20:45
*** gsimondo1 has joined #openstack-containers20:46
*** armaan has joined #openstack-containers20:47
*** gsimondon has quit IRC20:48
*** zhubingb_ has quit IRC20:49
*** dardelean_ has quit IRC21:06
*** dardelean_ has joined #openstack-containers21:06
*** flwang has quit IRC21:07
*** dardelean_ has quit IRC21:11
*** gsimondo1 has quit IRC21:11
*** flwang has joined #openstack-containers21:19
*** armaan has quit IRC21:25
*** vijaykc4 has quit IRC21:27
*** flwang1 has joined #openstack-containers21:52
*** janki has joined #openstack-containers21:55
*** gsimondon has joined #openstack-containers22:18
*** gsimondon has quit IRC22:24
*** rcernin has joined #openstack-containers22:24
*** vijaykc4 has joined #openstack-containers22:25
*** vijaykc4 has quit IRC22:30
*** janki has quit IRC22:52
*** dardelean_ has joined #openstack-containers23:04
*** dardelean_ has quit IRC23:06
*** dardelean_ has joined #openstack-containers23:06
*** dardelean_ has quit IRC23:26
*** zhubingb_ has joined #openstack-containers23:48
*** markguz_ has quit IRC23:49
*** zhubingb_ has quit IRC23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!