Thursday, 2017-07-27

*** mtanino has quit IRC00:07
*** dixiaoli has joined #openstack-containers00:51
*** mdnadeem has joined #openstack-containers01:11
*** ricolin has joined #openstack-containers01:14
*** harlowja has quit IRC01:19
*** yamamoto_ has quit IRC01:31
*** yamamoto has joined #openstack-containers01:35
*** tonanhngo has joined #openstack-containers01:48
*** mtanino_ has joined #openstack-containers02:09
*** shu-mutou-AWAY is now known as shu-mutou02:22
*** dixiaoli has quit IRC03:09
*** harlowja has joined #openstack-containers03:31
*** akhil_jain has joined #openstack-containers03:33
*** daidv has joined #openstack-containers03:42
*** mtanino_ has quit IRC03:53
*** daidv has quit IRC04:19
*** akhil has joined #openstack-containers04:27
*** akhil_jain has quit IRC04:30
*** juggler has joined #openstack-containers04:31
*** chhavi has joined #openstack-containers04:31
*** tonanhngo has quit IRC04:44
openstackgerritMerged openstack/magnum-ui master: Imported Translations from Zanata  https://review.openstack.org/48737004:45
*** ykarel has joined #openstack-containers04:46
*** harlowja has quit IRC04:52
*** sudhirag098 has joined #openstack-containers05:10
*** trinaths has joined #openstack-containers05:25
*** trinaths has left #openstack-containers05:25
*** trinaths has joined #openstack-containers05:29
*** vishwana_ has quit IRC05:40
*** vishwanathj has joined #openstack-containers05:41
*** hongbin has joined #openstack-containers05:47
*** hishh has joined #openstack-containers05:49
*** janki has joined #openstack-containers05:50
*** trinaths has quit IRC05:51
*** trinaths has joined #openstack-containers05:56
*** yuanying has joined #openstack-containers05:57
*** mjura has joined #openstack-containers06:02
strigazihi mkrai, can you approve the osc patches? we need to cut a release later today. worst case tomorrow06:04
*** ramishra has quit IRC06:06
*** ramishra has joined #openstack-containers06:08
*** sudhirag098 has quit IRC06:09
*** armaan has joined #openstack-containers06:17
*** dsariel has joined #openstack-containers06:18
*** armaan has quit IRC06:21
magicboiz<strigazi>: morning06:30
openstackgerrityatin proposed openstack/magnum master: Copy service configurations also  https://review.openstack.org/48687106:32
*** kzaitsev_ws has joined #openstack-containers06:33
*** tonanhngo has joined #openstack-containers06:34
magicboiz<strigazi>: "journalctl -u etcd" -> https://pastebin.com/uzVywwaC06:36
*** tonanhngo has quit IRC06:39
*** armaan has joined #openstack-containers06:45
*** trinaths has quit IRC06:51
*** trinaths1 has joined #openstack-containers06:51
*** kzaitsev_ws has quit IRC06:56
strigazimagicboiz multimaster?07:03
*** hongbin has quit IRC07:04
magicboiz<strigazi>: yes. 3 master nodes, and master_lb_enabled enabled07:06
magicboiz<strigazi>: "etcdctl ls  / --recursive" -> empty, it looks like etcd db is empty?07:07
strigazimagicboiz looks like the nodes can't find etcd07:07
strigazimagicboiz go to a master node and a worker node07:08
strigaziand show me07:08
strigaziwait for it07:08
magicboiz<strigazi>: https://pastebin.com/zta0P2g3 -> "level=debug msg="could not find network 6887a0c1bb2d9dc6668c33e8b03cd6d09c6474129276f0081330b89b5e4bbe23: client: etcd cluster is unavailable or misconfigured""07:08
strigazicat /etc/sysconfig/heat-params | grep ETCD_SERVER_IP07:10
strigazior grep ETCD_SERVER_IP /etc/sysconfig/heat-params :)07:10
magicboiz<strigazi>: LBaaS listener on port 2379 created as TYPE=HTTP. Why not TCP type?07:13
strigazimagicboiz There is high probability that we need a similar patch like kube for swarm multimaster07:13
*** rcernin has joined #openstack-containers07:13
* strigazi is going to a meeting07:14
magicboizETCD_SERVER_IP is well defined in /etc/sysconfig/heat-params, but I think that HAPROXY/Lbaas is expecting HTTP on port 2379, and it might be needed TCP instead07:14
strigazimagicboiz yeap, I think that's it07:18
magicboizhttps://github.com/openstack/magnum/blob/stable/ocata/magnum/drivers/swarm_fedora_atomic_v1/templates/cluster.yaml#L27807:18
magicboiz??07:18
strigazis/HTTP/{get_param: loadbalancing_protocol}/07:20
openstackgerritSpyros Trigazis (strigazi) proposed openstack/magnum master: Don't poll heat if no stack exists  https://review.openstack.org/48028607:29
openstackgerritLingyong Xu proposed openstack/magnum master: Use HostAddressOpt for opts that accept IP and hostnames  https://review.openstack.org/48772407:32
*** shu-mutou has quit IRC07:38
*** shu-mutou has joined #openstack-containers07:38
*** vijaykc4 has joined #openstack-containers08:01
openstackgerritMerged openstack/magnum stable/ocata: [k8s-fedora-atomic] fix multimaster cluster  https://review.openstack.org/48742508:09
*** tonanhngo has joined #openstack-containers08:15
openstackgerritSpyros Trigazis (strigazi) proposed openstack/magnum master: Extract kubernetes baremetal ports  https://review.openstack.org/46471008:18
*** tonanhngo has quit IRC08:19
*** mgoddard has joined #openstack-containers08:22
magicboiz<strigazi>: now loadbalancer listener is deployed with TCP type. But still don't work :( https://pastebin.com/W6nWx5jq08:23
strigazimagicboiz we need to add the virtual ip in the certs08:24
strigazimagicboiz like this https://review.openstack.org/#/c/487425/1/magnum/drivers/common/templates/kubernetes/fragments/make-cert.sh08:24
strigazimagicboiz here https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/swarm/fragments/make-cert.py#L7308:25
magicboiz<strigazi>: many things are being discovery these days... do you want me to open some bugs to track them?08:29
strigazimagicboiz yes please08:30
magicboiz<strigazi>: quick summary:08:31
magicboiz1. swarm_fedora_atomic_v1/templates/cluster.yaml -> incorrect etcd listener type08:31
magicboiz2. swarm/fragments/network-config-service.sh -> wrong reference to  FLANNEL_ETCD_KEY and FLANNEL_ETCD08:32
magicboiz3. k8s -> multimaster + LB does not expose service external IP08:34
magicboiz<strigazi>: ok for you?08:34
strigaziI don't get 3. which on is it?08:35
strigaziI think 3 works, it was a configuration on magnum.conf08:35
strigaziit is default to false on purpose08:35
strigazimagicboiz it was that the kubernetes services didn't get externalIP right?08:36
strigazi1 and 2 sound good08:37
magicboiz<strigazi>: k8s does not publish external-ip for services when deploying k8s cluster with LB enabled. <DimGR> gave a workaround for this08:38
DimGRthe workaround was a fix to get lbaas working08:39
strigazinot sure what was the fix08:39
DimGRto delete cm pod etc08:39
strigazimagicboiz you needed to delete the pod too?08:39
strigazimagicboiz did you do any other change to get lbaas to work?08:40
magicboiz<strigazi>: lbaas is not working yet.08:41
strigazimagicboiz why? what is the error?08:41
magicboizthe error was related with the service exposed-IP08:42
strigazimagicboiz, let's make things clearer. You are talking about the load balancer for kubernetes services or the load balancer for the kuberneets API?08:43
strigazimagicboiz, the fix for the kubernetes API is merged in ocata and will be released next week08:48
magicboizwhen you do "kubectl get services" I get always something like "nginx <ip> <pending>". That "<pending>" is correct?08:48
strigazimagicboiz right. so it is not for the kubernetes API08:48
strigazimagicboiz Now you run the controller manager as a pod or from the binary?08:49
strigazimagicboiz we need to see its log08:49
magicboiz<strigazi>: let me deploy a new k8s cluster. But controller manager runs as magnum deploys it.08:50
strigazimagicboiz ok, it should work08:51
magicboiz<strigazi>: All I test is to create pod and service for nginx as stated here: https://docs.openstack.org/magnum/latest/user/index.html08:51
strigazimagicboiz that is correct08:51
*** kzaitsev_ws has joined #openstack-containers08:51
strigaziJust have the config option enabled08:51
magicboiz<strigazi>: exactly, that's why I think there's something wrong somewhere :)08:52
strigazito get effect you need to restart the api and conductor08:52
magicboiz<strigazi>: which config option?08:52
strigaziIf you did already yesterday it is fine08:52
strigazicluster_user_trust08:52
strigazihttps://github.com/openstack/magnum/blob/master/magnum/conf/trust.py#L2108:53
magicboiz<strigazi>: you mean cluster_user_trust=True? yes, it's enabled from yesterday.08:53
strigazimagicboiz ok08:53
*** vijaykc4 has quit IRC09:08
*** salmankhan has joined #openstack-containers09:10
*** yolanda has quit IRC09:11
*** yolanda has joined #openstack-containers09:12
magicboiz<strigazi>: new info. Patch https://review.openstack.org/#/c/487425/ + "cluster_user_trust=True" has some new side effects:09:14
magicboizk8s WITHOUT TLS is never deployed by Heat (Cloud provider could not be initialized: could not init cloud provider..... /v3/auth/tokens: x509: failed to load system roots and no roots provided)09:14
magicboizk8s WITH TLS, is deployed but fails because master nodes show this error: GET /v1.24/images/gcr.io/google_containers/hyperkube:v1.5.3/json returned error: No such image: gcr.io/google_containers/hyperkube:v1.5.309:15
strigazimagicboiz the second error is not related to lbaas. Can you pul manuall that image?09:17
strigazimagicboiz the second error is not related to lbaas. Can you pul manually that image?09:17
*** vijaykc4 has joined #openstack-containers09:17
strigaziThe first error is weird. Can you show me /var/lib/cloud/instance/user-data.txt ?09:18
*** shu-mutou is now known as shu-mutou-AWAY09:20
openstackgerritRicardo Rocha proposed openstack/magnum master: Allow labels on cluster create  https://review.openstack.org/47334609:31
strigazimagicboiz without tls it does n't work we need to have certs for kubernetes.09:31
strigazimagicboiz with tls can you try to pull manually the images from gcr.io ?09:31
openstackgerritRicardo Rocha proposed openstack/magnum master: Allow labels on cluster create  https://review.openstack.org/47334609:31
magicboiz<strigazi>: with TLS enabled cluster, I'm also getting " controllermanager.go:280 .....  x509: failed to load system roots and no roots provided"09:32
strigazivery strange09:33
strigazimagicboiz can you show me /var/log/cloud-init-output.log ?09:36
*** juggler has quit IRC09:47
*** daidv has joined #openstack-containers09:49
*** masber has quit IRC09:55
openstackgerrityatin proposed openstack/magnum master: Copy service configurations also  https://review.openstack.org/48687109:57
strigaziykarel ^^ ++09:58
*** yamamoto has quit IRC10:00
openstackgerritRicardo Rocha proposed openstack/python-magnumclient master: Add --labels for cluster-create  https://review.openstack.org/47334310:16
*** tonanhngo has joined #openstack-containers10:21
ykarelmagicboiz, where did you see this error: k8s WITHOUT TLS is never deployed by Heat (Cloud provider could not be initialized: could not init cloud provider..... /v3/auth/tokens: x509: failed to load system roots and no roots provided)10:25
*** tonanhngo has quit IRC10:26
*** vijaykc4 has quit IRC10:27
*** yamamoto has joined #openstack-containers10:31
*** vijaykc4 has joined #openstack-containers10:33
*** vijaykc4 has quit IRC11:00
magicboiz<ykarel>: in my environment11:04
magicboiz<ykarel>: I do cluster-create but heat never progress beyond master nodes creation....11:05
magicboiz<ykarel>: with TLS, heat goes ahead and create minions11:05
*** dsariel_ has joined #openstack-containers11:09
*** vijaykc4 has joined #openstack-containers11:15
openstackgerritOpenStack Proposal Bot proposed openstack/magnum-ui master: Imported Translations from Zanata  https://review.openstack.org/48781211:19
strigaziykarel magicboiz it is a problem with the certs mounted in the controller manager11:19
strigaziIf you add this: http://paste.openstack.org/show/616704/ it works11:20
strigaziykarel magicboiz or if we remove the the mounts for /etc/ssl/certs from the controller manager11:22
*** vishwanathj has quit IRC11:26
*** vishwanathj has joined #openstack-containers11:26
*** armaan has quit IRC11:53
*** armaan has joined #openstack-containers11:53
strigazi.n11:54
*** vijaykc4 has quit IRC11:54
strigazimkrai, apologies for pinging again. Can you have a look at the OSC client patches? https://review.openstack.org/#/q/status:open+project:openstack/python-magnumclient+branch:master+topic:bp/openstackclient-support11:57
*** armaan has quit IRC12:09
*** armaan has joined #openstack-containers12:09
*** catintheroof has joined #openstack-containers12:11
*** yamamoto has quit IRC12:14
*** mdnadeem has quit IRC12:23
*** dsariel_ has quit IRC12:27
*** dsariel has quit IRC12:28
*** chhavi has quit IRC12:32
openstackgerritMerged openstack/magnum master: Extract kubernetes baremetal ports  https://review.openstack.org/46471012:38
*** dsariel has joined #openstack-containers12:47
*** mgoddard has quit IRC12:48
*** askb has quit IRC12:50
*** mgoddard has joined #openstack-containers12:54
*** yamamoto has joined #openstack-containers13:14
*** yamamoto has quit IRC13:21
*** vijaykc4 has joined #openstack-containers13:23
*** armaan has quit IRC13:37
*** armaan has joined #openstack-containers13:38
strigaziykarel fyi  Spyros Trigazis (strigazi) proposed openstack-infra/project-config master: [magnum] make legacy swarm -nv  https://review.openstack.org/48785813:47
*** mtanino_ has joined #openstack-containers13:49
*** akhil has quit IRC13:53
openstackgerritSpyros Trigazis (strigazi) proposed openstack/magnum master: WIP create kubernetes-dashboard using the heat-agent  https://review.openstack.org/46881613:57
openstackgerritSpyros Trigazis (strigazi) proposed openstack/magnum master: WIP Leverage heat-container-agent for monitoring  https://review.openstack.org/46881613:57
*** ashishb_ has joined #openstack-containers14:01
*** ashishb has quit IRC14:04
mkraistrigazi: Hi14:11
ykarelstrigazi, Ok14:13
strigazimkrai hi14:13
mkraistrigazi: I approved the OSC patches. I hope you have already tested it :)14:14
*** janki has quit IRC14:14
mkraistrigazi: Don't have much time to test14:14
*** armaan has quit IRC14:14
strigazimkrai tested against our prod magnum deployemnt and in devstack14:15
*** armaan has joined #openstack-containers14:15
strigazimkrai yatin tested as well14:15
*** trinaths1 has left #openstack-containers14:15
strigaziykarel https://review.openstack.org/#/c/487361/ let's take this too and I'll add the unit tests and a ci too.14:16
mkraistrigazi: Ok thanks14:17
strigazimkrai the first 4 patches have also 100% test coverage14:18
openstackgerritMerged openstack/python-magnumclient master: OSC 1/4 Add CT create and UT framework  https://review.openstack.org/44616014:18
openstackgerritMerged openstack/python-magnumclient master: OSC 2/4 Add Cluster Create and List  https://review.openstack.org/44617914:18
openstackgerritMerged openstack/python-magnumclient master: OSC 3/4 Add remaining CT commands  https://review.openstack.org/44619814:19
openstackgerritMerged openstack/python-magnumclient master: OSC 4/4 Add remaining cluster commands  https://review.openstack.org/44666014:19
*** armaan has quit IRC14:19
ykarelstrigazi, Ok14:20
openstackgerritMerged openstack/python-magnumclient master: OSC: Add cluster config command  https://review.openstack.org/48736114:25
*** funzo has quit IRC14:27
*** marst has joined #openstack-containers14:33
*** ykarel has quit IRC14:38
*** vishwanathj has quit IRC14:53
*** vishwanathj has joined #openstack-containers14:54
*** kzaitsev_ws has quit IRC14:56
*** vijaykc4 has quit IRC15:01
*** rcernin has quit IRC15:03
*** armaan has joined #openstack-containers15:07
*** mjura has quit IRC15:09
*** vishwanathj has quit IRC15:29
*** vishwanathj has joined #openstack-containers15:29
*** vishwana_ has joined #openstack-containers15:33
*** vishwanathj has quit IRC15:33
*** vishwana_ has quit IRC15:35
*** vishwanathj has joined #openstack-containers15:35
*** armaan has quit IRC15:44
*** armaan has joined #openstack-containers15:45
*** hongbin has joined #openstack-containers15:48
*** armaan has quit IRC15:49
*** vijaykc4 has joined #openstack-containers15:53
openstackgerritSpyros Trigazis (strigazi) proposed openstack/magnum master: swarm-ci: Catch ConnectionError instead of SSLError  https://review.openstack.org/48792015:55
mnaserstrigazi sorry for the afk-ness, i know you wanted to a cut a release, i had to get octavia+lbaas deployed first to be able to test lbaas16:17
mnaserim finishing that up today, so hopefully i'll have time to test lbaas without trusts, and test it with trusts and have a patch :)16:18
*** ramishra has quit IRC16:21
*** funzo has joined #openstack-containers16:21
*** funzo has quit IRC16:21
*** funzo has joined #openstack-containers16:21
*** hishh has quit IRC16:22
*** ricolin has quit IRC16:32
*** mgoddard has quit IRC16:40
openstackgerritMathieu Velten proposed openstack/magnum master: Launch k8s scheduler & controller-manager as system containers  https://review.openstack.org/48794316:55
openstackgerritSpyros Trigazis (strigazi) proposed openstack/magnum master: WIP Leverage heat-container-agent for monitoring  https://review.openstack.org/46881616:59
openstackgerritMathieu Velten proposed openstack/magnum master: Expose kube_version as a label  https://review.openstack.org/48794716:59
*** vijaykc4 has quit IRC17:03
*** harlowja has joined #openstack-containers17:09
*** vijaykc4 has joined #openstack-containers17:12
openstackgerritSpyros Trigazis (strigazi) proposed openstack/magnum master: WIP Leverage heat-container-agent for monitoring  https://review.openstack.org/46881617:13
*** rcernin has joined #openstack-containers17:16
*** mgoddard has joined #openstack-containers17:22
openstackgerritMathieu Velten proposed openstack/magnum master: Expose kube_version as a label  https://review.openstack.org/48794717:23
*** dsariel has quit IRC17:43
*** armaan has joined #openstack-containers17:45
*** armaan has quit IRC17:45
*** penick has joined #openstack-containers17:45
*** armaan has joined #openstack-containers17:46
*** armaan has quit IRC17:47
*** armaan has joined #openstack-containers17:48
*** magicboiz has quit IRC17:51
*** mgoddard has quit IRC18:02
*** jmlowe has quit IRC18:28
*** ricolin has joined #openstack-containers18:31
*** vijaykc4 has quit IRC18:36
*** ricolin has quit IRC18:39
*** armaan has quit IRC18:44
*** armaan has joined #openstack-containers18:45
*** salmankhan has quit IRC18:48
*** jmlowe has joined #openstack-containers18:57
*** armaan has quit IRC19:10
*** armaan has joined #openstack-containers19:11
openstackgerritMathieu Velten proposed openstack/magnum master: Expose kube_version as a label  https://review.openstack.org/48794719:17
openstackgerritOpenStack Proposal Bot proposed openstack/python-magnumclient master: Updated from global requirements  https://review.openstack.org/48812019:21
*** vishwanathj has quit IRC19:34
*** vishwanathj has joined #openstack-containers19:35
openstackgerritOpenStack Proposal Bot proposed openstack/magnum master: Updated from global requirements  https://review.openstack.org/48746919:47
openstackgerritOpenStack Proposal Bot proposed openstack/python-magnumclient master: Updated from global requirements  https://review.openstack.org/48812019:57
*** hishh has joined #openstack-containers20:19
*** jmlowe has quit IRC20:27
*** salmankhan has joined #openstack-containers20:30
*** salmankhan has quit IRC20:35
*** hishh has quit IRC20:39
*** vishwanathj has quit IRC21:01
*** vishwanathj has joined #openstack-containers21:02
*** yamamoto_ has joined #openstack-containers21:13
*** vishwanathj has quit IRC21:19
*** vishwanathj has joined #openstack-containers21:20
*** yamamoto_ has quit IRC21:20
*** vishwanathj has quit IRC21:27
*** vishwanathj has joined #openstack-containers21:28
*** vishwanathj has quit IRC21:28
*** vishwanathj has joined #openstack-containers21:29
*** armaan has quit IRC21:32
*** vishwanathj has quit IRC21:32
*** vishwanathj has joined #openstack-containers21:33
*** penick has quit IRC21:35
*** yamamoto has joined #openstack-containers22:04
*** mtanino_ has quit IRC22:06
*** rcernin has quit IRC22:26
*** catintheroof has quit IRC22:27
*** openstack has joined #openstack-containers22:44
*** ArchiFleKs has quit IRC22:45
*** hongbin has quit IRC23:03
*** penick has joined #openstack-containers23:05
*** ArchiFleKs has joined #openstack-containers23:07
*** kbyrne has quit IRC23:47
*** kbyrne has joined #openstack-containers23:48
*** catintheroof has joined #openstack-containers23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!