| *** hongbin has quit IRC | 00:18 | |
| *** ianychoi has joined #openstack-containers | 00:31 | |
| openstackgerrit | Merged openstack/magnum-ui master: Add release notes for Queens https://review.openstack.org/539161 | 00:51 |
|---|---|---|
| openstackgerrit | Merged openstack/magnum-ui master: Imported Translations from Zanata https://review.openstack.org/539105 | 00:51 |
| *** rcernin has joined #openstack-containers | 01:09 | |
| *** pengdake has joined #openstack-containers | 01:10 | |
| *** oikiki has quit IRC | 01:10 | |
| *** dave-mccowan has joined #openstack-containers | 01:13 | |
| *** hieulq_ has quit IRC | 01:15 | |
| *** ricolin has joined #openstack-containers | 01:19 | |
| *** ricolin has quit IRC | 01:21 | |
| *** hieulq_ has joined #openstack-containers | 01:27 | |
| *** vijaykc4 has joined #openstack-containers | 01:51 | |
| *** haint_ has joined #openstack-containers | 01:51 | |
| *** kiennt26 has joined #openstack-containers | 01:54 | |
| *** ramishra has joined #openstack-containers | 01:55 | |
| *** haint93 has quit IRC | 01:55 | |
| *** fragatina has quit IRC | 01:56 | |
| *** fragatina has joined #openstack-containers | 01:58 | |
| *** fragatina has quit IRC | 02:02 | |
| *** harlowja has quit IRC | 02:15 | |
| *** rcernin has quit IRC | 02:35 | |
| *** pengdake has quit IRC | 02:41 | |
| *** mnasiadka has joined #openstack-containers | 02:51 | |
| *** mnasiadka has quit IRC | 02:55 | |
| *** ykarel|away has joined #openstack-containers | 03:07 | |
| *** dpawar has joined #openstack-containers | 03:10 | |
| *** ricolin has joined #openstack-containers | 03:16 | |
| *** hongbin has joined #openstack-containers | 03:19 | |
| *** chris_hultin has quit IRC | 03:23 | |
| *** ykarel|away has quit IRC | 03:23 | |
| *** ykarel|away has joined #openstack-containers | 03:24 | |
| *** chris_hultin|AWA has joined #openstack-containers | 03:24 | |
| *** chris_hultin|AWA is now known as chris_hultin | 03:24 | |
| *** vijaykc4 has quit IRC | 03:33 | |
| *** ykarel|away has quit IRC | 03:34 | |
| *** pengdake has joined #openstack-containers | 03:50 | |
| *** flwang1 has quit IRC | 03:59 | |
| *** pengdake has quit IRC | 04:00 | |
| *** jianhua has quit IRC | 04:02 | |
| *** jianhua has joined #openstack-containers | 04:05 | |
| *** fragatina has joined #openstack-containers | 04:11 | |
| *** vijaykc4 has joined #openstack-containers | 04:15 | |
| *** shu-mutou-AWAY has quit IRC | 04:15 | |
| *** fragatina has quit IRC | 04:16 | |
| *** dpawar has quit IRC | 04:23 | |
| *** dpawar has joined #openstack-containers | 04:24 | |
| *** vijaykc4 has quit IRC | 04:25 | |
| *** dpawar has quit IRC | 04:28 | |
| *** ykarel|away has joined #openstack-containers | 04:31 | |
| *** dave-mccowan has quit IRC | 04:32 | |
| *** vijaykc4 has joined #openstack-containers | 04:46 | |
| *** rcernin has joined #openstack-containers | 04:51 | |
| *** harlowja has joined #openstack-containers | 04:51 | |
| *** rcernin has quit IRC | 04:51 | |
| *** rcernin has joined #openstack-containers | 04:52 | |
| *** jianhua has quit IRC | 04:58 | |
| *** jianhua has joined #openstack-containers | 05:03 | |
| *** fragatina has joined #openstack-containers | 05:08 | |
| *** fragatina has quit IRC | 05:09 | |
| *** pengdake has joined #openstack-containers | 05:12 | |
| *** zerick_ has quit IRC | 05:13 | |
| *** zerick has joined #openstack-containers | 05:15 | |
| *** dpawar has joined #openstack-containers | 05:26 | |
| *** hongbin has quit IRC | 05:26 | |
| *** mdnadeem has joined #openstack-containers | 05:41 | |
| *** vijaykc4 has quit IRC | 05:48 | |
| *** vijaykc4 has joined #openstack-containers | 05:51 | |
| *** threestrands has quit IRC | 05:51 | |
| *** threestrands has joined #openstack-containers | 06:01 | |
| *** daidv has quit IRC | 06:08 | |
| *** vijaykc4 has quit IRC | 06:11 | |
| *** jianhua has quit IRC | 06:14 | |
| *** jianhua has joined #openstack-containers | 06:16 | |
| *** pcichy has joined #openstack-containers | 06:19 | |
| *** trinaths has joined #openstack-containers | 06:19 | |
| *** lpetrut has joined #openstack-containers | 06:31 | |
| *** mjura has joined #openstack-containers | 06:42 | |
| *** dsariel has joined #openstack-containers | 06:47 | |
| *** lpetrut has quit IRC | 06:54 | |
| *** lpetrut has joined #openstack-containers | 06:54 | |
| *** harlowja has quit IRC | 07:01 | |
| *** pengdake has quit IRC | 07:01 | |
| *** itlinux has joined #openstack-containers | 07:02 | |
| *** pengdake has joined #openstack-containers | 07:03 | |
| *** zhenguo has joined #openstack-containers | 07:05 | |
| *** rcernin has quit IRC | 07:08 | |
| *** trinaths has quit IRC | 07:09 | |
| *** oikiki has joined #openstack-containers | 07:12 | |
| *** itlinux has quit IRC | 07:15 | |
| *** lpetrut has quit IRC | 07:21 | |
| *** pcichy has quit IRC | 07:23 | |
| *** mnasiadka has joined #openstack-containers | 07:25 | |
| *** lpetrut has joined #openstack-containers | 07:30 | |
| *** ykarel|away is now known as ykarel|lunch | 07:44 | |
| *** gbialas has joined #openstack-containers | 07:46 | |
| *** daidv has joined #openstack-containers | 07:46 | |
| *** pcaruana has joined #openstack-containers | 07:51 | |
| *** AlexeyAbashkin has joined #openstack-containers | 07:51 | |
| *** AlexeyAbashkin has quit IRC | 07:51 | |
| *** AlexeyAbashkin has joined #openstack-containers | 07:52 | |
| *** lpetrut has quit IRC | 07:53 | |
| *** oikiki has quit IRC | 08:05 | |
| *** b_bezak has joined #openstack-containers | 08:11 | |
| *** trinaths has joined #openstack-containers | 08:15 | |
| *** rcernin has joined #openstack-containers | 08:17 | |
| *** itlinux has joined #openstack-containers | 08:20 | |
| *** rcernin has quit IRC | 08:26 | |
| *** lpetrut has joined #openstack-containers | 08:28 | |
| *** janki has joined #openstack-containers | 08:31 | |
| *** lpetrut has quit IRC | 08:40 | |
| *** armaan has joined #openstack-containers | 08:40 | |
| *** ykarel|lunch is now known as ykarel | 08:44 | |
| *** dsariel has quit IRC | 08:47 | |
| *** pooja_jadhav has joined #openstack-containers | 08:49 | |
| *** janki is now known as janki|afk | 08:55 | |
| *** serlex has joined #openstack-containers | 09:03 | |
| *** rcernin has joined #openstack-containers | 09:05 | |
| openstackgerrit | Sayali Lunkad proposed openstack/magnum master: WIP: Add opensuse image building https://review.openstack.org/520063 | 09:06 |
| *** janki|afk is now known as janki | 09:21 | |
| *** vijaykc4 has joined #openstack-containers | 09:31 | |
| *** vijaykc4 has quit IRC | 09:49 | |
| *** salmankhan has joined #openstack-containers | 09:53 | |
| *** armaan has quit IRC | 09:53 | |
| *** zhenguo has quit IRC | 09:53 | |
| *** armaan has joined #openstack-containers | 09:54 | |
| *** mnasiadka has quit IRC | 09:55 | |
| *** ykarel is now known as ykarel|away | 10:06 | |
| *** lpetrut has joined #openstack-containers | 10:07 | |
| *** lpetrut_ has joined #openstack-containers | 10:09 | |
| *** ykarel|away has quit IRC | 10:11 | |
| *** dsariel has joined #openstack-containers | 10:11 | |
| *** lpetrut has quit IRC | 10:12 | |
| *** mnasiadka has joined #openstack-containers | 10:15 | |
| *** pc_m has quit IRC | 10:17 | |
| *** pengdake has quit IRC | 10:17 | |
| jianhua | did any guys used yum in fedora-atomic? i want to use tcpdump command in my k8s cluster,but i could not install yum in my fedora-atomic system | 10:21 |
| *** pc_m has joined #openstack-containers | 10:24 | |
| *** bhagyashris has joined #openstack-containers | 10:25 | |
| strigazi | jianhua: rpm-ostree install <your package> | 10:29 |
| armaan | strigazi: hello, Is there a need to open a port other than 9511 for magnum to work? | 10:35 |
| *** pbourke has quit IRC | 10:46 | |
| strigazi | armaan: You can use magnum in another port by changing the port in magnum.conf and updating the endpoints accordingly | 10:47 |
| *** pbourke has joined #openstack-containers | 10:48 | |
| armaan | strigazi: I have opened the port 9511 already and magnum api from outside world is working fine. I was wondering if internally any service tries to access any other port like 9090? | 10:49 |
| armaan | which also needs to be opened in the firewall. | 10:49 |
| strigazi | you don't have to open to the outside of your cloud | 10:50 |
| strigazi | the magnum api must be reachable from the vm in your cloud | 10:50 |
| strigazi | not from outsie | 10:50 |
| strigazi | not from outside | 10:50 |
| armaan | strigazi: Please correct me if i am wrong, my assumption is that the VM running in the compute node should only be able to access 9511 on the controllers? and no other port? | 10:51 |
| strigazi | The vms also need to be able to contact: nova (for the kube-openstack cloud provider), keystone (tokens), heat (signal heat and software deployments), cinder ( if you want to use volumes) and neutron (for lbaas) | 10:52 |
| armaan | strigazi: If a customer makes an api call to create a cluster, i suppose that will go to port 9511 and we need to open port 9511 to the world? | 10:52 |
| *** belmoreira has joined #openstack-containers | 10:52 | |
| armaan | strigazi: ok, yes i have made sure that VM can access all those endpoints. | 10:53 |
| strigazi | armaan oh ok, you mean for users | 10:53 |
| armaan | strigazi: yes, we run a public cloud. | 10:53 |
| strigazi | armaan: yes, for users to contact magnum you need 9511 *only* | 10:53 |
| armaan | so customer need to issue api calls against port 9511 | 10:54 |
| armaan | okay. | 10:54 |
| armaan | strigazi: Do you have any suggestion on how can i tackle the liveness and dashboard endpoint problem in our Pike setup? If a python developer is needed to tackle it, i think we can allocate one to this issue? | 10:55 |
| strigazi | armaan try this: | 10:57 |
| * strigazi is finding the link | 10:57 | |
| strigazi | armaan: https://bugs.launchpad.net/magnum/+bug/1742420/comments/3 | 10:57 |
| openstack | Launchpad bug 1742420 in Magnum "Pod's can't communicate with kube-apiserver in multi master cluster" [Undecided,In progress] - Assigned to Spyros Trigazis (strigazi) | 10:57 |
| armaan | reading the bug report ... | 10:58 |
| *** trinaths has quit IRC | 11:00 | |
| *** dardelean has joined #openstack-containers | 11:01 | |
| *** jmlowe has quit IRC | 11:02 | |
| armaan | damn, i am hitting that "User data too large" bug again after rebuilding repo :/ | 11:07 |
| *** AlexeyAbashkin has quit IRC | 11:09 | |
| *** vijaykc4 has joined #openstack-containers | 11:10 | |
| strigazi | rebuilding repo? | 11:10 |
| armaan | strigazi: Since stable/pike is currently broken, i am running pike up to this commit d0f08306150f2f14a5951d0916538471fb028752 | 11:11 |
| armaan | strigazi: https://github.com/openstack/magnum/commit/d0f08306150f2f14a5951d0916538471fb028752 | 11:12 |
| armaan | and to do that i have to reclone the git repo and rebuild the venvs with openstack-ansible playbooks | 11:12 |
| armaan | strigazi: I think it was ykarel who told me how to fix the "User data too large" issue, trying to find out that conversation ... | 11:14 |
| armaan | btw this is the error message: http://paste.openstack.org/show/658132/ | 11:15 |
| *** AlexeyAbashkin has joined #openstack-containers | 11:15 | |
| armaan | strigazi: i guess this is the bug https://bugs.launchpad.net/magnum/+bug/1650538 | 11:16 |
| openstack | Launchpad bug 1650538 in Magnum "User data too large" [Undecided,Fix released] - Assigned to Spyros Trigazis (strigazi) | 11:16 |
| strigazi | https://github.com/openstack/magnum/commits/stable/pike see commit "Leverage heat-container-agent for monitoring" | 11:16 |
| strigazi | if you have a problem with the heat agent comment out https://github.com/openstack/magnum/blob/stable/pike/magnum/drivers/k8s_fedora_atomic_v1/templates/kubemaster.yaml#L466-L482 | 11:18 |
| armaan | strigazi: that commit leads to a bug | 11:18 |
| strigazi | Which bug? | 11:18 |
| armaan | strigazi: https://bugs.launchpad.net/magnum/+bug/1744362 | 11:18 |
| openstack | Launchpad bug 1744362 in Magnum "heat-container-agent fails to communicate with Keystone." [Medium,Confirmed] | 11:18 |
| armaan | oh i can comment out that, good to know that ... | 11:19 |
| armaan | ok rebuilding the repo upto commit a342f92062ef7bacf0ea1b07d3732eaa6ee5a5e1 | 11:26 |
| armaan | https://github.com/openstack/magnum/commit/a342f92062ef7bacf0ea1b07d3732eaa6ee5a5e1 | 11:26 |
| strigazi | armaan: ok | 11:27 |
| strigazi | armaan: in the bug, you posted this log: http://paste.openstack.org/show/647025/ | 11:28 |
| armaan | strigazi: yes? | 11:28 |
| strigazi | from that node, the keystone endpoint is not reachable? | 11:28 |
| armaan | yep | 11:28 |
| armaan | because that is the internal endpoint | 11:29 |
| strigazi | oh | 11:29 |
| strigazi | crap :( | 11:29 |
| strigazi | but the public is reachable right? | 11:29 |
| armaan | strigazi: yes, public is reachable. | 11:29 |
| strigazi | we'll fix it | 11:29 |
| armaan | thanks :) | 11:32 |
| vabada | strigazi is this https://review.openstack.org/#/c/529098/9 one of the patches to be tested? the one you mentione from oikiki? | 11:35 |
| *** mdnadeem_ has joined #openstack-containers | 11:43 | |
| *** pcichy has joined #openstack-containers | 11:44 | |
| *** mdnadeem has quit IRC | 11:44 | |
| *** mdnadeem has joined #openstack-containers | 11:45 | |
| *** vijaykc4 has quit IRC | 11:46 | |
| strigazi | yes | 11:46 |
| strigazi | and https://review.openstack.org/#/c/534364/ | 11:46 |
| *** vijaykc4 has joined #openstack-containers | 11:46 | |
| vijaykc4 | i think we should target OSP 14 right | 11:46 |
| strigazi | vabada: yes, and https://review.openstack.org/#/c/534364/ | 11:47 |
| *** mdnadeem_ has quit IRC | 11:48 | |
| *** yamamoto_ has quit IRC | 11:48 | |
| vabada | good, I'll test them today | 11:48 |
| vabada | thanks | 11:48 |
| strigazi | vijaykc4: to whom is your message for? | 11:50 |
| vijaykc4 | strigazi, sorry my bad wrong channel | 11:50 |
| *** salmankhan has quit IRC | 11:50 | |
| *** dpawar has quit IRC | 11:50 | |
| *** salmankhan has joined #openstack-containers | 11:51 | |
| strigazi | vijaykc4: np :) | 11:51 |
| gokhan | hi strigazi, about openstack-ca patch , when will you plan to cherry pick this pactch on pike? I tested it on pike and it is working successfully. | 11:55 |
| *** ykarel has joined #openstack-containers | 11:55 | |
| gokhan | and also this patch https://review.openstack.org/#/c/524151/ ? | 11:55 |
| *** itlinux has quit IRC | 11:55 | |
| strigazi | gokhan I'll take care of these today | 12:00 |
| *** gabrielvinha has quit IRC | 12:02 | |
| gokhan | strigazi, ok thanks. also on pike there are 2 problems. First is kubernates-dashboard is not working like armaan said. and the second heat-agent is not working because of this bug https://bugs.launchpad.net/magnum/+bug/1744362. | 12:05 |
| openstack | Launchpad bug 1744362 in Magnum "heat-container-agent fails to communicate with Keystone." [Medium,Confirmed] | 12:05 |
| *** dsariel has quit IRC | 12:07 | |
| *** threestrands has quit IRC | 12:08 | |
| *** dave-mccowan has joined #openstack-containers | 12:08 | |
| ykarel | ricolin, hi. have you checked ^^ | 12:10 |
| *** janki has quit IRC | 12:10 | |
| ricolin | ykarel, yeah, so my plan needs some adjust to avoid some error, haven got time for it this week due to release job (try to survive from gate stable issue), will keep working on it ASAP | 12:17 |
| ykarel | ricolin, Ok Thanks | 12:19 |
| ykarel | ricolin, BTW is there some workaround that can be used till then | 12:20 |
| *** dsariel has joined #openstack-containers | 12:21 | |
| *** yamamoto has joined #openstack-containers | 12:22 | |
| *** yamamoto has quit IRC | 12:22 | |
| *** yamamoto has joined #openstack-containers | 12:23 | |
| openstackgerrit | Sayali Lunkad proposed openstack/magnum master: WIP: Add opensuse image building https://review.openstack.org/520063 | 12:32 |
| *** rcernin has quit IRC | 12:58 | |
| armaan | strigazi: I applied that patch https://review.openstack.org/#/c/533593/10/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-master.sh@26 | 13:07 |
| strigazi | armaan: only that line I guess, right? | 13:08 |
| strigazi | or the full patch? | 13:08 |
| armaan | strigazi: damn, all the 3 lines :( | 13:08 |
| armaan | and now containers are not starting on the minion node | 13:08 |
| strigazi | we need only that line, the other lines also enable RBAC which requires more changes. | 13:09 |
| *** dpawar has joined #openstack-containers | 13:09 | |
| strigazi | You could try it in a running cluster though | 13:10 |
| armaan | strigazi: cluster creation complete, but no container on minion http://paste.openstack.org/show/658141/ | 13:11 |
| armaan | strigazi: ok, let me fix that. | 13:11 |
| *** vijaykc4 has quit IRC | 13:16 | |
| *** vijaykc4 has joined #openstack-containers | 13:17 | |
| *** vijaykc4 has quit IRC | 13:17 | |
| *** astellwag has joined #openstack-containers | 13:19 | |
| *** janki has joined #openstack-containers | 13:28 | |
| *** vijaykc4 has joined #openstack-containers | 13:28 | |
| *** mnasiadka has quit IRC | 13:33 | |
| openstackgerrit | Sayali Lunkad proposed openstack/magnum master: WIP: Add opensuse image building https://review.openstack.org/520063 | 13:34 |
| *** tobberydberg has quit IRC | 13:41 | |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack/magnum stable/pike: Add openstack_ca_file configuration option https://review.openstack.org/539535 | 13:42 |
| strigazi | gokhan: ^^ | 13:43 |
| gokhan | strigazi, thanks :) | 13:49 |
| openstackgerrit | Merged openstack/magnum master: ci: Add redirection from /v2 to /identity/v2 https://review.openstack.org/496229 | 13:50 |
| *** yolanda has quit IRC | 14:01 | |
| *** itlinux has joined #openstack-containers | 14:02 | |
| *** yolanda has joined #openstack-containers | 14:03 | |
| *** yamamoto has quit IRC | 14:04 | |
| *** jmlowe has joined #openstack-containers | 14:04 | |
| openstackgerrit | Sayali Lunkad proposed openstack/magnum master: WIP: Add opensuse image building https://review.openstack.org/520063 | 14:05 |
| *** pcaruana has quit IRC | 14:05 | |
| *** yamamoto has joined #openstack-containers | 14:06 | |
| *** yamamoto has quit IRC | 14:11 | |
| armaan | strigazi: No change, same liveness issue. Here is the cloud-init output from minion, please have a look at Line 78, http://paste.openstack.org/show/658153/ | 14:13 |
| strigazi | armaan segmentation fault :-O | 14:14 |
| strigazi | armaan: which script is /var/lib/cloud/instance/scripts/part-008 ? | 14:14 |
| armaan | strigazi: https://github.com/openstack/magnum/blob/stable/pike/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh | 14:16 |
| strigazi | are you using: Fedora-Atomic-26-20170723.0.x86_64.qcow2 ? | 14:19 |
| strigazi | which line is line 8 in /var/lib/cloud/instance/scripts/part-008 ? | 14:19 |
| armaan | strigazi: atomic install --storage ostree --system --system-package=no --name=kube-proxy docker.io/openstackmagnum/kubernetes-proxy:${KUBE_TAG} | 14:20 |
| openstackgerrit | Sayali Lunkad proposed openstack/magnum master: WIP: Add opensuse image building https://review.openstack.org/520063 | 14:21 |
| *** pcaruana has joined #openstack-containers | 14:21 | |
| strigazi | armaan this is a new problem right? I don't know if it has a side effect the liveness issue | 14:21 |
| strigazi | can you ask in #atomic and ping me there? | 14:22 |
| strigazi | armaan: ^^ | 14:22 |
| armaan | strigazi: roger that. | 14:23 |
| armaan | strigazi: I only get this from #cat /etc/system-release "Fedora release 26 (Twenty Six)" | 14:24 |
| strigazi | armaan: atomic --version | 14:25 |
| armaan | strigazi: 1.18.1 | 14:25 |
| strigazi | and rpm-ostree status | 14:26 |
| armaan | strigazi: http://paste.openstack.org/show/658159/ | 14:27 |
| armaan | strigazi: not sure if this is a new issue. I may have missed it earlier | 14:28 |
| *** mdnadeem has quit IRC | 14:32 | |
| armaan | strigazi: Can folks at CERN test the stable/pike on a multi node deployment? | 14:32 |
| ykarel | armaan, just to check: is minion ready in, kubectl get nodes | 14:32 |
| armaan | ykarel: Yep, http://paste.openstack.org/show/658161/ | 14:33 |
| ykarel | armaan, ack | 14:33 |
| ykarel | and you still have endpoint issue? | 14:34 |
| ykarel | kubernetes-dashboard not getting endpoint | 14:34 |
| armaan | ykarel: Yes, and it is driving me nuts :( | 14:35 |
| ykarel | ohh :( | 14:36 |
| serlex | ykarel are you running k8s or swarm with magnum? | 14:36 |
| ykarel | serlex, both, but currently don't have an environment | 14:38 |
| *** yamamoto has joined #openstack-containers | 14:38 | |
| serlex | i see, I'm running 1.6 k8s atm. I can see the kube-proxy kube-controller-manager etc in "ps auxf"...anyway to see them via kubectl | 14:39 |
| serlex | "kubectl -n kube-system get po" is showing coredns and dashboard | 14:39 |
| armaan | serlex: I think those are in containers now | 14:43 |
| armaan | serlex: atomic containers list --no-trunc | 14:43 |
| strigazi | armaan try to create a cluster with docker-storage-driver overlay | 14:43 |
| armaan | strigazi: ok. | 14:44 |
| strigazi | armaan never saw this segfault | 14:44 |
| *** mnasiadka has joined #openstack-containers | 14:45 | |
| *** yamamoto has quit IRC | 14:47 | |
| *** yamamoto has joined #openstack-containers | 14:48 | |
| armaan | strigazi: Just confirmed that the template already had that set. http://paste.openstack.org/show/658168/ | 14:48 |
| *** yamamoto has quit IRC | 14:48 | |
| *** pcaruana has quit IRC | 14:48 | |
| *** yamamoto has joined #openstack-containers | 14:49 | |
| strigazi | armaan ok, then to eliminate the space issue, remove docker-volume-size | 14:49 |
| armaan | strigazi: Do you think this could be behind the liveness issue as well? | 14:52 |
| strigazi | armaan I'm now sure, perhaps | 14:53 |
| strigazi | armaan: you had one master? | 14:53 |
| armaan | strigazi: yes | 14:53 |
| serlex | ok thanks ykarel | 14:54 |
| serlex | still trying to get to the bottom of "/var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/6a214668-faec-4e6f-98e1-6115ef138300 does not exist" | 14:54 |
| armaan | ykarel: IIRC, you were able to get the dashboard endpoints with a single machine deployment? but you had to make some change? | 14:54 |
| ykarel | armaan yes | 14:54 |
| ykarel | serlex, i faced this but can't remember the fix now | 14:55 |
| ykarel | serlex, did you checked kubelet logs on minion node where pod is running? | 14:56 |
| serlex | i have and looking at it right now, seems very noisy, trying to identify the relevant failures | 14:58 |
| armaan | ykarel: IIRC the change was related to endpoints? | 14:58 |
| ykarel | armaan, yes i also remember the same | 14:58 |
| ykarel | let me check if i have noted the other changes | 14:59 |
| ykarel | i did | 15:00 |
| *** vijaykc4 has quit IRC | 15:00 | |
| ykarel | yes wc-notify also failed, but after endpoint change it worked | 15:01 |
| armaan | ykarel: great, could you please share those | 15:01 |
| armaan | oh | 15:01 |
| armaan | ykarel: i have this: http://paste.openstack.org/show/658170/ | 15:03 |
| armaan | ykarel: IIRC you changed the public endpoints of keystone and magnum? | 15:04 |
| ykarel | and heat, nova | 15:05 |
| armaan | ykarel: Changed the public endpoints of keystone, magnum, heat and nova from https to http? | 15:07 |
| ykarel | armaan, yes | 15:07 |
| armaan | ykarel: ok, does that mean that it is a ssl related issue? | 15:08 |
| ykarel | armaan, i don't think that | 15:08 |
| ykarel | i changed endpoint just to check that if that was dashboard issue | 15:08 |
| ykarel | and i saw errors related to ssl in my case, that's why i changed | 15:09 |
| serlex | ykarel...could you have a quick glance? https://pastebin.ubuntu.com/26495580/ | 15:09 |
| ykarel | armaan, for ex: kube-controller-manager failed due to Error: Jan 21 06:03:10 k8s-cluster-idkrik4qobks-master-0.novalocal runc[4840]: F0121 06:03:10.111442 1 controllermanager.go:176] error building controller context: cloud provider could not be initialized: could not init cloud provider "openstack": Post https://192.168.100.15:5000/v3/auth/tokens: x509: cannot validate certificate for 192.168.100.15 because it doesn't contain any IP | 15:09 |
| ykarel | SANs | 15:09 |
| serlex | or any other contributor | 15:10 |
| ykarel | also cert creation failed, but in you case you haven't seen these errors. right? | 15:10 |
| armaan | ykarel: how can i confirm? | 15:10 |
| ykarel | armaan, check kube-controller-manager, kubelet logs | 15:10 |
| *** pcaruana has joined #openstack-containers | 15:14 | |
| *** sheva_ has joined #openstack-containers | 15:15 | |
| *** itlinux has quit IRC | 15:18 | |
| ykarel | serlex, your magnum version? | 15:19 |
| ykarel | pike or master? | 15:19 |
| *** ispp has quit IRC | 15:19 | |
| serlex | magnum 2.7.0 | 15:20 |
| ykarel | server? | 15:20 |
| ykarel | magnum-api --version | 15:20 |
| serlex | ahh sorry 3.1.1 | 15:20 |
| *** dpawar has quit IRC | 15:22 | |
| ykarel | looks like ocata or even older, checking | 15:22 |
| ykarel | serlex, it's newton | 15:23 |
| ykarel | too old | 15:23 |
| serlex | yeah, i see it, ok no worries ykarel, no point debugging this further | 15:24 |
| serlex | apologies | 15:24 |
| *** armaan has quit IRC | 15:32 | |
| *** tobberydberg has joined #openstack-containers | 15:34 | |
| *** armaan has joined #openstack-containers | 15:34 | |
| *** armaan has quit IRC | 15:34 | |
| *** armaan has joined #openstack-containers | 15:35 | |
| *** sheva_ has quit IRC | 15:35 | |
| *** ispp has joined #openstack-containers | 15:36 | |
| *** yamamoto has quit IRC | 15:38 | |
| strigazi | armaan: is the cluster working now? | 15:40 |
| armaan | strigazi: cluster is up with same issues of liveness and no endpoints. | 15:41 |
| armaan | strigazi: http://paste.openstack.org/show/658174/ | 15:42 |
| strigazi | armaan with --insecure-bind-address=127.0.0.1 | 15:42 |
| strigazi | ? | 15:42 |
| armaan | strigazi: yes | 15:42 |
| strigazi | kubectl get cs | 15:42 |
| armaan | strigazi: http://paste.openstack.org/show/658175/ | 15:43 |
| strigazi | kubectl version | 15:44 |
| strigazi | and | 15:45 |
| strigazi | journalctl -u kube-controller-manager --no-paher | 15:45 |
| strigazi | journalctl -u kube-controller-manager --no-pager | 15:45 |
| armaan | strigazi: http://paste.openstack.org/show/658179/ | 15:46 |
| strigazi | journalctl -u kubelet --no-pager | 15:47 |
| strigazi | armaan on the node | 15:48 |
| strigazi | armaan: not the master | 15:48 |
| strigazi | and on the master do kubectl get no | 15:48 |
| *** yamamoto has joined #openstack-containers | 15:48 | |
| armaan | strigazi: oh ok | 15:49 |
| armaan | strigazi: journalctl -u kubelet --no-pager is throwing a lot of output | 15:52 |
| strigazi | journalctl -u kubelet --no-pager | grep -i register | 15:52 |
| *** yamamoto has quit IRC | 15:53 | |
| strigazi | armaan aslo | 15:56 |
| armaan | ok | 15:56 |
| strigazi | kubectl -n kube-system get po | 15:56 |
| strigazi | kubectl -n kube-system logs <the coredns pod> | 15:56 |
| armaan | kubectl -n kube-system get po -- | 15:56 |
| armaan | on master? | 15:57 |
| strigazi | armaan yes | 15:57 |
| armaan | ok | 15:57 |
| armaan | strigazi: journalctl -u kubelet --no-pager http://paste.openstack.org/show/658183/ | 15:57 |
| strigazi | all kubectl are done on the master node or anywhere where kubectl is configured | 15:57 |
| armaan | strigazi: journalctl -u kubelet --no-pager | grep -i register http://paste.openstack.org/show/658184/ | 15:57 |
| armaan | got it. | 15:57 |
| strigazi | I don't understand the node is registered and the components are running | 15:58 |
| strigazi | let's see what coredns is saying | 15:58 |
| armaan | strigazi: http://paste.openstack.org/show/658185/ | 15:59 |
| strigazi | armaan: kubectl -n kube-system edit deployment coredns | 16:00 |
| strigazi | change the tag in the image field from 011 to 1.0.5 | 16:00 |
| strigazi | then wait to see if the pod starts | 16:01 |
| strigazi | then do: kubectl -n kube-system logs <the coredns pod> -f | 16:01 |
| armaan | ok | 16:02 |
| armaan | strigazi: http://paste.openstack.org/show/658191/ | 16:03 |
| armaan | strigazi: looks like coredns has stopped crashing | 16:03 |
| strigazi | endpoints? | 16:05 |
| armaan | strigazi: no endpoint | 16:05 |
| armaan | strigazi: http://paste.openstack.org/show/658193/ | 16:05 |
| ykarel | strigazi, regarding https://bugs.launchpad.net/bugs/1746510, i think there was recently an issue with glance + eventlet as well | 16:06 |
| openstack | Launchpad bug 1746510 in Magnum "Kubernetes client is incompatible with evenlet and breaks the periodic tasks" [Undecided,New] | 16:06 |
| strigazi | ykarel the problem is with evenelet and multiprocessing | 16:06 |
| strigazi | armaan coredns is restarting, it restarted once already | 16:07 |
| *** gbialas has quit IRC | 16:07 | |
| strigazi | try again to het the logs, wait with -f | 16:07 |
| armaan | strigazi: yeah, i started celeberating too early :( | 16:07 |
| *** gbialas has joined #openstack-containers | 16:07 | |
| strigazi | or try "watch kubectl ..." | 16:08 |
| *** kiennt26 has quit IRC | 16:08 | |
| ykarel | strigazi, Ok | 16:08 |
| armaan | strigazi: this "watch kubectl -n kube-system logs coredns-3478615533-w34l1" ? | 16:08 |
| *** daidv has quit IRC | 16:09 | |
| strigazi | armaan yes | 16:09 |
| armaan | ok | 16:09 |
| strigazi | coredns starts, we should see some logs | 16:09 |
| *** belmoreira has quit IRC | 16:10 | |
| *** kiennt26 has joined #openstack-containers | 16:10 | |
| *** dsariel has quit IRC | 16:11 | |
| *** daidv has joined #openstack-containers | 16:11 | |
| *** dardelean has quit IRC | 16:11 | |
| *** gbialas has quit IRC | 16:12 | |
| *** dardelean has joined #openstack-containers | 16:12 | |
| armaan | strigazi: About this message "Liveness probe failed: Get http://:8080/health: dial tcp :8080: getsockopt: connection refused" --> which connection got refused here? | 16:13 |
| strigazi | armaan this comes from the dashboaed? | 16:14 |
| armaan | strigazi: yes | 16:14 |
| strigazi | armaan can you do kubectl get nodes? | 16:15 |
| armaan | strigazi: Line 60 http://paste.openstack.org/show/658194/ | 16:15 |
| armaan | strigazi: kubectl get nodes http://paste.openstack.org/show/658196/ | 16:16 |
| *** dardelean has quit IRC | 16:16 | |
| armaan | strigazi: coredns, failed to pull image http://paste.openstack.org/show/658197/ | 16:18 |
| *** belmoreira has joined #openstack-containers | 16:19 | |
| *** janki has quit IRC | 16:22 | |
| *** serlex has left #openstack-containers | 16:23 | |
| *** b_bezak has quit IRC | 16:25 | |
| *** ykarel is now known as ykarel|away | 16:25 | |
| *** b_bezak has joined #openstack-containers | 16:25 | |
| *** dsariel has joined #openstack-containers | 16:27 | |
| *** vijaykc4 has joined #openstack-containers | 16:29 | |
| *** ykarel|away has quit IRC | 16:29 | |
| *** b_bezak has quit IRC | 16:30 | |
| *** ricolin has quit IRC | 16:32 | |
| *** pcaruana has quit IRC | 16:39 | |
| *** daidv has quit IRC | 16:52 | |
| *** kiennt26 has quit IRC | 16:52 | |
| *** kiennt26 has joined #openstack-containers | 16:53 | |
| *** daidv has joined #openstack-containers | 16:53 | |
| *** b_bezak has joined #openstack-containers | 16:54 | |
| *** mjura has quit IRC | 17:01 | |
| *** harlowja has joined #openstack-containers | 17:02 | |
| *** yamamoto has joined #openstack-containers | 17:03 | |
| *** ramishra has quit IRC | 17:03 | |
| *** belmoreira has quit IRC | 17:08 | |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack/python-magnumclient master: Now cluster-template-update works for "labels" https://review.openstack.org/496136 | 17:11 |
| *** dsariel has quit IRC | 17:13 | |
| *** yamamoto has quit IRC | 17:16 | |
| *** AlexeyAbashkin has quit IRC | 17:23 | |
| *** b_bezak has quit IRC | 17:33 | |
| *** dsariel has joined #openstack-containers | 17:34 | |
| *** b_bezak has joined #openstack-containers | 17:34 | |
| *** flwang1 has joined #openstack-containers | 17:37 | |
| *** b_bezak has quit IRC | 17:38 | |
| *** zenirc369 has joined #openstack-containers | 17:42 | |
| *** lpetrut_ has quit IRC | 17:52 | |
| *** oikiki has joined #openstack-containers | 18:04 | |
| *** mnasiadka has quit IRC | 18:06 | |
| oikiki | strigazi: Saw and will get those send_cluster_metric changes in today. Almost finished #2 clearing cached certs on cluster delete, I just have to fix a test now. | 18:10 |
| *** hongbin has joined #openstack-containers | 18:12 | |
| *** mnasiadka has joined #openstack-containers | 18:17 | |
| *** AlexeyAbashkin has joined #openstack-containers | 18:17 | |
| flwang1 | oikiki: oh, nice, looking forward to reviewing it | 18:18 |
| oikiki | flwang1: thanks :) | 18:18 |
| *** vijaykc4 has quit IRC | 18:19 | |
| flwang1 | strigazi: do you know who is maintaining the https://hub.docker.com/r/openstackmagnum | 18:19 |
| flwang1 | oikiki: my pleasure | 18:19 |
| *** mnasiadka has quit IRC | 18:21 | |
| *** pcichy has quit IRC | 18:23 | |
| *** pcichy has joined #openstack-containers | 18:24 | |
| *** mnasiadka has joined #openstack-containers | 18:31 | |
| *** mnasiadka has quit IRC | 18:36 | |
| *** flwang1 has quit IRC | 18:37 | |
| *** mnasiadka has joined #openstack-containers | 18:46 | |
| *** dsariel has quit IRC | 18:47 | |
| *** mnasiadka has quit IRC | 18:51 | |
| *** AlexeyAbashkin has quit IRC | 18:51 | |
| *** harlowja has quit IRC | 18:52 | |
| *** mnasiadka has joined #openstack-containers | 19:01 | |
| *** mnasiadka has quit IRC | 19:06 | |
| *** freerunner has quit IRC | 19:06 | |
| *** freerunner has joined #openstack-containers | 19:08 | |
| *** dardelean has joined #openstack-containers | 19:12 | |
| *** dardelean has quit IRC | 19:16 | |
| *** mnasiadka has joined #openstack-containers | 19:17 | |
| *** lpetrut_ has joined #openstack-containers | 19:20 | |
| *** mnasiadka has quit IRC | 19:21 | |
| *** freerunner has quit IRC | 19:29 | |
| *** salmankhan has quit IRC | 19:33 | |
| *** mnasiadka has joined #openstack-containers | 19:33 | |
| *** harlowja has joined #openstack-containers | 19:37 | |
| *** mnasiadka has quit IRC | 19:37 | |
| *** freerunner has joined #openstack-containers | 19:43 | |
| *** mnasiadka has joined #openstack-containers | 19:48 | |
| *** mnasiadka has quit IRC | 19:53 | |
| *** armaan has quit IRC | 20:02 | |
| *** mnasiadka has joined #openstack-containers | 20:03 | |
| *** armaan has joined #openstack-containers | 20:03 | |
| *** mnasiadka has quit IRC | 20:08 | |
| *** freerunner has quit IRC | 20:17 | |
| *** mnasiadka has joined #openstack-containers | 20:18 | |
| *** mnasiadka has quit IRC | 20:22 | |
| *** freerunner has joined #openstack-containers | 20:23 | |
| *** mnasiadka has joined #openstack-containers | 20:26 | |
| *** mnasiadka has quit IRC | 20:31 | |
| *** mnasiadka has joined #openstack-containers | 20:32 | |
| *** mnasiadka has quit IRC | 20:37 | |
| *** flwang1 has joined #openstack-containers | 20:38 | |
| *** freerunner has quit IRC | 20:39 | |
| *** freerunner has joined #openstack-containers | 20:41 | |
| *** oikiki has quit IRC | 20:46 | |
| *** oikiki has joined #openstack-containers | 20:47 | |
| *** mnasiadka has joined #openstack-containers | 20:48 | |
| *** pcichy has quit IRC | 20:49 | |
| *** AJaeger has left #openstack-containers | 20:52 | |
| *** mnasiadka has quit IRC | 20:52 | |
| *** salmankhan has joined #openstack-containers | 20:58 | |
| *** zenirc369 has quit IRC | 20:59 | |
| *** lpetrut_ has quit IRC | 21:00 | |
| *** mnasiadka has joined #openstack-containers | 21:02 | |
| *** salmankhan has quit IRC | 21:02 | |
| *** mnasiadka has quit IRC | 21:07 | |
| *** armaan has quit IRC | 21:07 | |
| *** mnasiadka has joined #openstack-containers | 21:17 | |
| openstackgerrit | Kirsten G. proposed openstack/magnum master: Add send_cluster_metrics configuration parameter https://review.openstack.org/529098 | 21:20 |
| *** mnasiadka has quit IRC | 21:22 | |
| oikiki | flwang1 strigazi slunkad : does it matter if my author date is really far away from my current commit date? for the final tweaks to the patch i just pushed, my original author date is dec 19th but my current commit date is the jan 31st... or do we not care? do we ever update the author date on a patch? | 21:24 |
| *** mnasiadka has joined #openstack-containers | 21:32 | |
| *** mnasiadka has quit IRC | 21:36 | |
| *** salmankhan has joined #openstack-containers | 21:46 | |
| *** mnasiadka has joined #openstack-containers | 21:47 | |
| *** mnasiadka has quit IRC | 21:51 | |
| *** dardelean has joined #openstack-containers | 21:52 | |
| *** armaan has joined #openstack-containers | 21:59 | |
| *** mnasiadka has joined #openstack-containers | 22:02 | |
| *** mnasiadka has quit IRC | 22:07 | |
| *** mnasiadka has joined #openstack-containers | 22:17 | |
| flwang1 | oikiki: it doesn't matter totally | 22:18 |
| oikiki | flwang1: thank you, just wanted to make sure :) | 22:21 |
| *** mnasiadka has quit IRC | 22:21 | |
| *** dardelean has quit IRC | 22:24 | |
| *** mnasiadka has joined #openstack-containers | 22:32 | |
| *** mnasiadka has quit IRC | 22:36 | |
| *** salmankhan has quit IRC | 22:44 | |
| *** mnasiadka has joined #openstack-containers | 22:47 | |
| *** mnasiadka has quit IRC | 22:51 | |
| *** mnasiadka has joined #openstack-containers | 23:02 | |
| *** mnasiadka has quit IRC | 23:06 | |
| *** mnasiadka has joined #openstack-containers | 23:17 | |
| *** mnasiadka has quit IRC | 23:22 | |
| *** oikiki has quit IRC | 23:22 | |
| *** oikiki has joined #openstack-containers | 23:22 | |
| *** rcernin has joined #openstack-containers | 23:28 | |
| *** mnasiadka has joined #openstack-containers | 23:32 | |
| *** mnasiadka has quit IRC | 23:37 | |
| *** zenirc369 has joined #openstack-containers | 23:40 | |
| *** pengdake has joined #openstack-containers | 23:44 | |
| *** mnasiadka has joined #openstack-containers | 23:47 | |
| *** rcernin has quit IRC | 23:50 | |
| *** mnasiadka has quit IRC | 23:51 | |
| *** pengdake has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!