| *** salv-orlando has joined #openstack-kuryr | 00:23 | |
| *** salv-orlando has quit IRC | 00:27 | |
| *** yamamoto has joined #openstack-kuryr | 00:59 | |
| *** yamamoto has quit IRC | 01:04 | |
| *** salv-orlando has joined #openstack-kuryr | 01:23 | |
| *** salv-orlando has quit IRC | 01:28 | |
| *** hongbin has joined #openstack-kuryr | 01:41 | |
| *** yamamoto has joined #openstack-kuryr | 02:00 | |
| *** yamamoto has quit IRC | 02:05 | |
| *** salv-orlando has joined #openstack-kuryr | 02:24 | |
| *** hongbin_ has joined #openstack-kuryr | 02:28 | |
| *** salv-orlando has quit IRC | 02:28 | |
| *** hongbin has quit IRC | 02:31 | |
| *** yamamoto has joined #openstack-kuryr | 03:01 | |
| *** yamamoto has quit IRC | 03:06 | |
| *** salv-orlando has joined #openstack-kuryr | 03:25 | |
| *** salv-orlando has quit IRC | 03:30 | |
| *** yamamoto has joined #openstack-kuryr | 04:02 | |
| *** yamamoto has quit IRC | 04:08 | |
| *** hongbin_ has quit IRC | 04:22 | |
| *** salv-orlando has joined #openstack-kuryr | 04:26 | |
| *** salv-orlando has quit IRC | 04:30 | |
| *** snapiri has joined #openstack-kuryr | 04:38 | |
| *** gcheresh_ has joined #openstack-kuryr | 04:50 | |
| *** yboaron has joined #openstack-kuryr | 05:01 | |
| *** yamamoto has joined #openstack-kuryr | 05:04 | |
| *** yamamoto has quit IRC | 05:08 | |
| *** janonymous has joined #openstack-kuryr | 05:22 | |
| *** salv-orlando has joined #openstack-kuryr | 05:26 | |
| *** salv-orlando has quit IRC | 05:31 | |
| *** janki has joined #openstack-kuryr | 05:51 | |
| *** yamamoto has joined #openstack-kuryr | 06:05 | |
| *** yamamoto has quit IRC | 06:11 | |
| *** natanbro has joined #openstack-kuryr | 06:27 | |
| *** salv-orlando has joined #openstack-kuryr | 06:27 | |
| *** salv-orlando has quit IRC | 06:32 | |
| *** gcheresh_ has quit IRC | 06:47 | |
| *** yboaron has quit IRC | 06:50 | |
| *** yboaron has joined #openstack-kuryr | 06:59 | |
| *** yamamoto has joined #openstack-kuryr | 07:06 | |
| *** yboaron has quit IRC | 07:08 | |
| *** yamamoto has quit IRC | 07:13 | |
| *** gcheresh_ has joined #openstack-kuryr | 07:18 | |
| *** salv-orlando has joined #openstack-kuryr | 07:28 | |
| *** salv-orlando has quit IRC | 07:33 | |
| *** apuimedo has joined #openstack-kuryr | 07:42 | |
| apuimedo | livelace-link: only openshift 3.10 is going to support kuryr-kubernetes deployment | 08:05 |
|---|---|---|
| apuimedo | if you were using 3.10 openshift-ansible rpms | 08:05 |
| apuimedo | there's still some fixes missing that are only on master or on PR | 08:05 |
| apuimedo | we are getting the release ready | 08:05 |
| *** yamamoto has joined #openstack-kuryr | 08:08 | |
| *** yamamoto has quit IRC | 08:14 | |
| *** livelace has joined #openstack-kuryr | 08:16 | |
| *** yboaron has joined #openstack-kuryr | 08:16 | |
| *** salv-orlando has joined #openstack-kuryr | 08:29 | |
| *** salv-orlando has quit IRC | 08:33 | |
| *** salv-orlando has joined #openstack-kuryr | 08:51 | |
| *** pmannidi has quit IRC | 08:52 | |
| *** yamamoto has joined #openstack-kuryr | 09:10 | |
| *** yamamoto has quit IRC | 09:16 | |
| *** livelace-link has quit IRC | 10:09 | |
| *** livelace-link has joined #openstack-kuryr | 10:12 | |
| *** yamamoto has joined #openstack-kuryr | 10:12 | |
| openstackgerrit | Merged openstack/kuryr-tempest-plugin master: Updated from global requirements https://review.openstack.org/555219 | 10:15 |
| *** salv-orlando has quit IRC | 10:17 | |
| *** salv-orlando has joined #openstack-kuryr | 10:17 | |
| *** yamamoto has quit IRC | 10:17 | |
| *** salv-orlando has quit IRC | 10:22 | |
| *** yboaron_ has joined #openstack-kuryr | 10:27 | |
| *** yboaron has quit IRC | 10:29 | |
| *** yboaron has joined #openstack-kuryr | 10:32 | |
| *** yboaron_ has quit IRC | 10:34 | |
| *** garyloug has joined #openstack-kuryr | 10:53 | |
| *** atoth has quit IRC | 10:56 | |
| *** yamamoto has joined #openstack-kuryr | 11:14 | |
| *** yamamoto has quit IRC | 11:19 | |
| *** yboaron_ has joined #openstack-kuryr | 11:30 | |
| *** yboaron has quit IRC | 11:30 | |
| *** yamamoto has joined #openstack-kuryr | 11:39 | |
| openstackgerrit | Yossi Boaron proposed openstack/kuryr-kubernetes master: OCP-Router: add support for external Ingress Controller's LB creation https://review.openstack.org/564051 | 12:10 |
| *** atoth has joined #openstack-kuryr | 12:10 | |
| *** kzaitsev_pi has quit IRC | 12:23 | |
| *** yamamoto has quit IRC | 12:25 | |
| *** yamamoto has joined #openstack-kuryr | 12:26 | |
| *** maysamacedos has joined #openstack-kuryr | 12:30 | |
| *** kzaitsev_pi has joined #openstack-kuryr | 12:30 | |
| *** yboaron has joined #openstack-kuryr | 12:31 | |
| *** yboaron_ has quit IRC | 12:33 | |
| *** kzaitsev_pi has quit IRC | 12:39 | |
| *** kzaitsev_pi has joined #openstack-kuryr | 12:40 | |
| *** janonymous has quit IRC | 12:42 | |
| *** yboaron_ has joined #openstack-kuryr | 12:50 | |
| *** yboaron has quit IRC | 12:53 | |
| *** atoth has quit IRC | 13:08 | |
| *** salv-orlando has joined #openstack-kuryr | 13:09 | |
| *** salv-orlando has quit IRC | 13:10 | |
| *** salv-orlando has joined #openstack-kuryr | 13:10 | |
| *** gcheresh_ has quit IRC | 13:12 | |
| *** atoth has joined #openstack-kuryr | 13:23 | |
| *** rh-jelabarre has quit IRC | 13:38 | |
| openstackgerrit | Yossi Boaron proposed openstack/kuryr-kubernetes master: OCP-Router: Ingress controller support https://review.openstack.org/536387 | 13:42 |
| *** garyloug has quit IRC | 13:45 | |
| *** rh-jelabarre has joined #openstack-kuryr | 13:46 | |
| *** natanbro has quit IRC | 13:55 | |
| *** livelace has quit IRC | 14:02 | |
| *** hongbin has joined #openstack-kuryr | 14:05 | |
| *** livelace has joined #openstack-kuryr | 14:05 | |
| *** atoth has quit IRC | 14:08 | |
| *** janki has quit IRC | 14:16 | |
| *** jchhatbar has joined #openstack-kuryr | 14:16 | |
| *** jchhatbar has quit IRC | 14:17 | |
| *** jchhatbar has joined #openstack-kuryr | 14:17 | |
| *** atoth has joined #openstack-kuryr | 14:22 | |
| openstackgerrit | Yossi Boaron proposed openstack/kuryr-kubernetes master: OCP-Router: Ingress controller and L7 router unit tests https://review.openstack.org/564123 | 14:34 |
| *** premsankar has joined #openstack-kuryr | 14:36 | |
| *** jchhatbar has quit IRC | 14:45 | |
| *** rh-jelabarre has quit IRC | 15:10 | |
| *** yboaron_ has quit IRC | 15:21 | |
| *** janki has joined #openstack-kuryr | 15:22 | |
| *** salv-orlando has quit IRC | 15:29 | |
| *** salv-orlando has joined #openstack-kuryr | 15:29 | |
| *** salv-orlando has quit IRC | 15:34 | |
| *** janki has quit IRC | 15:39 | |
| *** janki has joined #openstack-kuryr | 15:41 | |
| *** janki has quit IRC | 15:44 | |
| *** janki has joined #openstack-kuryr | 15:45 | |
| *** janki has quit IRC | 15:52 | |
| *** janki has joined #openstack-kuryr | 15:59 | |
| *** maysamacedos has quit IRC | 16:56 | |
| *** salv-orlando has joined #openstack-kuryr | 17:08 | |
| livelace | apuimedo, Ping | 17:10 |
| apuimedo | livelace: pong | 17:17 |
| livelace | apuimedo, Sorry for interruption, but I see that CNI daemon have an error with initial starting | 17:20 |
| apuimedo | livelace: do you have a stacktrace? | 17:21 |
| livelace | apuimedo, https://paste.fedoraproject.org/paste/psSZzzWysGDG5BKdTqGBZg | 17:21 |
| livelace | As you can see we got CONTAINERID using --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt | 17:22 |
| livelace | But after that we got an SSL error, during access k8s API | 17:23 |
| livelace | But if I run another one kuryr-daemon inside that container with different ports I get: | 17:23 |
| livelace | https://paste.fedoraproject.org/paste/mVChr~AE3aTsZzLsBGXwlQ | 17:24 |
| livelace | There is no SSL problem anymore | 17:24 |
| livelace | And I don't understand why | 17:24 |
| livelace | *yet | 17:24 |
| apuimedo | this is really strange | 17:25 |
| livelace | During deployment of CNI daemons I set ssl_verify_server_crt = False | 17:26 |
| livelace | But it doesn't help | 17:26 |
| apuimedo | livelace: maybe it is only used in the actual client | 17:27 |
| apuimedo | not in the health check | 17:27 |
| * apuimedo checking | 17:27 | |
| livelace | There are k8s = clients.get_kubernetes_client() everywhere | 17:28 |
| livelace | And k8s_client has response = requests.get(url, cert=self.cert, | 17:28 |
| livelace | verify=self.verify_server, | 17:28 |
| livelace | headers=header) | 17:28 |
| apuimedo | that's right | 17:29 |
| *** maysamacedos has joined #openstack-kuryr | 17:30 | |
| livelace | And the second CNI daemon inside the same container/daemonset looks fine | 17:30 |
| livelace | When I started it, webconsole container got IP-address | 17:31 |
| apuimedo | could you check the /proc/num_proc/cmdline of the working and the non-working? | 17:31 |
| livelace | original: /usr/bin/kuryr-daemon --config-file /etc/kuryr/kuryr.conf | 17:34 |
| livelace | second: /usr/bin/kuryr-daemon --config-file /tmp/1.conf | 17:34 |
| apuimedo | livelace: can you paste me a diff of the config files? | 17:34 |
| livelace | the second has config where I just changed port numbers | 17:34 |
| livelace | apuimedo, https://paste.fedoraproject.org/paste/xgOyGeQnF~mIO1JPCrCifw | 17:35 |
| apuimedo | livelace: did you try which of the three changes makes a difference? | 17:38 |
| livelace | apuimedo, Do think that ports make sense ? | 17:38 |
| apuimedo | I would try with only the 8091 change to see if it is just that | 17:39 |
| *** salv-orlando has quit IRC | 17:39 | |
| *** salv-orlando has joined #openstack-kuryr | 17:40 | |
| *** salv-orlando has quit IRC | 17:44 | |
| apuimedo | livelace: since the cni daemon runs on the host namespace, did you check if something else runs on 8090? | 18:09 |
| apuimedo | or is it just a race? | 18:09 |
| *** gcheresh_ has joined #openstack-kuryr | 18:09 | |
| livelace | apuimedo, Of course I check, I wouldn't have any running cni daemons/contollers, if they all listen the same port | 18:10 |
| apuimedo | so, did you do the previous check I asked you of starting the second daemon only changing the health server port? | 18:11 |
| livelace | apuimedo, Yes, I set port number 8091 in config and in deployment config | 18:13 |
| apuimedo | and just with this change it worked? | 18:14 |
| apuimedo | without touching the bind_address settings? | 18:14 |
| livelace | No. There is the same problem | 18:15 |
| apuimedo | ok, so then the problem will be with the bind_address, most likely | 18:15 |
| livelace | Now, I'm trying to remove any readiness/liveness checks from daemonset | 18:15 |
| apuimedo | for downstream we don't use the cni health checks | 18:16 |
| apuimedo | cause it didn't make it into stable/queens iirc | 18:16 |
| livelace | Hm | 18:17 |
| livelace | https://paste.fedoraproject.org/paste/O6Di7KevEqsOg2kJPwzuUw | 18:18 |
| livelace | I have removed readiness/liveness checks from daemonset | 18:19 |
| livelace | I didn't change any port number, I stay it 8091 | 18:19 |
| apuimedo | livelace: which image is this? | 18:19 |
| livelace | latest | 18:19 |
| apuimedo | kuryr/cni:latest | 18:19 |
| apuimedo | ok | 18:19 |
| apuimedo | well, I mean, it makes sense that if the error is in the CNI health check it would not happen if the kubelet does not probe it | 18:20 |
| livelace | apuimedo, I'm going to deploy cluster from the beginning with disabled readiness/liveness | 18:22 |
| apuimedo | livelace: enable_kuryr_cni_probes is the setting to set to false | 18:22 |
| apuimedo | livelace: how long does the deployment take for you? | 18:23 |
| livelace | apuimedo, I know, I explored those things enough :) | 18:23 |
| livelace | apuimedo, About 30 minutes | 18:24 |
| livelace | apuimedo, I can write about results tomorrow, I decided to have working installation :) | 18:25 |
| apuimedo | livelace: perfect | 18:25 |
| apuimedo | let me know how it goes | 18:25 |
| apuimedo | I'll leave IRC on | 18:25 |
| livelace | apuimedo, Ok, thank you for your answers | 18:25 |
| apuimedo | if you get results earlier, maybe I can look at the problem tonight | 18:25 |
| apuimedo | no, thank you for the issue report. It is much appreciated! | 18:25 |
| livelace | Ok | 18:25 |
| *** livelace has quit IRC | 18:26 | |
| *** gcheresh_ has quit IRC | 18:29 | |
| *** salv-orlando has joined #openstack-kuryr | 18:40 | |
| *** salv-orlando has quit IRC | 18:45 | |
| *** janki has quit IRC | 18:52 | |
| *** atoth has quit IRC | 19:00 | |
| *** livelace has joined #openstack-kuryr | 19:12 | |
| livelace | apuimedo, First glance - it's working. deployment was born container, container has IP-address, service was created in Openstack, that service responds :) | 19:16 |
| livelace | Deletion working fine, Lbass was deleted, port is free. | 19:17 |
| livelace | I think, smoke test passed :) | 19:17 |
| livelace | But, more things I can say when I deploy production cluster and launch many containers and virtual machines on it. | 19:18 |
| livelace | apuimedo, Thanks again! | 19:18 |
| *** livelace has quit IRC | 19:18 | |
| *** gcheresh_ has joined #openstack-kuryr | 19:28 | |
| *** salv-orlando has joined #openstack-kuryr | 19:41 | |
| *** salv-orlando has quit IRC | 19:46 | |
| *** yboaron_ has joined #openstack-kuryr | 20:08 | |
| *** yboaron_ has quit IRC | 20:29 | |
| *** gcheresh_ has quit IRC | 20:33 | |
| *** salv-orlando has joined #openstack-kuryr | 20:42 | |
| *** salv-orlando has quit IRC | 20:46 | |
| *** maysamacedos has quit IRC | 21:25 | |
| *** salv-orlando has joined #openstack-kuryr | 21:43 | |
| *** salv-orlando has quit IRC | 21:48 | |
| *** salv-orlando has joined #openstack-kuryr | 22:36 | |
| *** rh-jelabarre has joined #openstack-kuryr | 22:51 | |
| *** hongbin has quit IRC | 22:59 | |
| *** maysamacedos has joined #openstack-kuryr | 23:09 | |
| *** pmannidi has joined #openstack-kuryr | 23:15 | |
| *** salv-orlando has quit IRC | 23:28 | |
| *** salv-orlando has joined #openstack-kuryr | 23:29 | |
| *** salv-orlando has quit IRC | 23:33 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!