| *** maysamacedos has quit IRC | 00:00 | |
| *** yamamoto has quit IRC | 00:03 | |
| *** rh-jelabarre has joined #openstack-kuryr | 00:08 | |
| *** salv-orlando has joined #openstack-kuryr | 00:11 | |
| *** salv-orlando has quit IRC | 00:16 | |
| *** s1061123 has quit IRC | 00:46 | |
| *** s1061123 has joined #openstack-kuryr | 00:50 | |
| *** yamamoto has joined #openstack-kuryr | 00:59 | |
| *** yamamoto has quit IRC | 01:05 | |
| *** salv-orlando has joined #openstack-kuryr | 01:11 | |
| *** salv-orlando has quit IRC | 01:15 | |
| *** hongbin has joined #openstack-kuryr | 01:21 | |
| *** phuoc has joined #openstack-kuryr | 02:24 | |
| *** yamamoto has joined #openstack-kuryr | 03:01 | |
| *** rh-jelabarre has quit IRC | 03:06 | |
| *** yamamoto has quit IRC | 03:07 | |
| *** hongbin has quit IRC | 03:54 | |
| *** yamamoto has joined #openstack-kuryr | 04:04 | |
| *** yamamoto has quit IRC | 04:09 | |
| *** pc_m has quit IRC | 04:29 | |
| *** dims has quit IRC | 04:30 | |
| *** pc_m has joined #openstack-kuryr | 04:33 | |
| *** dims has joined #openstack-kuryr | 04:35 | |
| *** gcheresh has joined #openstack-kuryr | 04:39 | |
| *** janki has joined #openstack-kuryr | 04:54 | |
| *** yamamoto has joined #openstack-kuryr | 05:05 | |
| *** CrayZee has joined #openstack-kuryr | 05:07 | |
| *** jchhatbar has joined #openstack-kuryr | 05:09 | |
| *** yamamoto has quit IRC | 05:12 | |
| *** janki has quit IRC | 05:13 | |
| *** ispp has joined #openstack-kuryr | 05:33 | |
| *** isssp has quit IRC | 05:36 | |
| *** snapiri- has joined #openstack-kuryr | 05:53 | |
| *** isssp has joined #openstack-kuryr | 05:53 | |
| *** ispp has quit IRC | 05:54 | |
| *** ispp has joined #openstack-kuryr | 05:55 | |
| *** CrayZee has quit IRC | 05:56 | |
| *** isssp has quit IRC | 05:58 | |
| *** openstackgerrit has quit IRC | 06:04 | |
| *** yamamoto has joined #openstack-kuryr | 06:09 | |
| *** yamamoto has quit IRC | 06:14 | |
| *** openstackgerrit has joined #openstack-kuryr | 06:52 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-tempest-plugin master: Skip service tests https://review.openstack.org/577821 | 06:52 |
|---|---|---|
| *** yamamoto has joined #openstack-kuryr | 06:56 | |
| phuoc | Hi, I tried kuryr-kubernetes by using devstack | 07:05 |
| phuoc | but I can not create pod | 07:05 |
| phuoc | I want to know this is an issue in kuryr-kubernetes now, or it is just in my environment :) | 07:06 |
| phuoc | thanks | 07:06 |
| *** aojea_ has joined #openstack-kuryr | 07:18 | |
| ltomasbo | hi phuoc, can you provide a bit more information? what type of error are you seeing? are you using Octavia? | 07:19 |
| *** pcaruana has joined #openstack-kuryr | 07:20 | |
| *** aojea_ has quit IRC | 07:20 | |
| dmellado | phuoc: any local.conf around? :D | 07:21 |
| phuoc | https://github.com/openstack/kuryr-kubernetes/blob/master/devstack/local.conf.sample | 07:23 |
| phuoc | I use this local.conf | 07:23 |
| phuoc | I think this is problem with Octavia | 07:24 |
| openstackgerrit | Peng Liu proposed openstack/kuryr-kubernetes master: Implement multi-vif driver https://review.openstack.org/575629 | 07:24 |
| ltomasbo | phuoc, yes, you need to add something like this to your local.conf: | 07:26 |
| phuoc | the amphora's IP address in k8s-service-net should be the same with cluster IP address | 07:27 |
| phuoc | Am I wrong? | 07:27 |
| ltomasbo | OCTAVIA_AMP_IMAGE_FILE="/tmp/test-only-amphora-x64-haproxy-centos-7.qcow2" | 07:27 |
| ltomasbo | OCTAVIA_AMP_IMAGE_SIZE=3 | 07:27 |
| ltomasbo | OCTAVIA_AMP_IMAGE_NAME="test-only-amphora-x64-haproxy-centos-7" | 07:27 |
| ltomasbo | and download to /tmp the qcow2 image before stacking: | 07:27 |
| ltomasbo | wget https://tarballs.openstack.org/octavia/test-images/test-only-amphora-x64-haproxy-centos-7.qcow2 | 07:27 |
| ltomasbo | phuoc, no, it should not be the same | 07:28 |
| phuoc | ltomasbo, thank you ^^ | 07:28 |
| openstackgerrit | Peng Liu proposed openstack/kuryr-kubernetes master: [WIP] Implement NPWG multi-vif driver https://review.openstack.org/578009 | 07:28 |
| ltomasbo | clusterIP will be one of the allowed address pairs that the amphora VM accepts | 07:28 |
| ltomasbo | phuoc, ^^ | 07:28 |
| openstackgerrit | Peng Liu proposed openstack/kuryr-kubernetes master: [WIP] Implement NPWG multi-vif driver https://review.openstack.org/578009 | 07:29 |
| phuoc | I have another question, Will this issue be fixed soon? :) | 07:31 |
| phuoc | because I can wait for 2 or 3 days | 07:34 |
| ltomasbo | phuoc, what issue? the octavia thing? that is not an issue on kuryr-kubernetes side I believe, so I cannot answer | 07:34 |
| ltomasbo | dmellado, do you have any info on that? | 07:34 |
| dmellado | what issue? | 07:35 |
| dmellado | phuoc: ? | 07:36 |
| dmellado | if you mean the amphora qcow2 image | 07:36 |
| phuoc | dmellado, yes | 07:37 |
| dmellado | that's unrelated to kuryr-kubernetes but rather to Otavia and DIB | 07:37 |
| phuoc | I got it | 07:37 |
| dmellado | basically, unless you provide a precreated image like that | 07:37 |
| dmellado | OCTAVIA_AMP_IMAGE_FILE and so | 07:37 |
| dmellado | DIB fails to generate the Octavia image, thus devstack fails | 07:37 |
| dmellado | I guess I'll update the sample local.conf, but if you check what do we do on the upstream infra is basically just the same | 07:38 |
| dmellado | https://github.com/openstack/kuryr-kubernetes/blob/master/playbooks/get_amphora_tarball.yaml | 07:38 |
| phuoc | is there problem with ubuntu xenial ? | 07:41 |
| *** yboaron_ has joined #openstack-kuryr | 07:43 | |
| phuoc | Ok, I got that | 07:45 |
| phuoc | thank you guys ^^ | 07:45 |
| *** pmannidi has quit IRC | 07:51 | |
| *** salv-orlando has joined #openstack-kuryr | 08:22 | |
| *** salv-orlando has quit IRC | 08:25 | |
| *** salv-orlando has joined #openstack-kuryr | 08:26 | |
| openstackgerrit | Genadi Chereshnya proposed openstack/kuryr-tempest-plugin master: Test restart of CNI and Controller kuryr pods https://review.openstack.org/552498 | 08:32 |
| *** salv-orlando has quit IRC | 08:40 | |
| *** garyloug has joined #openstack-kuryr | 08:49 | |
| *** jchhatbar is now known as janki | 09:01 | |
| openstackgerrit | Merged openstack/kuryr-tempest-plugin master: Skip service tests https://review.openstack.org/577821 | 09:20 |
| openstackgerrit | Shachar Snapiri proposed openstack/kuryr-kubernetes master: Explicitly set threaded=False when starting Flask https://review.openstack.org/577760 | 09:23 |
| openstackgerrit | Shachar Snapiri proposed openstack/kuryr-kubernetes master: Remove redundant line from dragonflow config https://review.openstack.org/577656 | 09:27 |
| *** salv-orlando has joined #openstack-kuryr | 09:41 | |
| *** salv-orlando has quit IRC | 09:46 | |
| *** jchhatbar has joined #openstack-kuryr | 10:17 | |
| *** janki has quit IRC | 10:19 | |
| *** salv-orlando has joined #openstack-kuryr | 10:42 | |
| dulek | dmellado: Crap, skipping tests isn't enough because it's resourceSetup that fails… :P | 10:45 |
| dmellado | dulek: yeah :\ | 10:46 |
| dmellado | I'm thinking about maybe setting a whole setting for this on tempest for now | 10:46 |
| dmellado | such as in the config | 10:46 |
| *** salv-orlando has quit IRC | 10:47 | |
| *** yamamoto has quit IRC | 10:58 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-tempest-plugin master: Explicitly disable service tests by default https://review.openstack.org/578062 | 10:59 |
| dmellado | dulek: ^^ | 10:59 |
| dmellado | that should handle it | 10:59 |
| dmellado | dammit | 10:59 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Explicitly set threaded=False when starting Flask https://review.openstack.org/577760 | 11:03 |
| *** yamamoto has joined #openstack-kuryr | 11:07 | |
| *** yamamoto has quit IRC | 11:30 | |
| *** yamamoto has joined #openstack-kuryr | 11:33 | |
| *** atoth has joined #openstack-kuryr | 11:38 | |
| *** rh-jelabarre has joined #openstack-kuryr | 11:38 | |
| openstackgerrit | Shachar Snapiri proposed openstack/kuryr-kubernetes master: Remove redundant line from dragonflow config https://review.openstack.org/577656 | 12:13 |
| dulek | yboaron_: Any progress with IP conflict investigation? | 12:22 |
| dulek | yboaron_: I've started looking and I see kuryr-kubernetes is doing multiple POST to /load-balancer/v2.0/lbaas/loadbalancers | 12:23 |
| dulek | yboaron_: It looks a bit like LB gets created, then some change is detected on Service and controller tries to create the LB again. | 12:25 |
| yboaron_ | dulek, well, not yet, lemme check it | 12:25 |
| dulek | yboaron_: This is the run I'm looking at: http://logs.openstack.org/60/577760/1/check/kuryr-kubernetes-tempest-daemon-containerized-octavia/a4a1048/ | 12:25 |
| dulek | yboaron_: Might be easier to communicate if we look on the same logs. | 12:26 |
| yboaron_ | dulek, OK, 10x, checking it .. | 12:26 |
| dulek | yboaron_: See e.g. this: http://logs.openstack.org/60/577760/1/check/kuryr-kubernetes-tempest-daemon-containerized-octavia/a4a1048/controller/logs/screen-o-api.txt.gz#_Jun_25_12_26_00_143396 | 12:28 |
| dulek | yboaron_: This is request ending with 201. | 12:28 |
| dulek | Then below there's a traceback for another POST that gets the IP conflict. | 12:29 |
| dulek | It clearly looks like Kuryr tries to create the LB twice. I guess that it's done with the same IP. | 12:29 |
| yboaron_ | Kuryr uses the IP (cluster IP) allocated by K8S | 12:30 |
| dulek | yboaron_: Ah, okay, though I don't think that invalidates my hypothesis? | 12:32 |
| yboaron_ | dulek, I think that for some reason the pool/subnet separation between K8S service IPAM and OCTAVIA amaphora allocation - not configured properly | 12:33 |
| dulek | yboaron_: Hm, okay, that should be pretty easy to check, lemme try. | 12:34 |
| dulek | yboaron_: Isn't it like Octavia just creates the amphora using the IP we've provided it with? | 12:39 |
| yboaron_ | dulek, Octavia creates another port from this subnet | 12:40 |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Add SR-IOV capabilities to VIF handler https://review.openstack.org/512280 | 12:41 |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Add SR-IOV binding driver to CNI https://review.openstack.org/512281 | 12:41 |
| *** maysamacedos has joined #openstack-kuryr | 12:46 | |
| yboaron_ | dulek, I can see that 'kuryr-kubernetes-tempest-octavia' gate is green, so I assume that it should be related to subnet pool separation | 12:48 |
| yboaron_ | dulek, Did u see the failure also on non containerized gate ? | 12:48 |
| dulek | yboaron_: Yeah, the failure is transient and shows up on every Octavia gate. | 12:49 |
| dulek | yboaron_: I'll try to find an example. | 12:49 |
| dulek | yboaron_: Hm, I can't find it now to be honest… | 12:53 |
| dulek | Maybe you're right. | 12:53 |
| yboaron_ | dulek, NP, I"ll dig at the latest zuul results of patch#577760 | 12:54 |
| *** yamamoto has quit IRC | 13:01 | |
| *** yamamoto has joined #openstack-kuryr | 13:29 | |
| *** snapiri- has quit IRC | 13:41 | |
| *** hongbin has joined #openstack-kuryr | 13:42 | |
| *** rh-jelabarre has quit IRC | 13:46 | |
| dulek | yboaron_: Okay, you were right! | 13:48 |
| dulek | yboaron_: I think I have the root cause. | 13:48 |
| dulek | yboaron_: kuryr-controller gets restarted. If we're unlucky we end up creating second LB with same IP. | 13:48 |
| yboaron_ | dulek, : BTW, checking https://review.openstack.org/#/c/577760/ latest Zuul failures, seems that tempest fails because 'service_tests_enabled' option is missing | 13:49 |
| dulek | yboaron_: I guess that's dmellado's fault. :) | 13:49 |
| dulek | yboaron_: There are two things to fix this. First - we should definitely fix the restarting issue. | 13:49 |
| dulek | yboaron_: But secondly - ensure_loadbalancer should check for old LB being created and delete/reuse it. | 13:49 |
| yboaron_ | dulek, yes, that's correct | 13:50 |
| dmellado | dulek: heh | 13:51 |
| dmellado | what did I do? | 13:51 |
| dmellado | xD | 13:51 |
| yboaron_ | dmellado, take a look at http://logs.openstack.org/60/577760/3/check/kuryr-kubernetes-tempest-octavia/a26cac0/testr_results.html.gz | 13:51 |
| yboaron_ | dmellado, 'oslo_config.cfg.NoSuchOptError: no such option service_tests_enabled in group [kuryr_kubernetes]' | 13:52 |
| dmellado | damn | 13:52 |
| dmellado | more and more bullshit of this | 13:52 |
| dmellado | xD | 13:52 |
| yboaron_ | dulek, A far as I remember in case kuryr-controller being restarted, LBaaS handler should not call LBaaS driver cause of : https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/controller/handlers/lbaas.py#L287 | 13:55 |
| dulek | yboaron_: I assume _is_lbaas_spec_in_sync returns false because some stuff wasn't finished yet? | 13:56 |
| yboaron_ | dulek, but I agree that for robustness LBaaS driver should handle this case | 13:56 |
| yboaron_ | dulek, most probably you are right | 13:56 |
| dulek | yboaron_: It's super tricky to write idempotent code for such case. | 13:57 |
| ltomasbo | dulek, yboaron_: I also see that most (if not all) the calls to lbaas client are not try/catched | 13:57 |
| dulek | yboaron_: But there's worse thing - I don't know at all why kuryr-controller gets restarted because we don't have logs for previous restarts. :D | 13:57 |
| ltomasbo | so, if they fail, the controller dies | 13:57 |
| dulek | ltomasbo: Not controller, just Handler. | 13:58 |
| ltomasbo | the handler dies and the kuryr-controller crashes | 13:58 |
| ltomasbo | besause the lbaas driver failing, right? | 13:58 |
| dulek | ltomasbo: We have RetryHandler in place to catch exceptions. | 13:58 |
| yboaron_ | dulek, we support HA, isn't it? standby should resume the work .. :-) | 13:58 |
| dulek | yboaron_: No HA in gate yet. ;) | 13:59 |
| yboaron_ | dulek, just kidding | 13:59 |
| dulek | Seems like readinessProbe fails… | 14:00 |
| *** jchhatbar is now known as janki | 14:01 | |
| ltomasbo | in my setup is the alive probe failing | 14:01 |
| ltomasbo | and that restarts the kuryr-controller | 14:01 |
| dulek | "http://158.69.95.158:8090/ready: dial tcp 158.69.95.158:8090: getsockopt: connection refused" | 14:01 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-tempest-plugin master: Explicitly disable service tests by default https://review.openstack.org/578062 | 14:01 |
| dulek | Liveness does the same. | 14:01 |
| dmellado | dulek: yboaron_ | 14:02 |
| dmellado | my fault | 14:02 |
| ltomasbo | this is what I see in my env: | 14:02 |
| dmellado | forgot to amend that | 14:02 |
| dmellado | xD | 14:02 |
| dulek | ltomasbo: Okay, awesome, so I can reproduce that locally? | 14:02 |
| dmellado | dulek: by going to ltomasbo house | 14:02 |
| ltomasbo | http://paste.openstack.org/show/724312/ | 14:02 |
| dmellado | and logging in | 14:02 |
| dmellado | xD | 14:02 |
| ltomasbo | it is not the root, as it restarted the controller and I lost the initial cause | 14:03 |
| ltomasbo | but this is the state that is left, and why the controller is not getting over it after restarts | 14:03 |
| *** rh-jelabarre has joined #openstack-kuryr | 14:03 | |
| dulek | ltomasbo: Oh my, what the heck is that. | 14:03 |
| ltomasbo | it cannot find the created loadbalancer when restarting it seems | 14:04 |
| dulek | ltomasbo: Restarting here could happen because of handler exploding a few times and liveness probe failing. | 14:04 |
| ltomasbo | perhaps it is octavia breaking us... | 14:04 |
| dulek | But - that's not what I see in the gate. | 14:04 |
| dulek | In the gate I see connection refused. | 14:04 |
| yboaron_ | dmellado, 10x for the update | 14:04 |
| yboaron_ | dulek, what/where logs are u checking/digging into? | 14:05 |
| dulek | yboaron_: kubelet. E.g. http://logs.openstack.org/60/577760/1/check/kuryr-kubernetes-tempest-daemon-containerized-octavia/a4a1048/controller/logs/screen-kubelet.txt.gz#_Jun_25_12_00_19_490424 | 14:06 |
| dulek | gcheresh: Hey man, by accident we've created ultimate kuryr-controller restarting machine in the gate. | 14:07 |
| dulek | gcheresh: We just don't know how it works yet. | 14:07 |
| dulek | gcheresh: But it already found issues in our code. xD | 14:08 |
| ltomasbo | lol | 14:09 |
| *** gcheresh has quit IRC | 14:10 | |
| dulek | Hey, looks like kubelet is trying port 8090, while the probes are on 8082 by default. We might have some mismatch here. | 14:11 |
| *** gcheresh has joined #openstack-kuryr | 14:11 | |
| ltomasbo | 8090 was for cni, right? | 14:11 |
| ltomasbo | and 8082 for controller, right? | 14:11 |
| ltomasbo | dulek, ^^ | 14:11 |
| dulek | ltomasbo: mhm. There's no mismatch. Just CNI health checks fail as well. | 14:12 |
| ltomasbo | dulek, I created an env with devstack, and I see both kuryr pods in bad status | 14:16 |
| ltomasbo | kube-system kuryr-cni-ds-hcvkd 0/1 Running 10 31m | 14:16 |
| ltomasbo | kube-system kuryr-controller-7db7c944fb-gh9vt 0/1 CrashLoopBackOff 13 31m | 14:16 |
| ltomasbo | and yes, I'm getting connection refused on port 8082 | 14:17 |
| ltomasbo | (on kubelet) | 14:18 |
| dulek | ltomasbo: Both pods are on host networking. | 14:18 |
| ltomasbo | Jun 26 10:17:33 kuryr-i24c-02.mpc.lab.eng.bos.redhat.com hyperkube[39681]: I0626 10:17:33.549062 39681 prober.go:111] Readiness probe for "kuryr-controller-7db7c944fb-gh9vt_kube-system(03a6f212-7947-11e8-84a0-842b2b2d5e58):controller" failed (failure): Get http://10.16.144.154:8082/ready: dial tcp 10.16.144.154:8082: getsockopt: connection refused | 14:18 |
| dulek | ltomasbo: Check out if you can reach that from localhost. | 14:18 |
| ltomasbo | curl 10.16.144.154:8082 | 14:19 |
| ltomasbo | curl: (7) Failed connect to 10.16.144.154:8082; Connection refused | 14:19 |
| ltomasbo | and yes, both are on hostnetworking | 14:20 |
| dulek | ltomasbo: Flask update broke another thing? | 14:20 |
| ltomasbo | yes, this is with your patch applied | 14:21 |
| dulek | ltomasbo: This looks fine: http://logs.openstack.org/60/577760/1/check/kuryr-kubernetes-tempest-daemon-containerized-octavia/a4a1048/controller/logs/kubernetes/pod_logs/kube-system-kuryr-cni-ds-pjmdd.txt.gz#_2018-06-25_12_40_03_674 | 14:21 |
| ltomasbo | Umm, in my case it is not working... | 14:22 |
| ltomasbo | and I see both containers crashing... | 14:22 |
| dulek | ltomasbo: Yes, yes, sure. I mean the logs from server initialization seem fine. | 14:23 |
| ltomasbo | Starting CNI health check server. | 14:23 |
| ltomasbo | * Serving Flask app "cni-health-daemon" (lazy loading) | 14:23 |
| ltomasbo | * Environment: production | 14:23 |
| ltomasbo | WARNING: Do not use the development server in a production environment. | 14:23 |
| ltomasbo | Use a production WSGI server instead. | 14:23 |
| ltomasbo | * Debug mode: off | 14:23 |
| ltomasbo | I also see the cni health check server starting | 14:23 |
| dulek | ltomasbo: Yeah, this looks fine, isn't it? | 14:23 |
| ltomasbo | but the controller dies after a while | 14:23 |
| ltomasbo | with no reason... | 14:24 |
| ltomasbo | these are the last 2 lines from the log | 14:24 |
| ltomasbo | 2018-06-26 14:17:26.565 37 INFO werkzeug [-] * Running on http://127.0.0.1:8090/ (Press CTRL+C to quit) | 14:24 |
| ltomasbo | 2018-06-26 14:17:31.676 29 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/dispatch.py:62 | 14:24 |
| ltomasbo | 2018-06-26 14:18:00.858 29 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/dispatch.py:62 | 14:24 |
| ltomasbo | and similarly for the kuryr-controller | 14:25 |
| ltomasbo | 2018-06-26 14:24:10.383 1 INFO kuryr_kubernetes.controller.service [-] Service 'KuryrK8sService' stopping | 14:26 |
| dulek | ltomasbo: The controller isn't dying. It's getting killed by kubelet due to liveness/readiness check being unreachable. | 14:26 |
| ltomasbo | 2018-06-26 14:24:10.387 1 INFO kuryr_kubernetes.controller.service [-] Service 'KuryrK8sService' stopped | 14:26 |
| ltomasbo | not even that I think | 14:26 |
| ltomasbo | let me remove the readiness/liveness probe to avoid those restarts | 14:26 |
| dulek | ltomasbo: Why? It gets SIGTERM, so starts shutting down. | 14:26 |
| dulek | ltomasbo: Good idea! | 14:26 |
| ltomasbo | dulek, you are right, seems it is working without the probes | 14:30 |
| yboaron_ | dulek, ltomasbo: could it be that the LBaaS Handler/driver exception (due to not being able to ensure already created LB ) is the root cause for all of this ? | 14:33 |
| yboaron_ | changing: | 14:33 |
| yboaron_ | https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/controller/drivers/lbaasv2.py#L404 | 14:33 |
| yboaron_ | to : | 14:33 |
| yboaron_ | except (n_exc.Conflict, n_exc.InternalServerError) | 14:33 |
| yboaron_ | dulek, ltomasbo : should fix the LBaaS driver ensure_LB for already created loadbalancer | 14:33 |
| dulek | yboaron_: I don't think it's the root cause. The root cause is that healthchecks endpoint is unreachable. | 14:34 |
| dulek | ltomasbo: Maybe it's time for netstat and checking if 8082 and 8090 are opened? | 14:34 |
| ltomasbo | dulek, not in my env... | 14:36 |
| ltomasbo | yboaron_, I think that will help with the issue, at least the one I'm hitting on my other env | 14:37 |
| yboaron_ | ltomasbo, what issue is it? | 14:37 |
| ltomasbo | problem at ensure_loadbalancer | 14:37 |
| ltomasbo | I saw t he loadbalancer was either created or in pending creation adn the controller could not start due to that | 14:38 |
| ltomasbo | probably it got restarted by some reason in between and then it cannot proceed further | 14:38 |
| ltomasbo | yboaron_, though, in my case I'm getting the error when performing the find at line 405 | 14:39 |
| ltomasbo | so, probably that should also be try/catch | 14:39 |
| yboaron_ | ltomasbo, yes, I assume controller restart and LBaaS state annotation weren't saved, otherwise the LB handler should stop at should_ignore | 14:40 |
| ltomasbo | I guess so | 14:40 |
| ltomasbo | and perhaps it was restarted by the probes | 14:40 |
| ltomasbo | that could explain why it didn't happen for juriarte, as he has the env without probes... | 14:41 |
| *** rh-jelabarre has quit IRC | 14:41 | |
| ltomasbo | but, in my case, deleting the svc and the loadbalancer, and then restarting the kuryr-controller fixed the issue | 14:41 |
| yboaron_ | dmellado, ping | 14:58 |
| dmellado | yboaron_: pong what's up? | 14:58 |
| yboaron_ | dmellado, all well , 10x | 14:58 |
| dmellado | \o/ | 14:59 |
| yboaron_ | dmellado, I want to add the openshift python client in the tempest plugin | 14:59 |
| yboaron_ | dmellado, https://github.com/openshift/openshift-restclient-python/blob/4de49b9c3c700ad3b8556b27aa948396ea4de55c/README.md | 14:59 |
| dmellado | yboaron_: should be done in a fairly similar way as I did with the k8s one | 14:59 |
| yboaron_ | dmellado, Yes , I saw that you just updated the requirements.txt file | 15:00 |
| dmellado | you can even make it depend on a config | 15:00 |
| dmellado | like | 15:00 |
| dmellado | default container_foo = k8s | 15:00 |
| dmellado | but then on the base | 15:00 |
| dmellado | it'd create either k8s or openshift base on what you'd like to | 15:01 |
| dmellado | and the openshift gates would override that | 15:01 |
| dmellado | oh, well | 15:01 |
| dmellado | The OpenShift client depends on the Kubernetes Python client, and as part of the installation process, the Kubernetes (K8s) client is automatically installed. | 15:01 |
| dmellado | it won't really hurt | 15:02 |
| yboaron_ | dmellado, Yep, so, should I add something also to requirements.txt ? | 15:03 |
| dmellado | yboaron_: sure, if the client is on pypi I suggest that you just add that to the requirementes | 15:03 |
| dmellado | requirements | 15:03 |
| yboaron_ | dmellado, I have the openshift python client in my local env, for ocp gate | 15:03 |
| dmellado | https://pypi.org/project/openshift/ | 15:03 |
| yboaron_ | dmellado, OK, 10x! | 15:04 |
| dmellado | yboaron_: you can just add like that | 15:05 |
| dmellado | but also check that it doesn't raise the k8s client req | 15:05 |
| dmellado | and after that I'd propose to add that to upper-requirements as well | 15:05 |
| dmellado | https://github.com/openstack/requirements/blob/master/upper-constraints.txt | 15:05 |
| yboaron_ | dmellado, OK, 10x! | 15:06 |
| dmellado | yboaron_: yw! | 15:06 |
| dmellado | ping me if you need help or clarification ;) | 15:06 |
| dmellado | yboaron_: in any case | 15:06 |
| dmellado | think that if your needs wouldn't be just enough with the k8s one | 15:07 |
| dmellado | thanks! | 15:07 |
| yboaron_ | dmellado, NP | 15:07 |
| dmellado | damn, the patch for skipping failed | 15:12 |
| dmellado | ltomasbo: could you take a look, I'll need to go soon-ish | 15:12 |
| *** janki has quit IRC | 15:13 | |
| ltomasbo | dmellado, seems it is missing the service_tests_enabled option at kuryr_kubernetes... | 15:14 |
| ltomasbo | oslo_config.cfg.NoSuchOptError: no such option service_tests_enabled in group [kuryr_kubernetes] | 15:14 |
| dmellado | ltomasbo: yeah, but I added it | 15:14 |
| dmellado | https://review.openstack.org/#/c/578062/2/kuryr_tempest_plugin/plugin.py | 15:15 |
| ltomasbo | dmellado, https://review.openstack.org/#/c/578062/2/kuryr_tempest_plugin/config.py | 15:21 |
| dmellado | ltomasbo: just realized it | 15:21 |
| dmellado | I need a coffee or time out | 15:21 |
| ltomasbo | dmellado, ^^ perhaps you should change service_tests by service_test_enabled... | 15:21 |
| ltomasbo | xD | 15:21 |
| dmellado | exactly | 15:22 |
| dmellado | will amend | 15:22 |
| dmellado | and then go away | 15:22 |
| dmellado | xD | 15:22 |
| ltomasbo | :D | 15:23 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-tempest-plugin master: Explicitly disable service tests by default https://review.openstack.org/578062 | 15:24 |
| dmellado | ltomasbo: dulek yboaron_ all tight | 15:24 |
| dmellado | right | 15:24 |
| dmellado | now it works | 15:24 |
| dmellado | could you please make it sure so those changes | 15:24 |
| dmellado | goes their way when this is done | 15:24 |
| dmellado | I mean https://review.openstack.org/#/c/577760/ | 15:24 |
| * dmellado goes away | 15:25 | |
| dmellado | ttyl! | 15:25 |
| dmellado | g'night y'all | 15:25 |
| *** garyloug_ has joined #openstack-kuryr | 15:29 | |
| *** garyloug has quit IRC | 15:32 | |
| *** garyloug__ has joined #openstack-kuryr | 15:36 | |
| *** yboaron_ has quit IRC | 15:37 | |
| *** garyloug_ has quit IRC | 15:39 | |
| *** garyloug__ has quit IRC | 15:41 | |
| *** pcaruana has quit IRC | 15:42 | |
| *** garyloug__ has joined #openstack-kuryr | 15:43 | |
| *** janki has joined #openstack-kuryr | 16:39 | |
| *** CrayZee has joined #openstack-kuryr | 16:49 | |
| *** CrayZee has quit IRC | 16:51 | |
| *** garyloug__ has quit IRC | 17:05 | |
| *** yamamoto has quit IRC | 17:12 | |
| *** yamamoto has joined #openstack-kuryr | 17:14 | |
| *** yamamoto has quit IRC | 17:20 | |
| openstackgerrit | Merged openstack/kuryr-tempest-plugin master: Explicitly disable service tests by default https://review.openstack.org/578062 | 17:59 |
| *** yamamoto has joined #openstack-kuryr | 18:16 | |
| *** yamamoto has quit IRC | 18:20 | |
| *** yamamoto has joined #openstack-kuryr | 18:21 | |
| *** dmellado has quit IRC | 18:32 | |
| *** gcheresh has quit IRC | 18:49 | |
| *** atoth has quit IRC | 18:53 | |
| *** janki has quit IRC | 19:04 | |
| *** gcheresh has joined #openstack-kuryr | 19:48 | |
| *** gcheresh has quit IRC | 20:20 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-tempest-plugin master: Disable load balancer service scenario https://review.openstack.org/578195 | 20:27 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Explicitly set threaded=False when starting Flask https://review.openstack.org/577760 | 20:30 |
| *** maysamacedos has quit IRC | 20:45 | |
| *** irenab has quit IRC | 21:51 | |
| *** irenab has joined #openstack-kuryr | 21:52 | |
| *** lihi has quit IRC | 22:22 | |
| *** lihi has joined #openstack-kuryr | 22:25 | |
| *** hongbin has quit IRC | 22:36 | |
| *** threestrands has joined #openstack-kuryr | 22:38 | |
| *** threestrands has quit IRC | 22:38 | |
| *** threestrands has joined #openstack-kuryr | 22:38 | |
| openstackgerrit | Merged openstack/kuryr-tempest-plugin master: Disable load balancer service scenario https://review.openstack.org/578195 | 22:47 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!