| *** rh-jelabarre has quit IRC | 01:24 | |
| *** garyloug has quit IRC | 01:45 | |
| *** JJ-Club has joined #openstack-kuryr | 02:09 | |
| *** hongbin has joined #openstack-kuryr | 02:44 | |
| *** hongbin has quit IRC | 04:55 | |
| *** irenab has joined #openstack-kuryr | 05:43 | |
| *** gcheresh_ has joined #openstack-kuryr | 06:15 | |
| *** yboaron_ has joined #openstack-kuryr | 06:29 | |
| *** ccamposr has joined #openstack-kuryr | 07:22 | |
| *** openstackgerrit has quit IRC | 07:29 | |
| *** gcheresh_ has quit IRC | 07:50 | |
| *** openstackgerrit has joined #openstack-kuryr | 08:08 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Add needed driver and handler for policy gate https://review.openstack.org/624604 | 08:08 |
|---|---|---|
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-tempest-plugin master: Add network policy security group rules tests https://review.openstack.org/603618 | 08:12 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-tempest-plugin master: Add network policy security group rules tests https://review.openstack.org/603618 | 08:15 |
| *** gcheresh_ has joined #openstack-kuryr | 08:16 | |
| *** yboaron_ has quit IRC | 08:18 | |
| *** yboaron_ has joined #openstack-kuryr | 09:07 | |
| *** garyloug has joined #openstack-kuryr | 09:43 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Add needed driver and handler for policy gate https://review.openstack.org/624604 | 09:45 |
| *** gkadam has joined #openstack-kuryr | 10:05 | |
| *** gkadam has quit IRC | 10:05 | |
| openstackgerrit | Genadi Chereshnya proposed openstack/kuryr-tempest-plugin master: Working with noop for vif_pool_driver https://review.openstack.org/621411 | 10:25 |
| openstackgerrit | Maysa de Macedo Souza proposed openstack/kuryr-kubernetes master: Add support to match expressions in Network Policies https://review.openstack.org/624009 | 10:51 |
| *** maysams has joined #openstack-kuryr | 10:51 | |
| dulek | yboaron_: Ping! | 12:23 |
| dulek | yboaron_: I'm trying to tackle python 3.6 gate errors. I think timeouts on that gate are more often. | 12:24 |
| dulek | yboaron_: When you take a look at this log: http://logs.openstack.org/84/623184/1/gate/kuryr-kubernetes-tempest-daemon-octavia-py36/346910f/controller/logs/screen-kuryr-kubernetes.txt.gz#_Dec_11_14_17_39_648533 | 12:24 |
| dulek | And look for kuryr-service-471643875 - it seems that LBaaS spec is getting annotated, but no state ever is. | 12:24 |
| dulek | Also I can't find any try to create an LB with IP of 10.1.0.143, which corresponds to that svc. | 12:25 |
| dulek | Looks exactly as if it's being ignored. Any idea why can that happen? | 12:25 |
| dulek | Ooooh, maybe _has_pods is failing somehow…? This might be it… | 12:28 |
| yboaron_ | dulek, pong | 12:31 |
| dulek | yboaron_: Some info above. :) | 12:32 |
| yboaron_ | dulek, yep, checking | 12:32 |
| dulek | Oooh, this code is awful: https://github.com/openstack/kuryr-kubernetes/blob/300dc36b062aaa2c5eb1c3ee2f8c041e1e33a99d/kuryr_kubernetes/controller/handlers/lbaas.py#L323-L326 | 12:32 |
| dulek | But _has_pods should be fine. | 12:33 |
| dulek | yboaron_: port['name']? | 12:36 |
| dulek | yboaron_: https://github.com/openstack/kuryr-kubernetes/blob/300dc36b062aaa2c5eb1c3ee2f8c041e1e33a99d/kuryr_kubernetes/controller/handlers/lbaas.py#L312 | 12:36 |
| dulek | yboaron_: I cannot see 'name' in ports inside subsets, inside endpoints: http://logs.openstack.org/84/623184/1/gate/kuryr-kubernetes-tempest-daemon-octavia-py36/346910f/controller/logs/kubernetes/endpoints.txt.gz | 12:36 |
| yboaron_ | dulek, IIRC, name is optional | 12:37 |
| dulek | Okay. So looking at endpoint definition this check should pass as well… | 12:38 |
| yboaron_ | dulek, Does all service creation fail in py36 gate? | 12:39 |
| yboaron_ | dulek, I meant all services | 12:39 |
| dulek | yboaron_: I think that some succeed. | 12:39 |
| dulek | yboaron_: | 12:40 |
| dulek | 2018-12-11 14:24:12.320 26470 INFO kuryr_tempest_plugin.tests.scenario.base [-] Look for openstack.org/kuryr-lbaas-state string in ep=kuryr-service-168770576 annotation | 12:40 |
| dulek | 2018-12-11 14:24:16.555 26472 INFO kuryr_tempest_plugin.tests.scenario.base [-] We received 2 replies from prot=TCP;10.1.0.132:80 - connectivity was veified! | 12:40 |
| dulek | E.g. | 12:40 |
| dulek | Oh, it may be only a timeout waiting for LB or member to be created? | 12:41 |
| dulek | yboaron_: So lbaas-spec is annotated on endpoint on 14:17:39. | 12:48 |
| dulek | yboaron_: But no LB creation happens after that date in log of Octavia API. | 12:48 |
| yboaron_ | dulek, we should see the LB creation POST request at OCTAVIA | 12:49 |
| *** rh-jelabarre has joined #openstack-kuryr | 12:50 | |
| dulek | yboaron_: That's what I'm looking for. Just a sec, I'll try to match all the POST's I see to a K8s svc. | 12:50 |
| yboaron_ | dulek, first service handler annotates the lbaas spec at endpoint, and after that the endpoint handler should do rest of the work | 12:50 |
| yboaron_ | dulek, by rest of the work I mean LB,listener, bla bla creation | 12:51 |
| dulek | yboaron_: Yes, so I only see 3 POST's on LB's in octavia-api logs. | 12:53 |
| dulek | yboaron_: One is the K8s API LB, two others are related to different services that passed connectivity test. | 12:53 |
| yboaron_ | dulek, do we have the endpoints spec anywhere? k8s logs? | 12:54 |
| dulek | yboaron_: The one for kuryr-service-471643875 is missing. | 12:54 |
| dulek | yboaron_: Sure: http://logs.openstack.org/84/623184/1/gate/kuryr-kubernetes-tempest-daemon-octavia-py36/346910f/controller/logs/kubernetes/endpoints.txt.gz | 12:54 |
| dulek | It's still there as it timed out on it. | 12:54 |
| dulek | So this one endpoint never got deleted. | 12:54 |
| yboaron_ | dulek, need to leave for 10 minsm will continue to look at this later | 12:57 |
| dulek | yboaron_: I've walked through the code and besides _should_ignore() there is nothing that could prevent POST from being set. | 13:06 |
| dulek | Other than endpoint notification not coming from K8s at all, of course. :P | 13:06 |
| dulek | yboaron_: _should_ignore() does some Python magic with any() and generators. Maybe Python 3 changes some behavior of those. | 13:07 |
| dulek | yboaron_: First I'll try to find such a failure on non-Python 3 gate. | 13:08 |
| dulek | If I won't find it, I'll add additional logs there and debug further. | 13:08 |
| openstackgerrit | Genadi Chereshnya proposed openstack/kuryr-tempest-plugin master: Testing port_pool_min and port_pool_max https://review.openstack.org/620060 | 13:08 |
| dulek | yboaron_: Okay, this is not tied to Python 3, we can see something similar here: http://logs.openstack.org/84/623184/1/check/kuryr-kubernetes-tempest-daemon-octavia/eac0c8c/controller/logs/kubernetes/endpoints.txt.gz | 13:26 |
| yboaron_ | dulek, this one is kubernetes endpoint, we create the LB for this service in deployment/plugin.sh | 13:30 |
| dulek | yboaron_: kuryr-service-1704367057 is not. | 13:31 |
| dulek | This run is pretty similar, but… we're missing a ton of Octavia API logs for some reason… | 13:31 |
| yboaron_ | dulek, that;s correct- but maybe it's related to py36 and the should_ignore code | 13:31 |
| dulek | yboaron_: Symptoms are the same… | 13:32 |
| yboaron_ | dulek, not sure that I followed all the details, do u see the same behavior also in non py3.6 gates? | 13:33 |
| dulek | yboaron_: Yes, on the same patch I've found kuryr-kubernetes-tempest-daemon-octavia run that has similar issue. | 13:33 |
| dulek | yboaron_: Aaaand… There's one pod that never got wired nor it's name appear in kuryr-controller log. | 13:34 |
| dulek | Daaamn, something's really screwed up. | 13:35 |
| yboaron_ | dulek, can u share the patch link? | 13:35 |
| dulek | Yeah, that pod never gets annotated. | 13:35 |
| dulek | yboaron_: You can always get the commit id from logs link: 623184 | 13:36 |
| dulek | yboaron_: So https://review.openstack.org/#/c/623184 | 13:36 |
| yboaron_ | dulek, OK!, 10x | 13:36 |
| dulek | Maybe watcher on controller dies on some point… | 13:37 |
| dulek | Hm, nope, services are watched fine. | 13:39 |
| dulek | It seems like it's loosing some of the notifications from K8s API. I wonder if there are etcd errors? | 13:39 |
| dulek | Nah, etcd looks pretty much okay. | 13:40 |
| yboaron_ | dulek, IIRc, K8S updates endpoints subset details only after relevant pods were wired, and we can see that subsets are updated at endpoints | 13:40 |
| *** garyloug_ has joined #openstack-kuryr | 13:40 | |
| dulek | yboaron_: But we see this stuff updated. | 13:40 |
| dulek | yboaron_: http://logs.openstack.org/84/623184/1/check/kuryr-kubernetes-tempest-daemon-octavia/eac0c8c/controller/logs/kubernetes/endpoints.txt.gz - it's pretty much updated. | 13:41 |
| dulek | Both pods are up and running. | 13:41 |
| yboaron_ | dulek, that's exactly what I meant, seems that pod handlers are fine | 13:42 |
| dulek | It's a different one that never got noticed by controller. | 13:42 |
| dulek | Waaaait, this one's pending… It's something with Kubernetes itself… | 13:43 |
| *** garyloug has quit IRC | 13:44 | |
| *** gcheresh_ has quit IRC | 14:00 | |
| dulek | Awwww, this seems to be etcd issue again. | 14:00 |
| *** garyloug_ has quit IRC | 14:01 | |
| yboaron_ | dulek, How do you know? | 14:01 |
| *** garyloug_ has joined #openstack-kuryr | 14:02 | |
| dulek | yboaron_: All the failed runs have higher number of timeouts in etcd and kubernetes-api logs. | 14:02 |
| dulek | yboaron_: I'm trying to confirm that and see if it's related to the cloud the gate is running on. | 14:02 |
| *** irclogbot_1 has quit IRC | 14:32 | |
| *** gcheresh_ has joined #openstack-kuryr | 14:38 | |
| *** irclogbot_1 has joined #openstack-kuryr | 14:42 | |
| *** irclogbot_1 has quit IRC | 14:55 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add support for podSelector https://review.openstack.org/623554 | 14:58 |
| *** garyloug_ has quit IRC | 15:03 | |
| *** garyloug_ has joined #openstack-kuryr | 15:04 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Testing etcd with highest IO priority https://review.openstack.org/624731 | 15:13 |
| *** maysams has quit IRC | 15:20 | |
| *** irclogbot_1 has joined #openstack-kuryr | 15:21 | |
| *** gcheresh_ has quit IRC | 15:41 | |
| *** maysams has joined #openstack-kuryr | 15:43 | |
| openstackgerrit | Yossi Boaron proposed openstack/kuryr-kubernetes master: Log k8s endpoints being ignored by Kuryr https://review.openstack.org/624749 | 16:15 |
| openstackgerrit | Merged openstack/kuryr-tempest-plugin master: Use kuryr-kubernetes-tempest-jobs project template https://review.openstack.org/623184 | 16:23 |
| *** yboaron_ has quit IRC | 16:55 | |
| *** ccamposr has quit IRC | 17:06 | |
| *** garyloug__ has joined #openstack-kuryr | 17:21 | |
| *** garyloug_ has quit IRC | 17:24 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add support for podSelector https://review.openstack.org/623554 | 17:51 |
| *** garyloug__ has quit IRC | 18:05 | |
| *** celebdor_ has left #openstack-kuryr | 19:55 | |
| *** yboaron_ has joined #openstack-kuryr | 20:56 | |
| *** yboaron_ has quit IRC | 21:24 | |
| *** s1061123 has joined #openstack-kuryr | 22:23 | |
| *** rh-jelabarre has quit IRC | 22:29 | |
| *** s1061123_ has quit IRC | 22:29 | |
| *** rh-jelabarre has joined #openstack-kuryr | 22:29 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!