| *** tonanhngo has joined #openstack-kuryr | 00:01 | |
| *** tonanhngo has quit IRC | 00:07 | |
| *** tonanhngo has joined #openstack-kuryr | 00:09 | |
| *** tonanhngo has quit IRC | 00:14 | |
| *** tonanhngo has joined #openstack-kuryr | 00:16 | |
| *** tonanhngo has quit IRC | 00:20 | |
| *** tonanhngo has joined #openstack-kuryr | 00:22 | |
| *** tonanhngo has quit IRC | 00:27 | |
| *** caowei has joined #openstack-kuryr | 00:33 | |
| *** gouthamr has quit IRC | 01:06 | |
| *** yedongcan has joined #openstack-kuryr | 01:09 | |
| *** hyunsun has joined #openstack-kuryr | 01:30 | |
| *** atoth has quit IRC | 01:32 | |
| *** limao has joined #openstack-kuryr | 01:39 | |
| *** tonanhngo has joined #openstack-kuryr | 02:01 | |
| *** tonanhngo has quit IRC | 02:06 | |
| *** tonanhngo has joined #openstack-kuryr | 02:06 | |
| *** hongbin has joined #openstack-kuryr | 02:09 | |
| *** kiennt has joined #openstack-kuryr | 02:10 | |
| *** tonanhngo has quit IRC | 02:12 | |
| *** gouthamr has joined #openstack-kuryr | 02:25 | |
| *** kiennt has quit IRC | 02:40 | |
| hongbin | limao: hey liping, just let you know that a bug was created for the issue we discussed this morning: https://bugs.launchpad.net/kuryr-libnetwork/+bug/1703698 | 03:10 |
|---|---|---|
| openstack | Launchpad bug 1703698 in kuryr-libnetwork "Couldn't get docker plugin to work" [Undecided,New] | 03:10 |
| *** aojea has joined #openstack-kuryr | 03:13 | |
| *** aojea has quit IRC | 03:17 | |
| *** yedongcan has quit IRC | 03:40 | |
| *** kiennt has joined #openstack-kuryr | 03:45 | |
| *** gouthamr has quit IRC | 03:46 | |
| *** yboaron_ has joined #openstack-kuryr | 04:15 | |
| *** kiennt has quit IRC | 04:27 | |
| *** yamamoto has joined #openstack-kuryr | 04:33 | |
| *** hongbin has quit IRC | 04:35 | |
| *** pcaruana has joined #openstack-kuryr | 04:41 | |
| *** kiennt has joined #openstack-kuryr | 04:45 | |
| limao | hongbin: coped | 04:52 |
| limao | hongbin: copied ... ;-) | 04:52 |
| *** pcaruana has quit IRC | 04:58 | |
| *** aojea has joined #openstack-kuryr | 05:26 | |
| *** tonanhngo has joined #openstack-kuryr | 05:28 | |
| *** kiennt has quit IRC | 05:29 | |
| *** janki has joined #openstack-kuryr | 05:40 | |
| *** aojea has quit IRC | 06:07 | |
| *** kiennt has joined #openstack-kuryr | 06:08 | |
| *** kiennt_ has joined #openstack-kuryr | 06:21 | |
| *** kiennt has quit IRC | 06:22 | |
| *** yedongcan1 has joined #openstack-kuryr | 06:23 | |
| *** tonanhngo has quit IRC | 06:28 | |
| *** tonanhngo has joined #openstack-kuryr | 06:32 | |
| *** tonanhngo has quit IRC | 06:34 | |
| *** kiennt_ has quit IRC | 06:43 | |
| *** kiennt has joined #openstack-kuryr | 06:44 | |
| *** kiennt has quit IRC | 06:53 | |
| *** kiennt has joined #openstack-kuryr | 06:55 | |
| *** tonanhngo has joined #openstack-kuryr | 07:03 | |
| *** kiennt has quit IRC | 07:03 | |
| *** limao has quit IRC | 07:05 | |
| *** kiennt has joined #openstack-kuryr | 07:05 | |
| *** limao has joined #openstack-kuryr | 07:05 | |
| *** yedongcan has joined #openstack-kuryr | 07:06 | |
| *** yedongcan1 has quit IRC | 07:08 | |
| *** limao has quit IRC | 07:10 | |
| *** tonanhngo has quit IRC | 07:20 | |
| *** kiennt has quit IRC | 07:22 | |
| *** ivc_ has quit IRC | 07:23 | |
| *** aojea has joined #openstack-kuryr | 07:25 | |
| *** kiennt has joined #openstack-kuryr | 07:25 | |
| *** pcaruana has joined #openstack-kuryr | 07:31 | |
| *** egonzalez has joined #openstack-kuryr | 07:34 | |
| *** kzaitsev_ws has joined #openstack-kuryr | 07:36 | |
| *** roeyc has joined #openstack-kuryr | 07:38 | |
| *** pcaruana has quit IRC | 07:41 | |
| *** pcaruana has joined #openstack-kuryr | 07:45 | |
| *** limao has joined #openstack-kuryr | 07:56 | |
| *** gouthamr has joined #openstack-kuryr | 08:02 | |
| *** garyloug has joined #openstack-kuryr | 08:09 | |
| *** gouthamr has quit IRC | 08:21 | |
| *** pcaruana has quit IRC | 08:33 | |
| *** limao has quit IRC | 08:40 | |
| *** limao has joined #openstack-kuryr | 08:41 | |
| *** wangbo has joined #openstack-kuryr | 08:43 | |
| *** limao has quit IRC | 08:44 | |
| *** limao has joined #openstack-kuryr | 08:44 | |
| *** pcaruana has joined #openstack-kuryr | 08:46 | |
| *** limao has quit IRC | 08:48 | |
| *** limao has joined #openstack-kuryr | 08:48 | |
| *** kural has joined #openstack-kuryr | 08:51 | |
| *** limao has quit IRC | 08:53 | |
| dmellado | ltomasbo: o/ | 08:54 |
| dmellado | I'm checking the lbaas issue that we were showing and basically this is due that the fake_svc_kuryr has no endpoint | 08:54 |
| dmellado | and this chunk of code looks for it | 08:54 |
| dmellado | https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/controller/handlers/lbaas.py#L146 | 08:54 |
| dmellado | I'll be trying to put a endpoint for it on the devstack part too and hopefully we'll be able to close this | 08:55 |
| dmellado | yboaron_: ^^ | 08:55 |
| ltomasbo | dmellado, o/ | 08:55 |
| dmellado | it could be handy for you too | 08:55 |
| yboaron_ | dmellado, OK , I'll check it , 10x | 08:55 |
| ltomasbo | dmellado, at a first glance I thought it was related to this patch: https://review.openstack.org/#/c/467241/ | 08:55 |
| ltomasbo | as it is the one creating them | 08:56 |
| dmellado | it is | 08:56 |
| dmellado | 50 +kind: Service | 08:56 |
| dmellado | 51 +apiVersion: v1 | 08:56 |
| dmellado | 52 +metadata: | 08:56 |
| dmellado | 53 + name: "${fake_svc_name}" 54 + namespace: kube-system | 08:56 |
| dmellado | 55 +spec: | 08:56 |
| dmellado | 56 + type: ClusterIP | 08:56 |
| dmellado | 57 + clusterIP: "${router_ip}" | 08:56 |
| dmellado | 58 + ports: | 08:56 |
| dmellado | 59 + - protocol: TCP | 08:57 |
| dmellado | 60 + port: 80 | 08:57 |
| dmellado | ooops thought it would be more compact | 08:57 |
| dmellado | if you check this part it doesn't create any endpoint | 08:57 |
| ltomasbo | dmellado, you had a devstack ready for this, right? | 08:57 |
| ltomasbo | perhaps we can remove this or modify it | 08:57 |
| dmellado | yep, I'll be trying to create the endpoint manually | 08:57 |
| ltomasbo | and quickly check | 08:57 |
| dmellado | and if it fixes it I'll submit a patch | 08:58 |
| dmellado | that was what I was saying xD | 08:58 |
| ltomasbo | great! | 08:58 |
| * dmellado goes first to my 1:1, let's bet on if it'll be moved or not xD | 08:58 | |
| ltomasbo | :D | 08:58 |
| *** tonanhngo has joined #openstack-kuryr | 09:21 | |
| *** limao has joined #openstack-kuryr | 09:22 | |
| *** tonanhngo has quit IRC | 09:23 | |
| *** limao has quit IRC | 09:23 | |
| *** limao has joined #openstack-kuryr | 09:24 | |
| *** limao has quit IRC | 09:28 | |
| *** pcaruana has quit IRC | 09:32 | |
| *** pcaruana has joined #openstack-kuryr | 09:32 | |
| *** limao has joined #openstack-kuryr | 09:48 | |
| *** caowei has quit IRC | 09:53 | |
| *** hyunsun has quit IRC | 10:08 | |
| *** kiennt has quit IRC | 10:12 | |
| *** limao has quit IRC | 10:19 | |
| *** wangbo has quit IRC | 10:31 | |
| irenab | dmellado, I also have issues with the fake-service in the nested ... | 11:10 |
| *** atoth has joined #openstack-kuryr | 11:18 | |
| *** yedongcan has left #openstack-kuryr | 11:19 | |
| dmellado | irenab: also with the endpoint? | 11:20 |
| irenab | dmellado, initially looking for the specific subnet name to allocate VIP from | 11:30 |
| dmellado | irenab: in my case it seems that the fake svc has no endpoint | 11:31 |
| dmellado | thus kuryr-k8s complains | 11:31 |
| dmellado | I'm having a look | 11:31 |
| irenab | for me it seems that devstack ignores the setting in local.conf | 11:34 |
| irenab | dmellado, do you use apuimedo heat patch? | 11:42 |
| dmellado | irenab: nope, I'm just using a plain 'gate' config | 11:42 |
| *** wangbo has joined #openstack-kuryr | 11:51 | |
| *** ivc has joined #openstack-kuryr | 12:05 | |
| irenab | dmellado, I think now I hit the same problem as you | 12:23 |
| dmellado | heh, 'awesome', I'm looking into it (multi-tasking as of now) | 12:23 |
| dmellado | will try to put up a patch later | 12:23 |
| irenab | dmellado, what is supposed to be the proper state? | 12:26 |
| irenab | do you have some workaround? | 12:26 |
| dmellado | irenab: I'm trying to work it around by creating an endpoint for that service | 12:26 |
| irenab | I see kuryr-svc-router in the kube-system namspace woth the last IP in the services subnet with no endpoints | 12:27 |
| dmellado | exactly | 12:29 |
| irenab | I do not see lbaas either | 12:32 |
| irenab | on neutron side | 12:33 |
| ltomasbo | irenab, lbaas is created as admin | 12:37 |
| ltomasbo | instead of demo user | 12:37 |
| *** yedongcan has joined #openstack-kuryr | 12:37 | |
| ltomasbo | but it is there (at least in my case) | 12:37 |
| ltomasbo | perhaps that is the problem... | 12:37 |
| irenab | ltomasbo, I checked for admin :-) | 12:38 |
| ltomasbo | not there either? | 12:39 |
| ltomasbo | umm | 12:39 |
| irenab | but still having the wierd error when trying to create pod: https://pastebin.com/9LiDFyJs | 12:41 |
| *** yamamoto has quit IRC | 12:41 | |
| irenab | I think this is not related to the lbaas issue | 12:41 |
| ltomasbo | umm | 12:43 |
| ltomasbo | was it you also sending this error before, right? | 12:43 |
| ltomasbo | or someone else shared with me too a few days ago | 12:43 |
| *** rwallner has joined #openstack-kuryr | 12:43 | |
| *** rwallner has quit IRC | 12:43 | |
| *** rwallner has joined #openstack-kuryr | 12:44 | |
| irenab | me | 12:44 |
| irenab | changed to CentOS, but still have the same issue | 12:45 |
| irenab | it is in the nested env | 12:45 |
| irenab | dmellado, ltomasbo checking the plugin.sh, seems that neutron side lbaas is not supposed to be present for the fake-router-service | 12:57 |
| kzaitsev_ws | folks, I'm seeing strange behaviour of neutron. when kuryr-k8s requests a port for the 1st time it gets a port with 'binding failed' in its vif_type | 13:02 |
| kzaitsev_ws | have anyone seen smth like that before? | 13:02 |
| irenab | kzaitsev_ws, BM or nested? | 13:08 |
| *** yamamoto has joined #openstack-kuryr | 13:08 | |
| *** yamamoto has quit IRC | 13:13 | |
| kzaitsev_ws | bm as usual | 13:19 |
| kzaitsev_ws | irenab: looks like my ovs-agent is down. or rather it's up, but neutron thinks it's down | 13:19 |
| ltomasbo | umm, that could be related to one of the neutron failures we are seeing somewhere else | 13:23 |
| *** tonanhngo has joined #openstack-kuryr | 13:23 | |
| ltomasbo | the openvswitch dies with a segfault and the neutron-agent fails to restart | 13:23 |
| ltomasbo | can you manually restart both of them? | 13:24 |
| ltomasbo | kzaitsev_ws, ^^ | 13:24 |
| *** wangbo has quit IRC | 13:25 | |
| *** tonanhngo has quit IRC | 13:25 | |
| kzaitsev_ws | ovs looks fine, I believe the problem might be with rabbit | 13:29 |
| kzaitsev_ws | as I see a lot of timeouts in the openvswitch logs | 13:30 |
| kzaitsev_ws | ovs itself looks ok btw, but restarting the agent helped | 13:30 |
| kzaitsev_ws | but | 13:30 |
| kzaitsev_ws | it doesn't report port as active | 13:30 |
| ltomasbo | it is connected to the VM/container? | 13:31 |
| ltomasbo | perhaps it failed to connect it | 13:31 |
| ltomasbo | we got a similar failure with the kubelet not plugging the container due to missing (not found) namespace | 13:31 |
| ltomasbo | and the port remained down | 13:31 |
| ltomasbo | even though the container was up and the ovs has the port connected | 13:31 |
| * kzaitsev_ws restarting stuff | 13:32 | |
| ltomasbo | :D | 13:34 |
| kzaitsev_ws | "Hello, it. Have you tried turning it off and on again?" (c) | 13:34 |
| ltomasbo | :D | 13:34 |
| ltomasbo | (almost) always work! | 13:35 |
| ltomasbo | let us know if that is the case! | 13:35 |
| *** yboaron_ has quit IRC | 13:38 | |
| *** garyloug has quit IRC | 13:39 | |
| *** garyloug has joined #openstack-kuryr | 13:39 | |
| *** kural_ has joined #openstack-kuryr | 13:39 | |
| *** kural has quit IRC | 13:43 | |
| dmellado | lol | 13:43 |
| dmellado | irenab: oh, hmmm.... | 13:44 |
| kzaitsev_ws | ugh | 13:49 |
| kzaitsev_ws | ok it can be 1.7 | 13:49 |
| kzaitsev_ws | has anyone tried 1.7 yet? =) | 13:49 |
| kzaitsev_ws | or it can be a problem in my code =/ since the cni is modified. however it used to work just fine on 1.6 and 1.5 branches =/ | 13:50 |
| *** yamamoto has joined #openstack-kuryr | 13:51 | |
| dmellado | kzaitsev_ws: what happened? | 13:56 |
| dmellado | again binding failed? | 13:56 |
| kzaitsev_ws | nope, that I solved with openvswitch agent restarts | 13:56 |
| kzaitsev_ws | now the CNI reports nothing to k8s. | 13:57 |
| kzaitsev_ws | it starts watching and immidiately stops | 13:57 |
| kzaitsev_ws | wathing api I mean | 13:58 |
| kzaitsev_ws | that can be caused either by api change in k8s or by my code. | 13:59 |
| kzaitsev_ws | the code on the minions haven't changed in a while, so I'm calling 1.7 the culprit. but will investigate | 14:00 |
| kzaitsev_ws | sidenote: my test cluster didn't feel good so we re-installed it. using kubeadm. and 1.7 is the version it installs | 14:01 |
| *** janki has quit IRC | 14:03 | |
| *** hongbin has joined #openstack-kuryr | 14:06 | |
| *** kzaitsev_ws has quit IRC | 14:09 | |
| *** kzaitsev_ws has joined #openstack-kuryr | 14:13 | |
| *** zengchen has joined #openstack-kuryr | 14:18 | |
| *** tonanhngo has joined #openstack-kuryr | 14:20 | |
| *** tonanhngo has quit IRC | 14:21 | |
| *** gouthamr has joined #openstack-kuryr | 14:35 | |
| *** roeyc has quit IRC | 15:12 | |
| *** yedongcan has left #openstack-kuryr | 15:29 | |
| kzaitsev_ws | omg | 15:36 |
| kzaitsev_ws | well the 1st problem was ovs-agent not working properly but now I've uncovered a bug in my multi-vif implementation in the CNI part %) | 15:37 |
| kzaitsev_ws | it's amazing that I've only hit it now! I should've been hit it day 1. | 15:39 |
| kzaitsev_ws | it also highlights the tests issue =( | 15:39 |
| *** zengchen has quit IRC | 15:40 | |
| *** aojea has quit IRC | 15:40 | |
| ltomasbo | what problem/error was? | 15:42 |
| *** yamamoto has quit IRC | 15:48 | |
| kzaitsev_ws | ltomasbo: before my patches the cni would always expect a single VIF, then if it's inactive it would expect it to become active and only then it would kill the api watcher | 15:49 |
| *** yamamoto has joined #openstack-kuryr | 15:49 | |
| kzaitsev_ws | now with my patch there may be multiple VIFs, so the logic has to be a bit more complex | 15:49 |
| kzaitsev_ws | for every API event. if there is at least 1 VIF that is not ready, then we should wait for more events. and only if every VIF turned to ready we can kill the watcher and stop recieving the events | 15:51 |
| *** yamamoto has quit IRC | 15:51 | |
| ltomasbo | ahh | 15:52 |
| ltomasbo | right | 15:52 |
| kzaitsev_ws | now the problem is — I decided, that it should be perfectly ok to ask for 0 VIFs. like if you're going to request some persistent volume for your pod and would only work with it? | 15:52 |
| ltomasbo | so, it stopped watching after the first one | 15:53 |
| kzaitsev_ws | nope =) | 15:53 |
| kzaitsev_ws | bear with me | 15:53 |
| ltomasbo | :D | 15:53 |
| kzaitsev_ws | or rather it depends on what you mean for 'stopped watching after the 1st one' =) | 15:53 |
| ltomasbo | :D | 15:54 |
| kzaitsev_ws | in fact yes. in my current implementation there is no way to decide if the Pod has just been created or that it has requested exactly 0 vifs (somehow, machanism is not in the patch intentionally) | 15:54 |
| kzaitsev_ws | so the watcher stopped after recieving the very 1st pod event =) the pod had no vifs annotation and the CNI thought that it is not requesting anything | 15:55 |
| kzaitsev_ws | so it killed the watcher %) | 15:56 |
| ltomasbo | ahh | 15:56 |
| ltomasbo | ok | 15:56 |
| *** egonzalez has quit IRC | 15:57 | |
| kzaitsev_ws | it didn't wait for event with vifs %) | 15:57 |
| kzaitsev_ws | funny thing is. I should've hit it long ago | 15:57 |
| ltomasbo | yep | 15:57 |
| ltomasbo | how do you managed to avoid it? | 15:57 |
| kzaitsev_ws | but seeminglly kuryr-k8s was super-fast and managed to launch the event with vif *before* cni started to watch for it | 15:58 |
| kzaitsev_ws | I guess... | 15:58 |
| ltomasbo | :D | 15:58 |
| kzaitsev_ws | actually it's possible to differentiate two cases — if there is no vifs key in the annotation — it's a brand new Pod | 15:59 |
| kzaitsev_ws | if it's empty — the Pod is somehow requesting 0 VIFs | 15:59 |
| kzaitsev_ws | but I'll extract this logic to a separate patch | 16:00 |
| kzaitsev_ws | not even sure we need this flexibility | 16:00 |
| *** roeyc has joined #openstack-kuryr | 16:15 | |
| *** openstackgerrit has joined #openstack-kuryr | 16:20 | |
| openstackgerrit | Kirill Zaitsev proposed openstack/kuryr-kubernetes master: Allow passing multiple VIFs to CNI https://review.openstack.org/471012 | 16:20 |
| *** roeyc has quit IRC | 16:51 | |
| *** yamamoto has joined #openstack-kuryr | 16:51 | |
| *** pcaruana has quit IRC | 16:56 | |
| openstackgerrit | Kirill Zaitsev proposed openstack/kuryr-kubernetes master: Add SR-IOV capabilities to VIF handler https://review.openstack.org/462455 | 16:56 |
| openstackgerrit | Kirill Zaitsev proposed openstack/kuryr-kubernetes master: Add SR-IOV binding driver to CNI https://review.openstack.org/462456 | 16:56 |
| *** yamamoto has quit IRC | 17:02 | |
| openstackgerrit | Kirill Zaitsev proposed openstack/kuryr-kubernetes master: [DRAFT] Allow requesting additional VIFs via annotation https://review.openstack.org/482544 | 17:05 |
| *** aojea has joined #openstack-kuryr | 17:16 | |
| openstackgerrit | Jaivish Kothari(janonymous) proposed openstack/kuryr-kubernetes master: [WIP] CNI Daemon template https://review.openstack.org/480028 | 17:17 |
| *** aojea has quit IRC | 17:21 | |
| *** tonanhngo has joined #openstack-kuryr | 17:31 | |
| *** tonanhngo has quit IRC | 17:33 | |
| *** tonanhngo has joined #openstack-kuryr | 17:36 | |
| *** aojea has joined #openstack-kuryr | 17:37 | |
| *** aojea has quit IRC | 17:42 | |
| *** yboaron_ has joined #openstack-kuryr | 17:46 | |
| *** kural_ has quit IRC | 17:57 | |
| *** yamamoto has joined #openstack-kuryr | 18:04 | |
| *** kzaitsev_ws has quit IRC | 18:06 | |
| *** yamamoto has quit IRC | 18:20 | |
| *** yamamoto has joined #openstack-kuryr | 18:21 | |
| *** yamamoto has quit IRC | 18:26 | |
| *** atoth has quit IRC | 18:59 | |
| *** aojea has joined #openstack-kuryr | 19:03 | |
| *** yamamoto has joined #openstack-kuryr | 19:16 | |
| *** yamamoto has quit IRC | 19:54 | |
| *** yboaron_ has quit IRC | 20:42 | |
| *** yamamoto has joined #openstack-kuryr | 20:54 | |
| *** yamamoto has quit IRC | 21:01 | |
| *** rwallner has quit IRC | 21:04 | |
| *** aojea has quit IRC | 21:23 | |
| *** aojea has joined #openstack-kuryr | 21:32 | |
| *** aojea has quit IRC | 21:53 | |
| *** mattmceuen has joined #openstack-kuryr | 21:53 | |
| *** yuval has quit IRC | 22:35 | |
| *** yuval has joined #openstack-kuryr | 22:37 | |
| *** limao has joined #openstack-kuryr | 22:46 | |
| *** limao_ has joined #openstack-kuryr | 22:49 | |
| *** limao has quit IRC | 22:51 | |
| *** yamamoto has joined #openstack-kuryr | 22:52 | |
| *** mattmceuen has quit IRC | 23:00 | |
| *** limao_ has quit IRC | 23:08 | |
| *** limao has joined #openstack-kuryr | 23:09 | |
| *** ivc has quit IRC | 23:10 | |
| *** limao has quit IRC | 23:14 | |
| *** ivc has joined #openstack-kuryr | 23:15 | |
| *** tonanhngo has quit IRC | 23:27 | |
| *** tonanhngo has joined #openstack-kuryr | 23:32 | |
| *** garyloug has quit IRC | 23:35 | |
| *** tonanhngo has quit IRC | 23:37 | |
| *** mattmceuen has joined #openstack-kuryr | 23:39 | |
| *** hongbin has quit IRC | 23:40 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!