| *** karimb has quit IRC | 00:18 | |
| *** zengchen has quit IRC | 00:22 | |
| *** c00281451 has joined #openstack-kuryr | 00:34 | |
| *** aojea has joined #openstack-kuryr | 00:47 | |
| *** salv-orlando has joined #openstack-kuryr | 00:48 | |
| *** limao has joined #openstack-kuryr | 00:48 | |
| *** aojea has quit IRC | 00:51 | |
| *** salv-orlando has quit IRC | 00:52 | |
| *** aojea has joined #openstack-kuryr | 01:48 | |
| *** salv-orlando has joined #openstack-kuryr | 01:48 | |
| *** c00281451 has quit IRC | 01:52 | |
| *** aojea has quit IRC | 01:52 | |
| *** salv-orlando has quit IRC | 01:53 | |
| *** hongbin has joined #openstack-kuryr | 01:54 | |
| *** yamamoto has joined #openstack-kuryr | 01:56 | |
| *** limao has quit IRC | 02:14 | |
| *** limao_ has joined #openstack-kuryr | 02:14 | |
| *** wangbo has joined #openstack-kuryr | 02:28 | |
| *** gouthamr has joined #openstack-kuryr | 02:32 | |
| *** gouthamr has quit IRC | 02:42 | |
| *** gouthamr has joined #openstack-kuryr | 02:42 | |
| *** salv-orlando has joined #openstack-kuryr | 02:50 | |
| *** salv-orlando has quit IRC | 02:54 | |
| *** limao_ has quit IRC | 03:03 | |
| *** limao has joined #openstack-kuryr | 03:05 | |
| *** limao has quit IRC | 03:06 | |
| *** limao_ has joined #openstack-kuryr | 03:06 | |
| *** limao has joined #openstack-kuryr | 03:09 | |
| *** limao has quit IRC | 03:09 | |
| *** limao has joined #openstack-kuryr | 03:10 | |
| *** limao_ has quit IRC | 03:11 | |
| *** limao has quit IRC | 03:12 | |
| *** limao has joined #openstack-kuryr | 03:12 | |
| *** limao has quit IRC | 03:16 | |
| *** limao has joined #openstack-kuryr | 03:17 | |
| *** limao has quit IRC | 03:21 | |
| *** openstackgerrit has quit IRC | 03:22 | |
| *** limao has joined #openstack-kuryr | 03:22 | |
| *** limao has quit IRC | 03:22 | |
| *** goncalo has quit IRC | 03:25 | |
| *** phuoc_ has quit IRC | 03:31 | |
| *** goncalo has joined #openstack-kuryr | 03:32 | |
| *** yamamoto_ has joined #openstack-kuryr | 03:39 | |
| *** hongbin has quit IRC | 03:40 | |
| *** yamamoto has quit IRC | 03:42 | |
| *** aojea has joined #openstack-kuryr | 03:49 | |
| *** salv-orlando has joined #openstack-kuryr | 03:50 | |
| *** aojea has quit IRC | 03:54 | |
| *** salv-orlando has quit IRC | 03:55 | |
| *** kiennt26 has quit IRC | 03:57 | |
| *** kiennt26 has joined #openstack-kuryr | 03:57 | |
| *** janki has joined #openstack-kuryr | 04:36 | |
| *** vikasc has quit IRC | 04:46 | |
| *** kiennt26 has quit IRC | 04:47 | |
| *** limao has joined #openstack-kuryr | 04:47 | |
| *** aojea has joined #openstack-kuryr | 04:50 | |
| *** salv-orlando has joined #openstack-kuryr | 04:51 | |
| *** aojea has quit IRC | 04:54 | |
| *** salv-orlando has quit IRC | 04:57 | |
| *** vikasc has joined #openstack-kuryr | 04:59 | |
| *** wangbo has quit IRC | 05:09 | |
| *** yboaron__ has joined #openstack-kuryr | 05:10 | |
| *** salv-orlando has joined #openstack-kuryr | 05:15 | |
| *** pmannidi has quit IRC | 05:15 | |
| *** limao has quit IRC | 05:23 | |
| *** limao has joined #openstack-kuryr | 05:24 | |
| *** janki has quit IRC | 05:25 | |
| *** limao has quit IRC | 05:29 | |
| *** gouthamr has quit IRC | 05:34 | |
| *** aojea has joined #openstack-kuryr | 05:51 | |
| *** wangbo has joined #openstack-kuryr | 05:51 | |
| *** aojea has quit IRC | 05:55 | |
| *** kiennt26 has joined #openstack-kuryr | 06:17 | |
| *** janki has joined #openstack-kuryr | 06:24 | |
| *** salv-orlando has quit IRC | 06:33 | |
| *** salv-orlando has joined #openstack-kuryr | 06:35 | |
| *** openstackgerrit has joined #openstack-kuryr | 06:40 | |
| openstackgerrit | Yossi Boaron proposed openstack/kuryr-kubernetes master: Eliminate wrong ERROR report (in kuryr log file) when service of type LoadBalancer type is deleted https://review.openstack.org/512670 | 06:40 |
|---|---|---|
| *** pcaruana has joined #openstack-kuryr | 06:45 | |
| *** yboaron__ has quit IRC | 07:19 | |
| *** danil has joined #openstack-kuryr | 07:26 | |
| *** caowei has joined #openstack-kuryr | 07:29 | |
| *** aojea has joined #openstack-kuryr | 07:38 | |
| *** karimb has joined #openstack-kuryr | 07:41 | |
| *** egonzalez has joined #openstack-kuryr | 07:46 | |
| *** aojea has quit IRC | 07:48 | |
| *** wangbo has quit IRC | 07:49 | |
| *** wangbo has joined #openstack-kuryr | 07:50 | |
| *** caowei has quit IRC | 08:04 | |
| *** caowei has joined #openstack-kuryr | 08:05 | |
| *** wangbo has quit IRC | 08:06 | |
| *** jerms has quit IRC | 08:07 | |
| *** jerms has joined #openstack-kuryr | 08:07 | |
| *** wangbo has joined #openstack-kuryr | 08:08 | |
| *** phuoc_ has joined #openstack-kuryr | 08:17 | |
| *** wangbo has quit IRC | 08:23 | |
| *** wangbo has joined #openstack-kuryr | 08:27 | |
| *** yboaron__ has joined #openstack-kuryr | 08:30 | |
| *** wangbo has quit IRC | 08:31 | |
| *** yamamoto_ has quit IRC | 08:37 | |
| *** yboaron_ has joined #openstack-kuryr | 08:38 | |
| *** leyal has quit IRC | 08:38 | |
| *** yboaron__ has quit IRC | 08:39 | |
| *** yamamoto has joined #openstack-kuryr | 08:40 | |
| *** yamamoto has quit IRC | 08:40 | |
| *** aojea has joined #openstack-kuryr | 08:44 | |
| *** leyal has joined #openstack-kuryr | 08:45 | |
| *** garyloug has joined #openstack-kuryr | 08:45 | |
| *** wangbo has joined #openstack-kuryr | 08:46 | |
| *** aojea has quit IRC | 08:48 | |
| openstackgerrit | Yossi Boaron proposed openstack/kuryr-kubernetes master: Closes-Bug: #1714204 https://review.openstack.org/512636 | 08:57 |
| openstack | bug 1714204 in kuryr-kubernetes "Delete service/deployment causes exception" [Medium,Triaged] https://launchpad.net/bugs/1714204 - Assigned to yossib (yossi-boaron-1234) | 08:57 |
| openstackgerrit | Yossi Boaron proposed openstack/kuryr-kubernetes master: Eliminate wrong ERROR report when service of type LoadBalancer type is deleted https://review.openstack.org/512670 | 09:02 |
| *** yamamoto has joined #openstack-kuryr | 09:07 | |
| *** pmannidi has joined #openstack-kuryr | 09:09 | |
| *** yamamoto has quit IRC | 09:11 | |
| *** yamamoto has joined #openstack-kuryr | 09:17 | |
| *** wangbo has quit IRC | 09:17 | |
| *** yamamoto has quit IRC | 09:21 | |
| *** wangbo has joined #openstack-kuryr | 09:26 | |
| *** wangbo has quit IRC | 09:28 | |
| *** yamamoto has joined #openstack-kuryr | 09:28 | |
| *** caowei has quit IRC | 09:28 | |
| *** egonzalez has quit IRC | 09:31 | |
| *** wangbo has joined #openstack-kuryr | 09:33 | |
| apuimedo | dulek: how's the cni eexist investigation going? | 09:33 |
| dulek | apuimedo: Awful, as I'm just finding new bugs. | 09:34 |
| *** yamamoto_ has joined #openstack-kuryr | 09:34 | |
| dulek | apuimedo: To get things tested after a failed run I sometimes need to restack, as there seems to be one more bug: | 09:34 |
| dulek | If a vif never got plugged correctly, when deleting that pod kubelet will call CNI, CNI will spawn watcher thread and this thread never gets it's WATCH notification. | 09:35 |
| dulek | So everything fails with timeouts constantly. | 09:35 |
| apuimedo | dulek: yeah | 09:36 |
| apuimedo | and I learned of another bug today thinking in the shower | 09:36 |
| apuimedo | if you are waiting for the port to go active and in the meantime the pod is terminated | 09:37 |
| apuimedo | booom | 09:37 |
| apuimedo | we don't cancel that thread | 09:37 |
| apuimedo | so we just timeout | 09:37 |
| dulek | apuimedo: Yup, very similar thing, isn't it? | 09:37 |
| apuimedo | similar | 09:37 |
| dulek | apuimedo: I'd paraphrase a saying - there are 2 things that's hard in computer programming and distributed systems. ;) | 09:37 |
| *** yamamoto has quit IRC | 09:37 | |
| dulek | (and off-by-one errors) | 09:37 |
| apuimedo | the good tihng is that with daemonized cni it is easier to fix it :-) | 09:37 |
| apuimedo | dulek: and async | 09:38 |
| apuimedo | fucking callbacks | 09:38 |
| apuimedo | xD | 09:38 |
| apuimedo | dulek: so, the game plan should be | 09:38 |
| apuimedo | in the cni daemon we should probably keep in a data structure the events we've seen | 09:39 |
| apuimedo | the key should maybe be the pod id | 09:39 |
| apuimedo | and the value a list with the last events seen | 09:39 |
| apuimedo | so if when add to network is called we already saw the damn vif | 09:40 |
| apuimedo | we don't need to wait again | 09:40 |
| dulek | apuimedo: That's feasible once daemon will be watching /pods?host=host | 09:40 |
| apuimedo | if a delete action comes | 09:40 |
| apuimedo | dulek: that should happen ASAP, yes | 09:40 |
| apuimedo | it is of the utmost importance to change to that | 09:40 |
| apuimedo | oh, the value should be a tuple of the list of events and active thread | 09:41 |
| apuimedo | so it can be cancelled | 09:41 |
| apuimedo | so we should have a handler that on cancelling the thread it cleans up the resources | 09:41 |
| dulek | apuimedo: I think I've drafter something like that, just haven't saved it into a dict (event was saved in the handler). | 09:42 |
| dulek | apuimedo: So here's the issue: Watcher expects oslo.service.ThreadGroup instance. This means eventlet. | 09:42 |
| apuimedo | dulek: I know | 09:42 |
| dulek | apuimedo: Question is - do we fear eventlet so much, that it's worth to reimplement Watcher to be less eventlet-dependent? | 09:42 |
| dulek | I've tried running with eventlet.monkey_patch() and I see that pyroute2 fails. | 09:43 |
| dulek | But there's this: https://github.com/craneworks/python-pyroute2/blob/master/pyroute2/config/eventlet.py | 09:43 |
| dulek | (I should probably find it in Neutron as author states it's from there) | 09:44 |
| apuimedo | dulek: maybe you can write to svinota and put me in CC | 09:44 |
| *** aojea has joined #openstack-kuryr | 09:45 | |
| apuimedo | dulek: I wonder if we really gain much by using evenlet compared to just having pthreads | 09:45 |
| dulek | apuimedo: I doubt it, but I'm not sure if some standard lib have same interface as oslo.service? | 09:46 |
| apuimedo | very doubtful | 09:46 |
| dulek | apuimedo: Good news - I think I've overcame issue with IPDB. | 09:47 |
| apuimedo | you mean the eexist? Or with eventlet? | 09:47 |
| dulek | eexist. | 09:47 |
| apuimedo | how? | 09:48 |
| dulek | Okay, I didn't. :P I don't really know what's happening here, but it started to manifest again. | 09:48 |
| apuimedo | :-) | 09:48 |
| dulek | I thought chaning https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/cni/binding/base.py#L31-L40 to return new IPDB instance for each thread will help. | 09:48 |
| *** aojea has quit IRC | 09:49 | |
| apuimedo | shouldn't if it is due to timeouts | 09:49 |
| *** egonzalez has joined #openstack-kuryr | 09:50 | |
| dulek | apuimedo: You're right. But I'm seeing 2 types of errors. Second one is KeyError: u'tapdc9c0043-a2' - and for this IPDB per thread helps. | 09:50 |
| dulek | I'll dig a bit more… I'm really getting sick of BaseHTTPServer, I blame it for the timeouts. | 09:51 |
| apuimedo | dulek: wanna do gRPC? | 09:52 |
| apuimedo | it's in openstack/requirements | 09:52 |
| apuimedo | or you want to use the same wsgi as neutron and keystone? | 09:53 |
| dulek | apuimedo: Hard to decide. gRPC should do it. And if k8s is moving to that for CNI communication… | 09:54 |
| apuimedo | dulek: that's my thought | 09:55 |
| *** yamamoto_ has quit IRC | 09:56 | |
| *** yboaron__ has joined #openstack-kuryr | 10:01 | |
| *** pmannidi has quit IRC | 10:02 | |
| *** yboaron_ has quit IRC | 10:03 | |
| *** kiennt26 has quit IRC | 10:05 | |
| *** wangbo has quit IRC | 10:22 | |
| *** yamamoto has joined #openstack-kuryr | 10:26 | |
| *** yboaron_ has joined #openstack-kuryr | 10:34 | |
| *** yboaron__ has quit IRC | 10:36 | |
| *** egonzalez has quit IRC | 10:39 | |
| *** aojea has joined #openstack-kuryr | 10:45 | |
| *** aojea has quit IRC | 10:50 | |
| *** caowei has joined #openstack-kuryr | 10:52 | |
| *** atoth has joined #openstack-kuryr | 11:12 | |
| openstackgerrit | Yossi Boaron proposed openstack/kuryr-kubernetes master: Fix KeyError exception upon service/deployment deletion https://review.openstack.org/512636 | 11:19 |
| *** rwallner has joined #openstack-kuryr | 11:27 | |
| *** caowei has quit IRC | 11:31 | |
| *** wangbo has joined #openstack-kuryr | 11:34 | |
| *** wangbo has quit IRC | 11:43 | |
| *** aojea has joined #openstack-kuryr | 11:46 | |
| *** aojea has quit IRC | 11:50 | |
| *** wangbo has joined #openstack-kuryr | 11:55 | |
| *** phuoc_ has quit IRC | 11:58 | |
| *** yamamoto has quit IRC | 11:59 | |
| *** salv-orlando has quit IRC | 12:26 | |
| *** salv-orlando has joined #openstack-kuryr | 12:28 | |
| *** aojea has joined #openstack-kuryr | 12:47 | |
| *** aojea has quit IRC | 12:51 | |
| *** yamamoto has joined #openstack-kuryr | 12:59 | |
| *** ajo has quit IRC | 13:03 | |
| *** yamamoto has quit IRC | 13:07 | |
| dulek | apuimedo: Okay, simple question… How do you think a single file socket behaves with multiple upcoming TCP connections? | 13:08 |
| dulek | apuimedo: Because I'm now starting to think that this is the main issue - I don't get any timeouts where there aren't multiple requests from kubelet coming simultaneously. | 13:09 |
| *** wangbo has quit IRC | 13:20 | |
| openstackgerrit | Merged openstack/kuryr master: Allow multiple binding drivers https://review.openstack.org/508778 | 13:23 |
| *** gouthamr has joined #openstack-kuryr | 13:26 | |
| *** danil has quit IRC | 13:34 | |
| *** limao has joined #openstack-kuryr | 13:41 | |
| *** limao_ has joined #openstack-kuryr | 13:48 | |
| *** limao has quit IRC | 13:49 | |
| *** yamamoto has joined #openstack-kuryr | 13:54 | |
| *** yamamoto has quit IRC | 13:55 | |
| openstackgerrit | Merged openstack/kuryr-libnetwork master: Add installation guide for kuryr-libnetwork https://review.openstack.org/509655 | 13:58 |
| *** salv-orlando has quit IRC | 14:11 | |
| *** salv-orlando has joined #openstack-kuryr | 14:11 | |
| *** garyloug has quit IRC | 14:15 | |
| *** garyloug has joined #openstack-kuryr | 14:15 | |
| *** salv-orlando has quit IRC | 14:16 | |
| *** janki has quit IRC | 14:25 | |
| *** limao_ has quit IRC | 14:25 | |
| *** limao has joined #openstack-kuryr | 14:25 | |
| *** tonygunk has joined #openstack-kuryr | 14:43 | |
| apuimedo | dulek: afaik it should work | 14:46 |
| apuimedo | let me check | 14:46 |
| dulek | apuimedo: Mhm, I think so too after some research. I'm now looking if it isn't client's fault. | 14:47 |
| apuimedo | ok | 14:47 |
| *** aojea has joined #openstack-kuryr | 14:48 | |
| dulek | apuimedo: One of the funnier bugs I'm getting is kubelet/CNI driver starting to use all CPU. Restart doesn't help… | 14:49 |
| *** aojea has quit IRC | 14:53 | |
| *** yamamoto has joined #openstack-kuryr | 14:55 | |
| *** ajo has joined #openstack-kuryr | 15:00 | |
| apuimedo | :/ | 15:01 |
| apuimedo | dulek: maybe you can upgrade to latest hyperkube | 15:02 |
| apuimedo | gotta go pick up the kid from school | 15:02 |
| apuimedo | will check on things later | 15:02 |
| dulek | apuimedo: I'll try switching to normal INET socket first… If that doesn't help I'm puzzled. | 15:02 |
| dulek | apuimedo: Because that would mean the problem is… elsewhere? | 15:03 |
| *** yamamoto has quit IRC | 15:05 | |
| *** limao has quit IRC | 15:09 | |
| *** pcaruana has quit IRC | 15:19 | |
| *** hongbin has joined #openstack-kuryr | 15:27 | |
| dulek | apuimedo: Sweet mother of TPC/IP stacks - it works now! | 15:42 |
| dulek | apuimedo: I'll just switch the server to bind to 127.0.0.1:CONF.port_number and client to call that. Looks like it'll help. | 15:44 |
| dulek | apuimedo: That will be easiest way for now, then we can evaluate grpc or whatever. | 15:44 |
| *** yboaron_ has quit IRC | 15:44 | |
| apuimedo | dulek: how much did that help? | 15:45 |
| *** aojea has joined #openstack-kuryr | 15:49 | |
| *** aojea has quit IRC | 15:54 | |
| *** salv-orlando has joined #openstack-kuryr | 16:12 | |
| *** salv-orlando has quit IRC | 16:17 | |
| *** salv-orlando has joined #openstack-kuryr | 16:25 | |
| *** aojea has joined #openstack-kuryr | 16:50 | |
| *** aojea has quit IRC | 16:54 | |
| *** leyal has quit IRC | 17:04 | |
| *** leyal has joined #openstack-kuryr | 17:05 | |
| *** garyloug has quit IRC | 17:14 | |
| *** aojea has joined #openstack-kuryr | 17:22 | |
| *** phuoc_ has joined #openstack-kuryr | 18:02 | |
| *** tonygunk has quit IRC | 18:24 | |
| *** aojea has quit IRC | 18:30 | |
| *** aojea has joined #openstack-kuryr | 19:16 | |
| *** irenab has left #openstack-kuryr | 19:17 | |
| *** aojea has quit IRC | 19:21 | |
| *** jdavis has joined #openstack-kuryr | 19:28 | |
| *** atoth has quit IRC | 19:28 | |
| *** salv-orlando has quit IRC | 19:31 | |
| *** salv-orlando has joined #openstack-kuryr | 19:35 | |
| openstackgerrit | Hongbin Lu proposed openstack/kuryr-libnetwork master: [WIP] Introduce sriov port driver https://review.openstack.org/500436 | 19:40 |
| *** salv-orlando has quit IRC | 20:02 | |
| *** salv-orlando has joined #openstack-kuryr | 20:02 | |
| *** salv-orlando has quit IRC | 20:07 | |
| *** openstackgerrit has quit IRC | 20:17 | |
| *** salv-orlando has joined #openstack-kuryr | 20:18 | |
| *** rwallner has quit IRC | 20:31 | |
| *** rwallner has joined #openstack-kuryr | 20:33 | |
| *** rwallner_ has joined #openstack-kuryr | 20:34 | |
| *** rwallner has quit IRC | 20:37 | |
| *** rwallner_ has quit IRC | 20:38 | |
| *** gouthamr has quit IRC | 20:43 | |
| *** jdavis has quit IRC | 21:11 | |
| *** salv-orlando has quit IRC | 21:16 | |
| *** salv-orlando has joined #openstack-kuryr | 21:18 | |
| *** gouthamr has joined #openstack-kuryr | 21:20 | |
| *** salv-orlando has quit IRC | 21:21 | |
| *** salv-orlando has joined #openstack-kuryr | 21:23 | |
| *** rwallner has joined #openstack-kuryr | 22:05 | |
| *** rwallner has quit IRC | 22:09 | |
| *** salv-orlando has quit IRC | 22:33 | |
| *** gouthamr has quit IRC | 22:41 | |
| *** rwallner has joined #openstack-kuryr | 22:44 | |
| *** rwallner has quit IRC | 22:44 | |
| *** rwallner has joined #openstack-kuryr | 22:45 | |
| *** rwallner has quit IRC | 22:48 | |
| *** jdavis has joined #openstack-kuryr | 23:12 | |
| *** jdavis has quit IRC | 23:17 | |
| *** hongbin has quit IRC | 23:21 | |
| *** karimb has quit IRC | 23:21 | |
| *** salv-orlando has joined #openstack-kuryr | 23:34 | |
| *** salv-orlando has quit IRC | 23:39 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!