| *** salv-orlando has quit IRC | 00:05 | |
| *** salv-orlando has joined #openstack-kuryr | 00:06 | |
| *** yamamoto has joined #openstack-kuryr | 00:08 | |
| *** salv-orlando has quit IRC | 00:10 | |
| *** aojea has joined #openstack-kuryr | 00:45 | |
| *** aojea has quit IRC | 00:49 | |
| *** caowei has joined #openstack-kuryr | 01:14 | |
| *** kiennt26 has joined #openstack-kuryr | 01:34 | |
| *** salv-orlando has joined #openstack-kuryr | 02:07 | |
| *** vikasc has joined #openstack-kuryr | 02:12 | |
| *** salv-orlando has quit IRC | 02:12 | |
| *** dougbtv__ has quit IRC | 02:45 | |
| *** aojea has joined #openstack-kuryr | 02:46 | |
| *** pc_m has quit IRC | 02:50 | |
| *** pc_m has joined #openstack-kuryr | 02:50 | |
| *** alraddarla has quit IRC | 02:50 | |
| *** dmellado has quit IRC | 02:51 | |
| *** leifmadsen has quit IRC | 02:51 | |
| *** aojea has quit IRC | 02:52 | |
| *** oanson has quit IRC | 02:53 | |
| *** leifmadsen has joined #openstack-kuryr | 02:57 | |
| *** dmellado has joined #openstack-kuryr | 02:57 | |
| *** oanson has joined #openstack-kuryr | 03:00 | |
| *** salv-orlando has joined #openstack-kuryr | 03:08 | |
| *** salv-orlando has quit IRC | 03:13 | |
| *** premsankar has quit IRC | 03:14 | |
| *** premsankar has joined #openstack-kuryr | 03:15 | |
| *** nhlfr has quit IRC | 03:15 | |
| *** Guest49950 has quit IRC | 03:15 | |
| *** alraddarla has joined #openstack-kuryr | 03:26 | |
| *** huats has joined #openstack-kuryr | 03:53 | |
| *** huats_ has quit IRC | 03:56 | |
| *** salv-orlando has joined #openstack-kuryr | 04:09 | |
| *** salv-orlando has quit IRC | 04:14 | |
| *** aojea has joined #openstack-kuryr | 04:32 | |
| *** aojea has quit IRC | 04:38 | |
| *** salv-orlando has joined #openstack-kuryr | 05:10 | |
| *** salv-orlando has quit IRC | 05:14 | |
| *** deepika has joined #openstack-kuryr | 05:38 | |
| *** deepika is now known as Guest57502 | 05:38 | |
| *** nhlfr has joined #openstack-kuryr | 05:47 | |
| *** janki has joined #openstack-kuryr | 06:03 | |
| *** salv-orlando has joined #openstack-kuryr | 06:12 | |
| *** salv-orl_ has joined #openstack-kuryr | 06:13 | |
| *** salv-orlando has quit IRC | 06:22 | |
| *** yamamoto has quit IRC | 07:19 | |
| *** yamamoto has joined #openstack-kuryr | 07:21 | |
| *** yamamoto has quit IRC | 07:26 | |
| *** salv-orl_ has quit IRC | 07:37 | |
| *** salv-orlando has joined #openstack-kuryr | 07:37 | |
| *** salv-orlando has quit IRC | 07:42 | |
| *** yboaron_ has joined #openstack-kuryr | 07:51 | |
| *** premsankar has quit IRC | 07:55 | |
| *** danil has joined #openstack-kuryr | 08:15 | |
| *** danil has quit IRC | 08:16 | |
| *** yamamoto has joined #openstack-kuryr | 08:24 | |
| dulek | ltomasbo: Hi, I've sent a CNI daemon commit with updated timeouts yesterday. If that's not a lot of trouble I'd appreciate if you could give it a spin. | 08:25 |
|---|---|---|
| dulek | I've also tried to create 50 pods at once instead of 30 and everything broke. I guess it's natural there's a limit somewhere, especially with just a single kuryr-kubernetes. | 08:26 |
| *** yamamoto has quit IRC | 08:34 | |
| *** salv-orlando has joined #openstack-kuryr | 08:36 | |
| *** jistr is now known as jistr|mtg | 08:48 | |
| *** garyloug has joined #openstack-kuryr | 09:04 | |
| *** yamamoto has joined #openstack-kuryr | 09:04 | |
| irenab | dulek, what do you mean by everything broke | 09:07 |
| dulek | irenab: Controller wasn't catching up with annotating VIFs, so multiple CNI requests timed out. Same on the side of CNI daemon not being able to answer all the requests from CNI driver in time. | 09:08 |
| dulek | irenab: So I've ended up with maybe half of the pods running. | 09:09 |
| irenab | dulek, sounds bad .. | 09:09 |
| dulek | irenab: And rest of them in various error states. | 09:09 |
| dulek | irenab: I can try to run that without CNI daemon, let me try. | 09:10 |
| *** yamamoto has quit IRC | 09:13 | |
| dulek | I need to restack first… | 09:15 |
| *** yamamoto has joined #openstack-kuryr | 09:25 | |
| *** lihi has quit IRC | 09:29 | |
| *** yamamoto has quit IRC | 09:31 | |
| *** lihi has joined #openstack-kuryr | 09:32 | |
| *** yamamoto has joined #openstack-kuryr | 09:38 | |
| dulek | irenab: When run without daemon it's a bit better, but still 5/50 pods failed. I cannot even identify errors as it's mostly IPDB raising RuntimeError without info or some brs missing on host. | 09:46 |
| dulek | I'll check what happens with CNI daemon on fresh stack. | 09:46 |
| irenab | I hoped that Daemon wil improve the situation | 09:47 |
| dulek | irenab: Unfortunately it looks like it uncovers some other problems. :( | 09:50 |
| irenab | dulek, this is good, since now problems must be fixed | 09:51 |
| irenab | as long as the root of the problem can be found ... | 09:52 |
| irenab | I expected k8s to try to recreate the failed Pod | 09:52 |
| dulek | irenab: It tries that for some time but if errors are recurring it eventually stops. | 09:53 |
| dulek | irenab: The problem is that in our case we're doing mostly the same between runs - VIF stays the same., | 09:53 |
| irenab | maybe poolling at CNI Daemon cannot be delayed and must be part of the solution .. | 09:54 |
| irenab | but lets see what you will be able to figure out as the source of the problems you ran into | 09:55 |
| *** jistr|mtg is now known as jistr | 10:15 | |
| *** yamamoto has quit IRC | 10:26 | |
| *** yamamoto has joined #openstack-kuryr | 10:31 | |
| *** yamamoto has quit IRC | 10:55 | |
| *** yamamoto has joined #openstack-kuryr | 11:14 | |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Add VIF-Handler And Drivers Design approach https://review.openstack.org/513715 | 11:16 |
| *** yamamoto has quit IRC | 11:38 | |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Add VIF-Handler And Drivers Design approach https://review.openstack.org/513715 | 11:51 |
| *** salv-orlando has quit IRC | 11:58 | |
| *** salv-orlando has joined #openstack-kuryr | 12:14 | |
| *** salv-orlando has quit IRC | 12:27 | |
| *** salv-orlando has joined #openstack-kuryr | 12:27 | |
| *** yamamoto has joined #openstack-kuryr | 12:31 | |
| *** salv-orlando has quit IRC | 12:32 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements https://review.openstack.org/516936 | 13:15 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements https://review.openstack.org/516937 | 13:16 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements https://review.openstack.org/509781 | 13:16 |
| *** salv-orlando has joined #openstack-kuryr | 13:28 | |
| *** yboaron__ has joined #openstack-kuryr | 13:28 | |
| *** yboaron_ has quit IRC | 13:30 | |
| *** salv-orlando has quit IRC | 13:32 | |
| *** danil has joined #openstack-kuryr | 13:56 | |
| *** danil has quit IRC | 13:56 | |
| *** danil has joined #openstack-kuryr | 13:57 | |
| *** yboaron_ has joined #openstack-kuryr | 14:00 | |
| *** atoth has joined #openstack-kuryr | 14:02 | |
| *** yboaron__ has quit IRC | 14:03 | |
| *** salv-orlando has joined #openstack-kuryr | 14:09 | |
| *** janki has quit IRC | 14:17 | |
| *** salv-orlando has quit IRC | 14:41 | |
| *** salv-orlando has joined #openstack-kuryr | 14:42 | |
| *** salv-orlando has quit IRC | 14:47 | |
| *** salv-orlando has joined #openstack-kuryr | 14:49 | |
| *** yamamoto has quit IRC | 15:05 | |
| *** yamamoto has joined #openstack-kuryr | 15:07 | |
| *** yamamoto has quit IRC | 15:11 | |
| *** alraddarla has quit IRC | 15:14 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Prevent pyroute2.IPDB threads leaking https://review.openstack.org/517406 | 15:15 |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: CNI split - introducing CNI daemon https://review.openstack.org/515186 | 15:15 |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Support kuryr-daemon when running containerized https://review.openstack.org/518024 | 15:15 |
| *** alraddarla_ has joined #openstack-kuryr | 15:17 | |
| dulek | irenab, ltomasbo: I've uploaded patch with some more optimizations. Switching from thread-based server to process-based increases the performance and I no longer see CommitException raised by pyrout2.ipdb. | 15:21 |
| dulek | In order to do that I needed to move from using multiprocessing.Queue to multiprocessing.Manager. But that's just an implementation detail, I think. | 15:22 |
| *** yamamoto has joined #openstack-kuryr | 15:22 | |
| *** yamamoto has quit IRC | 15:22 | |
| yboaron_ | dulek, aren't you in PTO today :-) | 15:29 |
| dulek | yboaron_: Hm. Why should I? :D | 15:43 |
| yboaron_ | dulek, that's what lpeer said at the meeting today | 15:43 |
| yboaron_ | the DFG meeting | 15:44 |
| *** yamamoto has joined #openstack-kuryr | 16:23 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: DNM: Trying to run gate tests with CNI Daemon https://review.openstack.org/509765 | 16:23 |
| *** yamamoto has quit IRC | 16:32 | |
| *** yboaron_ has quit IRC | 16:33 | |
| *** yamamoto has joined #openstack-kuryr | 17:27 | |
| *** yamamoto has quit IRC | 17:33 | |
| *** salv-orlando has quit IRC | 20:12 | |
| *** salv-orlando has joined #openstack-kuryr | 20:12 | |
| *** salv-orlando has quit IRC | 20:17 | |
| *** salv-orlando has joined #openstack-kuryr | 20:20 | |
| *** aojea has joined #openstack-kuryr | 20:35 | |
| *** danil has quit IRC | 20:35 | |
| *** jdavis has joined #openstack-kuryr | 21:08 | |
| *** garyloug has quit IRC | 21:28 | |
| *** aojea has quit IRC | 22:12 | |
| *** dougbtv__ has joined #openstack-kuryr | 22:14 | |
| *** dougbtv__ has quit IRC | 22:50 | |
| *** salv-orlando has quit IRC | 23:00 | |
| *** salv-orlando has joined #openstack-kuryr | 23:01 | |
| *** salv-orlando has quit IRC | 23:06 | |
| *** jdavis has quit IRC | 23:38 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!