Tuesday, 2017-11-14

*** aojea has joined #openstack-kuryr00:02
*** aojea has quit IRC00:07
openstackgerritHongbin Lu proposed openstack/kuryr-libnetwork master: Allow RequiresMACAddress to be configurable  https://review.openstack.org/51951000:09
*** hongbin has quit IRC00:23
*** salv-orlando has joined #openstack-kuryr00:38
*** salv-orlando has quit IRC00:43
*** caowei has joined #openstack-kuryr01:03
*** kiennt26 has joined #openstack-kuryr01:37
*** salv-orlando has joined #openstack-kuryr01:39
*** salv-orlando has quit IRC01:44
*** kiennt26 has quit IRC01:54
*** aojea has joined #openstack-kuryr02:03
*** aojea has quit IRC02:07
*** kiennt26 has joined #openstack-kuryr02:20
*** salv-orlando has joined #openstack-kuryr02:40
*** salv-orlando has quit IRC02:44
*** hongbin has joined #openstack-kuryr02:53
*** salv-orlando has joined #openstack-kuryr03:41
*** salv-orlando has quit IRC03:45
*** yamamoto has joined #openstack-kuryr03:52
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements  https://review.openstack.org/51909403:56
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements  https://review.openstack.org/51693604:00
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/51693704:00
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978104:00
*** aojea has joined #openstack-kuryr04:04
*** aojea has quit IRC04:11
*** kiennt26 has quit IRC04:13
*** salv-orlando has joined #openstack-kuryr04:41
*** salv-orlando has quit IRC04:46
openstackgerritKien Nguyen proposed openstack/kuryr-libnetwork master: Support shared kuryr subnetpool  https://review.openstack.org/51622804:56
openstackgerritKien Nguyen proposed openstack/kuryr-libnetwork master: Use specific Exception  https://review.openstack.org/51953704:56
*** caowei has quit IRC04:59
*** hongbin has quit IRC05:01
*** yboaron has joined #openstack-kuryr05:15
*** caowei has joined #openstack-kuryr05:32
*** janki has joined #openstack-kuryr05:34
*** salv-orlando has joined #openstack-kuryr05:42
*** salv-orlando has quit IRC05:47
*** janki has quit IRC06:05
*** aojea has joined #openstack-kuryr06:06
*** aojea has quit IRC06:11
*** aojea has joined #openstack-kuryr06:19
*** janki has joined #openstack-kuryr06:20
*** aojea has quit IRC06:24
*** aojea has joined #openstack-kuryr06:28
*** salv-orlando has joined #openstack-kuryr06:30
*** aojea has quit IRC06:31
*** pmannidi has quit IRC07:03
*** janonymous has joined #openstack-kuryr07:20
*** salv-orlando has quit IRC07:36
*** salv-orlando has joined #openstack-kuryr07:36
*** salv-orlando has quit IRC07:40
*** salv-orlando has joined #openstack-kuryr07:47
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Fix kubelet retries issues  https://review.openstack.org/51840407:57
*** yamamoto has quit IRC08:09
*** pcaruana has joined #openstack-kuryr08:10
*** yamamoto has joined #openstack-kuryr08:12
*** yamamoto has quit IRC08:12
*** salv-orlando has quit IRC08:24
*** yamamoto has joined #openstack-kuryr08:41
*** salv-orlando has joined #openstack-kuryr08:42
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Fix kubelet retries issues  https://review.openstack.org/51840408:48
*** yamamoto has quit IRC08:50
*** garyloug has joined #openstack-kuryr08:59
*** salv-orlando has quit IRC09:20
*** salv-orlando has joined #openstack-kuryr09:22
*** salv-orlando has quit IRC09:26
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: CNI split - introducing CNI daemon  https://review.openstack.org/51518609:28
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Support kuryr-daemon when running containerized  https://review.openstack.org/51802409:29
*** yamamoto has joined #openstack-kuryr10:01
*** salv-orlando has joined #openstack-kuryr10:08
*** salv-orlando has quit IRC10:11
*** salv-orlando has joined #openstack-kuryr10:11
*** caowei has quit IRC10:16
*** yamamoto has quit IRC10:36
*** yamamoto has joined #openstack-kuryr10:36
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Prevent pyroute2.IPDB threads leaking  https://review.openstack.org/51740610:48
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: CNI split - introducing CNI daemon  https://review.openstack.org/51518610:48
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Add CNI daemon Tempest job  https://review.openstack.org/51961210:48
openstackgerritEyal Leshem proposed openstack/kuryr-kubernetes master: Network policy support spec  https://review.openstack.org/51923910:48
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add readiness probe to kuryr-controller pod  https://review.openstack.org/51850210:51
*** gouthamr has joined #openstack-kuryr11:10
*** salv-orlando has quit IRC11:19
*** salv-orlando has joined #openstack-kuryr11:19
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Add CNI daemon Tempest job  https://review.openstack.org/51961211:29
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add readiness probe to kuryr-controller pod  https://review.openstack.org/51850211:34
*** wangbo has joined #openstack-kuryr11:56
*** gouthamr has quit IRC12:05
*** reedip has quit IRC12:10
*** gouthamr has joined #openstack-kuryr12:20
*** atoth has joined #openstack-kuryr12:20
*** reedip has joined #openstack-kuryr12:23
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Prevent pyroute2.IPDB threads leaking  https://review.openstack.org/51740612:28
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: CNI split - introducing CNI daemon  https://review.openstack.org/51518612:28
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: CNI Daemon documentation  https://review.openstack.org/50938012:28
*** gouthamr has quit IRC12:53
*** garyloug has quit IRC13:01
*** gouthamr has joined #openstack-kuryr13:02
*** garyloug has joined #openstack-kuryr13:03
*** wangbo has quit IRC13:10
yboaron#link https://docs.google.com/document/d/1Z-tRv6nWDLTCLA-uCrrh9aLakGIxxQSRniaS3E4DHIs/edit?usp=sharing13:16
yboaronhi all , you can find in attached link a draft design document for openshift-route support  in kuryr context13:17
yboaronwill be glad to get your feedback13:17
*** gouthamr has quit IRC13:38
*** salv-orlando has quit IRC13:40
*** apuimedo has joined #openstack-kuryr13:41
apuimedodulek: what's the deal with the timeouts?13:41
*** salv-orlando has joined #openstack-kuryr13:41
dulekapuimedo: Hi! :)13:41
dulekapuimedo: It's a longer story…13:42
dulekSo I was testing CNI daemon with running 20-30-50 pods in parallel. I've started to encountered tracebacks in daemon logs.13:43
dulekMostly from pyroute2.13:43
dulekAnd pods that raised tracebacks were unpingable even if kubelet retried the request.13:44
dulekSo I started to fight with tracebacks - that's why you see those timeouts options added.13:44
dulekBut eventually me and ltomasbo realized, that tracebacks aren't the clue of the problem -  the clue was the fact that for failed requests kubelet may issue DEL after a successful ADD.13:45
dulekSo I've fixed that, but left the timeouts configurable anyway.13:45
dulekapuimedo: What do you think?13:46
apuimedodulek: fine13:53
apuimedosounds good13:53
apuimedoirenab: how far along are you in the review for that?13:53
*** gouthamr has joined #openstack-kuryr13:54
apuimedodulek: irenab: ltomasbo: https://www.openstack.org/videos/sydney-2017/how-to-build-scalable-reliable-and-stable-kubernetes-cluster-atop-openstack13:55
dulekapuimedo: You mean it's worth watching?13:56
*** yboaron has quit IRC14:01
apuimedo['[]\|14:02
dulekapuimedo: Sorry, I don't get that…14:03
apuimedodulek: that was my daugther messing with the keyboard14:05
apuimedowell, they took kuryr-kubernetes and modified it14:05
apuimedoso it is worth to watch14:05
apuimedoat least that part14:05
dulekapuimedo: Nice, I've almost thought it's an emoji. :)14:05
apuimedo:-)14:06
*** gouthamr has quit IRC15:13
*** gouthamr has joined #openstack-kuryr15:16
*** hongbin has joined #openstack-kuryr15:20
*** janki has quit IRC15:39
apuimedodulek: somehow I expected the readiness to be an http endpoint15:51
apuimedo:P15:51
dulekapuimedo: Ah, you're talking about ltomasbo's patch. Yeah, readiness can be an HTTP endpoint, /healthz is normally used.15:53
dulekapuimedo: But we'd need to set up HTTP server, I'm not sure if that's worth it?15:53
apuimedothat's what I had in mind15:54
apuimedobut for now15:55
apuimedowe can take it15:55
dulekapuimedo: I'm not sure if you've checked out our discussion from yesterday's meeting?15:57
dulekapuimedo: The problem I've raised is ltomasbo's patch will not stop kuryr-controller from processing requests.15:58
dulekapuimedo: It's just an indicator for the administrator.15:58
apuimedoI did not15:58
apuimedodulek: yeah. That much I know15:59
apuimedo:-)15:59
apuimedoI think that's fine15:59
apuimedofor now15:59
apuimedowe can make it configurable later15:59
apuimedobut I think it is quite consistent with the k8s semantics15:59
dulekapuimedo: Okay, at least ltomasbo was a bit confused about it and started to work on starting Watcher only after ports are reloaded.15:59
apuimedothat pods being ready at t=40 does not mean that they do nothing before15:59
*** janki has joined #openstack-kuryr16:00
dulekapuimedo: I agree with that. As long as we won't break anything by processing before reload?16:00
apuimedodulek: I know I may be too picky with these things...16:02
apuimedoHow about using /usr/bin/test instead of cat16:02
apuimedosince we have no interest in the content of the file?16:02
apuimedodulek: well, reload should not break existing entries16:02
apuimedoif it does, it is a bug16:02
dulekapuimedo: I'd say - use whatever is more portable?16:03
dulekapuimedo: I just wondered if doing ADD operation won't "steal" already allocated port from the pool if it wasn't reloaded into memory as reserved yet.16:03
apuimedotest is everywhere16:04
apuimedoeven on alpine base image16:04
apuimedo;-)16:04
dulekapuimedo: But I haven't deeply looked at port pool code, so I don't know.16:04
apuimedoltomasbo: you here?16:04
dulekapuimedo: Then test it is. :)16:04
apuimedodulek: but in any case16:04
ltomasboyep, now I'm here16:04
apuimedofollow-up patch for that16:04
ltomasboI can change it with test instead16:04
apuimedodon't modify it, dulek16:04
dulekapuimedo: Hm? Why should I?16:05
ltomasboapuimedo, dulek: it will not use existing ports if they are not in the pool16:05
apuimedodulek: cause I already triggered the workflow xD16:05
ltomasboso, if a request is sent before the ready, a new port creation will be triggered16:05
dulekapuimedo: Oh, okay, sure. :)16:06
ltomasboumm, I think I need to read the backlog...16:06
dulekltomasbo: Ah, so it's fine. I'd leave the readiness probe as is then.16:06
apuimedoltomasbo: the question is whether that may fuck up the recovery16:06
apuimedoor the recovery may take over the ports of the new request16:06
ltomasboapuimedo, you mean if we create a pod before the pool is loaded?16:07
apuimedoltomasbo: right16:07
ltomasboapuimedo, it will not still a port, as it can only get existing ports from the pool, otherwise it will call neutron for a new port16:08
apuimedos/still/steal/ ?16:08
ltomasboyep16:08
apuimedoltomasbo: IIRC, the way you did it16:08
apuimedothe port pool will not be available until it is all processed16:09
apuimedodo I remember well?16:09
*** salv-orlando has quit IRC16:09
ltomasbono, I don't think so, it is including the ports sequentially16:10
ltomasboso, for the nested case, it will populate the ports pool by pool, i.e., trunk by trunk16:10
apuimedoltomasbo: that means, then, that a request may succeed while the ports are being recovered16:10
ltomasboa request will succeed anyway, the only different is whether it will get a port from the pool, or create a new one16:11
dulekltomasbo: I can imagine a race condition here:16:12
dulekltomasbo: 1. Recovery starts.16:12
dulekltomasbo: Okay, even better.16:12
ltomasbothe recovery action is a different eventlet thread16:12
dulekltomasbo: 1. ADD request comes, Neutron port gets created and starts to get annotated into pod.16:12
dulekltomasbo: 2. Recovery starts. Fetches all the ports from Neutron.16:13
ltomasbohttps://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/controller/drivers/vif_pool.py#L110-L11516:13
dulekltomasbo: 3. Port from point 1 gets added into pool as unreserved.16:13
ltomasbodulek, umm, let me think about that16:15
ltomasbonot sure if that case is possible (and it should not be if you wait for the readiness flag...)16:15
ltomasboso, it should not be an issue anyway16:16
ltomasbobut, we also check from the kubernetes side, if the port id is annotated in an existing pod16:16
ltomasbobefore recovering the ports16:16
dulekltomasbo: I don't really know, I would need to read your code more carefully. But if there's a race condition we should start Watcher only after recovery is finished.16:19
ltomasbodulek, agreed16:21
apuimedothis case is possible16:22
apuimedofor sure16:22
*** lihi has quit IRC16:22
apuimedobecause creating the port and it being annotated have a significant time difference16:23
*** lihi has joined #openstack-kuryr16:24
apuimedoltomasbo: dulek ^^16:27
dulekapuimedo, ltomasbo: Okay, so we know there's a potential bug in there.16:28
apuimedoyup16:29
*** salv-orlando has joined #openstack-kuryr16:31
ltomasbogot it!16:31
openstackgerritMerged openstack/kuryr-kubernetes master: Add readiness probe to kuryr-controller pod  https://review.openstack.org/51850216:31
apuimedodulek: ltomasbo: we should have health probes for the daemonset that keep track of the health of the requests connection to the k8s api16:34
apuimedoI'll add a bug for that16:34
*** pcaruana has quit IRC16:48
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add oslo_cache to default_subnet driver  https://review.openstack.org/51970417:04
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add oslo_cache to default_subnet driver  https://review.openstack.org/51970417:06
*** dougbtv__ has joined #openstack-kuryr17:08
*** janki has quit IRC17:10
*** yamamoto has quit IRC17:15
*** salv-orl_ has joined #openstack-kuryr17:20
*** yamamoto has joined #openstack-kuryr17:21
*** salv-orlando has quit IRC17:22
*** yamamoto has quit IRC17:26
*** yamamoto has joined #openstack-kuryr17:36
*** dougbtv__ has quit IRC17:39
*** yamamoto_ has joined #openstack-kuryr17:40
*** yamamoto has quit IRC17:41
*** salv-orl_ has quit IRC17:44
*** dougbtv__ has joined #openstack-kuryr17:45
*** yamamoto_ has quit IRC17:45
*** dougbtv_ has quit IRC17:48
*** salv-orlando has joined #openstack-kuryr17:52
*** yamamoto has joined #openstack-kuryr17:52
*** salv-orlando has quit IRC17:54
*** salv-orlando has joined #openstack-kuryr17:55
*** salv-orl_ has joined #openstack-kuryr17:58
*** janonymous has quit IRC18:00
*** salv-orlando has quit IRC18:01
*** yamamoto has quit IRC18:03
*** dougbtv__ has quit IRC18:12
*** salv-orl_ has quit IRC18:24
*** salv-orlando has joined #openstack-kuryr18:24
*** salv-orlando has quit IRC18:28
*** yamamoto has joined #openstack-kuryr18:29
*** garyloug has quit IRC18:33
*** yamamoto has quit IRC18:35
*** yamamoto has joined #openstack-kuryr18:36
*** dougbtv__ has joined #openstack-kuryr18:36
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements  https://review.openstack.org/51909418:45
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements  https://review.openstack.org/51693618:50
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/51693718:50
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978118:50
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements  https://review.openstack.org/51909418:56
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements  https://review.openstack.org/51693619:04
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/51693719:04
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978119:05
*** mestery has quit IRC19:10
*** mestery_ has joined #openstack-kuryr19:10
*** mestery_ is now known as mestery19:10
*** yamamoto has quit IRC19:11
*** yboaron has joined #openstack-kuryr19:14
*** aojea has joined #openstack-kuryr20:08
*** yamamoto has joined #openstack-kuryr20:12
*** pcaruana has joined #openstack-kuryr20:12
*** yamamoto has quit IRC20:23
*** gouthamr has quit IRC20:53
*** salv-orlando has joined #openstack-kuryr21:18
*** gouthamr has joined #openstack-kuryr21:23
*** aojea has quit IRC21:24
*** aojea has joined #openstack-kuryr21:24
*** aojea has quit IRC21:29
*** gouthamr has quit IRC21:49
*** pcaruana has quit IRC22:02
*** yboaron has quit IRC22:04
*** pcaruana has joined #openstack-kuryr22:06
*** pcaruana has quit IRC22:07
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements  https://review.openstack.org/51909422:19
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements  https://review.openstack.org/51693622:22
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/51693722:22
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978122:22
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-tempest-plugin master: Updated from global requirements  https://review.openstack.org/51978222:22
*** aojea has joined #openstack-kuryr22:25
*** gouthamr has joined #openstack-kuryr22:26
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements  https://review.openstack.org/51909422:26
*** aojea has quit IRC22:29
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements  https://review.openstack.org/51693622:30
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/51693722:30
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978122:30
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-tempest-plugin master: Updated from global requirements  https://review.openstack.org/51978222:30
*** pmannidi has joined #openstack-kuryr23:08
*** aojea has joined #openstack-kuryr23:26
*** aojea has quit IRC23:31
*** gouthamr has quit IRC23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!