Monday, 2017-11-13

*** caowei has joined #openstack-kuryr01:25
*** kiennt26 has joined #openstack-kuryr01:46
*** dougbtv__ has quit IRC01:57
*** hongbin has joined #openstack-kuryr02:06
*** janonymous has joined #openstack-kuryr02:24
*** salv-orlando has joined #openstack-kuryr02:48
*** salv-orl_ has quit IRC02:51
*** yamamoto has joined #openstack-kuryr04:01
*** caowei has quit IRC04:56
*** caowei has joined #openstack-kuryr05:29
*** hongbin has quit IRC05:30
*** janki has joined #openstack-kuryr05:33
irenabdulek, good morning06:49
openstackgerritEyal Leshem proposed openstack/kuryr-kubernetes master: Network policy support spec  https://review.openstack.org/51923906:50
openstackgerritEyal Leshem proposed openstack/kuryr-kubernetes master: Network policy support spec  https://review.openstack.org/51923906:56
*** yboaron has joined #openstack-kuryr07:25
*** pmannidi has quit IRC07:39
*** pcaruana has joined #openstack-kuryr08:10
*** salv-orlando has quit IRC08:11
*** salv-orlando has joined #openstack-kuryr08:11
*** salv-orlando has quit IRC08:15
*** yboaron_ has joined #openstack-kuryr08:16
*** yboaron has quit IRC08:18
dulekirenab: Hi, what's up?08:23
*** yboaron_ has quit IRC08:25
*** yboaron_ has joined #openstack-kuryr08:25
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Prevent pyroute2.IPDB threads leaking  https://review.openstack.org/51740608:27
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: CNI split - introducing CNI daemon  https://review.openstack.org/51518608:27
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Support kuryr-daemon when running containerized  https://review.openstack.org/51802408:27
*** salv-orlando has joined #openstack-kuryr08:36
openstackgerritEyal Leshem proposed openstack/kuryr-kubernetes master: Network policy support spec  https://review.openstack.org/51923909:27
*** garyloug has joined #openstack-kuryr09:30
*** caowei has quit IRC09:50
*** caowei has joined #openstack-kuryr09:52
*** yamamoto has quit IRC10:02
*** s1061123 has joined #openstack-kuryr10:03
*** s1061123_ has quit IRC10:04
irenabdulek, wanted to check waht tool you used to create sequence diagram for the Pod create flow10:06
dulekirenab: It's Inkscape, I've just modified already existing diagram.10:06
*** kiennt26 has quit IRC10:08
*** caowei has quit IRC10:13
ltomasboping dulek, I've just seen your comments on https://review.openstack.org/#/c/518502/10:18
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements  https://review.openstack.org/51909410:18
ltomasboI want to discuss a couple of things with you (as I'm not sure how to better fix it)10:18
dulekltomasbo: Let me just read those again. :)10:18
ltomasbothey are simple10:18
ltomasboabout the LOG message when the ports are loaded10:19
ltomasbobut that made me realize about one thing10:19
dulekltomasbo: Yup, okay. So you prefer no try-except?10:19
ltomasboregardless of setting the LOG.info inside or outside of the with10:19
ltomasbono, my question is about another related issue10:19
ltomasboif the with fail10:20
ltomasbothe kubernetes-controller pod will never be ready10:20
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements  https://review.openstack.org/51693610:20
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/51693710:20
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978110:20
ltomasboso, on the one hand, I'm not sure if the LOG should be in or out of the with10:20
ltomasboand yep, I agree I need to include a try-except, but perhaps with a LOG.Error if it is not possible to write the file10:21
ltomasbowhat do you think?10:21
dulekltomasbo: I agree with the ERROR level, being an admin I'd prefer to have it logged clearly why my pod isn't ready.10:21
dulekltomasbo: But I'd also prefer a log instead of pod being restarted constantly.10:22
ltomasboyep, so, I'll print the ERROR in the try-except, and the LOG.info before the with10:22
ltomasbodulek, thanks for the feedback, just wanted to double check with you10:23
dulekltomasbo: Works for me. No problem. :)10:23
ltomasbogreat!10:23
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add readiness probe to kuryr-controller pod  https://review.openstack.org/51850210:51
*** yamamoto has joined #openstack-kuryr10:52
ltomasbohi again dulek10:56
ltomasboonce that is polished and merged ^^, could you update the dockerhub containers?10:56
dulekltomasbo: Sure, apologies I haven't did that earlier, but I've had hope that CNI daemon will be merged earlier.10:57
dulekltomasbo: BTW I've found the problem with ADD-DEL-ADD-DEL sequence. Had you taken a look on newest version of the patch?10:58
ltomasboohh10:58
ltomasbono, I didn't10:58
ltomasboI'll do now! what was the problem (I was on PTO last Thu-Fri)10:59
dulekltomasbo: Main problem was that we're using names to identify pods.10:59
dulekltomasbo: And we cannot really use anything different as that's the only common value in both API and CNI requests.10:59
ltomasboumm11:00
dulekltomasbo: Now kubelet (more specifically CRI) uses CNI_CONTAINERID value.11:00
dulekltomasbo: It's internal, but this is the ultimate *container* id.11:00
ltomasboso the add from one pod, was mathcing with the del of other?11:00
dulekltomasbo: Exactly. Last DEL was actually for older CNI_CONTAINERID.11:01
ltomasboman, you are dealing with the hardest/trickiest bugs...11:01
ltomasbodulek, can we use the neutron port id to differentiate them?11:02
dulekltomasbo: I've fixed it by ignoring DEL requests that doesn't match CNI_CONTAINERID saved when doing last ADD request.11:02
ltomasboahh, you already fixed it!11:02
dulekltomasbo: Hm… Wouldn't it be the same?11:02
ltomasbogoing to take a look at the patch then11:03
ltomasbowell, neutron port ids should be unique11:03
dulekI mean - the problem is we're networking Pods, not containers. VIF annotation is tied to Pod, same is true for ports then.11:03
dulekCNI assumes each container (with unique CNI_CONTAINERID) will have a separate networking.11:04
ltomasboumm, I'm missing something11:04
ltomasbowe are doing 1pod-1port11:04
dulekYes.11:04
ltomasbowhat was the problem then? I assume we still want to do 1 pod 1 port (or many for the multi nic later on)11:05
ltomasboand hte problem happened with single container pods, right?11:05
dulekltomasbo: Yes.11:05
dulekltomasbo: Uhm. Okay, so main problem is that CRI/CNI is working on container level, not pod level.11:06
dulekltomasbo: Maybe an example:11:06
dulekltomasbo: 1. kubelet tries to spawn a container "foo" for pod "bar". CNI fails for that request.11:07
dulekltomasbo: 2. Therefore kubelet will send a CNI DEL request for container "foo" in pod "bar".11:07
ltomasboexample will help a lot! thanks! (and taking a look at your modifications to the patch...)11:07
ltomasbook11:07
dulekltomasbo: 3. Meanwhile a new container "foo2" is starting.11:07
dulekltomasbo: 4. If DEL request from point 2 gets retried - we don't want to unplug container foo2, but only foo.11:08
ltomasbook, got it11:09
ltomasboso it was deleting the wrong container inside the right pod...11:09
ltomasbo(I need more coffee after the long weekend...)11:09
dulekltomasbo: For kubelet it's perfectly normal as it works on containers, not pods.11:10
dulekltomasbo: In Kuryr we're networking whole pods, so it's kind of Kuryr architecture flaw.11:10
ltomasbowell, AFAIK, kubernetes is suppose to network pods, not containers either11:11
ltomasboso, we are aligned with that11:11
dulekltomasbo: Kubernetes as a whole - yes, AFAIR there's an assumption that all containers in a pod use single IP.11:12
ltomasboyep, as well as they 'schedulable' items are pods, not containers11:12
dulekltomasbo: But that isn't true for CNI/CRI interfaces, that are more universal and are designed to deal with containers, not pods.11:12
*** atoth has quit IRC11:13
ltomasboI thought the containers in a multi-container pod where meant to share the IP (and communicate among them as normal processes)11:13
ltomasbo*were11:14
dulekltomasbo: Yes, yes, they are. Just on the low level kubelet invokes CNI commands assuming they work on containers, not whole pods.11:15
dulekltomasbo: So it has an assumption that DEL request for podA, containerA will not unplug podA, containerB.11:15
dulekltomasbo: And in Kuryr case it would as we're using names to identify which VIF to unplug.11:16
ltomasbodulek, and for a 2 container pod, the cni is meant to link the network to both containers? or just to one of them?11:16
ltomasboIMHO kuryr is providing neutron ports to pods, not to containers, but I understand the problems with the retries11:17
dulekltomasbo: TBH I haven't checked that.11:17
dulekOkay, so retries were tricky. :)11:17
dulekNow it works much better, you can try to verify.11:17
dulekBut I've easily networked 50 pods on my env.11:18
ltomasbosure, I'll give it a try asap! thanks for the clear explanation!11:18
dulekSome timed out on waiting for VIF annotation, but now with restarts working correctly - it's fine. :)11:18
duleks/restarts/retries11:18
ltomasbogreat!11:19
*** yamamoto has quit IRC11:51
*** atoth has joined #openstack-kuryr11:59
*** yamamoto has joined #openstack-kuryr12:08
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add readiness probe to kuryr-controller pod  https://review.openstack.org/51850212:09
irenabdulek, ltomasbo I was trying to catch up on your discussion regarding CNI/CRI kubelet calls. I am still not sure I understand what is the issue kuryr has12:20
irenabdulek, is there any difference in what kuryr does for the forst Pod Container plug versus second one?12:21
ltomasboirenab, let me try to explain it (to also double check if I got it right)12:22
dulekirenab: No, there isn't a difference. VIF annotation stays the same, maybe only netns id changes.12:22
ltomasbowhen there are multiple pods being created, some actions may fail due to timeouts12:22
ltomasboand the problem is that when the add operation for a container in a pod fails there will be a retry for a new container inside the pod12:23
ltomasbobut the add action for the second container may come before the del action for the failed container12:23
irenabin this case the container id will be different, right?12:23
ltomasboyep, that is what dulek has fixed, so that if the id is different, the del action is skipped12:24
irenabsince it failed to wire it before?12:24
irenabkuryr failed to wire the first container, therefor it is not present?12:25
dulekirenab: Yup, failures where seen most often when ~50 pods were created at once, as kuryr-controller wasn't catching up with annotation pods in time.12:25
dulek(probably Neutron being slow was also an issue)12:25
irenabdulek, agree with ltomasbo , you revealed very interesting issues12:26
dulekirenab: I really hope that's the last one…12:26
irenabdulek, I won't count o it :-)12:26
ltomasboI wonder if there are other race conditions where we trigger the events in the wrong order12:27
dulekBTW I've described the issue in a bug: https://bugs.launchpad.net/kuryr-kubernetes/+bug/173148512:27
openstackLaunchpad bug 1731485 in kuryr-kubernetes "Kuryr ignores CNI_CONTAINERID when serving requests" [High,In progress] - Assigned to Michal Dulko (michal-dulko-f)12:27
dulekirenab: Ah, you've already saw it.12:27
irenabyes, but somehow it was a bit confusing why kuryr disconnects the pod12:28
dulekhttps://bugs.launchpad.net/kuryr-kubernetes/+bug/1731486 - this one is also interesting. Guys from k8s sig-network pointed it out to me.12:28
openstackLaunchpad bug 1731486 in kuryr-kubernetes "Kuryr ignores namespaces" [Undecided,New]12:28
dulekirenab: Here's the k8s guys explanation of logic behind those retries: "Yeah. In general you can't rely on the caller remembering what happened last time - due to process crashes it may not have received the result.12:29
dulekSo you always have to be prepared for the caller to send you something you didn't expect.12:29
dulekAnd kubelet takes advantage of this: it deliberately doesn't remember certain things and speculatively calls DEL because it knows you have to deal with it."12:29
irenabwe need to have namespace name to be either tagged on port or added to the port name12:30
dulekirenab: Yup! And added to watcher URL-s to make sure we're watching on correct namespace.12:30
dulekAnyway that's separate issue.12:30
ltomasbohow will namespaces fit openstack resources? could it be just different projects?12:31
irenabdulek, and you previously hoped you dealed with last issue... Seems you just started12:31
ltomasbothat way you will not need to add tags/names (and skip neutron calls...)12:31
dulekirenab: I have something with higher priority today, but I can definitely grab that bug when I'll have some more time.12:32
irenabltomasbo, this is the topic we discussed previously and din't reach consensus. Same project can be mapped to number of namespaces12:32
dulekltomasbo: I'm not sure if I follow. The only issue we have is that Kuryr assumes names are unique. And they aren't, (namespace, name) pairs are.12:33
dulekltomasbo: I don't think we need to provide isolation of namespaces - please note that kubernetes doesn't separate networking of different namespaces either.12:33
ltomasboirenab, but we can have tenants being in different projects, and the projects being mapped to namespaces (1-1)12:33
irenabwe will have to deal with it once start Network Policies implementation, since policies are aplicable per namespace12:34
ltomasboumm12:34
dulekirenab: +1!12:34
ltomasboperhaps now that we have more info about the network policies12:34
ltomasboit will be nice to have a call and discuss about the namespaces again12:34
irenabthere is a spec for Network Policies that leyal published today12:35
irenabhttps://review.openstack.org/#/c/519239/12:35
ltomasboI'm a bit scare about the amount of calls to neutron...12:35
irenabltomasbo, agree with you, we will need to check what can be batched12:36
ltomasboyep12:36
irenabltomasbo, +1 on renew the discussion on multiple namespaces. I think we probably need some tests that involve multi ns applications12:38
dulekoc12:39
dulekAww, wrong window. :)12:39
ltomasbo+1 on the tests too12:40
ltomasboxD12:40
*** yboaron_ has quit IRC13:23
*** yboaron has joined #openstack-kuryr13:24
*** yamamoto has quit IRC13:24
*** yboaron has quit IRC13:31
*** yboaron has joined #openstack-kuryr13:31
dulekMeeting time?14:01
irenablets start14:07
irenabkuryr meeting is at openstack-meeting-4, sorry for the delay14:08
irenabltomasbo: dulek leyal joining the meeting?14:09
ltomasboups, yep14:09
*** garyloug has quit IRC14:18
*** yamamoto has joined #openstack-kuryr14:25
*** yamamoto has quit IRC14:34
*** janki has quit IRC14:41
*** hongbin has joined #openstack-kuryr15:05
*** salv-orlando has quit IRC15:15
*** salv-orlando has joined #openstack-kuryr15:15
*** salv-orlando has quit IRC15:20
*** yboaron has quit IRC15:39
ltomasbodulek, I can confirm you were right!15:40
ltomasboeven with the pod in status 'NotReady', I can create/scale pods15:40
dulekltomasbo: Well, it's not that I'm happy of making your task harder. :P15:41
ltomasboxD15:41
*** yboaron has joined #openstack-kuryr15:41
*** gouthamr_ has joined #openstack-kuryr15:41
ltomasbodulek, nice catch! let's see how I can fix it...15:41
dulekltomasbo: I'd leave the probe exactly as it is - that's still useful piece of information for the admin if we document it.15:42
ltomasboit was too easy to be true...15:42
*** gouthamr_ is now known as gouthamr15:42
dulekltomasbo: And meanwhile kuryr-controller should not do Watcher.start until all the ports are recovered.15:42
ltomasbodulek, yep, I agree that it a good to have, but I'll try to figure out (in a different patch) how to achieve the other behavior too15:42
dulekCool! :)15:43
yboaronltomasbo, I have a question regarding :  https://review.openstack.org/#/c/518502/ is relevant15:58
ltomasboyboaron, tell me15:58
yboaronDoes readinessProbe  is added to yaml only for devstack ?15:59
yboaronFrom what I saw , it's  added only at  devstack/lib/kuryr_kuberenets , but maybe I'm missing something ..16:00
ltomasboyboaron, that is a deployment configuration, it is added to devstack if you deploy with it, and I'll try to add it to openstack-ansible if you deploy it with that: https://github.com/openshift/openshift-ansible/pull/606216:00
ltomasbobut you are right, perhaps I need to update some documentation for manual installation about this16:01
ltomasboor are you missing it in a different place?16:01
yboaronltomasbo, Yes as you said , thought a bout kuryr-k8s user , that would like to utilize ports pool16:02
yboaronin non devstack environment16:02
ltomasboyboaron, we only have tools for devstack installation at the kuryr-kubernetes repo, so that is why I only included that way there16:03
ltomasbobut you are right, I should add some note/info somewhere about this...16:04
yboaronOK , got it - devstack is all our world !  :-) thanks16:04
ltomasboxD16:05
ltomasboyboaron: https://github.com/openstack/kuryr-kubernetes/blob/master/doc/source/installation/containerized.rst16:05
ltomasboyboaron, will that be a good place to include that information?16:05
yboaronltomasbo, looks like that right place16:07
yboaronthe16:07
ltomasbogreat! thanks! I'll add it there then16:07
yboaronltomasbo, thanks16:08
*** salv-orlando has joined #openstack-kuryr16:16
*** salv-orlando has quit IRC16:21
*** yboaron has quit IRC16:24
dulekyboaron, ltomasbo: We have a script generating yamls for general k8s use: https://docs.openstack.org/kuryr-kubernetes/latest/installation/containerized.html#generating-kuryr-resource-definitions-for-kubernetes16:37
dulekThose use the same functions as DevStack, so adding probe to DevStack will add it to scripts as well.16:37
*** pcaruana has quit IRC16:49
*** salv-orlando has joined #openstack-kuryr17:09
*** dougbtv__ has joined #openstack-kuryr17:14
*** janonymous has quit IRC17:20
*** dougbtv__ has quit IRC18:12
*** aojea has joined #openstack-kuryr18:21
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Prevent pyroute2.IPDB threads leaking  https://review.openstack.org/51740618:34
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: CNI split - introducing CNI daemon  https://review.openstack.org/51518618:34
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Support kuryr-daemon when running containerized  https://review.openstack.org/51802418:34
*** aojea has quit IRC19:14
*** atoth has quit IRC20:09
*** salv-orlando has quit IRC20:28
*** salv-orlando has joined #openstack-kuryr20:28
*** salv-orlando has quit IRC20:33
*** aojea has joined #openstack-kuryr20:33
*** salv-orlando has joined #openstack-kuryr21:00
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements  https://review.openstack.org/51909421:06
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements  https://review.openstack.org/51909421:08
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements  https://review.openstack.org/51909421:08
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements  https://review.openstack.org/51909421:09
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements  https://review.openstack.org/51909421:09
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements  https://review.openstack.org/51693621:10
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/51693721:10
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978121:10
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements  https://review.openstack.org/51693621:10
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/51693721:10
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978121:10
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/51693721:10
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978121:11
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements  https://review.openstack.org/51693621:12
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/51693721:12
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978121:12
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements  https://review.openstack.org/51693621:17
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/51693721:17
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978121:17
*** fkautz_ has joined #openstack-kuryr21:46
*** fkautz has quit IRC21:51
*** nhlfr has quit IRC21:51
*** fkautz_ is now known as fkautz21:51
*** Guest57502 has quit IRC22:00
*** aojea has quit IRC22:01
*** aojea has joined #openstack-kuryr22:02
*** aojea has quit IRC22:06
*** Guest54320 has joined #openstack-kuryr22:10
*** dougbtv__ has joined #openstack-kuryr22:10
*** nhlfr has joined #openstack-kuryr22:15
*** dougbtv__ has quit IRC22:29
*** pmannidi has joined #openstack-kuryr22:46
*** gouthamr has quit IRC23:09
openstackgerritHongbin Lu proposed openstack/kuryr-libnetwork master: Introduce sriov port driver  https://review.openstack.org/50043623:21
*** salv-orlando has quit IRC23:37
*** salv-orlando has joined #openstack-kuryr23:38
openstackgerritHongbin Lu proposed openstack/kuryr-libnetwork master: Add debug statement for response data  https://review.openstack.org/51950723:40
*** salv-orlando has quit IRC23:42

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!