| *** hongbin has joined #openstack-kuryr | 01:16 | |
| *** ccamposr__ has joined #openstack-kuryr | 02:27 | |
| *** ccamposr has quit IRC | 02:29 | |
| *** hongbin has quit IRC | 04:21 | |
| openstackgerrit | Nayan Deshmukh proposed openstack/kuryr-kubernetes master: Count cni DEL failure as well to mark it unhealthy https://review.opendev.org/668594 | 04:42 |
|---|---|---|
| *** gcheresh_ has joined #openstack-kuryr | 04:45 | |
| *** gcheresh_ has quit IRC | 04:54 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes stable/stein: Fix adding pods with host networking to svc in L2 mode https://review.opendev.org/668608 | 06:28 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes stable/rocky: Fix adding pods with host networking to svc in L2 mode https://review.opendev.org/668609 | 06:29 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes stable/stein: Support None from Octavia get_api_major_version() https://review.opendev.org/668614 | 06:44 |
| *** gcheresh_ has joined #openstack-kuryr | 06:57 | |
| *** ccamposr has joined #openstack-kuryr | 07:02 | |
| *** ccamposr__ has quit IRC | 07:04 | |
| aperevalov | hi, has anybody encountered with a problem when k8s watch api was too slow? I found that time since kuryr-controller annotates VIF after port status became ACTIVE, till cni-daemon receives this VIF, sometimes up to 16 seconds. But average range is 4-15 seconds for this operation. There wasn't overload of k8s at this time. | 07:33 |
| ltomasbo | aperevalov, nested or neutron vif driver? | 07:39 |
| aperevalov | neutron_vif, but I think it doesn't matter | 07:39 |
| aperevalov | it's about RPC through k8s (etcd) watch API | 07:40 |
| ltomasbo | aperevalov, ahh, you mean after the controller actually sets the port to active, right? | 07:40 |
| ltomasbo | until the kuryr-cni reads it, and finishes | 07:40 |
| aperevalov | right, looks like watch API in k8s is not working immediately. Or I can't tweak it correctly. | 07:41 |
| aperevalov | I understand that the main advantage of using k8s as RPC was keeping state, for example, after CNI-Daemon or controller restart, we can continue to work. But it feels like too slow RPC. | 07:43 |
| aperevalov | or RPC is not right term here, just interconnection. | 07:43 |
| dulek | aperevalov: The point is we're polling Neutron until port becomes ACTIVE. | 07:55 |
| dulek | aperevalov: And that's probably the longest wait. | 07:55 |
| dulek | aperevalov: It's like this: pod created->notification to kuryr-controller->VIF creation->annotation->kuryr-daemon notices annotation->kuryr-daemon binds port->kuryr-controller polls Neutron API until port becomes ACTIVE->kuryr-controller annotates pod marking that port is active->kuryr-daemon returns to CNI->kubelet marks pod as having an IP. | 07:57 |
| *** pcaruana has joined #openstack-kuryr | 08:01 | |
| *** maysams has joined #openstack-kuryr | 08:02 | |
| ltomasbo | aperevalov, dulek: I understood that aperevalov was refering to the last part, from when kuryr-controller annotates the port as active (not when it annotates it the first time with status down) | 08:08 |
| ltomasbo | till the kuryr-cni realises is active and exits | 08:08 |
| ltomasbo | if your at considering from t he first annotation, then the waiting is normal and it will be related to the time it takes neutron to actaully set the port to active | 08:09 |
| dulek | ltomasbo: The last part shouldn't differ too much from other parts, e.g. waiting for kuryr-controller to notice the pod. | 08:13 |
| ltomasbo | yep | 08:13 |
| dulek | Maybe CNI (like K8s CNI) screws this up and it takes it longer to return to kubelet? | 08:13 |
| aperevalov | what is the usual time in your installation between annotation of active state till CNI-Daemon reads it from k8s? | 08:38 |
| ltomasbo | aperevalov, I guess dulek's question is if you are measuring since the pod gets annotated with the port, or since the annorations gets updated to reflect the new (ACTIVE) status of the pod | 08:49 |
| ltomasbo | port | 08:49 |
| aperevalov | I told about interval since point of time when annotation of ACTIVE state in kuryr-controller happend, till CNI-Daemon was notified (and yes returned output to cni-plugin -> kubelet). | 08:52 |
| dulek | aperevalov: I never observed a longer delay there. | 08:53 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-tempest-plugin master: Ensure empty/fail replies are not counted https://review.opendev.org/668528 | 08:53 |
| aperevalov | ok, I'll do a synthetic test, by curl ;) to make sure the root cause in k8s. | 08:54 |
| dulek | aperevalov: Maybe you have something strange in this config option: CONF.cni_daemon.vif_annotation_timeout ? | 08:54 |
| dulek | aperevalov: Ah no, sorry, it seems like the polling time is one second there, hardcoded. | 08:55 |
| *** danil has joined #openstack-kuryr | 08:59 | |
| dulek | aperevalov, danil: You're aware of this issue: https://bugs.launchpad.net/kuryr-kubernetes/+bug/1835039 ? | 10:25 |
| openstack | Launchpad bug 1835039 in kuryr-kubernetes "CNI returns just one interface when actually setting multiple up" [Medium,Confirmed] | 10:25 |
| *** aperevalov has quit IRC | 10:37 | |
| *** aperevalov1 has joined #openstack-kuryr | 10:48 | |
| aperevalov1 | dulek: no I didn't see this issue, k8s support chain of cni plugins, and get IP from the latest in chain. Also need to clarify what kind of CNI spec version ;) | 10:49 |
| dulek | Eh, korzen isn't even here… | 10:55 |
| dulek | aperevalov1: It's one of the latest, 0.4 probably. My colleague has issues with that + virtlet. | 10:56 |
| aperevalov1 | Also, I introduced bug there in 607a249e10f59aebc64fcada1d4e9da693624acc or it was later. The bug is following when cniplugin prints nonempty sandbox it means the name of the interface inside container, if empty it means name of interface on the host. Last time I saw CNI output it seems it was vice versa... hmm, I'll check it now. | 11:02 |
| *** korzen has joined #openstack-kuryr | 11:04 | |
| aperevalov1 | oops, Interfaces:[{Name:tap70d0746d-ac ... Sandbox:d4438f1937254d789d8ba1952297d34fd0786daa7bb93b313252514002b86e58}], ... | 11:04 |
| aperevalov1 | Name of the interface as on the host, need put empty sandbox in this case or eth0 as container's interface name | 11:04 |
| aperevalov1 | It wasn't affected k8s, until I removed nsenter approach from there. | 11:05 |
| korzen | I see that multiple interfaces are working with kubelet, but I am integrating kuryr with Virtlet, and there are drogons ;) | 11:07 |
| korzen | I have made some changes to return multiple ifs but then I have spotted that tapXX is returned, instead of ethX | 11:09 |
| aperevalov1 | yes, tapXX when sandbox is empty | 11:09 |
| korzen | virtlet is smart enough not to trust CNI info and is reworking the CNI output | 11:13 |
| korzen | so I ended up with 4 interfaces: tapX, tapY, eth0, eth1 instead of 2 ;) | 11:13 |
| korzen | instead of two interfaces | 11:14 |
| aperevalov1 | ok, I asked Dan Williams about such case, he said it's not a problem for k8s to work with such result. | 11:17 |
| korzen | my understanding is that kuryr should return ethX interfaces with sandbox fill with id | 11:21 |
| aperevalov1 | yes, it's correct | 11:21 |
| *** rh-jelabarre has joined #openstack-kuryr | 11:39 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Speed up pools prepopulation on namespace creation https://review.opendev.org/668661 | 12:05 |
| *** korzen has quit IRC | 12:47 | |
| *** maysams has quit IRC | 13:27 | |
| *** ndesh has quit IRC | 13:55 | |
| *** danil has quit IRC | 14:06 | |
| *** maysams has joined #openstack-kuryr | 14:26 | |
| *** gcheresh_ has quit IRC | 14:55 | |
| *** aperevalov1 has quit IRC | 14:59 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-tempest-plugin master: Ensure empty/fail replies are not counted https://review.opendev.org/668528 | 15:25 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-tempest-plugin master: Ensure empty/fail replies are not counted https://review.opendev.org/668528 | 15:32 |
| *** gcheresh_ has joined #openstack-kuryr | 17:11 | |
| *** gcheresh_ has quit IRC | 17:51 | |
| openstackgerrit | Maysa de Macedo Souza proposed openstack/kuryr-tempest-plugin master: Test recreate namespace and pod https://review.opendev.org/668737 | 18:41 |
| openstackgerrit | Maysa de Macedo Souza proposed openstack/kuryr-kubernetes master: Fix fail to recreate namespace when previous KuryrNet CRD is not deleted https://review.opendev.org/666850 | 18:57 |
| *** gcheresh_ has joined #openstack-kuryr | 18:59 | |
| openstackgerrit | Maysa de Macedo Souza proposed openstack/kuryr-kubernetes master: Fix fail to recreate namespace when previous KuryrNet CRD is not deleted https://review.opendev.org/666850 | 19:04 |
| *** gcheresh_ has quit IRC | 19:30 | |
| *** maysams has quit IRC | 20:31 | |
| *** pcaruana has quit IRC | 20:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!