| *** kiseok7 has joined #openstack-kuryr | 01:05 | |
| openstackgerrit | pengyuesheng proposed openstack/kuryr-kubernetes master: Bump the openstackdocstheme extension to 1.20 https://review.opendev.org/672875 | 01:37 |
|---|---|---|
| *** maysams has quit IRC | 02:08 | |
| *** ndesh has joined #openstack-kuryr | 02:37 | |
| *** gkadam has joined #openstack-kuryr | 03:40 | |
| *** gcheresh has joined #openstack-kuryr | 05:14 | |
| *** threestrands has joined #openstack-kuryr | 05:44 | |
| *** brault has joined #openstack-kuryr | 06:25 | |
| *** brault has quit IRC | 06:39 | |
| *** takamatsu has quit IRC | 07:17 | |
| *** brault has joined #openstack-kuryr | 07:19 | |
| *** brault has quit IRC | 07:23 | |
| *** brault has joined #openstack-kuryr | 07:23 | |
| *** maysams has joined #openstack-kuryr | 07:27 | |
| *** pcaruana has quit IRC | 07:35 | |
| *** pcaruana has joined #openstack-kuryr | 08:13 | |
| *** takamatsu has joined #openstack-kuryr | 08:32 | |
| *** takamatsu_ has joined #openstack-kuryr | 08:48 | |
| *** threestrands has quit IRC | 08:48 | |
| *** takamatsu has quit IRC | 08:49 | |
| *** takamatsu_ has quit IRC | 09:30 | |
| *** takamatsu has joined #openstack-kuryr | 09:37 | |
| *** ndesh has quit IRC | 09:57 | |
| *** takamatsu has quit IRC | 10:08 | |
| *** takamatsu has joined #openstack-kuryr | 10:24 | |
| *** pcaruana has quit IRC | 10:45 | |
| aperevalov | we decided to implement direct RPC before any other performance improvements. I didn't yet create separate blueprint. | 11:13 |
| dulek | aperevalov: Let me try to get apuimedo here… | 11:14 |
| aperevalov | but I see here some different approaches: 1) use oslo.messaging. This approach intend to use 3rd part messaging server like rabbitmq, but not from OpenStack's controller, cause it could be unavailable from k8s cluster. | 11:15 |
| aperevalov | 2) use own listening loop, which for example implemented by iohttp or similar framework, it this approach there is no messaging bus. | 11:17 |
| dulek | aperevalov: What had you already used? | 11:17 |
| dulek | aperevalov: To test this. | 11:18 |
| aperevalov | each of approach has its drawbacks as well as its own advantages. | 11:18 |
| dulek | ltomasbo: ^ | 11:18 |
| aperevalov | I tried iohttp, but not yet integrated it into kuryr-controller. | 11:18 |
| aperevalov | I think we need to figure out with key requirements for this feature. | 11:19 |
| aperevalov | Is it performance or robustness? | 11:19 |
| dulek | aperevalov: So you don't know if direct communication improves anything? | 11:19 |
| aperevalov | yes, I didn't yet measured it, but I measured different parts of launch time process. | 11:20 |
| *** celebdor has joined #openstack-kuryr | 11:21 | |
| celebdor | Hi | 11:21 |
| dulek | celebdor: o/ | 11:21 |
| aperevalov | and I measured in artificial test the time "kubectl --watch" reacts to pods modification. And it's not so huge time. | 11:21 |
| celebdor | sorry, for some reason my irc client loses auth and I get kicked out all the damn time | 11:22 |
| dulek | celebdor: So summary is that aperevalov was looking at implementing direct communication between kuryr-controller and kuryr-daemon. | 11:22 |
| dulek | Instead of waiting for pods to get annotated with vifs. | 11:22 |
| celebdor | that's the same that I saw in the sydney summit some team in China was doing | 11:23 |
| aperevalov | yes, but I want to keep saving annotation into k8s, for fallback case, e.g. when kuryr-controller is restarting. | 11:23 |
| celebdor | how big is the advantage? | 11:23 |
| dulek | I think we don't know yet. | 11:25 |
| aperevalov | it's hard to say now, I saw in my measurements that cni-daemon obtains annotation after a one or two seconds. | 11:25 |
| celebdor | you mean one or two seconds after the controller POSTs the annotation? | 11:26 |
| aperevalov | and comparable to "kubectl in --watch mode" it was always slower. | 11:26 |
| aperevalov | yes | 11:26 |
| celebdor | so kubectl sees the annotation way earlier? | 11:26 |
| celebdor | how much earlier approximately? | 11:26 |
| dulek | celebdor: kubectl does not got through Octavia LB. | 11:27 |
| dulek | celebdor: It should really matter that much, but hey - it's Octavia. | 11:27 |
| ltomasbo | lol | 11:27 |
| aperevalov | time was synchonized. It was better to prepare numbers. No it wasn't Octavia in that setup as well as another load balancer. | 11:28 |
| aperevalov | it would be betther for me to prepare some document result, but I don't have it now. | 11:29 |
| celebdor | dulek: the connection to an API in kube controller would also be via clusterip... | 11:29 |
| aperevalov | celebdor, no it's not necessary (for test environment) if they both in the same subnet. | 11:31 |
| dulek | celebdor: The idea is not to use HTTP and services, just talk to pod's IP. | 11:31 |
| aperevalov | so my traffic goes directly w/o proxing/routing... | 11:31 |
| dulek | That's one of my issues with the idea - it's breaking some K8s paradigm, that pods communicate through services. | 11:31 |
| celebdor | talk in which protocol? | 11:32 |
| dulek | I see RabbitMQ was tossed in the discussion. | 11:33 |
| dulek | Well, that's not really a good idea to depend on another service. | 11:33 |
| aperevalov | as I remember you also using hostNetwork for kuryr-controller and kuryr-daemon. So both of this service is not obligated to use k8s Service's endpoints address. | 11:34 |
| dulek | iohttp - I'm not familiar with that really. | 11:34 |
| dulek | Right, it's hostNetworking. | 11:34 |
| aperevalov | it is one process/thread async library for rpc. | 11:35 |
| dulek | aperevalov, celebdor: My another problem with that approach is that the same issue is solved by daemon VIF choice idea. | 11:35 |
| dulek | As we make kuryr-controller only manage the pools and kuryr-daemon just does everything on its own. | 11:36 |
| aperevalov | Also I heard a suggestion from my co-workers just to write directly to etcd. | 11:36 |
| aperevalov | what is "daemon VIF choice idea"? is it enhancement for pools we discussed before? | 11:37 |
| dulek | aperevalov: Yup. | 11:38 |
| celebdor | etcd would have to be a separate etcd | 11:38 |
| celebdor | it is not usually allowed to touch the k8s etcd | 11:38 |
| celebdor | dulek: daemon vif choice requires the controller handling the SG after the port assignment, doesn't it? | 11:39 |
| dulek | celebdor: Yes, we need that to support SG's. | 11:39 |
| dulek | s/SG's/NP's | 11:40 |
| celebdor | in general, accessing rabbit is problematic from the vm network | 11:40 |
| celebdor | that's why I asked a fuck ton of times the Neutron community to expose a watch endpoint for port events | 11:40 |
| celebdor | we even had an intern write a service that reads the bus and exposes as a watch endpoint | 11:40 |
| aperevalov | celebdor, but neutron rpc is some kind of watching API for ports. | 11:41 |
| celebdor | which, as I said, is often not allowed to reach from the VMs in the kubernetes deployed on VMs case | 11:41 |
| aperevalov | I forgot, it was through mq | 11:42 |
| dulek | aperevalov: Yes, so we see nested deployments as main use case for Kuryr. | 11:42 |
| dulek | aperevalov: And in those allowing VM's (which run K8s masters, nodes, so Kuryr) to access Neutron's RabbitMQ is… crazy. | 11:43 |
| dulek | Wouldn't be allowed on any sane env where you don't 100% trust the code. | 11:43 |
| dulek | aperevalov: So maybe we should take a step back and ask about your use case? What are you building? | 11:43 |
| aperevalov | yes it's true, | 11:44 |
| celebdor | dulek: your golang CNI does not create the vlan/veth interfaces, does it? | 11:44 |
| aperevalov | we have both bm and vm use cases on the same installation | 11:44 |
| celebdor | it only asks the cni daemon to do it, IIRC | 11:44 |
| aperevalov | so we don't have so general case as public cloud | 11:45 |
| celebdor | right | 11:45 |
| dulek | celebdor: It does not, just data processing and forwarding is done there. | 11:45 |
| celebdor | we did demo bm and vm together last year | 11:46 |
| celebdor | https://github.com/openshift/kuryr-kubernetes/blob/master/kuryr_cni/main.go#L41-L44 confused me | 11:47 |
| aperevalov | our master runs in vm, but it's not nova-instance. So every minion and master in the same control plane network. | 11:47 |
| celebdor | aperevalov so on the BM machines I suppose you run the neutron agents and so on, right? | 11:47 |
| dulek | celebdor: Ha, good point, I actually copied that from the skeleton but it shouldn't really be necessary. | 11:47 |
| aperevalov | celebdor, you right | 11:47 |
| celebdor | :D | 11:48 |
| celebdor | aperevalov good | 11:48 |
| aperevalov | so kuryr-controlle's hostNetwork IP is reachable from bm nodes | 11:48 |
| celebdor | right | 11:49 |
| aperevalov | and the idea was: 1) connect from kuryr-daemon directly to kuryr-controller | 11:49 |
| aperevalov | 2) or to separate MQ on the master for only this purpose (own oslo.messaging) | 11:49 |
| celebdor | In general I'm not opposed to cni daemons connecting to a controller | 11:49 |
| celebdor | as long as it's not reusing the OpenStack rabbitmq | 11:50 |
| aperevalov | yes, it should be own/separate MQ | 11:50 |
| aperevalov | or even as I wrote in 1) w/o MQ | 11:50 |
| aperevalov | MQ such as rabbitmq has useful features such as 100% deliverable, even if kuryr-daemon will restared/or failed. | 11:52 |
| aperevalov | but MQ can increase notification time | 11:53 |
| dulek | aperevalov: And it's a dependency. | 11:53 |
| dulek | I don't really love dependencies. | 11:54 |
| celebdor | does kubectl see the notifications soon enough for your purpose? | 11:54 |
| aperevalov | I think, yes it shows stable time when it prints results. | 11:55 |
| dulek | aperevalov: So maybe there's some issue with how Python networking code is looking for those notifications? | 11:56 |
| aperevalov | I also that it was due to k8s watching API handling in kuryr-controller/daemon | 11:56 |
| dulek | Maybe it's just a matter of some polling time tweaking? | 11:56 |
| aperevalov | is it a polling mode in kuryr, I thought it was blocking read. | 11:58 |
| celebdor | can you explain "I also that it was due to k8s watching API handling in kuryr-controller/daemon" some more? | 11:58 |
| aperevalov | I also thought it was due to problems in handling watch API | 11:59 |
| dulek | aperevalov: Yes, but we use requests library for watching the API. Maybe it does some buffering and only flushes every 1 second or something like this? | 11:59 |
| dulek | aperevalov: It's worth checking first IMO. | 12:00 |
| aperevalov | dulek, ok | 12:00 |
| celebdor | dulek: aperevalov: I think that's the first thing to check out. watch for the pod annotations | 12:01 |
| *** rh-jelabarre has joined #openstack-kuryr | 12:01 | |
| celebdor | using kubectl and record how late we see those | 12:01 |
| celebdor | if that's good enough, I'm not opposed to have cni daemon changed to use go client instead | 12:02 |
| celebdor | for example | 12:02 |
| dulek | celebdor, aperevalov: It would be best to just write a client in Python and golang and measure the difference. | 12:02 |
| celebdor | yup | 12:02 |
| celebdor | agreed | 12:02 |
| dulek | aperevalov: Then if that will not work, I think I'd prefer to use some other protocol than RabbitMQ - grpc maybe? | 12:03 |
| celebdor | grpc is not a message broker though | 12:04 |
| aperevalov | ok, as I see K8sClient's watch just post a request periodically, but not listen a socket. Ok I'll strace golang's client. | 12:04 |
| aperevalov | where to write updates on it? | 12:05 |
| aperevalov | in blueprint I created? | 12:05 |
| celebdor | for example | 12:06 |
| celebdor | sounds like a good place | 12:06 |
| dulek | aperevalov: Just a thought here. You're aware that we're waiting for pod annotations in a loop that has a period of 1 second? | 12:06 |
| dulek | aperevalov: https://github.com/openstack/kuryr-kubernetes/blob/0c2d88d8d39420f80e333c51cd87c286f4c7c4c8/kuryr_kubernetes/cni/plugins/k8s_cni_registry.py#L114-L118 | 12:06 |
| dulek | celebdor: Do we really need a message broker? If waiting for message times out just fall back to reading the annotation. | 12:07 |
| celebdor | I'm not saying we need it | 12:07 |
| celebdor | I'm just saying that it's a very different thing | 12:08 |
| celebdor | gotta step out for a while to feed the kids | 12:09 |
| celebdor | ttyl | 12:09 |
| aperevalov | dulek, thanks, right now I know. | 12:09 |
| *** takamatsu has quit IRC | 12:10 | |
| *** pcaruana has joined #openstack-kuryr | 12:10 | |
| dulek | aperevalov: And just in case - same happens when waiting for it to become active: https://github.com/openstack/kuryr-kubernetes/blob/0c2d88d8d39420f80e333c51cd87c286f4c7c4c8/kuryr_kubernetes/cni/plugins/k8s_cni_registry.py#L69-L78 | 12:11 |
| dulek | Maybe, just maybe, you only need to tweak those? | 12:11 |
| aperevalov | As I remember we discussed with ltomasbo, he suggested to not wait all VIF, just wait for main VIF. Probably first VIF. | 12:16 |
| *** takamatsu has joined #openstack-kuryr | 12:31 | |
| aperevalov | I did strace of kubectl. Looks like it's not possible to attach file into launchpad. So I put it into etherpad, please see (https://etherpad.openstack.org/p/strace_of_kubectl) | 12:51 |
| aperevalov | so we can see: it does epoll on socket 3 (4th it's epoll's fd). There is no requests to get updates. It just awake when new bytes in fd 3 appears. So it follows another approach. What do you think about this approach and implementing it in k8s_client. I know python is not fully multithreaded. | 12:53 |
| celebdor | worth a try | 13:03 |
| *** gkadam has quit IRC | 13:28 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Unset --admission-control when starting K8s API https://review.opendev.org/673819 | 13:54 |
| *** takamatsu has quit IRC | 14:28 | |
| aperevalov | celebdor: "worth a try" what about it was? reducing number of VIF to wait for or epoll approach in golang? | 14:50 |
| celebdor | "implementing it in k8s_client" | 14:52 |
| celebdor | in your message from an hour ago | 14:52 |
| aperevalov | yes ) | 14:58 |
| *** gdwornicki has joined #openstack-kuryr | 15:16 | |
| *** gcheresh has quit IRC | 15:17 | |
| *** takamatsu has joined #openstack-kuryr | 16:47 | |
| *** takamatsu has quit IRC | 16:59 | |
| *** aperevalov has quit IRC | 17:00 | |
| *** gdwornicki has quit IRC | 17:36 | |
| *** gdwornicki has joined #openstack-kuryr | 17:39 | |
| *** gkadam has joined #openstack-kuryr | 17:43 | |
| *** gdwornicki has quit IRC | 18:04 | |
| *** gdwornicki has joined #openstack-kuryr | 18:08 | |
| *** altlogbot_2 has quit IRC | 18:19 | |
| *** altlogbot_0 has joined #openstack-kuryr | 18:20 | |
| *** gkadam has quit IRC | 18:23 | |
| *** takamatsu has joined #openstack-kuryr | 18:43 | |
| *** gdwornicki has quit IRC | 19:07 | |
| *** gdwornicki has joined #openstack-kuryr | 19:22 | |
| *** takamatsu has quit IRC | 19:47 | |
| *** takamatsu has joined #openstack-kuryr | 21:40 | |
| *** takamatsu has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!