| opendevreview | Roman Dobosz proposed openstack/kuryr-kubernetes master: Fixes for latest changes on Neutron devstack. https://review.opendev.org/c/openstack/kuryr-kubernetes/+/794200 | 09:13 |
|---|---|---|
| opendevreview | Merged openstack/kuryr-kubernetes master: Update the Service Creation Process https://review.opendev.org/c/openstack/kuryr-kubernetes/+/795363 | 09:57 |
| opendevreview | Sunday Mgbogu proposed openstack/kuryr-kubernetes master: Kuryr Kubernetes Loadbalancers Reconciliation Design https://review.opendev.org/c/openstack/kuryr-kubernetes/+/796234 | 10:17 |
| opendevreview | Roman Dobosz proposed openstack/kuryr-kubernetes master: Minor formatting corrections. https://review.opendev.org/c/openstack/kuryr-kubernetes/+/797869 | 10:17 |
| opendevreview | Michał Dulko proposed openstack/kuryr-kubernetes master: Add original exception to log abou dead component https://review.opendev.org/c/openstack/kuryr-kubernetes/+/797876 | 11:05 |
| opendevreview | Michał Dulko proposed openstack/kuryr-kubernetes master: Add original exception to log about dead component https://review.opendev.org/c/openstack/kuryr-kubernetes/+/797876 | 11:07 |
| opendevreview | Roman Dobosz proposed openstack/kuryr-kubernetes master: Fixes for latest changes on Neutron devstack. https://review.opendev.org/c/openstack/kuryr-kubernetes/+/794200 | 11:07 |
| digitalsimboja | Hello! | 12:34 |
| digitalsimboja | @maysams, @ltomasbo: I am reworking the reconciliation flow and wish to share my line of thoughts around that? | 12:34 |
| opendevreview | Sunday Mgbogu proposed openstack/kuryr-kubernetes master: Kuryr Kubernetes Loadbalancers Reconciliation Design https://review.opendev.org/c/openstack/kuryr-kubernetes/+/796234 | 12:57 |
| digitalsimboja | Please take a look above | 12:58 |
| digitalsimboja | Thanks | 12:58 |
| digitalsimboja | Here is how I view it: | 12:58 |
| digitalsimboja | If OpenStack resource is deleted or modified, KuryrLoadBalancer which occasionally 'pools' the OpenStack resources notices | 12:59 |
| digitalsimboja | And sends a patch to k8s lb CRD with empty status | 13:00 |
| ltomasbo | digitalsimboja, checking | 13:02 |
| ltomasbo | digitalsimboja, some inputs there as I see it | 13:02 |
| ltomasbo | 1) add Handler to the KuryrLoadBalancer box | 13:03 |
| ltomasbo | 2) perhaps this needs to be modified by something more meaningful "LB deleted/modified/klb ev()" | 13:03 |
| digitalsimboja | noting... | 13:04 |
| ltomasbo | I assume that means the KLB Handler is doing some "get loadbalancers" and then it detects if there are some missing | 13:04 |
| digitalsimboja | sure | 13:04 |
| ltomasbo | also, the ServiceHandler/EndpointHandler rows there look wrong to me | 13:05 |
| ltomasbo | the spec is not removed/lost, so they do not need to recreate them | 13:05 |
| digitalsimboja | Thats my confusion honestly | 13:05 |
| ltomasbo | they are only involved when the service/endpoints gets created/deleted/modified, i.e., when k8s actions take place, not when OpenStack action take place (deletion of loadbalancer, as in your case here) | 13:05 |
| digitalsimboja | They are alredy holding the spec | 13:06 |
| ltomasbo | digitalsimboja, perhaps you can try to write that sequence diagram for the service creation first, shared with us to ensure you got it right, and then, we go for updating this one for the reconciliation | 13:07 |
| ltomasbo | it would be a great addition to the documentation to have such a figure for the service/endpoints creation | 13:07 |
| digitalsimboja | That would make sense | 13:07 |
| ltomasbo | and I think it will help you with the flow too | 13:07 |
| digitalsimboja | Sure! Because we are more or less doing something like a reverse of the k8s event in this case? | 13:08 |
| ltomasbo | not really | 13:09 |
| ltomasbo | we just need to checkout that OpenStack resource is missing and trigger the recreation (same flow as when creating the service) | 13:09 |
| ltomasbo | and as you already have there, the recreation is triggered by removing the status section | 13:10 |
| ltomasbo | so, you are missing how/where to detect the OpenStack resource is missing | 13:10 |
| ltomasbo | then the second step, how to trigger the recreation you already know about it (removing the status on the related KLB CRD) | 13:10 |
| digitalsimboja | My thoughts around detecting missing OpenStack resource: I believe this can be handled by the KuryrLoadBalancerHandler since it had a way of patching to k8s CRD with empty status | 13:13 |
| digitalsimboja | So I am wondering the reconcile mechanism should be implemented right there sort of | 13:13 |
| digitalsimboja | So am thinking of doing something like this: | 13:14 |
| digitalsimboja | on the KuryrLoadBalancerHandler | 13:14 |
| digitalsimboja | get_loadbalancer ev() | 13:14 |
| digitalsimboja | Then compare with loadbalancer_crd | 13:15 |
| digitalsimboja | and then trigger the recreation is anything is missing | 13:15 |
| ltomasbo | there is some problems with that approach | 13:15 |
| digitalsimboja | But the get_loadbalancer ev() should call Octavia API after some time | 13:15 |
| ltomasbo | 1) the handler is executed for specific KLB CRD (i.e., for specific services) | 13:16 |
| ltomasbo | so, it is not something that runs in the background regularly | 13:16 |
| ltomasbo | 2) the OpenStack calls are usually on the drivers... so I'm more inclined to add that on the lbaasv2 driver instead (but we will need to circle back to removing the CRD status as you mentioned, so not completely sure about this | 13:17 |
| ltomasbo | what is get_loadbalancer_ev()? | 13:17 |
| ltomasbo | ev stands for events? | 13:17 |
| digitalsimboja | clients.get_loadbalancer_clients sort of | 13:18 |
| ltomasbo | there is no events in the OpenStack Octavia API you can subscribe to... the call will be kind of "give me all the loadbalancers" | 13:18 |
| ltomasbo | and then you will need to process all the KLB CRDs status, and check if the loadbalancer ID in the CRD status is on the retrieved information about all those loadbalancers on OpenStack side | 13:18 |
| digitalsimboja | soemthing like this would get me the entire loadbalancers I think: | 13:19 |
| digitalsimboja | def get_octavia_version(self): | 13:19 |
| digitalsimboja | lbaas = clients.get_loadbalancer_client() | 13:19 |
| digitalsimboja | lbaas should get me the loadbalancer to talk to Octavia? | 13:19 |
| digitalsimboja | the client sorry | 13:19 |
| digitalsimboja | If that is the case then I need to figure out how to regularly make the call to fetch the loadbalancers from OpenStack using the client? | 13:20 |
| ltomasbo | yep, there is some background threads in other drivers doing something similar | 13:21 |
| ltomasbo | perhaps you can take a look at them, and add that on the lbaasv2 driver | 13:21 |
| digitalsimboja | perfect | 13:21 |
| ltomasbo | but would be nice if you also get someone else inputs on this | 13:22 |
| digitalsimboja | sure! | 13:22 |
| digitalsimboja | So for now, I would need to modify the flow to reflect that lbaasv2 driver should call Octavia API to fetch all the loadbalancers | 13:24 |
| digitalsimboja | then | 13:24 |
| ltomasbo | for now I would try to write a similar image to the one you have for the service creation flow | 13:26 |
| ltomasbo | to help you understand the creation flow (and to later improve documentation with that) | 13:26 |
| ltomasbo | then, we can re-take the existing one, and have kind of a mix betwee nthe creating flow and the one you have | 13:27 |
| ltomasbo | as the difference will be the triggering mechanism, but the second part should be exactly the same | 13:27 |
| digitalsimboja | Perfect | 13:27 |
| digitalsimboja | For now let me check other drivers and see how they regularly call OpenStack? | 13:32 |
| ltomasbo | vif_pool does some of those | 13:35 |
| ltomasbo | before the lbaas has also some code related to that, let me try to find it on an old repo | 13:35 |
| ltomasbo | actually, we had it on the handler init https://github.com/openshift/kuryr-kubernetes/blob/release-3.11/kuryr_kubernetes/controller/handlers/lbaas.py#L175 | 13:37 |
| ltomasbo | which means, perhaps you are right and the best place is the KuryrLoadBalancerHandler | 13:37 |
| ltomasbo | actually, that function was to clean up loadbalancers, and you can take ideas from there, as it was using the OpenStack API to get the OpenStack loadbalancers... | 13:39 |
| ltomasbo | digitalsimboja, ^ | 13:39 |
| ltomasbo | digitalsimboja, I wrote some information that could be useful for you, not sure if you saw it | 13:42 |
| ltomasbo | as you disconnected right after | 13:42 |
| digitalsimboja | No I did not | 13:43 |
| ltomasbo | <ltomasbo> vif_pool does some of those | 13:43 |
| ltomasbo | <ltomasbo> before the lbaas has also some code related to that, let me try to find it on an old repo | 13:43 |
| ltomasbo | <ltomasbo> actually, we had it on the handler init https://github.com/openshift/kuryr-kubernetes/blob/release-3.11/kuryr_kubernetes/controller/handlers/lbaas.py#L175 | 13:43 |
| ltomasbo | <ltomasbo> which means, perhaps you are right and the best place is the KuryrLoadBalancerHandler | 13:43 |
| ltomasbo | <ltomasbo> actually, that function was to clean up loadbalancers, and you can take ideas from there, as it was using the OpenStack API to get the OpenStack loadbalancers... | 13:43 |
| ltomasbo | digitalsimboja, ^ | 13:43 |
| digitalsimboja | Thanks | 13:43 |
| maysams | reading... | 13:46 |
| maysams | ltomasbo: regarding adding the extra thread, in the example you share it would only run upon restart of the controller meaning it would only get the Services and lbs at that specific moment | 13:57 |
| maysams | it might happens that some services are create after the controller is up and not taken into account? | 13:57 |
| ltomasbo | unless the thread is running every X mins, right? and redoing the actions, right? | 13:58 |
| maysams | I just thought if it would make sense to rely on the reconciliation mechanism we have in Kuryr, but this might increase the number of calls to Octavia | 13:58 |
| ltomasbo | the example I pasted was just to be executed at the begining (kuryr-controller start), but we can have another one that never finishes... and rerun the same loop every 10 mins or so | 13:59 |
| maysams | this one done once | 13:59 |
| ltomasbo | right, that will mean for each existing KLB CRD there will be a call to Octavia API, right? | 14:00 |
| maysams | my pasting is somehow not working here... hm | 14:00 |
| ltomasbo | that won't scale really well I'm afraid | 14:00 |
| maysams | yeah | 14:01 |
| maysams | digitalsimboja: as Luis mentioned the service and endpoints handlers does not have impact during reconciliation. In Kuryr, the communication would be between kuryrloadbalancer CRD and lbaasv2 | 14:05 |
| digitalsimboja | Understood! | 14:06 |
| digitalsimboja | They are only involved when an event occurs on K8s, so I have to scrap those | 14:06 |
| maysams | yes | 14:06 |
| digitalsimboja | Now coming back to the real problem, I think what we need is a task schedular sort of that runs every x mins? | 14:07 |
| digitalsimboja | That is makes a call to OpenStack every x mins | 14:07 |
| maysams | yes, have you checked the Link Luis provided? | 14:07 |
| digitalsimboja | I am looking at it yeah | 14:08 |
| maysams | that is a good examples on how to fetch data every x time in a separete thread | 14:08 |
| digitalsimboja | Also, the vif_pool driver uses the eventlet.spawn method | 14:17 |
| digitalsimboja | Taking a look | 14:17 |
| opendevreview | Michał Dulko proposed openstack/kuryr-kubernetes master: Use correct logger when setting pyroute2 log level https://review.opendev.org/c/openstack/kuryr-kubernetes/+/797956 | 15:04 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!