| *** kmadac has quit IRC | 02:05 | |
| *** kmadac has joined #openstack-kuryr | 02:07 | |
| *** rh-jelabarre has quit IRC | 02:37 | |
| *** hongbin has joined #openstack-kuryr | 03:03 | |
| *** gcheresh has joined #openstack-kuryr | 03:45 | |
| *** gcheresh has quit IRC | 03:51 | |
| *** hongbin has quit IRC | 04:05 | |
| *** gcheresh has joined #openstack-kuryr | 05:06 | |
| *** dulek has quit IRC | 05:56 | |
| *** ccamposr has joined #openstack-kuryr | 06:03 | |
| *** dulek has joined #openstack-kuryr | 06:05 | |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Update sriov neutron ports with pci info https://review.openstack.org/642704 | 06:05 |
|---|---|---|
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Support sriovdp arbitrary resource names https://review.openstack.org/642491 | 06:05 |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Add PodResources service client https://review.openstack.org/651580 | 06:05 |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: [WIP] DPDK in baremetal containers using SR-IOV https://review.openstack.org/651581 | 06:05 |
| *** maysams has joined #openstack-kuryr | 07:13 | |
| *** maysams has quit IRC | 07:40 | |
| *** maysams has joined #openstack-kuryr | 07:44 | |
| *** alisanhaji has joined #openstack-kuryr | 08:04 | |
| *** brault has joined #openstack-kuryr | 08:28 | |
| alisanhaji | Hi dulek, so do you have an idea about how to make kuryr-cni work with a containerized kubelet? | 08:39 |
| dulek | alisanhaji: Can you show me the definition of that kubelet pod/container? Like what host volumes it has? | 08:40 |
| alisanhaji | dulek: the config.json of kubelet runc container says that are mounted /proc /dev /sys /etc/kubernetes /var/lib | 08:40 |
| dulek | alisanhaji: Hm, so no /var/run? How does it talk with Docker then? | 08:41 |
| dulek | alisanhaji: Wait, it's runc, so possibly there's a socket in /var/lib? | 08:45 |
| dulek | It must be like that. | 08:45 |
| alisanhaji | Here is the complete definition of kubelet container: http://paste.openstack.org/show/749160/ | 08:45 |
| dulek | alisanhaji: Okay, there's /run, probably it's where the socket we should talk to is… | 08:47 |
| dulek | alisanhaji: Can you check if it has Python binary? | 08:47 |
| alisanhaji | Indeed their is the docker.sock in /run but no docker binary | 08:48 |
| alisanhaji | no python binary either | 08:50 |
| dulek | alisanhaji: Okay, so there are two issues we would need to fix to make this work. One is that we hardcode /var/run/docker.sock - we would need to make it configurable or fetch it from somewhere. I guess kubelet does the same, so that one is easy. | 08:51 |
| dulek | alisanhaji: The more complicated, but not hopeless thing would be to switch `docker exec` to some cool `curl` call to that socket… | 08:51 |
| dulek | Oh, third issue, " | python -c "\${finder}" docker\`" - this pipe won't work. | 08:52 |
| dulek | Which, kinda sucks, because I don't want to write JSON processing in bash… | 08:52 |
| alisanhaji | I see what you want to do, but I am not sure how much time it would cost... :-( | 08:55 |
| dulek | alisanhaji: 1. is trivial, 2. ranges from easy to hard :P, 3. might be easy if Docker API offers some filtering. | 08:57 |
| dulek | alisanhaji: Oh crap, but is there even curl in that container? | 08:57 |
| alisanhaji | Just checked, and it does not... | 09:00 |
| dulek | :) | 09:00 |
| dulek | Okay, it's actually good. | 09:00 |
| dulek | alisanhaji: Because at this point it will be easier to just rewrite kuryr-cni in golang and inject a binary like a proper CNI plugin should do. | 09:01 |
| dulek | Since we dropped support for no kuryr-daemon, it actually isn't that much work. | 09:01 |
| dulek | alisanhaji: It's hard for me to prioritize this work at the moment, but we could definitely make it a goal for the cycle. | 09:02 |
| alisanhaji | dulek: that would be great! No worries I can get back to working on kuryr with magnum when it's ready | 09:03 |
| dulek | ltomasbo, dmellado: What do you think. ^ Talking about rewriting that part that calls kuryr-daemon to golang and injecting a binary as CNI plugin instead of a hacky bash script. | 09:03 |
| dulek | alisanhaji: I suspect that kubelet container has host networking? | 09:04 |
| alisanhaji | yes indeed | 09:05 |
| dulek | alisanhaji: Okay, so that should work. :) | 09:06 |
| dulek | alisanhaji: I'll propose a blueprint later on. | 09:06 |
| alisanhaji | dulek: Awesome, I will keep tracking the blueprints, thanks for the help :-D | 09:09 |
| *** danil has joined #openstack-kuryr | 09:10 | |
| dmellado | hey dulek, well, I wouldn't mind but I'd love checking a blueprint on that | 09:47 |
| dmellado | maysams: pong | 09:47 |
| * dmellado just landed... | 09:47 | |
| maysams | hey dmellado | 09:47 |
| maysams | dmellado: ohh I hope you have had a pleasant flight :) | 09:48 |
| dmellado | yep! my flight got delayed a bit but I'll survive xD | 09:49 |
| maysams | dmellado: XD | 09:50 |
| dmellado | so what's up? :D | 09:50 |
| dmellado | I was reading the backlog regarding the golang stuff, but I might be missing something more around | 09:50 |
| maysams | dmellado: I was gonna ask you if you could rebase your patch https://review.openstack.org/#/c/645139/ when you have sometime, because my dependants ps needed rebase | 09:51 |
| dmellado | maysams: sure thing, I'll tackle that so I don't block you ;) | 09:52 |
| maysams | dmellado: thank you ;) | 09:52 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Add ipBlock support to NP https://review.openstack.org/645139 | 09:55 |
| dmellado | maysams: there you go:P | 09:55 |
| maysams | dmellado: Thanks!! | 09:57 |
| dulek | dmellado, ltomasbo, alisanhaji: https://blueprints.launchpad.net/kuryr-kubernetes/+spec/golang-kuryr-cni | 10:46 |
| * dmellado reading | 10:47 | |
| *** maysams is now known as maysams|afk | 11:14 | |
| *** danil has quit IRC | 11:17 | |
| dulek | dmellado, ltomasbo, maysams|afk: What do you think about this folks: bug/1824332 ? | 11:18 |
| dulek | Hm, bot isn't linking? bug 1824332 | 11:18 |
| openstack | bug 1824332 in kuryr-kubernetes "Loadbalancer doesn't get recreated when deleted" [Undecided,New] https://launchpad.net/bugs/1824332 | 11:19 |
| dulek | Here you go. ^\ | 11:19 |
| dmellado | dulek: let's see | 11:19 |
| dulek | This was reported by livelace yesterday. I reproduced it very easily. | 11:19 |
| dmellado | so if we have a service and we delete the lb | 11:19 |
| dmellado | then boom? | 11:19 |
| dulek | dmellado: No, Kuryr seems fine until it notices some change, e.g. scale-up. | 11:20 |
| dmellado | hmmm I see | 11:24 |
| dmellado | dulek: well, as you mention it shouldn't happen, but I'd prefer to honor that and fix it | 11:25 |
| dmellado | I'll try reproducing it as well | 11:25 |
| dulek | dmellado: If you have setup hanging around it's easy. ;) | 11:26 |
| dmellado | yep, doing it just now xD | 11:26 |
| dulek | I'd really like to get ltomasbo's and maysams's opinions as they worked with LB code more closely. Because maybe it's crazy difficult. :P | 11:26 |
| dulek | dmellado: Ah, you can recover from the crash loop by deleting the svc. | 11:26 |
| dmellado | dulek: lol, thanks for the hint xD | 11:27 |
| *** danil has joined #openstack-kuryr | 11:37 | |
| *** rh-jelabarre has joined #openstack-kuryr | 12:05 | |
| *** maysams|afk is now known as maysams | 12:55 | |
| *** danil has quit IRC | 12:58 | |
| alisanhaji | dulek: thanks for the blueprint :) | 13:11 |
| *** pcaruana has quit IRC | 13:20 | |
| *** celebdor has quit IRC | 13:30 | |
| *** pcaruana has joined #openstack-kuryr | 13:42 | |
| ltomasbo | dulek, what whas the question? | 13:48 |
| ltomasbo | *was | 13:48 |
| *** gkadam has joined #openstack-kuryr | 14:02 | |
| *** gkadam has quit IRC | 14:03 | |
| maysams | dulek: I managed to reproduce the bug on mentioned | 14:13 |
| maysams | s/on/you | 14:13 |
| maysams | https://launchpad.net/bugs/1824332 | 14:14 |
| openstack | Launchpad bug 1824332 in kuryr-kubernetes "Loadbalancer doesn't get recreated when deleted" [Undecided,New] | 14:14 |
| maysams | dulek: this occurs because the annotations is not being removed when the lbaas is deleted | 14:15 |
| maysams | dulek: I believe this is not difficult to fix | 14:15 |
| maysams | dulek; and I agree with you that the lbaas should get recreated | 14:19 |
| dulek | ltomasbo: Your opinion about the bug mentioned above. ^ | 14:20 |
| dulek | maysams: We don't watch on loadbalancers on Octavia API, but you're right that probably deleting the annotations (only the State, I guess) when we notice that LB is gone should fix this. | 14:23 |
| dulek | Actually some OpenStack API poller thread wouldn't be too hard to implement. Or even better - thread that would inject "on_present" events into the queue. | 14:25 |
| maysams | dulek, "on_present" events for what resources of what api? | 14:35 |
| dulek | maysams: Hm, probably Endpoints on K8s API? | 14:36 |
| maysams | dulek, I think we could remove the annotation when we get a not_found or recreate the lbaas | 14:36 |
| dulek | maysams: Well, with LBaaS go all the other resources, so this isn't something we can easily do in one call, I think. | 14:37 |
| maysams | dulek, hmmm true | 14:37 |
| dulek | maysams: But yes, on "not found" we can remove LBaaSState and this will trigger an on_present and Kuryr will do its work by creating a new one. | 14:38 |
| dulek | maysams: And this if fine, I was just looking for a way to achieve recreation even if Service or Endpoint events are not triggered. | 14:39 |
| ltomasbo | dulek, regarding that bug | 14:40 |
| ltomasbo | I believe the loadbalancer is recreated if it gets deleted, right? | 14:40 |
| maysams | dulek, I sEE | 14:40 |
| maysams | see* | 14:40 |
| ltomasbo | the code calls create, and if it is there, then it tries to find it and load its information | 14:41 |
| dulek | ltomasbo: No, I tried doing `openstack delete loadbalancer --cascade <id>`, then scaled the env and Kuryr got into crash loop. | 14:41 |
| dulek | (due to healthchecks of course) | 14:41 |
| dulek | ltomasbo: In this case it wasn't looking for LB, I think, because it only needed to add members on scale-up | 14:42 |
| ltomasbo | umm, ahh, ok | 14:42 |
| dulek | Oh, it's _wait_for_provisioning that's failing… | 14:42 |
| ltomasbo | dulek, I think it just tries to recreate it when the kuryr-controller is restarted | 14:43 |
| ltomasbo | and I guess the annotation is not matching | 14:43 |
| dulek | ltomasbo: Restarting kuryr-controller wasn't triggering a recreation on my env. | 14:43 |
| dulek | Maybe we just broke that somehow. | 14:44 |
| ltomasbo | dulek, probably becuase the annotations were already there... | 14:44 |
| ltomasbo | anyway, not sure if going and killing openstack reousrces manually should be supported either... | 14:45 |
| ltomasbo | it is a nice thing to have, but... | 14:45 |
| dulek | ltomasbo: livelace hit that when switching to Octavia from LBaaS v2. | 14:45 |
| ltomasbo | not always possible | 14:45 |
| dulek | ltomasbo: Well, it's definitely the K8s way of doing things - I check the state, if it doesn't match the reality I fix the reality. | 14:46 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Ensure port_range_min is optional https://review.openstack.org/651805 | 14:47 |
| *** gcheresh has quit IRC | 14:48 | |
| ltomasbo | dulek, yes | 14:49 |
| maysams | ltomasbo: Is this the bug genadi mentioned today? ^ | 14:50 |
| ltomasbo | dulek, but for instance, in kubernetes, if you delete the machine object | 14:50 |
| ltomasbo | a new one is created | 14:50 |
| ltomasbo | but if you remove the openstack VM, but leave the machine object there, nothing will happen | 14:50 |
| ltomasbo | as k8s is only looking for k8s resources | 14:50 |
| ltomasbo | maysams, yes | 14:51 |
| maysams | ltoamsbo, okay | 14:51 |
| ltomasbo | maysams, though I think I read it from dulek and dmellado | 14:51 |
| ltomasbo | sorry I was at a meeting and was only connecting at times | 14:51 |
| dulek | maysams, ltomasbo: Yes, it was found by dmellado and gcheresh. | 14:51 |
| dulek | ltomasbo: Hm, so… VM is created from Machine objects by Operator? | 14:52 |
| ltomasbo | dulek, great! and yes, it must have been due to my previous patch removing that default | 14:52 |
| maysams | ltomasbo, that's fine | 14:52 |
| dulek | ltomasbo: If so, it'll get recreated - operators apart from watching from changes also periodically trigger "reconciliation" even without events. | 14:52 |
| dulek | s/from/for | 14:53 |
| ltomasbo | dulek, ahh, ok, maybe I didn't wait enough | 14:53 |
| ltomasbo | but in that case, we would need to implement something similar | 14:53 |
| ltomasbo | kuryr-reconciliation | 14:53 |
| ltomasbo | I agree we should move to use more and more CRDs to have the information about the openstack resources, and trigger the needed actions in case they are missing | 14:54 |
| dulek | ltomasbo: Ha, yes - both LBaaS structs should be only one Kuryr CRD that has spec and status. :) | 14:54 |
| dulek | ltomasbo: Okay, but I think you convinced me that this isn't really top priority. | 14:56 |
| dulek | livelace issue was somehow special - he switched from LBaaS v2 to Octavia. | 14:56 |
| dulek | I mean we can fix the bug, so on Service scale it'll work normally and recreate everything. | 14:58 |
| dulek | But to do proper "reconciliation" is harder and not top thing we need. | 14:58 |
| ltomasbo | dulek, to me, it sounds like a great thing to have, but I would prioritice other things | 14:59 |
| dulek | ltomasbo: Fair enough, let's keep the bug hanging and we'll get back to it. | 15:00 |
| ltomasbo | perhaps we can start moving to CRDs when we can, and new features can be 'forced' to have it there from the beginning | 15:00 |
| ltomasbo | anyway, let me check the code... not sure if the code should already handle lbaas creation in that case already | 15:01 |
| ltomasbo | dulek, seems if the lbaas_state annotations are there, we don't retrigger ensure_loadbalancer: https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/controller/handlers/lbaas.py#L562-L575 | 15:04 |
| ltomasbo | perhaps we need an extra find_loadblaancer to ensure that on updates... but that will slow down things for scaling/exposing actions | 15:05 |
| dulek | ltomasbo: Hm, that does make sense. | 15:05 |
| dulek | ltomasbo: We can always catch the 404 exception on member create and make sure that will delete the State annotation. | 15:06 |
| dulek | ltomasbo: And this will retrigger whole thing. | 15:06 |
| dulek | Can anything else trigger change in LB? I mean - can Service be modified? | 15:06 |
| ltomasbo | sounds like a nice workaround | 15:07 |
| ltomasbo | yes, we can expose different ports | 15:07 |
| ltomasbo | add an extra port exposed | 15:07 |
| ltomasbo | change the target... | 15:07 |
| dulek | ltomasbo: Credits for the workaround idea go to maysams. ;) | 15:07 |
| ltomasbo | :) | 15:07 |
| dulek | ltomasbo: Hm, so we should probably check failure modes for those as well and add deleting annotation there accordingly. | 15:08 |
| maysams | :-) | 15:08 |
| *** celebdor has joined #openstack-kuryr | 15:38 | |
| *** pcaruana has quit IRC | 15:42 | |
| *** premsankar has joined #openstack-kuryr | 16:01 | |
| *** ccamposr has quit IRC | 16:03 | |
| *** mrostecki has quit IRC | 16:09 | |
| aperevalov | dulek, hi! Are you here? | 16:10 |
| dulek | aperevalov: Yup, what's up? | 16:10 |
| aperevalov | oh, I just want to ask, did you faced with issue in HA. When one master goes down and CNI on minion can't reconnect to VIP, and initial k8s connection was made by VIP. Or you just covered haproxy use case? | 16:12 |
| aperevalov | now I tried through haproxy, the same result: Connection broken. | 16:14 |
| aperevalov | But after cni restart (w/o reconfiguring url) it's ok. So I assume our kuryr cni doesn't handle Connection broken correctly. | 16:14 |
| aperevalov | of course daemon was used ;) | 16:14 |
| *** mrostecki has joined #openstack-kuryr | 16:18 | |
| aperevalov | I cal show you a call stack | 16:19 |
| dulek | aperevalov: Hm, so daemon haven't reconnected? | 16:23 |
| aperevalov | yes, it didn't | 16:24 |
| dulek | aperevalov: Traceback would be useful here. It's master or stable/something? | 16:24 |
| aperevalov | https://etherpad.openstack.org/p/cni-ha-reconnect-problem | 16:25 |
| dulek | Okay, so this was fixed long time ago, it should recover from this. | 16:26 |
| dulek | Is it some old Kuryr? | 16:26 |
| aperevalov | probably yes, this build wasn't prepared by me. And I didn't yet check on master. Thanks! I'll recheck on master. | 16:27 |
| dulek | aperevalov: Okay, thanks! | 16:35 |
| *** aperevalov has quit IRC | 16:58 | |
| *** pcaruana has joined #openstack-kuryr | 17:05 | |
| *** maysams has quit IRC | 17:18 | |
| *** alisanhaji has quit IRC | 18:00 | |
| *** gmann is now known as gmann_afk | 18:15 | |
| *** pcaruana has quit IRC | 19:02 | |
| *** gmann_afk is now known as gmann | 19:27 | |
| *** gcheresh has joined #openstack-kuryr | 20:35 | |
| *** gcheresh has quit IRC | 20:48 | |
| *** premsankar has quit IRC | 23:11 | |
| *** kmadac has quit IRC | 23:21 | |
| *** irclogbot_1 has quit IRC | 23:24 | |
| *** kmadac has joined #openstack-kuryr | 23:27 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!