| *** irclogbot_3 has joined #openstack-kuryr | 00:17 | |
| *** celebdor has quit IRC | 00:36 | |
| *** hongbin has joined #openstack-kuryr | 02:22 | |
| *** hongbin has quit IRC | 04:12 | |
| *** spsurya has quit IRC | 04:31 | |
| *** lxkong has quit IRC | 04:32 | |
| *** spsurya has joined #openstack-kuryr | 04:34 | |
| *** lxkong has joined #openstack-kuryr | 04:34 | |
| *** janki has joined #openstack-kuryr | 05:04 | |
| *** dims has quit IRC | 06:39 | |
| *** dims has joined #openstack-kuryr | 06:41 | |
| *** pcaruana has joined #openstack-kuryr | 07:27 | |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Annotate nodes with pci info for direct ports https://review.openstack.org/642703 | 07:41 |
|---|---|---|
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Update sriov neutron ports with pci info https://review.openstack.org/642704 | 07:41 |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Annotate nodes with pci info for direct ports https://review.openstack.org/642703 | 07:55 |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Update sriov neutron ports with pci info https://review.openstack.org/642704 | 07:55 |
| *** celebdor has joined #openstack-kuryr | 08:08 | |
| *** maysams has joined #openstack-kuryr | 08:14 | |
| dulek | maysams: Hey, so do you have an idea why the LB member pod isn't added the ingress rule? | 08:23 |
| dulek | maysams: Is that an overlook from our side? Misunderstanding of NP spec? Some bug? | 08:23 |
| maysams | dulek, I'm reproducing that test locally | 08:23 |
| maysams | dulek, to analyze it better | 08:24 |
| dulek | maysams: Cool, do you want the pod definitions? | 08:24 |
| maysams | dulek, Yup | 08:25 |
| dulek | maysams: http://pastebin.test.redhat.com/742258 | 08:27 |
| dulek | maysams: This is result of `get all -o yaml`, note that there's a Service definition on the bottom. | 08:27 |
| maysams | dulek: Right, thanks | 08:28 |
| dulek | maysams: Ah, you either need coredns to be running or just change the command in line 55 to call service IP instead of svc-server.network-policy-965. | 08:28 |
| maysams | dulek: okay | 08:29 |
| dulek | maysams: Ah, also note that I changed client-a definition to retry 10000000 times instead of 5. :D | 08:30 |
| maysams | dulek: XDD okay | 08:35 |
| maysams | dulek, could you also get the namespace definition? | 08:41 |
| dulek | maysams: Sure: http://pastebin.test.redhat.com/742264 | 08:44 |
| dulek | maysams: finalizers? What's that? | 08:44 |
| maysams | dulek, finalizers? First time I see it | 08:45 |
| *** alisanhaji has joined #openstack-kuryr | 08:45 | |
| dulek | maysams: Okay, that's nothing too important really, just says that before ns can be deleted k8s needs to purge all of it's resources. | 08:46 |
| dulek | s/it's/its | 08:46 |
| maysams | dulek: yup yup, just googled it | 08:47 |
| maysams | dulek: do you have some secret defined? | 09:14 |
| dulek | maysams: http://paste.openstack.org/raw/748229/ | 09:16 |
| dulek | maysams: But it should get created along with namespace automatically, I think. | 09:17 |
| maysams | dulek, interesting.. It was not created, when I created the new namespace. I only have the default one | 09:19 |
| dulek | maysams: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/framework.go#L216-L231 | 09:21 |
| dulek | This is how it creates the Namespace. Nothing out of ordinary there, IMO. | 09:22 |
| *** celebdor has quit IRC | 09:48 | |
| maysams | dulek: did your server pod ever got not ready? | 09:51 |
| maysams | dulek: because it could not access port 80 and 81 | 09:52 |
| dulek | maysams: Uh, I think I lost your first message due to some disconnect. | 09:54 |
| maysams | right | 09:54 |
| maysams | dulek: I asked if your server pod ever got not ready due to not being able to access port 80 and 81 | 09:54 |
| dulek | maysams: No, it worked fine. | 09:55 |
| dulek | That would be weird, really… Can you share the log? | 09:55 |
| maysams | well it's running but not ready | 09:56 |
| maysams | so I just described the pod | 09:56 |
| maysams | and got | 09:56 |
| maysams | aah, I just saw the log Error from server (BadRequest): a container name must be specified for pod server, choose one of: [server-container-80 server-container-81] | 09:56 |
| maysams | dulek: sry.. I will try to fix that | 09:57 |
| dulek | maysams: `kubectl -n <namespace> logs <pod-name> -c server-container-80` | 09:58 |
| dulek | :) | 09:58 |
| dulek | maysams: It's just complaining that it doesn't know logs of which container from the pod you want. | 09:59 |
| *** maysams has quit IRC | 10:06 | |
| *** celebdor has joined #openstack-kuryr | 10:06 | |
| *** maysams has joined #openstack-kuryr | 10:57 | |
| *** spsurya has quit IRC | 10:59 | |
| *** spsurya has joined #openstack-kuryr | 11:00 | |
| maysams | dulek: sorry I needed to be afk for a few min | 11:05 |
| kmadac3 | @dulek: Hi yesterday I added support for revisions into macvlan driver and it helped. Are you ok, that I create change to gerit? | 11:37 |
| dulek | kmadac3: Definitely! I'm happy to review it. Please just run `tox -epep8,py27` before submitting to make sure tests are passing. | 11:38 |
| dmellado | kmadac3: please do so! | 11:43 |
| dmellado | yep, and soon enough, py3* as well, please | 11:44 |
| dmellado | kmadac3: if you haven't done before, please take a look at https://docs.openstack.org/infra/manual/developers.html | 11:44 |
| dmellado | thanks! | 11:44 |
| kmadac3 | @dulek: ok I will do it asap | 11:49 |
| *** pcaruana has quit IRC | 11:53 | |
| *** rh-jelabarre has joined #openstack-kuryr | 12:12 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Add ipBlock support to NP https://review.openstack.org/645139 | 12:41 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Add ipBlock support to NP https://review.openstack.org/645139 | 12:43 |
| *** pcaruana has joined #openstack-kuryr | 12:54 | |
| *** altlogbot_3 has quit IRC | 13:01 | |
| *** irclogbot_3 has quit IRC | 13:01 | |
| *** irclogbot_3 has joined #openstack-kuryr | 13:02 | |
| *** altlogbot_1 has joined #openstack-kuryr | 13:02 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Add ipBlock support to NP https://review.openstack.org/645139 | 13:38 |
| *** ccamposr has joined #openstack-kuryr | 14:06 | |
| *** alisanhaji has quit IRC | 14:27 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Fix cri-o gate https://review.openstack.org/644867 | 14:50 |
| *** altlogbot_1 has quit IRC | 15:21 | |
| *** altlogbot_2 has joined #openstack-kuryr | 15:25 | |
| *** irclogbot_3 has quit IRC | 15:30 | |
| *** irclogbot_0 has joined #openstack-kuryr | 15:32 | |
| *** irclogbot_0 has quit IRC | 15:36 | |
| *** irclogbot_1 has joined #openstack-kuryr | 15:37 | |
| *** ccamposr has quit IRC | 15:43 | |
| dulek | maysams: Good news! | 16:08 |
| maysams | dulek: tell me | 16:08 |
| dulek | maysams: It does work with Amphora. | 16:08 |
| dulek | So at least it's only ovn provider quirks. | 16:08 |
| maysams | dulek: but I think it's still wrong | 16:09 |
| dulek | And as it isn't a feature yet, we don't need to worry about it today IMO. | 16:09 |
| dulek | maysams: Oh well. :P | 16:09 |
| maysams | could you share the client sg? | 16:09 |
| maysams | dulek: ^ | 16:10 |
| dulek | maysams: Oh crap, I might still have the rule that I created to fix this on my env. Okay, let me try again. | 16:10 |
| maysams | okay | 16:10 |
| dulek | maysams: Okay, it worked even after I deleted that rule. | 16:12 |
| maysams | dulek: I think it's working because of the default egress rle | 16:13 |
| maysams | rule | 16:13 |
| dulek | maysams: http://paste.openstack.org/raw/748252/ | 16:14 |
| dulek | maysams: This is on client-a port. | 16:14 |
| dulek | maysams: First rule seems to open it for the server, which is good, isn't it? | 16:14 |
| maysams | dulek: yes.. but we are trying to access the svc | 16:15 |
| maysams | in the clietn pod | 16:15 |
| dulek | maysams: Yes, which is fine, client has egress rule allowing it to call the svc, right? | 16:15 |
| maysams | so.. that rule should be with the remote_ip refering to the svc, not to the server | 16:16 |
| dulek | maysams: Ah, I see. | 16:16 |
| maysams | dulek: does that makes sense? | 16:16 |
| maysams | dulek, I'm trying to figure it out if this is wrong or not :P | 16:17 |
| dulek | maysams: It called the svc IP - "svc-server.network-policy-4081 (10.1.0.150:80) open" | 16:17 |
| dulek | maysams: It does make sense to me. But somehow it works… | 16:17 |
| dulek | maysams: Ah, there are rules allowing egress everywhere? | 16:17 |
| *** gaoyan has joined #openstack-kuryr | 16:17 | |
| maysams | yup | 16:17 |
| dulek | maysams: Want me to check what happens if I delete them? | 16:17 |
| maysams | yess :) | 16:18 |
| dulek | maysams: Okay, let's see. Where are those rules coming from? | 16:19 |
| dulek | maysams: I mean those two egress ones that I just deleted? | 16:19 |
| dulek | maysams: Because it's not the default SG, it's definitely an SG created by kuryr-controller. | 16:19 |
| dulek | maysams: Yep, it lost ability to connect. | 16:20 |
| maysams | dulek: good.. so now we know the problem | 16:20 |
| dulek | maysams: Shall I add a rule allowing it to call the LB IP? | 16:21 |
| maysams | yupp, please | 16:21 |
| *** gaoyan has quit IRC | 16:21 | |
| dulek | openstack security group rule create --egress --protocol tcp --remote-ip 10.1.0.150 60edc57f-29b1-4f18-ace3-07713df196bf | 16:22 |
| *** janki has quit IRC | 16:22 | |
| maysams | that's right | 16:23 |
| dulek | maysams: "svc-server.network-policy-4081 (10.1.0.150:80) open" | 16:23 |
| dulek | Let me delete it and try again. | 16:23 |
| dulek | maysams: Yep, it stops working again. So I guess we see the issue now? | 16:25 |
| maysams | dulek: definitely | 16:26 |
| maysams | dulek: That network policy is restricting egress access to all namespaces in a specific port | 16:27 |
| maysams | dulek: and the approach I'm using in the patch is adding a new SG rule with the remote_ip of the pod that has a container matching the named port | 16:27 |
| maysams | dulek: That's why we have that IP and not a rule allowing from everywhere | 16:28 |
| dulek | maysams: Wait, I think I'm still testing with your patch. | 16:29 |
| maysams | dulek: you mean that specific test or the others tests? | 16:30 |
| dulek | maysams: At the moment I only run that specific test. | 16:32 |
| maysams | dulek, Okay. Btw, thanks a lot for checking these things ;) | 16:33 |
| dulek | maysams: Oh crap, I don't know what am I running anymore. :P Let me check again… | 16:33 |
| maysams | XDD | 16:33 |
| dulek | maysams: I do have your patch applied and I think I had it all the time. | 16:36 |
| dulek | maysams: (I've checked the code inside the container) | 16:37 |
| maysams | dulek, which is kinda good, right? Because we found the error | 16:38 |
| maysams | dulek, the reason why the test was not passing | 16:39 |
| dulek | maysams: I guess? :) Any idea why those default allow-all-egress are added in Octavia case but not the OVN case? | 16:39 |
| maysams | dulek: In OVN they are added as well | 16:39 |
| dulek | maysams: Ooooh, waaaait… | 16:39 |
| dulek | maysams: "sg_mode = create" - I have this on. | 16:40 |
| maysams | I have the update one | 16:40 |
| dulek | maysams: Okay, I'm not sure about this one, but whatever it's about VIP rules, not client. | 16:41 |
| maysams | dulek: yup.. btw with OVN the default egress rules was also present in the client pod http://paste.openstack.org/raw/748190/ | 16:41 |
| dulek | maysams: Hm… | 16:42 |
| dulek | maysams: I think what we observed is that client-a is failing to connect with OVN provider and working fine with Amphora? | 16:42 |
| maysams | dulek: you shared this with me yesterday ^ | 16:42 |
| dulek | Yes, yes, I remember. :) | 16:42 |
| maysams | dulek: yes.. and in both cases the default egress rule is created | 16:43 |
| dulek | maysams: So what's the difference…? | 16:43 |
| dulek | maysams: Okay, I think this is enough mysteries for Friday evening. I have the env, I can switch it back to OVN and we can analyze it again on Monday. | 16:44 |
| maysams | ahahah | 16:44 |
| maysams | dulek, sure thing | 16:44 |
| maysams | dulek: Thanks a lot :) | 16:44 |
| dulek | maysams: Thanks for taking this seriously. ;) Have a great weekend! | 16:45 |
| maysams | dulek, you too! | 16:45 |
| *** maysams has quit IRC | 16:50 | |
| *** celebdor has quit IRC | 16:58 | |
| *** gmann is now known as gmann_afk | 17:52 | |
| *** gmann_afk is now known as gmann | 18:11 | |
| *** celebdor has joined #openstack-kuryr | 18:40 | |
| *** pcaruana has quit IRC | 18:45 | |
| *** celebdor has quit IRC | 18:50 | |
| *** aperevalov has quit IRC | 19:00 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!