Thursday, 2019-03-21

*** janki has quit IRC02:44
*** rh-jelabarre has quit IRC03:21
*** janki has joined #openstack-kuryr04:00
*** janki has quit IRC05:20
*** janki has joined #openstack-kuryr05:51
*** gcheresh has joined #openstack-kuryr06:05
*** pcaruana has joined #openstack-kuryr06:11
*** ccamposr has joined #openstack-kuryr07:08
*** aperevalov has joined #openstack-kuryr08:02
dulekltomasbo: I've debugged that ERROR happening in kuryr-daemon logs. Want a surprise? Same happens in n-cpu logs on gates with VM's.08:03
dulekltomasbo: So it's more of OVN DevStack plugin bug. Who should I talk to about it?08:03
ltomasbodulek, umm08:04
ltomasbodulek, dalvarez most probably08:04
dulekltomasbo: Great, thanks!08:04
ltomasbodulek, have you tried to run the commands manually?08:05
dulekltomasbo: Yes, they don't work. This isn't that complicated really, it's only because OVN's DevStack plugin puts OVN on a file socket.08:05
ltomasbodulek, the command failing is similar to this one right: http://dani.foroselectronica.es/ovn-routing-and-ovn-trace-550/08:05
dulekWhile the default from kuryr.conf/nova.conf is 127.0.0.1:6640.08:06
ltomasboahh, ok08:06
dulekltomasbo: So it either needs to follow the default or [os_vif_ovs]ovsdb_i-don't-remember = <path to file socket>08:06
dulekor set.08:06
ltomasboahh, ok08:07
dulekAh and not OVN on file socket, OVS DB is put on file socket. :P08:08
* dulek is still drinking first coffee.08:08
ltomasbo:)08:08
*** openstackgerrit has quit IRC08:17
*** maysams has joined #openstack-kuryr08:21
*** gkadam has joined #openstack-kuryr08:33
*** gkadam has quit IRC08:33
*** alisanhaji has joined #openstack-kuryr08:50
*** celebdor has joined #openstack-kuryr09:07
*** pcaruana has quit IRC09:45
*** pcaruana has joined #openstack-kuryr09:46
*** alisanhaji has quit IRC10:40
dulekmaysams: Should I run the tests again?10:42
maysamsdulek: That you be really great :)10:43
dulekmaysams: Sure, no problem.10:43
maysamsdulek: I was trying to set up everything manually10:43
maysamsdulek: Thanks :D10:43
maysamss/you/would10:44
*** alisanhaji has joined #openstack-kuryr10:54
*** openstackgerrit has joined #openstack-kuryr11:04
openstackgerritMaysa de Macedo Souza proposed openstack/kuryr-kubernetes master: Add support for text ports on Network Policy Spec  https://review.openstack.org/64308011:04
openstackgerritDaniel Mellado proposed openstack/kuryr-kubernetes master: [WIP] Add ipBlock support to NP  https://review.openstack.org/64513911:36
dmelladomaysams: ltomasbo that's just some idea that came to me about how to handle ipblock11:42
dmelladofeel free to comment it there11:42
ltomasbodmellado, checking!11:43
ltomasboit may actually work! yes! I suppose you only need to cover the except case there11:45
ltomasbodmellado, ^^11:45
dmelladoyep, I'll be adding that and a few test and check11:45
ltomasbobut yep, that was the direction I was thinking too (without paying attention to the corner cases...)11:45
ltomasbodmellado, I guess we need to double check how that affects other stuff (services isolation, reaction to namespace, pod creation/re-labeling)11:46
dmelladoyep, totally, in any case as a first glance they shouldn't be affected as it's just another set of cidrs to add11:46
dmelladowhich would only affect the ipblock cidr11:47
dmelladobut let's see how the tests behave11:47
dmelladoin any case ltomasbo maysams11:48
dmelladoI saw that you did quite a few optimizations over my original code11:48
dmelladogood work! xD11:48
ltomasboyep, we hit a few corner cases... mostly about reacting to events11:48
ltomasbopods creation/relabeling may not be affected11:49
ltomasbonamespaces may the same11:49
ltomasboservices.... could be more tricky... we need to double check11:49
*** rh-jelabarre has joined #openstack-kuryr11:51
*** rh-jelabarre has quit IRC12:34
*** irclogbot_3 has quit IRC13:07
*** irclogbot_3 has joined #openstack-kuryr13:09
alisanhajiHi, I am trying to integrate Kuryr to Magnum, and I had a probelm with tiller deployment: http://paste.openstack.org/show/748168/13:16
alisanhajiKuryr kubernetes controller service didn't like the TargetPort as a string, it was expecting an int13:18
alisanhajiShould I file a bug report or was it already fixed?13:19
dulekalisanhaji: This should be fixed with https://review.openstack.org/#/c/641598/.13:23
*** altlogbot_0 has quit IRC13:23
dulekalisanhaji: Very recent thing. Do you use the container images from DockerHub?13:23
*** altlogbot_1 has joined #openstack-kuryr13:24
alisanhajidulek: oh great, yes I do use the container images13:26
alisanhajiVery recent indeed :D13:27
*** janki has quit IRC13:28
dulekalisanhaji: I can try to update the DockerHub image for you, it'll just take a while, I have terrible Internet connection here.13:31
*** gaoyan has joined #openstack-kuryr13:35
alisanhajiThat would be great, thanks! It's not urgent I am still having a misconfiguring LBaaS so I first need to deal with it, but it would be much easier for me if I could have the latest images to get everything working together13:35
alisanhajidulek: misconfigured*13:36
kmadac3Hi, I have deployed kuryr into VM with macvlan driver. I'm experiencing incorrect behavior. When I do simple deployment of 3 pods at one, those 3 pods are created, but IP/mac pair of only last created pod is added to allowed_address_pairs of neutron port. It seems like a bug to me. I checked the controller code, added couple of log messages and it seems that all 3 calls to get info about port to13:37
kmadac3neutron are done at almost same time and allowed_address_pairs is empty at that moment, so only last update adds ip/mac to it.13:37
*** altlogbot_1 has quit IRC13:39
*** altlogbot_3 has joined #openstack-kuryr13:40
dulekkmadac3: Ah, seems like a race condition, isn't it?13:41
kmadac3@dulek: is it seems so13:41
dulekkmadac3: We could use etags to fight it *if* Neutron API is supporting them.13:41
kmadac3@dulek: should I create a bug for that?13:43
dulekkmadac3: Sure, but I doubt we have anyone who's going to grab this at the moment.13:43
dulekkmadac3: Wait, but shouldn't this happen for ip vlan case as well?13:44
kmadac3@dulek: ok, I have some python development background so I can try to fix it13:44
*** irclogbot_3 has quit IRC13:45
dulekkmadac3: That's the fastest way IMO. I'm happy to help here. Let me check if Neutron API has etags.13:45
kmadac3@dulek: great, I found something here: https://specs.openstack.org/openstack/api-wg/guidelines/etags.html13:46
*** irclogbot_2 has joined #openstack-kuryr13:46
dulekkmadac3: Yes, it's exactly this stuff and it seems Neutron does support it.13:46
kmadac3@dulek: perfect, as etags are to me, i will need to study a bit, but I will do my best to implement it13:49
dulekkmadac3: Neutron's implementation may differ a bit as I look at it, but maybe I'm wrong.13:49
kmadac3@dulek: * as etags are new to me13:49
dulekkmadac3: I don't have a cloud ready to check how it works at the moment, but will try this evening.13:50
kmadac3@dulek: no problem, take your time13:50
*** rh-jelabarre has joined #openstack-kuryr13:50
dulekkmadac3: So Kuryr's implementation would just need to make sure to send If-Match along with this request: https://github.com/openstack/kuryr-kubernetes/blob/88e38e8e9493757d5db4da2b69dcc033b2f7465b/kuryr_kubernetes/controller/drivers/nested_macvlan_vif.py#L143-L14613:52
*** jistr is now known as jistr|call13:52
dulekkmadac3: And also make sure it is retried a few times.13:52
kmadac3@dulek: thanks, I will try to implement it there13:54
*** jistr|call is now known as jistr14:00
maysamsdulek: ping14:05
maysamsdulek, did you have a chance to run the tests again with my ps?14:05
dulekmaysams: Yes. And no. :P14:15
dulekmaysams: I have some troubles with etcd on my env and tests are failing due to that.14:15
dulekmaysams: Will need to recreate it, but will only be able to do that later in the evening or tomorrow morning.14:16
maysamsdulek, ahh okay14:16
maysamsno worries, ty!14:16
dulekalisanhaji: The new kuryr-controller is pushed, it should include ltomasbo's fix.14:16
ltomasbodulek, what fix?14:21
dulekltomasbo: named ports in services.14:24
ltomasboahh, ok!14:24
ltomasboso you have updated the upstream images?14:24
*** gaoyan has quit IRC14:25
alisanhajidulek: great! Just as I was going to pull them, thanks14:31
openstackgerritDaniel Mellado proposed openstack/kuryr-kubernetes master: [WIP] Add ipBlock support to NP  https://review.openstack.org/64513914:31
kmadac3@dulek: I checked the response headers in 'openstack port show' output and there is no etag response header. Does it mean, that neutron doesnt support etag? I have Rocky release deployed and here the output: http://paste.openstack.org/show/748172/14:32
*** altlogbot_3 has quit IRC14:35
*** altlogbot_3 has joined #openstack-kuryr14:36
*** irclogbot_2 has quit IRC14:38
*** irclogbot_0 has joined #openstack-kuryr14:39
celebdorkmadac3: etag?14:46
celebdorI never heard of etags :P14:47
*** gcheresh has quit IRC14:50
*** gaoyan has joined #openstack-kuryr14:50
*** gaoyan has quit IRC14:51
dulekltomasbo: Yes.15:13
dulekalisanhaji: No problem. :)15:13
dulekkmadac3: I'll check in a moment, my env is up now. :)15:13
dulekcelebdor: We have some lost update issue on mcvlan nested driver.15:14
dulekcelebdor: And etags could help. It's something like resourceVersion on K8s.15:14
dulekmaysams: http://paste.openstack.org/raw/748180/15:29
dulekmaysams: I'll post kuryr-controller log in a second.15:29
maysamsdulek, okay15:33
dulekmaysams: I've mailed it to you, sorry, didn't found a proper way to present a 2MB file. :P15:39
dulekmaysams: Use the timestamp from the test to navigate - failure happened before 15:21:04.844.15:39
dulek(it may be way before)15:39
dulek(like ~5-10 minutes)15:40
alisanhajidulek: Now everything is running, I can get magnum k8s to use kuryr, but when I schedule a pod, I have this error: NoMatches: No 'kuryr_kubernetes.vif_translators' driver found, looking for u'binding_failed'15:40
maysamsdulek: Thanks a lot!15:40
dulekalisanhaji: Ah right, I saw that one day…15:40
dulekalisanhaji: This is in which Kuryr pod?15:41
alisanhajiWhen I look into neutron port, I find that the kuryr ones are DOWN and they have a binding_vif_type as binding_failed15:41
alisanhajidulek: in the controller15:42
dulekalisanhaji: Okay, I think you should look into neutron logs.15:42
dulekalisanhaji: And I'm not sure which one. :P15:42
dulekalisanhaji: But it's probably port creation failure, IIRC last time someone asked about such error it was Neutron fautl.15:43
dulekfault15:43
alisanhajithe server logs only tell me that it failed to bind the port15:44
alisanhajithe neutron-agent get the RPC message for the new resource Port, but doesn't do anything15:44
maysamsdulek: What sg drivers do you have enabled on your env?15:46
dulekmaysams: Uh, oh. Maybe it's misconfigured?15:46
dulekmaysams: pod_security_groups_driver = policy15:47
dulekIt seems okay?15:47
maysamsyup15:47
dulekalisanhaji: Sorry, I don't know Neutron well enough to help here. :(15:48
maysamsdulek, and service_security_groups_driver ?15:48
dulekalisanhaji: But when I googled neutron binding_failed it seems to have some ideas.15:48
dulekmaysams: Same.15:49
maysamsokay, thanks15:49
dulekmaysams: http://paste.openstack.org/raw/748183/15:49
maysamsdulek, that's correct15:50
maysamsty15:50
alisanhajidulek: No worries, I will try to debug and see why it fails, thanks15:51
dulekmaysams: Ah, one thing. If you'll see something very strange, like an K8s event missing from kuryr-controller logs, don't assume you're wrong.15:59
dulekmaysams: I've had those issues with etcd, so maybe they're still manifesting.15:59
dulekAnd lost events were a sign of that on the gate.15:59
maysamsdulek: I see lots of issues regarding missing annotation on namespace16:00
maysamsdulek: Is that the one you were talking about?16:01
dulekmaysams: No, no, I mean etcd failures may lead to some events not being passed to kuryr-controller's watcher.16:01
maysamsdulek, right16:01
dulekmaysams: But I've made some precautions to make sure etcd is fine, so it shouldn't happen. Just want to make sure you won't spend a day debugging, because you rule it out. ;)16:02
dulekmaysams: And those missing annotations on namespace are because tests create a new namespace.16:02
maysamsdulek: hahah ok16:02
dulekAnd do everything there.16:02
maysamsyup yup16:02
dulekSo obviously kuryr-controller needs some time to annotate it with the subnet created for it.16:02
*** premsankar has joined #openstack-kuryr16:03
dulekkmadac3: Okay, I've consulted that with Neutron folks: https://developer.openstack.org/api-ref/network/v2/#revisions16:08
dulekkmadac3: They don't have etags, they have revision_number.16:09
dulekkmadac3: And when doing PUT you can add "If-Match: <number>" to headers.16:09
dulekAnd request will fail if revision was changed.16:09
dulekkmadac3: It's even easier: def update_port(self, port, body=None, revision_number=None)16:11
dulekSo update_port from here: https://github.com/openstack/kuryr-kubernetes/blob/88e38e8e9493757d5db4da2b69dcc033b2f7465b/kuryr_kubernetes/controller/drivers/nested_macvlan_vif.py#L143-L14616:11
dulekWill accept revision_number parameter :)16:11
maysamsdulek: Did deleted some security groups? I saw that the controller was over quota and then it got ready again16:14
dulekltomasbo: This solves networking-ovn tracebacks: https://review.openstack.org/#/c/645096/16:14
dulekmaysams: I bumped up the quota number, I was building from your patch and it haven't included my fix for that.16:15
dulekmaysams: So I did that manually after the cluster was up/.16:15
maysamsdulek, right16:15
dulekmaysams: I can run that test again, kill it while it's waiting for that pod and get the SG's and rules.16:16
maysamsdulek, that would be great.. I'm thinking if the namespace used in the tests was created before you bumped the quota16:17
dulekmaysams: No, it definitely was not.16:18
maysamsok16:18
dulekmaysams: Okay, let's see…16:32
dulekmaysams: Server is 10.1.1.242. client-a that cannot connect, but should is 10.1.1.25416:32
dulekI have sg-network-policy-965-allow-client-a-via-named-port-egress-rule.16:32
dulekmaysams: And those are the rules: http://paste.openstack.org/raw/748186/16:33
dulekmaysams: SG rule 0f1cce87-1673-4d24-b594-d84c17f83510 group is egress…16:34
maysamsright, seems that it's creating the wrong rule16:36
dulekmaysams: Nah, it doesn't look too bad yet IMO.16:37
maysamsyet?16:37
dulekmaysams: So LB VIP port has this attached: http://paste.openstack.org/raw/748188/16:37
dulekYet as I'm going through it. ;)16:37
dulekmaysams: If I read that SG correctly it means that it allows ingress access from everywhere at port 80 and 81, right?16:38
maysamsdulek, yup16:38
dulekAnd that seems fine.16:38
dulekNow the client-a pod…16:38
dulekmaysams: http://paste.openstack.org/raw/748190/16:39
dulekmaysams: This is on client-a port.16:39
kmadac3@dulek: sorry i was afk for a while. thanks for making things clear to me :) Hopefully it'll be enough for me to fix the issue.16:40
dulekmaysams: That egress looks okay as well, isn't it?16:41
maysamsdulek: yes, it does16:42
dulekkmadac3: It should be. Just pass port['revision_number'] to _update_port_address_pairs, add it to neutron.update_port and make sure it'll retry on 412 status code.16:43
dulekkmadac3: Where retry = fetching the port again and doing this: https://github.com/openstack/kuryr-kubernetes/blob/88e38e8e9493757d5db4da2b69dcc033b2f7465b/kuryr_kubernetes/controller/drivers/nested_macvlan_vif.py#L97-L11316:44
dulekmaysams: My only idea is that remote_ip_prefix='10.1.1.242/32' doesn't work as we thing?16:44
duleks/thing/think16:44
dulekMaybe 10.1.1.242 doesn't fall into it, because it's not usable address.16:45
dulekBut that's too bold, I wouldn't bet a cent on it…16:45
maysamsdulek: how it would not be a usable address?16:46
dulekmaysams: Good point, with /32 it is usable. :P16:46
maysamsdulek, yup16:48
maysamsdulek: could you check if the subnet got created for the namespace being used?16:57
dulekmaysams: Sure, just a second.16:57
dulekmaysams: cfdb78ed-a727-414b-b876-3f6c0fcdb098 | ns/network-policy-965-subnet | 4345f0b0-257f-47fb-8e9b-0da559c9fb52 | 10.1.1.192/2616:57
dulekLooks pretty okay?16:57
ltomasbodulek, nice!16:58
ltomasbo(the ovn fix)16:58
dulekmaysams: I think that the server pod port doesn't allow any ingress.16:58
dulekmaysams: http://paste.openstack.org/raw/748192/16:59
dulekmaysams: server pod's port has those SG's.16:59
dulekmaysams: I don't see anything that would allow traffic from client-a or the LB?17:00
alisanhajidulek: when kubelet tries to run kuryr-cni, I get an error: /opt/cni/bin/kuryr-cni: line 40: docker: command not found17:02
maysamsdulek: me neither17:02
maysamsltomasbo: May you confirm if we only create a rule with remote_group_ig when namespace sgs driver is enabled:17:02
dulekalisanhaji: Hm, no docker on host with kubelet? Why? cri-o?17:02
dulekalisanhaji: Nah, you wouldn't see that in kubelet log if you had cri-o. So why no docker? :D17:03
alisanhajidulek: Docker is present, but maybe the PATH is not present during kuryr-cni script17:04
dulekalisanhaji: Ah, might be, but I'm not sure what I can do here. We do `docker exec` to execute kuryr-cni inside our kuryr-cni container, so we have all the configs, etc.17:05
dulekmaysams: Sorry, I don't understand. I don't want to reconfigure the env, it seems a bit fragile. ;)17:06
dulekmaysams: I can try creating the SG rule myself and see if it helps.17:06
maysamsdulek: No worries.. I will analyze better those SG you shared with me17:07
maysamsthanks17:07
alisanhajidulek: if you are running docker inside docker, I think I will need to mount /usr/bin with cni DaemonSet17:07
dulekmaysams: This helped: openstack security group rule create --ingress --protocol tcp --remote-ip 10.1.1.233 1b76ff78-52df-42da-88c8-020c4999a97e17:09
dulekalisanhaji: It's not inside docker, it's run by kubelet.17:09
dulekOh waaaait, your kubelet is inside docker container?17:09
dulekNah, that shouldn't be, right? :P17:09
dulekalisanhaji: So if kuryr-cni script is run by kubelet we assume it's run on the host.17:10
dulekAnd by kuryr-cni I mean /opt/cni/bin/kuryr-cni17:10
dulekalisanhaji: Now tricky part is that this script only finds the kuryr-cni container through docker/runc and does docker/runc exec.17:10
alisanhajino kubelet is not running inside a VM17:11
*** maysams has quit IRC17:11
dulekalisanhaji: So where is it running?17:12
alisanhajiOh, it's actually running through runc17:14
dulekalisanhaji: Ah, so that error was a false positive?17:14
dulekalisanhaji: I really want to help here, but I need to go now. You can bug dmellado and ltomasbo about it if they're still online. Or maybe even celebdor if he still reads IRC. ;)17:15
dulekGuys, this is about Kuryr integration with Magnum, so pretty cool stuff in terms of K8s on OpenStack.17:15
dmelladohi o/17:16
dmelladoI was about to call it a day too, what's on? ;)17:16
dulekdmellado: If it's the case I don't keep you, go enjoy playing with the kid. ;)17:18
alisanhajidulek: well I still have the error of not finding docker17:18
alisanhajiit occurs over and over17:18
dulekalisanhaji: Can you put full log somewhere?17:18
dulekalisanhaji: I mean full log of running kuryr-cni, not full kubelet log.17:19
dmelladoso what's the issue? kuryr-cni is not running?17:19
dmelladoon which context?17:19
dulekdmellado: For some reason on Magnum it keeps bugging alisanhaji with docker binary being not found.17:19
dulekThat's the script by the way: https://github.com/openstack/kuryr-kubernetes/blob/master/cni_ds_init#L12-L5317:20
alisanhajihere: http://paste.openstack.org/show/748193/17:20
alisanhajidulek: thanks17:20
alisanhajiwhy does it need to run kuryr-cni another time?17:21
dulekalisanhaji: YOu mean through `docker exec`? Because the main kuryr-cni is in Python and Python on the host isn't granted, also we don't have kuryr.conf there, etc.17:24
dulekIt's easiest way, unfortunately Python apps aren't binaries that you can just put on host.17:24
dulekalisanhaji: Okay, so it seems like this fails: https://github.com/openstack/kuryr-kubernetes/blob/master/cni_ds_init#L4117:25
dulekalisanhaji: And script falls back to use Docker.17:25
dulekAnd doesn't find it for some reason.17:25
alisanhajidulek: I see17:25
dulekalisanhaji: Can't really help here, you need to figure out why kubelet processes doesn't see docker. And if there's a valid reason behind it we can discuss how to stop depending on it.17:26
dulekOkay, really need to go now. ;)17:26
dulekHave a great evening!17:26
alisanhajidulek: ok I will try to, thanks for the help17:28
*** alisanhaji has quit IRC17:30
*** gcheresh has joined #openstack-kuryr17:32
*** ccamposr has quit IRC17:36
*** gmann is now known as gmann_afk17:43
*** gcheresh has quit IRC17:58
*** maysams has joined #openstack-kuryr18:02
openstackgerritDaniel Mellado proposed openstack/kuryr-kubernetes master: [WIP] Add ipBlock support to NP  https://review.openstack.org/64513918:37
*** gmann_afk is now known as gmann18:40
*** premsankar has quit IRC19:02
*** mrostecki has quit IRC19:35
*** mrostecki has joined #openstack-kuryr19:42
*** mrostecki has quit IRC19:43
*** mrostecki has joined #openstack-kuryr19:45
*** celebdor has quit IRC20:19
*** rh-jelabarre has quit IRC21:04
*** pcaruana has quit IRC21:33
*** celebdor has joined #openstack-kuryr21:33
*** maysams has quit IRC21:38
*** irclogbot_0 has quit IRC22:05

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!