Tuesday, 2016-11-29

*** hongbin has quit IRC00:02
*** yuanying_ has joined #openstack-kuryr00:03
*** limao has joined #openstack-kuryr00:29
openstackgerritdengshaolin proposed openstack/fuxi: Retrieve volumes from cinder with matched metadata  https://review.openstack.org/39044800:34
openstackgerritdengshaolin proposed openstack/fuxi: Fix indexing operation for OrderDict in py3.4 and py3.5  https://review.openstack.org/40395800:40
*** shashank_hegde has quit IRC00:46
*** tonanhngo has quit IRC01:01
*** tonanhngo has joined #openstack-kuryr01:10
*** tonanhngo has quit IRC01:14
openstackgerritbailin.zhang proposed openstack/kuryr-libnetwork: Fix the file permissions of README.rst  https://review.openstack.org/39651901:20
*** shashank_hegde has joined #openstack-kuryr01:21
*** yamamot__ has joined #openstack-kuryr01:41
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi: Updated from global requirements  https://review.openstack.org/37374501:42
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork: Updated from global requirements  https://review.openstack.org/35197601:44
*** yamamot__ has quit IRC01:44
*** shashank_hegde has quit IRC01:56
*** shashank_hegde has joined #openstack-kuryr01:57
*** yamamot__ has joined #openstack-kuryr02:04
*** shashank_hegde has quit IRC02:20
*** hongbin has joined #openstack-kuryr02:22
*** reedip has quit IRC02:29
*** reedip has joined #openstack-kuryr02:32
*** yuanying_ has quit IRC03:00
*** oanson has joined #openstack-kuryr03:07
*** yamamo___ has joined #openstack-kuryr03:22
*** yamamot__ has quit IRC03:26
*** yuanying has joined #openstack-kuryr03:59
*** tonanhngo has joined #openstack-kuryr04:08
*** tonanhngo has quit IRC04:09
*** jgriffith is now known as jgriffith_away04:21
*** tonanhngo has joined #openstack-kuryr04:21
*** tonanhngo has quit IRC04:22
*** hongbin has quit IRC04:24
*** oanson has quit IRC04:30
*** reedip has quit IRC04:42
*** rock has quit IRC04:47
*** tonanhngo has joined #openstack-kuryr05:04
*** shashank_hegde has joined #openstack-kuryr05:05
*** tonanhngo has quit IRC05:06
*** shashank_hegde has quit IRC05:12
*** shashank_hegde has joined #openstack-kuryr05:35
*** shashank_hegde has quit IRC06:16
*** irenab has joined #openstack-kuryr06:24
*** shashank_hegde has joined #openstack-kuryr06:27
irenabvikasc, hi06:38
openstackgerritIlya Chukhnakov proposed openstack/kuryr-kubernetes: [WIP] CNI & VIFBridge binding  https://review.openstack.org/40403806:39
*** oanson has joined #openstack-kuryr06:41
ivc_irenab, apuimedo, ping06:43
irenabivc_, pong06:43
ivc_irenab, apuimedo, just uploaded a working CNI  https://review.openstack.org/404038 (still needs some cleanup, routes handling, tests and will also be split in smaller patches)06:44
irenabivc_, I will take a look and try to run it06:45
irenabivc_, can you please post your local.conf?06:45
ivc_irenab, apuimedo, also tried some 'stress-tests' by running "kubectl run nginx --image=nginx --replicas=30" (takes ~2min) and "kubectl delete deployment nginx" (works fine, but seems to exceed k8s timeout with some very unexpected results as it gets called second time by kubelet after pods are deleted and there's no annotations)06:47
ivc_irenab, http://paste.openstack.org/show/590746/06:48
ivc_irenab, but you also need to 'cd /opt/stack/kuryr-kubernetes && sudo pip install -r requirements.txt' to get rid of 'kuryr.lib' issues06:49
irenabivc_, thanks06:49
irenabivc_, regarding the second time delete, worth to check what k8s devs expected in such case. Looks like in our case its just 'success' operation06:54
ivc_irenab, that behaviour is really weird. everything is cleaned up properly on the first run, but for some reason k8s _sometimes_ calls cni again (and then we fail with our own timeout since '?watch=true' gets stuck)06:59
irenabivc_, maybe watch should be kicked upon condition?07:00
irenabdid you try to ask guys on k8s slack?07:00
ivc_irenab, what sort of condition do you suggest? like check pod with 'get' first?07:01
irenabactually, I would expect kubelet to take care of it ...07:01
*** yamamo___ has quit IRC07:02
irenablet me take a look on the code07:02
ivc_you mean kubelet code?07:02
irenabyes, that it will not fire CNI for the false action07:03
irenabchecking if pod exists sounds like workaround07:04
irenabalso it assumes that pod is removed from the k8s data store after kebelet 'approves' its deleted07:05
irenabkubelet07:05
irenabivc_, do you know how the k8s Pod delete flow goes?07:05
ivc_irenab, gotta dig into kubelet source07:06
irenabivc_, same for me :-)07:06
ivc_irenab, what i'm afraid of is that probably there's no contract between cni and kube-api that when cni is called pod can be queried from kube-api07:07
irenabivc_, starting to think the same07:08
ivc_given the async nature of k8s it might be totally possible and then we are screwed07:08
irenabbut still do not understand how you cpuld get pods without annotations07:08
irenabit either with annotations or does not exist07:09
ivc_nope i'm not getting pods without annotations :) i'm not getting pods at all :)07:09
irenabaha07:09
irenabnow it makes sense07:09
irenabso assuming there is no contact between Controller part and net plugin part on when pod is removed from the data model07:10
ivc_yep it does seems so07:10
irenabCNI should try to keep its sources local07:10
ivc_we can probably still can get away with our solution with daemon-based cni07:11
irenabyes, also thinking of it07:12
irenabactually why do we need k8s api for the delete case?07:12
ivc_we need vif details07:12
ivc_so we can os_vif.unplug07:13
irenabthe best way would to get it with other args07:13
ivc_otherwise we'll have leftover hybrid bridges07:13
ivc_no way we can get it with other args07:13
ivc_i mean with non-daemon cni07:14
irenabI was thinking about adding extra power to kubelet07:14
irenabprobably longer term07:14
irenabor we must keep local cache07:15
irenabcontainer id -> vif details07:15
irenabprobably what you mean by daemon07:16
ivc_yep. we need daemon anyway. with the current approach there's one hell of a storm of privsep helpers spawned by os_vif07:17
ivc_and we planned it anyway07:18
*** yuanying has quit IRC07:20
irenabso the CNI Add/Del will just send request to daemon and block for reply?07:21
ivc_yeah07:21
*** pcaruana has joined #openstack-kuryr07:21
*** saneax has joined #openstack-kuryr07:21
irenabits just annoying that for Del we need it only for the extra vif info07:23
ivc_its not extra07:24
irenabI mean which is not pushed to CNI07:24
ivc_its part of vif. think of ovs-hybrid case at least - there's that linux bridge07:24
ivc_ah, right07:24
ivc_well, k8s cni-plugin is still experimental, so maybe it could be customized the way we need07:26
*** yuanying has joined #openstack-kuryr07:26
ivc_the best place for our data is the netcfg json that is streamed to cni.stdin07:27
irenabI need some time to go over the code and try to dig into k8s code. Maybe we can consult with k8s guys07:27
irenabivc_, I agree07:27
irenaband get it in 'push' from kubelet07:27
ivc_irenab, i wonder why did we not find out that k8s behaviour with midokura's PoC07:29
*** dimak has joined #openstack-kuryr07:37
*** yamamot__ has joined #openstack-kuryr07:39
irenabivc_, I do not think it was properly tested on scale + maybe previous k8s version behaved differently07:42
*** lezbar has joined #openstack-kuryr08:08
*** lezbar has quit IRC08:17
*** yuanying has quit IRC08:17
janonymousalraddarla_: cool, you could try https://review.openstack.org/#/c/394788/, https://review.openstack.org/#/c/394251/ , https://review.openstack.org/#/c/393996/ if that's fine08:19
*** lezbar has joined #openstack-kuryr08:19
*** irenab has quit IRC08:37
*** irenab has joined #openstack-kuryr08:39
openstackgerritMerged openstack/kuryr: Nested-Containers: vlan driver  https://review.openstack.org/36199308:49
apuimedoivc_: "the best place for our data is the netcfg json that is streamed to cni.stdin" what makes you say that?08:50
*** shashank_hegde has quit IRC08:50
*** shashank_hegde has joined #openstack-kuryr08:59
*** shashank_hegde has quit IRC08:59
*** pmannidi has quit IRC09:00
*** garyloug has joined #openstack-kuryr09:01
*** lmdaly has joined #openstack-kuryr09:02
irenabapuimedo, did you read the discussion we had with ivc_ this morning?09:11
*** gsagie has joined #openstack-kuryr09:40
*** ram____ has joined #openstack-kuryr09:44
*** lezbar has quit IRC09:44
*** openstackgerrit has quit IRC09:48
*** openstackgerrit has joined #openstack-kuryr09:48
apuimedoirenab: I did09:50
apuimedoI'm a bit ill so maybe I didn't get all to perfection09:50
apuimedobut I thought the netcfg json that is passed to stdin is more for static configuration09:50
apuimedoso it would not serve for passing ovs-hybrid bridge names nor uuids09:51
apuimedoIf we want to avoid needing watch for delete we could just put data on tempfiles like /run/kuryr/uuid_pod09:51
apuimedoirenab: ivc_: ^^09:51
irenabapuimedo, when do you suggest to place the tempfiles? On Add?09:57
apuimedojust before returning success on add09:57
irenabapuimedo, workaround, but possible. so daemon is not needed10:01
irenabapuimedo, rest and get well soon!10:01
apuimedoirenab: nah...10:03
apuimedono rest10:03
apuimedoirenab: https://www.youtube.com/watch?v=jB2nWiEO9T010:04
apuimedolike this10:04
apuimedoivc_: what do you think of the files?10:04
*** reedip has joined #openstack-kuryr10:11
*** limao has quit IRC10:11
apuimedoirenab: ivc_: vikasc: https://review.openstack.org/#/c/404124/10:18
openstackgerritMerged openstack/kuryr-libnetwork: Updated from global requirements  https://review.openstack.org/35197610:23
*** ram____ has quit IRC10:44
openstackgerritMerged openstack/kuryr-libnetwork: Build docker busybox image locally  https://review.openstack.org/39995811:42
*** yamamot__ has quit IRC11:46
apuimedovikasc: https://review.openstack.org/39545311:51
*** vikasc has quit IRC11:56
*** gsagie has quit IRC12:14
*** lmdaly has quit IRC12:30
apuimedoivc_: irenab: I just posted a gate that tries to install devstack https://review.openstack.org/#/c/404191/12:31
apuimedoof course, it will fail until we solve the kuryr-lib release12:31
apuimedowhich I also submitted12:31
*** limao has joined #openstack-kuryr12:45
*** garyloug has quit IRC12:51
ivc_apuimedo, thats fantastic!12:59
*** limao_ has joined #openstack-kuryr13:04
ivc_apuimedo, regarding the 'files' approach, i'm not sure. one part of me thinks its terribly UGLY, but on the other hand its quite simple.13:05
ivc_apuimedo, but i'd prefer to avoid it if we can. imo the problem would solve itself as soon as we get to 'recycling' ports/vifs13:07
*** limao has quit IRC13:07
ivc_apuimedo, irenab, and our current cni approach is ineffective at best anyway (our cni is fat&slow, multiple IPDB instances flood netlink with unneded spam, os-vif spawns a separate sudo/rootwrap process on each cni exec) so we'll need to do something about it anyway even if it wasn't for the issue with de-sync between cni and k8s-api13:14
*** yamamoto has joined #openstack-kuryr13:20
*** yamamoto_ has joined #openstack-kuryr13:22
*** yamamoto has quit IRC13:25
irenabivc_, 'one part of me' ?13:25
irenabivc_, I think you are in favor of the daemon from the day one.13:27
ivc_irenab, well, not exactly 'from the day one', but i am :)13:31
irenabivc_, for recycling ports/vifs, what do you have in mind? per Clusrter or per node?13:32
ivc_irenab, performance-wise per-node would be best imo13:35
ivc_irenab, best solution would be if we could keep container-side interface and relocate it into some 'recycling' namespace(s)13:39
irenabivc_, neutron related, you distribute ports across nodes?13:41
ivc_irenab, but i need to deal with services/lbaas first, so wdyt if we merge the current 'fat' cni (it still requires some tweaks and maybe has to be split in multiple patches)13:41
irenabivc_, with files as apuimedo suggested?13:42
ivc_irenab, no, without files13:42
irenabivc_, we can. I think we keep kuryr-k8s experimental anyway13:42
irenabivc_, so how delete is resolved?13:42
ivc_irenab, it works now, you just see some unwanted messages in k8s logs13:43
irenabhow vif is removed ?13:43
alraddarla_janonymous, i'll take care of the three you mentioned. thanks for letting me help :)13:43
ivc_irenab, https://review.openstack.org/#/c/404038/1/kuryr_kubernetes/cni/binding/base.py@5713:44
irenabivc_, in case pod is not in k8s data model, will CNI be able to clean the host properly?13:44
ivc_irenab, during the first call to cni pod is available, but k8s calls cni second time in some cases13:45
apuimedoivc_: we should still find out why that second call happens13:46
apuimedoirenab: did you check on k8s?13:46
irenabivc_, not sure we can assume if during the first time Pod is in k8s data model.13:47
irenabapuimedo, not yet13:48
apuimedoI still prefer if we saved the trip to k8s :P13:48
apuimedolet's say that the split small exec / daemon solves it13:49
irenabapuimedo, agree on saving the trip for delete. Not sure we can do it for Add13:49
apuimedoin that solution, I assume the daemon keeps a dict of all the vifs for the host13:49
apuimedothen when it receives the request from the executable, it doesn't need to go to k8s anymore, since it has the data from watch13:50
apuimedoif it doesn't have data, it should either 'get' or timeout13:50
ivc_irenab, apuimedo, my point is that if you consider the 'fat' cni approach only as a temporary one (as i do) that will allow us to continue with testing (and we don't run 100s of pods for tests, right?) and services. and once we get to functionally-complete impl, we rework cni into daemon/exec13:50
apuimedowhich is what happens now13:50
apuimedoI'm talking about the lean cni now13:51
irenabivc_, I am ok with you proposal for taking steps13:51
irenabapuimedo, what do you think?13:51
apuimedothe only satisfactory solution for lean cni (split) that I can see is that the dict has a big size with LRU eviction. On first delete we just None the entry, and then when adding more, we collect the oldest Noned entries13:52
apuimedoirenab: I'm okay with taking steps13:52
apuimedoI'm just thinking out loud whether files still make sense with a lean split cni13:52
irenabapuimedo, define lean split13:53
ivc_irenab, daemon+exec13:53
apuimedoright13:53
ivc_apuimedo, i'd prefer if we did not persist state on filesystem if that could be avoided13:54
irenabexec called by kubelet and transfers call to daemon, blocks till gets answer?13:54
ivc_irenab, yes13:54
irenabapuimedo, so you considered Daemon to maintain files?13:54
ivc_irenab, apuimedo, and no os-vif/oslo.<anything> in exec for sure13:54
apuimedoirenab: no. With the split13:55
apuimedoI was considering keeping a big dictionary on memory13:55
irenabfiles versus store in mem?13:55
apuimedothat keeps data for all the pods that get scheduled to the host13:55
irenabthis whagt daemon will maintain13:56
apuimedoyup13:59
apuimedoand I was proposing a dict with LRU eviction that allows us to track double delete invokation13:59
apuimedoand saying that it is still more complex than a files based approach14:00
apuimedobut I guess we can try both14:00
*** lezbar has joined #openstack-kuryr14:01
ivc_apuimedo, i don't think we need LRU eviction. its just a matter of how much memory-per pod our data structures will require and how many pods you can expect per node14:02
apuimedoivc_: I understand the concern about keeping files14:02
apuimedoI've dealt with its pains in the past14:02
apuimedoivc_: the LRU is to keep track for older non-existing things14:03
ivc_apuimedo, not necessary. if we do not delete container-side interface but move it into some kuryr-managed netns, we could use it as source of truth14:05
apuimedowhy would we not delete the ifaces when DELETE is called?14:06
ivc_to get lightning-fast add on recycled ports14:07
ivc_*ADD14:07
apuimedooh, that14:07
apuimedoright14:07
apuimedoit flew out of my mind14:07
apuimedo:P14:07
ivc_:P14:07
apuimedogotta think about it a bit more14:07
ivc_apuimedo, it would be similar to pre-creating ports in advance (something that was discussed already iirc)14:12
apuimedoyes, yes14:12
ivc_apuimedo, irenab, also consider that it could be an option to get rid of annotations and instead use some sort of controller --> cni-daemon rpc14:16
irenabivc_, apuimedo : did you run devstack lately?14:16
ivc_irenab, with or without the cni patch?14:17
irenabivc_, I am starting to realise wheat 'part of me' means :-)14:17
irenabivc_, actually patch is not relevant, it fails for k8s-api14:17
ivc_irenab, i think i've forgot to update kuryr.conf cni config (you need to add "kuryr_conf": "/etc/kuryr/kuryr.conf")14:17
irenabdevstack does not get there far... k8s-api fails with Invalid Auth Config14:18
apuimedoirenab: I didn't run it with the cni patch14:18
apuimedo:O14:18
irenabapuimedo, even before this14:18
ivc_irenab, i think you had the same error before14:19
irenabNo RSA key provided14:19
irenabivc_, I think it was nsenter, so added it a priori14:20
irenabapuimedo, now with test you add , we will have an indication if its supposed to work14:23
ivc_irenab, well the test clearly states that it does not work :)14:24
*** lmdaly has joined #openstack-kuryr14:26
ivc_irenab, apuimedo, btw maybe what we are seeing with cni could actually be a bug in k8s and its cni plugin14:31
irenabivc_, I want to reproduce it in my devstack and post question on k8s slack14:33
ivc_irenab, cool14:34
irenabonce devstack is happy to serve me ...14:34
ivc_:)14:34
ivc_irenab, you'll also need to update cni/net.d/kuryr.conf with { "name": "kuryr", "type": "kuryr", "debug": true, "kuryr_conf": "/etc/kuryr/kuryr.conf" }14:35
ivc_irenab, and probably something else :) i need to restack on fresh env to cleanup the deployment part of the patch14:37
irenabivc_, share the details once you get some more gotchas :-()14:37
ivc_irenab, i'm gonna do a clean restack today and will try to update the patch so it is usable14:38
irenabivc_, thanks. I will continue tomorrow14:41
*** irenab has quit IRC14:45
*** garyloug has joined #openstack-kuryr14:47
kristianjfHey guys, new to channel. I was going through some of your slides/doc but having trouble finding which openstack release is scheduled for k8s-cni integration.15:04
kristianjfCan anyone point me in the right direction?15:05
ivc_kristianjf, you can probably get an official statement from apuimedo15:16
kristianjfivc_: thanks!15:17
*** dougbtv has joined #openstack-kuryr15:17
*** hongbin has joined #openstack-kuryr15:22
*** saneax is now known as saneax-_-|AFK15:23
*** tonanhngo has joined #openstack-kuryr15:33
openstackgerritHongbin Lu proposed openstack/fuxi: Add devstack plugin for Fuxi  https://review.openstack.org/40224415:39
*** oanson has quit IRC15:46
openstackgerritDarla Ahlert proposed openstack/kuryr-libnetwork: Unittest with mock  https://review.openstack.org/39478815:52
openstackgerritDarla Ahlert proposed openstack/kuryr-libnetwork: Unittests with mock  https://review.openstack.org/39425116:00
*** dimak has quit IRC16:28
*** limao_ has quit IRC16:36
*** pcaruana has quit IRC16:36
*** akanksha__ has joined #openstack-kuryr16:38
*** akanksha__ is now known as ApheleiaS16:41
openstackgerritDarla Ahlert proposed openstack/kuryr-libnetwork: Unittests with Mock and add a TestCase  https://review.openstack.org/39399616:54
dougbtvany chance anyone has seen kuryr being used with openshift? e.g. openshift on openstack with a shared neutron network between the two?16:58
dougbtvcurious if there's any documentation on trying to get that up, if possible.16:59
*** shashank_hegde has joined #openstack-kuryr17:02
*** shashank_hegde has quit IRC17:12
*** shashank_hegde has joined #openstack-kuryr17:16
*** neil_ has joined #openstack-kuryr17:16
*** neiljerram has quit IRC17:17
*** diogogmt has joined #openstack-kuryr17:31
*** neil_ has quit IRC17:44
*** irenab has joined #openstack-kuryr17:48
*** shashank_hegde has quit IRC17:51
*** kristianjf has quit IRC17:52
*** dougbtv has quit IRC17:58
*** neil_ has joined #openstack-kuryr18:00
*** jgriffith_away is now known as jgriffith18:05
*** irenab has quit IRC18:11
*** tonanhngo has quit IRC18:14
*** lmdaly has quit IRC18:20
*** shashank_hegde has joined #openstack-kuryr18:21
*** shashank_hegde has quit IRC18:25
openstackgerritDarla Ahlert proposed openstack/kuryr-libnetwork: Unittests with Mock and add a TestCase  https://review.openstack.org/39399618:27
openstackgerritDarla Ahlert proposed openstack/kuryr-libnetwork: Unittests with Mock and add a TestCase  https://review.openstack.org/39399618:34
*** jgriffith is now known as jgriffith_away18:40
*** shashank_hegde has joined #openstack-kuryr18:50
*** garyloug has quit IRC19:00
*** tonanhngo has joined #openstack-kuryr19:14
*** oanson has joined #openstack-kuryr19:27
*** lezbar has quit IRC19:47
*** jgriffith_away is now known as jgriffith19:56
*** dougbtv has joined #openstack-kuryr19:57
alraddarla_janonymous, ping ?20:36
*** diogogmt has quit IRC20:40
*** diogogmt has joined #openstack-kuryr20:42
*** oanson has quit IRC20:43
*** ajo has quit IRC21:16
*** ajo has joined #openstack-kuryr21:17
*** ajo has quit IRC21:43
*** diogogmt has quit IRC21:46
*** ajo has joined #openstack-kuryr21:46
*** diogogmt has joined #openstack-kuryr22:12
*** ajo has quit IRC22:40
*** ajo has joined #openstack-kuryr22:41
*** diogogmt has quit IRC23:01
*** yuanying has joined #openstack-kuryr23:16
openstackgerritHongbin Lu proposed openstack/fuxi: Add basic fullstack volume tests  https://review.openstack.org/40394123:18
openstackgerritHongbin Lu proposed openstack/fuxi: Add basic fullstack volume tests  https://review.openstack.org/40394123:22
*** pmannidi has joined #openstack-kuryr23:35

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!