Friday, 2017-03-17

*** kzaitsev_mb has joined #openstack-kuryr00:20
*** kzaitsev_mb has quit IRC00:25
*** limao has joined #openstack-kuryr01:01
*** tonanhngo has quit IRC01:04
*** yedongcan1 has joined #openstack-kuryr01:11
*** tonanhngo has joined #openstack-kuryr01:15
*** tonanhngo has quit IRC01:19
*** kzaitsev_mb has joined #openstack-kuryr01:21
*** yedongcan1 has quit IRC01:45
*** yedongcan has joined #openstack-kuryr01:46
*** hongbin_ has joined #openstack-kuryr02:50
*** yuanying has quit IRC03:03
*** janonymous has joined #openstack-kuryr03:06
*** kzaitsev_mb has quit IRC04:18
*** hongbin_ has quit IRC04:39
*** kzaitsev_mb has joined #openstack-kuryr04:52
*** yuanying has joined #openstack-kuryr04:56
*** kzaitsev_mb has quit IRC04:56
*** janki has joined #openstack-kuryr05:14
*** kzaitsev_mb has joined #openstack-kuryr05:55
*** ltomasbo|away is now known as ltomasbo06:26
*** saneax-_-|AFK is now known as saneax06:27
*** janonymous has quit IRC06:54
*** kzaitsev_mb has quit IRC07:16
*** pcaruana has joined #openstack-kuryr07:43
*** kzaitsev_mb has joined #openstack-kuryr07:48
*** limao has quit IRC08:00
*** limao has joined #openstack-kuryr08:01
*** pmannidi has quit IRC08:04
*** kzaitsev_mb has quit IRC08:32
ltomasboping apuimedo08:33
ltomasbowe tried with a fedora installation08:33
ltomasboand the kubelet screen is running08:33
dmelladosame error on the kubelet (privsep again...)08:33
ltomasbobut we are getting the same error when trying to spawn a VM08:34
ltomasboa container08:34
ltomasboVMs work now08:34
openstackgerritJaivish Kothari(janonymous) proposed openstack/kuryr-libnetwork master: Adding Openstack support for Xenial  https://review.openstack.org/44689108:43
openstackgerritJaivish Kothari(janonymous) proposed openstack/kuryr-libnetwork master: Adding Openstack support for Xenial  https://review.openstack.org/44689108:44
*** janonymous has joined #openstack-kuryr08:45
*** yamamoto has quit IRC08:57
apuimedo:/09:09
apuimedoltomasbo: what changed for VMs to suddenly09:09
ltomasbowe reinstalled the server09:09
ltomasboand put a fedora25 instead of centos7.309:10
ltomasboVMs work fine, and know the kubelet screen command is there09:10
ltomasbobut the error is the same09:10
ltomasbowhat makes me think about if it is something we merged recently on kuryr is the kubectl describe pod output09:10
apuimedoltomasbo: can you paste again the error you get now?09:11
apuimedodmellado: I managed to build the packages09:11
apuimedoI didn't push them to copr yet though09:11
dmelladoapuimedo: \o/09:12
apuimedoand kuryr-lib is missing because it's from a separate repo09:12
ltomasbohttp://paste.openstack.org/show/603092/09:12
ltomasboalso, in the kubelet log, not sure why/where it is trying to get something on eth009:15
ltomasboNetworkPlugin cni failed on the status hook for pod 'busybox-sleep1' - Unexpected command output Device "eth0" does not exist.09:15
ltomasboapuimedo, do you want to take a look yourself on the server?09:16
dmelladoapuimedo: please do so xD09:16
ltomasboI see ovs is 2.5.009:17
ltomasboyet the pip ovs wrappers are 2.7.009:17
ltomasboshould I try to downgrade those?09:17
ltomasboahh, so annoying, this was the simplest step before!!09:18
dmelladolol09:18
apuimedoltomasbo: eth0 does not exist because the cni plugin fails, I suppose09:20
apuimedoltomasbo: try to make the ovs wrappers match the version09:21
apuimedoalthough it is a bit weird that they'd not be backwards compatible09:21
ltomasboI believe they are (and the error is somewhere else)09:21
dmelladoapuimedo: ltomasbo I was told that they should xD09:23
dmelladowe do use 2.6 bindings with ovsd 2.509:23
apuimedoltomasbo: what is different from the other fedora VM you had working?09:23
*** neiljerram has joined #openstack-kuryr09:24
ltomasbothat is not working either09:25
ltomasboI had a centos7.3 server working09:25
ltomasbothere was no difference with what we had yesterday in this server09:25
ltomasboand now the only different is the fedora instead of centos09:25
*** limao has quit IRC09:25
dmelladosame fedora version, on a vagrant, works for me09:26
dmelladoonly difference is that the fedora-vagrant *has* an eth0 interface09:26
apuimedodmellado: and the VM doesn't09:27
apuimedo?09:27
dmelladoapuimedo: I mean the host09:27
apuimedohave you tried using "ip link" to change the name?09:27
apuimedoand see what happens?09:27
dmelladothat was going to be my next step, but prob will have to restack afterwards09:27
apuimedodmellado: why?09:28
apuimedoyou shouldn't09:28
apuimedothe interface is not going down, the index is hte same09:28
dmelladodon't you have to put the if down before changing names?09:28
dmelladoI thought so09:28
dmelladoi.e. ip link set foo down09:28
dmelladoip link set foo name bar09:28
dmelladoip link set bar up09:29
*** kzaitsev_mb has joined #openstack-kuryr09:29
apuimedoI thought you didn't09:30
apuimedolet me check09:30
apuimedodmellado: you're right09:31
apuimedodmellado: what does the VM use for network configuration? systemd-networkd?09:31
dmelladoand then prob I'll have to put a new route09:31
dmelladothe vagrant vm you mean?09:32
apuimedodmellado: the vagrant VM already has eth0 and it is working, isn't it?09:32
apuimedothe f25 is a physical host?09:32
dmelladoapuimedo: yep, f25 is a bare metal host09:33
dmelladowith 2 diff nics09:33
apuimedook09:33
apuimedocan't you just start a screen session on it09:33
apuimedoand on it put a script that does the set down, set name and set up09:34
apuimedoyou should regain connectivity without the need for re-stacking09:34
apuimedoalso... Which vif driver are you using?09:34
ltomasbowe did not specified any, so the generic one09:35
ltomasboit is an installation from upstream kuryr09:35
*** yamamoto has joined #openstack-kuryr09:35
apuimedoltomasbo: (I was just trying to figure out why the eth0 on the hypervisor would be relevant)09:36
ltomasboshould it be?09:37
ltomasbomy other server with centos7 is not using eth0 either09:37
ltomasboand it works09:37
apuimedoit should not09:37
apuimedoltomasbo: the best would be to go with the debugger on the os-vif code09:37
*** yamamoto has quit IRC09:38
apuimedoand see what the hell is going on09:38
apuimedo(and possibly compare it to when it is being used with nova-compute)09:38
*** yamamoto has joined #openstack-kuryr09:38
openstackgerritJaivish Kothari(janonymous) proposed openstack/kuryr-libnetwork master: Fix N536 hacking check  https://review.openstack.org/44691909:40
apuimedojanonymous: please, expand on the bug description https://bugs.launchpad.net/kuryr-libnetwork/+bug/167371609:41
openstackLaunchpad bug 1673716 in kuryr-libnetwork "Hacking check N536 Error" [Undecided,In progress] - Assigned to janonymous (janonymous)09:41
ltomasbocall for nova:09:41
ltomasboPlugging vif VIFOpenVSwitch(active=False,address=fa:16:3e:6d:84:42,bridge_name='br-int',has_traffic_filtering=True,id=694d75bb-81b6-4a32-970d-433c8deddb25,network=Network(9f60eacf-564a-4b57-9205-410ffbe7ec1f),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tap694d75bb-81')09:42
ltomasbocall for kubelet:09:42
ltomasboPlugging vif VIFOpenVSwitch(active=True,address=fa:16:3e:6a:71:7e,bridge_name='br-int',has_traffic_filtering=True,id=7e5918c8-3170-4ff3-ab8a-0829a665b8ab,network=Network(9f60eacf-564a-4b57-9205-410ffbe7ec1f),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap7e5918c8-31')09:42
*** yamamoto has quit IRC09:42
apuimedoltomasbo: is the tap device on the kubelet one somehow missing?09:43
ltomasboI just see 2 differences, the active=False/True and the preserve_on_delete09:43
janonymousapuimedo:added a gate link, tox -e pep8 fails on recent checkout..09:43
ltomasbolet me create one extra one and check (that was from the logs)09:44
apuimedook09:45
apuimedothanks09:45
ltomasboyep, tap is not being created09:47
apuimedothere you go09:48
apuimedo:-)09:48
apuimedoso pyroute2 is not creating the veth pair at all, or just not renaming it?09:48
ltomasbook, now, why??09:51
ltomasbopyroute2 0.4.1309:52
ltomasbomaybe I can try that by hand, to see if pyroute can create veths there09:52
apuimedoyes, please09:52
dmelladoif it doesn't, we're in deep ****09:52
dmelladoxd09:53
dmelladopyroute2==0.4.1309:53
dmelladoworks in the VM09:54
dmelladoso this is getting weirder and weirder...09:54
apuimedolet's wait for ltomasbo to report09:56
ltomasbo>>> import pyroute209:56
ltomasbo>>> ipdb = pyroute2.IPDB()09:56
ltomasbo>>> ipdb.create(ifname='test', peer='test-peer', kind='veth')09:56
ltomasbo{'index': 0, 'kind': 'veth', 'ipaddr': [], 'ports': [], 'peer': 'test-peer', 'ifname': 'test', 'vlans': [], 'ipdb_priority': 0, 'ipdb_scope': 'create'}09:56
ltomasbodoes not fail, but it does not create anything09:57
ltomasbopyroute2 0.4.13 too09:57
dmelladoltomasbo: let me try the same apparoach on my vm09:57
apuimedohttps://www.youtube.com/watch?v=KaqC5FnvAEc10:00
ltomasboweird enough10:01
ltomasboif I try to re-run the command10:01
ltomasboit generate an exception10:01
apuimedowhich?10:01
dmelladoit says it exists10:01
ltomasbointerface test exists!!10:01
dmelladoxD10:01
dmelladobut where?10:01
dmelladoxd10:01
apuimedo:O10:01
ltomasbowhere!!!10:01
apuimedoltomasbo: did you just run it on an ipython session?10:02
dmelladoapuimedo: nope, I just learnt10:02
dmelladothat ipython doesn't like having another IPDB around xD10:02
ltomasbo$ python10:02
ltomasboPython 2.7.13 (default, Jan 12 2017, 17:59:37)10:02
ltomasbo[GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux210:02
ltomasboType "help", "copyright", "credits" or "license" for more information.10:02
ltomasbo>>> import pyroute210:02
apuimedook10:02
apuimedokernel?10:03
ltomasbo4.9.13-201.fc25.x86_6410:03
apuimedook10:03
ltomasbowhere can I find that veth link??10:04
apuimedoltomasbo: can you create a veth pair with "ip link" ?10:04
*** svinota has joined #openstack-kuryr10:04
janonymousdiff netns?10:04
svinotaapuimedo, (here)10:05
apuimedothanks svinota10:05
dmelladojanonymous: it's just from raw python code10:05
dmelladoso not nentns :\10:05
dmelladosvinota: o/10:05
apuimedobasically, the issue is that somehow, ipdb does not create the veth pair10:05
apuimedoit reports success, but the interface is nowhere10:05
ltomasbobut it is supposed to be somewhere10:05
ltomasboas it reports success10:05
apuimedoltomasbo: if you do ipdb.interfaces10:05
svinotaapuimedo, I'm on a wired network and have to leave the room shortly, so may miss something. So excuse me beforehand10:06
apuimedodoes it show up?10:06
ltomasboand does not allow me to create it again10:06
apuimedoltomasbo: did you .commit() ?10:06
janonymousdmellado: ohh10:06
apuimedosvinota: no worries10:06
dmelladoltomasbo: how about trying 'ip link add foo type veth bar'10:06
dmelladoand check what happens10:06
apuimedo(it happens to me when I upgrade my kernel without keeping the old veth module10:06
apuimedo)10:06
apuimedo:P10:06
apuimedobut I doubt ltomasbo did that10:06
ltomasboit works with ip link dev addd10:08
dmelladoapuimedo: heh, creating a veth pair from cli seems to work10:08
svinotaapuimedo, is it possible to see the veth object in the ipdb after it is created? what the ipdb.interfaces[<name>].ipdb_scope ?10:08
ltomasbodmellado, what about the kernel stuff apuimedo mentioned? as you did the update10:08
dmelladoltomasbo: I just installed f2310:09
dmelladoupdated to f2510:09
dmelladoprior to install anything10:09
dmelladoso this shouldn't be happening10:09
dmelladoI mean, the new kenerl was up and running prior to run anything10:09
svinotaand if it's possible to see the whole object, it would be even better10:09
ltomasbo>>> pdb.interfaces['foo']10:10
ltomasbo{'family': 0, 'txqlen': 1000, 'ipdb_scope': 'system', 'index': 31, 'operstate': 'DOWN', 'num_tx_queues': 1, 'group': 0, 'carrier_changes': 1, 'ipaddr': [], 'neighbours': [], 'ifname': 'foo', 'promiscuity': 0, 'linkmode': 0, 'kind': 'veth', 'broadcast': 'ff:ff:ff:ff:ff:ff', 'link': 30, 'address': '86:7b:3b:6b:df:7e', 'vlans': [], 'ipdb_priority': 0, 'gso_max_segs': 65535, 'gso_max_size': 65536, 'qdisc': 'noop', 'mtu': 1500,10:10
ltomasbo'num_rx_queues': 1, 'carrier': 0, 'flags': 4098, 'ifi_type': 1, 'proto_down': 0, 'ports': []}10:10
ltomasbo>>> pdb.interfaces['test']                                                                    Traceback (most recent call last):10:10
ltomasbo  File "<stdin>", line 1, in <module>10:10
ltomasboKeyError: 'test'10:10
ltomasbofoo works (the one created from cli)10:10
*** aojea has joined #openstack-kuryr10:10
ltomasbothe one created with pyroute2 before is not there10:10
apuimedoltomasbo: did or did you not do ipdb.commit() after creating the iface?10:10
dmelladoI guess we didn't do that10:10
ltomasbo>>> pdb.create(ifname='test', peer='test-peer', kind='veth')10:11
ltomasbo{'index': 0, 'kind': 'veth', 'ipaddr': [], 'ports': [], 'peer': 'test-peer', 'ifname': 'test', 'vlans': [], 'ipdb_priority': 0, 'ipdb_scope': 'create'}10:11
ltomasbo>>> pdb.interfaces['test']                                                                    {'index': 0, 'kind': 'veth', 'ipaddr': [], 'ports': [], 'peer': 'test-peer', 'ifname': 'test', 'vlans': [], 'ipdb_priority': 0, 'ipdb_scope': 'create'}10:11
svinotaltomasbo, ipdb_scope = create means it's not committed10:11
svinotaor commit failed10:11
ltomasboapuimedo, I did not ipbd.commit()10:11
apuimedoheh10:11
apuimedotry with that then!10:11
*** kzaitsev_mb has quit IRC10:12
ltomasboperhaps I need to run python as sudo... :D10:12
ltomasbolet me try10:12
apuimedoalso10:13
ltomasbook10:14
ltomasbothat works10:14
apuimedovery well10:14
ltomasboand it creates the veth10:14
janonymous(Y)10:14
ltomasboso, then why it is not created for kuryr...10:14
apuimedothat's a good question10:15
ltomasbo:/10:15
dmelladoltomasbo: I bribed apuimedo10:15
dmelladoso he'll have a look10:15
svinotathere must be an exception in the logs, if the creation fails10:15
apuimedodmellado: I didn't agree to that yet10:15
apuimedoxD10:15
dmelladoapuimedo: but you will10:15
dmelladoyou know10:15
dmelladoxD10:15
ltomasbosvinota, what logs you refer to?10:15
ltomasbokubelet?10:16
dmelladootherwise recall about the add-jeff-as-a-reviewer10:16
dmelladoscript10:16
svinotaltomasbo, I don't work with kuryr so close, so you should know better :) but there are only two possible options — either ipdb creates the interface and actually receives a confirmation from the kernel, or it throws an exception10:17
ltomasbook, going to take a look with pdb to see if the cni gets some errors (though I don't see them on the kuryr logs)10:19
dmelladosvinota: so, the weird thing is that we couldn't get to get such exception, just the logs that we pasted before10:19
dmelladosvinota: http://paste.openstack.org/show/603092/10:19
apuimedoltomasbo: are you sure the interface was not created then?10:19
apuimedothe tap10:19
ltomasbomaybe it was and was deleted right away by kuryr10:20
ltomasbothat is what I'm going to check with pdb.set_trace...10:20
apuimedoltomasbo: oh, I thought you stopped it10:22
apuimedoltomasbo: if you can't trace, just put a system.exit() :P10:22
ltomasbo:D10:23
dmelladoawesome, this just gets yuo closer to the script adding gabert as a reviewer too10:23
dmelladoapuimedo:10:23
dmelladoxD10:23
apuimedo:-)10:24
apuimedodmellado: I'm trying to see if I can add ovsdbapp to os-vif10:24
apuimedoso we get rid of the command line crap10:24
dmelladothat'd be cool10:25
*** danil has joined #openstack-kuryr10:26
svinota(have to leave the room, back in 30 minutes)10:28
apuimedothanks svinota10:30
ltomasboapuimedo, the pdb.set_trace seems not to work or not to be reached10:31
ltomasbothis is the initial logs for kubelet when creating the container10:32
ltomasbo docker_manager.go:1901] Need to restart pod infra container for "busybox-sleep1_default(457eafd6-0b0d-11e7-8db9-0cc47a66a320)" because it is not found10:32
*** svinota has quit IRC10:32
ltomasbodocker_manager.go:746] Logging security options: {key:seccomp value:unconfined msg:}10:32
ltomasbodocker_manager.go:357] NetworkPlugin cni failed on the status hook for pod 'busybox-sleep1' - Unexpected command output Device "eth0" does not exist.10:32
ltomasbo with error: exit status 110:32
apuimedoltomasbo: this is our cni failing10:35
apuimedoand since it fails, it doesn't move the veth into the namespace and doesn't rename it10:35
*** yedongcan has left #openstack-kuryr10:36
ltomasbomakes sense10:37
ltomasboany idea how to proceed?10:37
apuimedoI'm going in to check10:39
*** yamamoto has joined #openstack-kuryr10:41
*** janki has quit IRC10:44
*** yuanying has quit IRC10:45
*** svinota has joined #openstack-kuryr10:47
*** kzaitsev_mb has joined #openstack-kuryr11:08
*** aojea_ has joined #openstack-kuryr11:28
*** aojea has quit IRC11:31
*** yuanying has joined #openstack-kuryr11:40
*** kzaitsev_mb has quit IRC11:42
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements  https://review.openstack.org/43860411:55
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/43196611:55
*** neiljerram has quit IRC11:55
*** neiljerram has joined #openstack-kuryr11:58
*** saneax is now known as saneax-_-|AFK12:27
*** kzaitsev_mb has joined #openstack-kuryr12:38
*** gsagie has joined #openstack-kuryr12:44
*** limao has joined #openstack-kuryr12:48
*** yamamoto has quit IRC12:55
*** yamamoto has joined #openstack-kuryr12:56
*** janonymous has quit IRC13:04
*** svinota has quit IRC13:16
*** danil has quit IRC13:20
apuimedoso it was os-vif/distgit13:20
apuimedosorry13:20
apuimedoos-vif/privsep13:20
apuimedono idea why yet13:21
apuimedobut, the good thing is that there's a workaround of commenting the vif plug13:21
apuimedo(which for ovs firewall is unnecessary13:21
apuimedoltomasbo: could you file the bug with kuryr-kubernetes and mention the workaround?13:21
*** kzaitsev_mb has quit IRC13:24
ltomasbosure, but do we know if if is actually a kuryr-kuberentes bug?13:27
*** limao_ has joined #openstack-kuryr13:27
ltomasboor os-vif? or somewhere else?13:27
ltomasboapuimedo, ^^13:27
apuimedoltomasbo: we don't. But for now. It pops up when using kuryr-kubernetes, so we file it there and we can move it when we find out more13:28
*** limao has quit IRC13:29
ltomasbook13:30
ltomasboI'll do that!13:30
apuimedothanks!13:31
*** apuimedo has quit IRC13:39
*** apuimedo has joined #openstack-kuryr14:05
*** hongbin has joined #openstack-kuryr14:08
*** kzaitsev_mb has joined #openstack-kuryr14:21
*** limao_ has quit IRC14:36
*** yamamoto has quit IRC14:43
*** yamamoto has joined #openstack-kuryr14:45
*** yamamoto has quit IRC14:50
*** neiljerram has quit IRC15:04
*** kzaitsev_mb has quit IRC15:07
*** yamamoto has joined #openstack-kuryr15:48
*** yamamoto has quit IRC15:56
*** kzaitsev_mb has joined #openstack-kuryr16:03
*** aojea_ has quit IRC16:28
*** vikasc has quit IRC16:41
ltomasbobug entry created: https://bugs.launchpad.net/kuryr-kubernetes/+bug/167383216:42
openstackLaunchpad bug 1673832 in kuryr-kubernetes "os_vif.plug() at cni fails with privsep issue when using ovs-firewall" [Undecided,New]16:42
*** vikasc has joined #openstack-kuryr16:54
*** kzaitsev_mb has quit IRC16:56
apuimedothanks ltomasbo17:02
openstackgerritAntoni Segura Puimedon proposed openstack/kuryr-kubernetes master: cni: Use ovsdbapp for ovs operations  https://review.openstack.org/44707317:13
openstackgerritAntoni Segura Puimedon proposed openstack/kuryr-kubernetes master: cni: Use ovsdbapp for ovs operations  https://review.openstack.org/44707317:15
*** pcaruana has quit IRC17:18
*** ltomasbo is now known as ltomasbo|away17:35
*** svinota has joined #openstack-kuryr17:53
*** kzaitsev_mb has joined #openstack-kuryr17:54
*** gsagie has quit IRC18:16
*** aojea has joined #openstack-kuryr18:22
*** aojea has quit IRC19:02
*** aojea has joined #openstack-kuryr19:14
*** aojea has quit IRC19:34
*** tonanhngo has joined #openstack-kuryr19:35
*** tonanhngo has quit IRC19:35
*** aojea has joined #openstack-kuryr19:40
*** tonanhngo has joined #openstack-kuryr19:46
*** svinota has quit IRC20:28
*** vikasc has quit IRC20:28
*** kzaitsev_mb has quit IRC20:32
*** neiljerram has joined #openstack-kuryr20:49
*** svinota has joined #openstack-kuryr21:05
*** svinota has quit IRC22:17
*** saneax-_-|AFK is now known as saneax22:19
*** svinota has joined #openstack-kuryr22:44
*** aojea has quit IRC23:10
*** svinota has quit IRC23:17

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!