Wednesday, 2016-04-06

*** Marga_ has joined #openstack-operators00:01
*** Marga_ has quit IRC00:02
*** Marga_ has joined #openstack-operators00:03
*** Marga_ has quit IRC00:03
*** Marga_ has joined #openstack-operators00:03
*** stanchan has joined #openstack-operators00:17
*** lhcheng has quit IRC00:42
*** esker has quit IRC00:43
*** esker has joined #openstack-operators00:47
*** mriedem has quit IRC01:20
*** lhcheng has joined #openstack-operators01:23
*** lhcheng_ has joined #openstack-operators01:24
*** lhcheng has quit IRC01:24
*** esker has quit IRC01:24
*** apoorvad has quit IRC01:35
*** ducttape_ has joined #openstack-operators01:38
*** ducttape_ has quit IRC01:53
*** busterswt has joined #openstack-operators01:55
*** _ducttape_ has joined #openstack-operators01:56
*** bustersw_ has joined #openstack-operators01:57
*** busterswt has quit IRC02:01
*** lhcheng_ has quit IRC02:01
*** harshs has joined #openstack-operators02:08
*** VW has joined #openstack-operators02:14
*** VW has quit IRC02:20
*** VW has joined #openstack-operators02:20
*** lhcheng has joined #openstack-operators02:30
*** lhcheng has quit IRC02:30
*** lhcheng has joined #openstack-operators02:30
*** bustersw_ has quit IRC02:33
*** harshs_ has joined #openstack-operators02:38
*** harshs has quit IRC02:40
*** harshs_ is now known as harshs02:40
*** _ducttape_ has quit IRC03:05
*** stanchan has quit IRC03:06
*** markvoelker has quit IRC03:26
*** esker has joined #openstack-operators03:34
*** esker has quit IRC03:52
*** resker has joined #openstack-operators03:52
*** VW has quit IRC03:55
*** esker has joined #openstack-operators03:57
*** resker has quit IRC03:59
*** esker has quit IRC04:05
*** esker has joined #openstack-operators04:07
*** markvoelker has joined #openstack-operators04:26
*** markvoelker has quit IRC04:32
*** armax has quit IRC05:01
*** simon-AS559 has joined #openstack-operators05:20
*** sekrit has quit IRC05:30
*** Marga_ has quit IRC05:34
*** Marga_ has joined #openstack-operators05:37
*** Marga_ has quit IRC05:38
*** Marga_ has joined #openstack-operators05:38
*** bvandenh has joined #openstack-operators05:38
*** admin0 has joined #openstack-operators05:41
*** Marga_ has quit IRC05:43
*** liverpooler has quit IRC05:44
*** saneax_AFK is now known as saneax05:44
*** admin0 has quit IRC05:45
*** Marga_ has joined #openstack-operators05:52
*** Marga_ has quit IRC05:56
*** harshs has quit IRC05:59
*** rcernin has joined #openstack-operators05:59
*** Marga_ has joined #openstack-operators06:01
*** fawadkhaliq has joined #openstack-operators06:02
*** vsedelnik has quit IRC06:04
*** lhcheng has quit IRC06:13
*** bvandenh has quit IRC06:13
*** vsedelnik has joined #openstack-operators06:14
*** bvandenh has joined #openstack-operators06:17
*** vsedelnik has quit IRC06:18
*** fawadkhaliq has quit IRC06:20
*** sekrit has joined #openstack-operators06:23
*** eddima has joined #openstack-operators06:26
*** markvoelker has joined #openstack-operators06:29
*** belmoreira has joined #openstack-operators06:30
*** simon-AS559 has quit IRC06:30
*** markvoelker has quit IRC06:35
*** simon-AS559 has joined #openstack-operators06:37
*** lhcheng has joined #openstack-operators07:00
*** openstackgerrit has quit IRC07:02
*** openstackgerrit has joined #openstack-operators07:03
*** Marga_ has quit IRC07:05
*** liverpooler has joined #openstack-operators07:05
*** pcaruana has joined #openstack-operators07:06
*** rcernin has quit IRC07:08
*** rcernin has joined #openstack-operators07:09
*** admin0 has joined #openstack-operators07:09
*** admin0 has quit IRC07:09
*** admin0 has joined #openstack-operators07:20
*** vsedelnik has joined #openstack-operators07:33
*** vsedelnik has quit IRC07:33
*** vsedelnik has joined #openstack-operators07:33
*** simon-AS559 has quit IRC07:37
*** goretoxo has joined #openstack-operators07:55
*** admin0 has quit IRC08:12
*** admin0 has joined #openstack-operators08:15
*** vsedelnik has quit IRC08:16
*** vsedelnik has joined #openstack-operators08:18
*** simon-AS559 has joined #openstack-operators08:24
*** simon-AS5591 has joined #openstack-operators08:25
*** simon-AS559 has quit IRC08:28
*** markvoelker has joined #openstack-operators08:30
*** simon-AS559 has joined #openstack-operators08:34
*** markvoelker has quit IRC08:35
*** simon-AS5592 has joined #openstack-operators08:35
*** simon-AS5591 has quit IRC08:36
*** vsedelnik has quit IRC08:37
*** simon-AS5591 has joined #openstack-operators08:37
*** haukebruno has joined #openstack-operators08:39
*** simon-AS5593 has joined #openstack-operators08:39
*** simon-AS5593 has quit IRC08:39
*** simon-AS559 has quit IRC08:39
*** simon-AS5592 has quit IRC08:40
*** vsedelnik has joined #openstack-operators08:41
*** simon-AS5591 has quit IRC08:41
*** vsedelnik has quit IRC08:42
*** vsedelnik has joined #openstack-operators08:42
*** lhcheng has quit IRC08:43
*** simon-AS559 has joined #openstack-operators08:43
*** derekh has joined #openstack-operators08:46
*** electrofelix has joined #openstack-operators09:05
*** Marga_ has joined #openstack-operators09:05
*** Marga_ has quit IRC09:10
*** zeih has joined #openstack-operators09:32
*** zeih has quit IRC09:44
*** bvandenh has quit IRC10:19
*** markvoelker has joined #openstack-operators10:31
*** bvandenh has joined #openstack-operators10:33
*** markvoelker has quit IRC10:36
*** zeih has joined #openstack-operators10:51
*** hieulq has joined #openstack-operators10:52
*** vsedelnik has quit IRC10:55
*** vsedelnik has joined #openstack-operators10:57
*** vsedelnik has quit IRC10:59
*** zioproto has quit IRC11:00
*** admin0_ has joined #openstack-operators11:04
*** zioproto has joined #openstack-operators11:06
*** admin0 has quit IRC11:07
*** admin0_ is now known as admin011:07
*** Marga_ has joined #openstack-operators11:15
*** hieulq_ has joined #openstack-operators11:15
*** Marga_ has quit IRC11:16
*** Marga_ has joined #openstack-operators11:16
*** hieulq has quit IRC11:17
*** vsedelnik has joined #openstack-operators11:26
*** zeih has quit IRC11:31
*** simon-AS559 has quit IRC11:32
*** zeih has joined #openstack-operators11:33
*** admin0 has quit IRC11:36
*** admin0 has joined #openstack-operators11:37
*** simon-AS559 has joined #openstack-operators11:52
*** busterswt has joined #openstack-operators11:55
*** busterswt has quit IRC11:59
*** snecklifter_ has joined #openstack-operators12:00
*** simon-AS5591 has joined #openstack-operators12:03
*** hieulq_ has quit IRC12:05
*** simon-AS5592 has joined #openstack-operators12:06
*** simon-AS559 has quit IRC12:06
*** Marga_ has quit IRC12:07
*** simon-AS5591 has quit IRC12:07
*** Marga_ has joined #openstack-operators12:09
*** markvoelker has joined #openstack-operators12:15
*** simon-AS5592 has quit IRC12:31
*** Marga_ has quit IRC12:36
*** simon-AS559 has joined #openstack-operators12:37
*** rarcea has joined #openstack-operators12:39
*** rarcea has quit IRC12:43
*** simon-AS559 has quit IRC13:00
*** haukebruno has quit IRC13:01
*** simon-AS559 has joined #openstack-operators13:03
*** bvandenh has quit IRC13:04
*** saneax is now known as saneax_AFK13:06
*** simon-AS5591 has joined #openstack-operators13:07
*** simon-AS559 has quit IRC13:08
*** Vinsh has quit IRC13:16
*** simon-AS5591 has quit IRC13:16
*** mriedem has joined #openstack-operators13:16
*** admin0 has quit IRC13:18
*** simon-AS559 has joined #openstack-operators13:23
*** vsedelnik has quit IRC13:25
*** simon-AS559 has quit IRC13:28
*** vsedelnik has joined #openstack-operators13:30
*** admin0 has joined #openstack-operators13:31
*** kstev has joined #openstack-operators13:32
*** bvandenh has joined #openstack-operators13:34
*** vsedelnik has quit IRC13:35
*** matrohon has joined #openstack-operators13:38
*** vsedelnik has joined #openstack-operators13:40
*** vinsh has joined #openstack-operators13:40
*** esker has quit IRC13:42
*** simon-AS559 has joined #openstack-operators13:43
*** matrohon has quit IRC13:53
*** matrohon has joined #openstack-operators13:53
*** simon-AS559 has quit IRC13:55
*** csoukup has joined #openstack-operators13:59
*** mriedem is now known as mriedem_meeting14:00
*** simon-AS559 has joined #openstack-operators14:00
*** saneax_AFK is now known as saneax14:02
*** electrofelix has quit IRC14:02
*** kstev has quit IRC14:05
*** krobzaur has joined #openstack-operators14:07
*** busterswt has joined #openstack-operators14:08
*** liverpooler has quit IRC14:10
*** busterswt has quit IRC14:13
*** snecklifter_ has quit IRC14:15
*** busterswt has joined #openstack-operators14:15
*** snecklifter_ has joined #openstack-operators14:15
*** snecklifter_ has quit IRC14:16
*** rpjr has joined #openstack-operators14:16
*** snecklifter has joined #openstack-operators14:16
*** zeih has quit IRC14:26
*** matrohon has quit IRC14:31
*** mriedem_meeting is now known as mriedem14:31
*** zul has quit IRC14:35
*** zeih has joined #openstack-operators14:36
*** electrofelix has joined #openstack-operators14:40
*** VW has joined #openstack-operators14:42
*** zeih has quit IRC14:47
*** zeih has joined #openstack-operators14:47
*** mdorman has joined #openstack-operators14:50
*** ducttape_ has joined #openstack-operators14:50
*** zeih has quit IRC14:51
*** zul has joined #openstack-operators15:01
*** kstev has joined #openstack-operators15:07
*** kstev has quit IRC15:11
*** kstev has joined #openstack-operators15:12
*** Marga_ has joined #openstack-operators15:16
*** belmoreira has quit IRC15:17
*** racedo has joined #openstack-operators15:17
*** SimonChung has quit IRC15:19
*** armax has joined #openstack-operators15:27
*** zul_ has joined #openstack-operators15:27
*** zul has quit IRC15:27
*** esker has joined #openstack-operators15:29
*** eddima has quit IRC15:33
*** admin0 has quit IRC15:37
*** armax_ has joined #openstack-operators15:41
*** armax has quit IRC15:42
*** armax_ is now known as armax15:42
*** rpjr has quit IRC15:46
*** electrofelix has quit IRC15:56
*** Rockyg has joined #openstack-operators15:56
*** vsedelnik has quit IRC15:57
*** vsedelnik has joined #openstack-operators15:58
*** simon-AS559 has quit IRC15:58
*** ducttape_ has quit IRC16:00
*** ducttape_ has joined #openstack-operators16:02
*** apoorvad has joined #openstack-operators16:06
*** xavpaice has quit IRC16:16
*** SimonChung has joined #openstack-operators16:17
*** xavpaice has joined #openstack-operators16:20
*** goretoxo has quit IRC16:27
*** Marga_ has quit IRC16:34
*** zul_ has quit IRC16:35
*** Marga_ has joined #openstack-operators16:37
*** haukebruno has joined #openstack-operators16:42
*** Marga_ has quit IRC16:44
*** lhcheng has joined #openstack-operators16:44
*** c_soukup has joined #openstack-operators16:46
*** admin0 has joined #openstack-operators16:49
*** csoukup has quit IRC16:49
*** derekh has quit IRC16:50
*** c_soukup has quit IRC16:50
*** zul has joined #openstack-operators16:51
*** zhangjn has quit IRC16:57
*** zhangjn has joined #openstack-operators16:58
*** saneax is now known as saneax_AFK17:00
*** piet has joined #openstack-operators17:03
*** esker has quit IRC17:05
*** harshs has joined #openstack-operators17:05
*** esker has joined #openstack-operators17:06
*** vsedelnik has quit IRC17:06
raginbajinSo, what we have found out while trying to run without L2_population is that we can't guarentee instances are avail at boot without it trying to communicate out.   So, we are digging back into our vif_error problem that we have tracked back to l2_population (atleast it occurs during that time).  We have found that there is code within the ovs_agent that during l2_pop if one of the other hypervisors does not respond correct17:11
raginbajinly back to the l2_pop call with either a exception that is not the only exact expception that the code is looking for it will kill the VM and basically put the neutron into a state where new vms can not be created.17:11
raginbajinThe only way out of it is to restart the neutronagent which kills the flows and causes an interruption.17:11
*** simon-AS559 has joined #openstack-operators17:11
raginbajinIt looks like it may be fixed in liberty but we are still going through the code for it.  We think this is what is causing our vif_error problems that prevents us from building new instances.17:12
raginbajinWe are building a test for it to see if we cause Neutron to go 100% cpu like we see and not respond anymore.17:13
raginbajinWhat causes the l2_pop to fail, we don't know yet, but we know it causes things to unwind and get us into this state.17:13
raginbajinJust for anyone that was dying to know some update to a few questions I've asked about lately17:13
*** admin0 has quit IRC17:15
*** simon-AS559 has quit IRC17:17
*** simon-AS559 has joined #openstack-operators17:17
*** eil397 has joined #openstack-operators17:18
*** Rockyg has quit IRC17:21
*** simon-AS559 has quit IRC17:22
claytonraginbajin: do you guys have metadata turned on?17:22
raginbajinWe do, but don't use it.. We use the CDROM instead as we saw early problems with it..17:22
claytonwe use metadata, so that might be why it works for us17:23
raginbajinahhhhhh17:23
raginbajinYes it would be.. We thought about how it was working.17:23
claytonbut DHCP would also probably be sufficent17:24
*** eil397 has left #openstack-operators17:24
claytonare you not using DHCP either?17:24
raginbajinWe do, but DHCP doesn't do it17:24
*** simon-AS559 has joined #openstack-operators17:24
raginbajinbecause not all the other instances will know about it17:24
raginbajinonce the DHCP request goes out as broadcast it goes back as  unicast17:24
claytonit's the traffic that comes back that builds the forwarding entries17:24
raginbajinand it talks unicast for the rest. So not everything will hear about it.17:24
claytonI think you have something else wrong though17:25
claytonthere there is no learned entry, it should get flooded to all vteps17:25
raginbajinThat wouldn't be surprising..17:25
raginbajinMaybe they changed the code in Liberty to flood all vteps.17:25
raginbajinWe only see that code for VLAN17:25
claytonwe're still on kilo till next week17:26
raginbajinbut run with l2_pop there right?17:26
*** admin0 has joined #openstack-operators17:26
claytonthe only changes in the flows for liberty are around arp anti-sproofing17:26
claytonyeah, true17:26
claytonbut I've looked at the flow tables for both17:26
raginbajinHmm it's very strange.. Did you turn l2_pop off everywhere including gateways, servers and computes?17:27
claytonyes17:27
claytoncan you post a gist of the output of ovs-ofctl dump-flows br-tun ?17:28
*** Marga_ has joined #openstack-operators17:28
raginbajinyeah one second.17:28
claytontable 22 should flood all BUM traffic to all vteps17:28
raginbajinI'm assuming you want it from a compute that has l2 totally turned off right?17:30
claytonthat should work17:30
raginbajinalright.. give me a few. I need to set that back up.17:30
claytonpreferably from a smaller environment :)17:30
raginbajinhehe.. yeah.. working on it now.17:33
*** esker has quit IRC17:39
*** david-lyle has quit IRC17:44
raginbajinI thinking about this, if we don't use metadata service to initiate any network traffic, then that wouldn't create any traffic to get an arp added.  DHCP is only going to respond in a unicast way so everyone doesn't know about it.. I guess I still don't get when a flood would happen.17:47
claytonlet's say you have an incoming ICMP, it will come into the router, there won't be an arp entry on the router, and it's going to flood the ARP traffic to all vxlan tunnels17:48
claytonone of them will respond and that should populate the same thing that L2 pop normally does17:48
claytondo you have arp responder turned on?17:49
*** SimonChung has quit IRC17:49
*** SimonChung1 has joined #openstack-operators17:49
raginbajinYou can't turn arp_responder on without turning on l2_pop17:49
claytonmaking sure :)17:49
claytondo you know if you're getting arp responses in the router namespace?17:50
raginbajinSo, if I'm on the brand new host that hasn't sent out any connectivity, and ping the router. Everything will work.17:52
raginbajinUp until that point, no one can connect to that box.17:52
claytonright, but the opposite direction should generate an ARP inside the private network, which should populate all your flows17:52
claytonyou should have something like this:17:53
clayton# ovs-ofctl dump-flows br-tun table=1017:54
claytonNXST_FLOW reply (xid=0x4):17:54
clayton cookie=0x86e68bf9735c84f1, duration=439104.776s, table=10, n_packets=1356583, n_bytes=106475933, idle_age=2, hard_age=65534, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0x86e68bf9735c84f1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:817:54
claytonwhen the traffic comes in the vxlan endpoint, it will create a new flow for the return traffic17:54
claytonthat's on the compute host17:54
claytonon the router side, it should be generaing arp traffic for any new compute hosts it's not seen before, and since that's broadcast it will get flooded to all compute hosts that are on that network17:55
*** rcernin has quit IRC17:56
*** mriedem has quit IRC17:56
raginbajinBut those will eventually expires17:57
claytononly if there is no traffic being passed, in which case the arp will expire also17:57
raginbajinThen you could have connectivity issues.17:58
*** ducttape_ has quit IRC17:58
raginbajinright..17:58
raginbajinThis network stuff is hurting my head.17:58
claytonheh17:58
claytonI spent a lot of time on this stuff earlier this year17:59
*** haukebruno has quit IRC18:00
*** simon-AS559 has quit IRC18:00
raginbajinI don't think we see a lot of gratuitous type of arps..18:00
raginbajinWe see only specific destination and sources, so I'm not sure how everyone else finds out about these mac's.18:00
claytonanyone that wants to talk to the host will have to arp for it's MAC address18:01
claytonand will expire the arp entry if it's not talking to it18:01
claytonthe learned flow will never expire if there is any traffic at all over a 5 minute time span18:01
raginbajinto that instance or any traffic?18:02
claytonthese flows are all per instance, per hypervisor18:02
*** david-lyle has joined #openstack-operators18:02
raginbajinand with l2_pop do those flows have that same 5 minute time span?18:03
claytonso if two instances talk to each other, they'll learn about each other via the flows that arp sets up, and those flows will stay up as long as they keep talking18:03
claytonif they stop talking, the entries in the arp cache will expire and they'll have to arp for each other again, which will learn new flows18:03
claytonany traffic between two instances will keep the learned flows alive18:03
raginbajinAnd that is pretty much invisble to the user then say if they ssh into the host or anything like that.18:04
claytonyes18:04
raginbajinor will they get host not found, and then have to try again to get it to work18:04
claytonit should just work18:04
raginbajinok18:04
claytonno, the packets are held while all this is in progress18:04
claytonthe arp flow here is not specific to openstack/vxlan/ovs/etc at all18:04
claytonthat's all normal18:04
*** mriedem has joined #openstack-operators18:11
*** piet has quit IRC18:15
*** ducttape_ has joined #openstack-operators18:16
*** haukebruno has joined #openstack-operators18:16
raginbajinHmm. I think we are going to do some more testing first.. I think we have probably maybe a config prob or software issue..18:17
raginbajinWhat version of OVS are you using? I wonder if its an older version that is not creating our tunnels when a new vm is built18:17
*** regXboi has joined #openstack-operators18:18
*** rockyg has joined #openstack-operators18:41
*** rarcea has joined #openstack-operators18:45
raginbajinclayton:  Ok I think I need more testing.. We turned off l2_pop on everything and we have it working as it should.. Which we have seen once or twoice, but not consistenly.. The only thing in the back of my mind is we did vxlan with icehouse and had an issue where eventually hosts were not able to talk to each other. The only way to resolve was to setup a cron to continiously ping each other.. Then things would work.18:46
*** maishsk has joined #openstack-operators19:05
mriedemops meeting in #openstack-meeting-419:10
*** rockyg has quit IRC19:15
*** ducttape_ has quit IRC19:17
*** simon-AS559 has joined #openstack-operators19:18
claytonmriedem: thanks19:18
*** simon-AS559 has quit IRC19:23
*** simon-AS559 has joined #openstack-operators19:23
*** haukebruno has quit IRC19:25
*** esker has joined #openstack-operators19:30
*** ducttape_ has joined #openstack-operators19:32
*** esker has quit IRC19:35
*** esker has joined #openstack-operators19:36
*** Long_yanG has joined #openstack-operators19:55
*** LongyanG has quit IRC19:58
*** maishsk has quit IRC19:58
*** piet has joined #openstack-operators19:59
*** maishsk has joined #openstack-operators20:00
*** maishsk_ has joined #openstack-operators20:04
*** mgagne_ is now known as mgagne20:04
*** maishsk has quit IRC20:05
*** maishsk has joined #openstack-operators20:07
*** maishsk_ has quit IRC20:08
*** vsedelnik has joined #openstack-operators20:09
*** rcernin has joined #openstack-operators20:09
*** rockyg has joined #openstack-operators20:13
*** pcaruana has quit IRC20:14
*** matrohon has joined #openstack-operators20:14
*** maishsk has quit IRC20:29
*** maishsk has joined #openstack-operators20:29
*** matrohon has quit IRC20:37
*** vsedelnik has quit IRC20:42
*** rockyg has quit IRC20:47
*** busterswt has quit IRC20:59
*** busterswt has joined #openstack-operators21:00
*** vsedelnik has joined #openstack-operators21:06
*** vsedelnik has quit IRC21:20
*** vsedelnik has joined #openstack-operators21:24
*** rcernin has quit IRC21:27
*** VW has quit IRC21:27
*** VW has joined #openstack-operators21:29
*** regXboi has quit IRC21:29
*** vsedelnik has quit IRC21:30
*** simon-AS559 has quit IRC21:31
*** busterswt has quit IRC21:36
*** piet has quit IRC21:40
*** kstev has quit IRC21:42
*** VW has quit IRC21:51
*** admin0 has quit IRC21:54
*** mriedem has quit IRC22:05
*** rarcea has quit IRC22:06
*** mdorman has quit IRC22:14
*** piet has joined #openstack-operators22:19
*** ducttape_ has quit IRC22:27
*** exedore6 has joined #openstack-operators22:27
*** piet has quit IRC22:32
*** piet has joined #openstack-operators22:33
*** exedore6 has quit IRC22:42
*** piet has quit IRC22:45
*** piet has joined #openstack-operators22:45
*** beddari has quit IRC22:49
*** exedore6 has joined #openstack-operators22:55
*** piet has quit IRC22:59
*** beddari has joined #openstack-operators23:02
*** exedore6 has quit IRC23:05
*** saneax_AFK is now known as saneax23:17
*** mriedem has joined #openstack-operators23:36

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!