Monday, 2016-03-21

« Sunday, 2016-03-20 Index Tuesday, 2016-03-22 »
*** yamamoto has joined #openstack-neutron-ovn00:56
*** chenli has joined #openstack-neutron-ovn01:03
*** yamamoto has quit IRC01:31
*** fzdarsky__ has joined #openstack-neutron-ovn02:08
*** fzdarsky_ has quit IRC02:11
*** yamamoto has joined #openstack-neutron-ovn02:32
*** yamamoto has quit IRC02:41
*** jckasper has joined #openstack-neutron-ovn03:18
*** jckasper has quit IRC03:21
*** jckasper has joined #openstack-neutron-ovn03:21
*** yamamoto has joined #openstack-neutron-ovn03:36
*** chenli has quit IRC04:00
openstackgerritZongKai LI proposed openstack/networking-ovn: Update ACL without compare for add/del sg rule  https://review.openstack.org/29360204:01
*** jckasper has quit IRC04:05
*** jckasper has joined #openstack-neutron-ovn04:07
*** jckasper has quit IRC04:10
*** yamamoto has quit IRC04:13
openstackgerritZongKai LI proposed openstack/networking-ovn: use utils.ovn_name to replace string joint  https://review.openstack.org/29507904:14
*** chenli has joined #openstack-neutron-ovn04:30
*** yamamoto has joined #openstack-neutron-ovn04:34
*** yamamoto has quit IRC04:37
*** yamamoto has joined #openstack-neutron-ovn04:40
*** yamamoto has quit IRC04:42
openstackgerritZongKai LI proposed openstack/networking-ovn: use utils.ovn_name to replace string joint  https://review.openstack.org/29507905:00
*** gangil has quit IRC05:07
*** xiaohhui has quit IRC05:13
*** xiaohhui has joined #openstack-neutron-ovn05:14
*** yamamoto has joined #openstack-neutron-ovn05:29
*** yamamoto has quit IRC05:52
*** yamamoto has joined #openstack-neutron-ovn05:59
*** pcaruana has quit IRC06:02
*** gangil has joined #openstack-neutron-ovn06:03
*** gangil has joined #openstack-neutron-ovn06:03
*** jckasper has joined #openstack-neutron-ovn06:11
*** jckasper has quit IRC06:15
*** openstackgerrit has quit IRC06:17
*** openstackgerrit_ is now known as openstackgerrit06:17
*** openstackgerrit_ has joined #openstack-neutron-ovn06:18
*** openstackgerrit_ is now known as openstackgerrit06:19
*** openstackgerrit_ has joined #openstack-neutron-ovn06:19
*** yamamoto has quit IRC06:22
*** yamamoto has joined #openstack-neutron-ovn06:41
*** chenli has quit IRC07:19
*** yamamoto has quit IRC08:01
*** yamamoto has joined #openstack-neutron-ovn08:02
*** yamamoto has quit IRC08:17
*** yamamoto has joined #openstack-neutron-ovn09:18
*** numans has joined #openstack-neutron-ovn09:23
*** yamamoto has quit IRC09:25
*** gangil has quit IRC09:56
*** Mic22 has joined #openstack-neutron-ovn10:03
*** jckasper has joined #openstack-neutron-ovn10:13
*** jckasper has quit IRC10:17
*** _Mic22 has joined #openstack-neutron-ovn10:52
*** Mic22 has quit IRC10:54
*** yamamoto has joined #openstack-neutron-ovn11:01
*** yamamoto has quit IRC11:04
*** yamamoto has joined #openstack-neutron-ovn11:10
*** _Mic22 has quit IRC11:14
*** numans_ has joined #openstack-neutron-ovn11:21
*** numans has quit IRC11:25
*** numans has joined #openstack-neutron-ovn11:37
*** yamamoto has quit IRC11:39
*** yamamoto has joined #openstack-neutron-ovn11:56
*** yamamoto has quit IRC12:02
*** asti has joined #openstack-neutron-ovn12:16
*** fzdarsky__ is now known as fzdarsky12:30
*** rtheis has joined #openstack-neutron-ovn12:42
*** jckasper has joined #openstack-neutron-ovn12:43
*** yamamoto has joined #openstack-neutron-ovn13:18
*** a_ta has joined #openstack-neutron-ovn13:58
*** brad_behle has joined #openstack-neutron-ovn14:02
*** pcaruana has joined #openstack-neutron-ovn14:05
*** yamamoto has quit IRC14:23
*** yamamoto has joined #openstack-neutron-ovn14:35
*** asti has quit IRC14:38
*** numans has quit IRC14:41
*** yamamoto has quit IRC14:48
*** asti has joined #openstack-neutron-ovn15:05
*** yamamoto has joined #openstack-neutron-ovn15:08
*** yamamoto has quit IRC15:09
*** yamamoto has joined #openstack-neutron-ovn15:10
*** yamamoto has quit IRC15:10
Sam-I-Amrussellb: moo?15:20
russellbSam-I-Am: howdy15:20
Sam-I-Amhey, got some time?15:20
Sam-I-Amhaving an ovn/ovs problem i cant figure out15:20
russellbi've got all the time that exists15:20
Sam-I-Amin this one deployment env, vms can't see past their local interfaces.15:21
Sam-I-Ami see traffic on the tap, but nothing seems to traverse br-int15:21
Sam-I-Amvms can't get dhcp. however, i can ping the dhcp server from anywhere that isn't a vm.15:21
russellbany errors in logs?  ovn-controller or ovs-vswitchd in particular15:22
Sam-I-Ami can only reproduce this issue in one particular environment15:22
russellband kernel log15:22
Sam-I-Ami see this in ovs log - ARN|system@ovs-system: failed to put[create] (Invalid argument) ufid:29b7a399-dd46-4fb9-9ffb-0864a327535a skb_priority(0/0),in_port(6),skb_mark(0/0),eth(src=fa:16:3e:51:e9:fc,dst=33:33:ff:51:e9:fc/00:00:00:00:00:00),eth_type(0x86dd),ipv6(src=::/::,dst=ff02::1:ff51:e9fc/::,label=0/0,proto=58,tclass=0/0,hlimit=255/0,frag=no),icmpv6(type=135,code=0),nd(target=fe80::f816:3eff:fe51:e9fc/::,sll=00:00:00:00:0015:22
russellbthat's a bad sign15:23
Sam-I-Amand these - sending OFPBMC_BAD_MASK error reply to OFPT_FLOW_MOD message15:23
Sam-I-Amlots of them15:23
russellbok15:23
russellbcan you pastebin the flow table?  ovs-ofctl -O OpenFlow13 dump-flows br-int15:24
Sam-I-Amone sec15:24
russellbfrom one node where it's puking like this15:24
Sam-I-Amhttp://paste.openstack.org/show/491295/15:25
Sam-I-Amthere's only one node, one vm, one dhcp agent15:25
Sam-I-Amvm should be on 172.16.1.15, dhcp server .1115:26
russellbi suspect you're not using a kernel module that includes conntrack support15:26
russellband so the flows are incompelte15:26
russellband packets just get dropped15:26
Sam-I-Amovn controller seems happy15:27
Sam-I-Amhuh...15:27
Sam-I-Amovs conntrack?15:27
russellbyes15:27
russellbwhen ovs-vswitchd starts, it emits some messages about features the datapath supports15:27
russellbit should say something about ct not being available15:28
russellbi don't remember the message15:28
russellbmajor hint is that there are no instances of the ct() action in the flow table15:28
russellband the BAD_MASK error you get is what happens if we program ct() flows but vswitchd doesn't support it15:28
russellbnot the friendlest failure mode here ...15:29
Sam-I-Amyep15:29
Sam-I-Amlooks like its missing conntrack support15:29
russellbyay for finding root cause at least15:30
Sam-I-Amyes, this is awesome15:30
Sam-I-Ami owe you a beer15:30
russellb:)15:30
Sam-I-Ammostly for reducing my health care bills15:30
Sam-I-Amhere's whats weird15:30
russellbheh15:30
Sam-I-Amwe're using the same custom built ubuntu packages everywhere15:30
Sam-I-Amon hardware and in vagrant it works fine, yet in CI it does not.15:31
Sam-I-Amthats what has been so bizarre is not being able to repro it anywhere else15:31
russellbdmesg | grep openvswitch15:31
russellbwhat do you get15:31
Sam-I-Am[    7.525793] openvswitch: Open vSwitch switching datapath15:31
Sam-I-Am[    7.875244] openvswitch: netlink: Flow get message rejected, Key attribute missing.15:31
Sam-I-Amthats it15:31
russellbyou're using the openvswitch module from your kernel, not the one built from the ovs tree15:32
russellbis that intentional?15:32
Sam-I-Amno, its not15:32
russellbok, that's the problem15:32
russellbwhen you use the one built from the ovs tree, the message will include the ovs version number15:32
russellb[ 1346.955785] openvswitch: Open vSwitch switching datapath 2.5.9015:32
russellbmine ^^^15:33
Sam-I-Amis that anytime ovs starts, or just during boot?15:34
Sam-I-Amtrying to compare with systems here that work15:34
russellbwhen the module loads15:34
Sam-I-Amhmmm15:36
Sam-I-Ami wonder why its not loading the right module in this particular place15:36
russellbloading it before you install the custom one?15:37
russellbi dunno15:37
*** yamamoto has joined #openstack-neutron-ovn15:37
Sam-I-Ami wonder if this cloud server is doing something weird with kernels15:38
Sam-I-Amtrying to remember how all of this stuff works15:39
Sam-I-Amits been a long time15:39
russellbi'm grabbing lunch soon, hopefully that at least gets you in the right direction15:40
*** yamamoto has quit IRC15:41
Sam-I-Amrussellb: ahhh, yeah... looks like there's no kernel headers. first thing to fix.15:42
*** yamamoto has joined #openstack-neutron-ovn15:42
Sam-I-Amrussellb: i'll stalk you later :)15:42
*** gongysh_ has joined #openstack-neutron-ovn15:44
Sam-I-Amrussellb: looks like compiler errors15:51
Sam-I-Amso its trying to build and failing15:51
*** gongysh_ has quit IRC15:55
Sam-I-Amrussellb: http://paste.openstack.org/show/491305/15:55
*** yamamoto has quit IRC15:56
*** yamamoto has joined #openstack-neutron-ovn16:00
*** yamamoto has quit IRC16:08
russellbSam-I-Am: that's the same failure we're getting in networking-ovn CI jobs right now16:16
russellbSam-I-Am: ubuntu backported some stuff that broke ovs16:16
*** banix has joined #openstack-neutron-ovn16:16
*** armax has joined #openstack-neutron-ovn16:16
Sam-I-Amrussellb: awesome16:18
russellbyou can downgrade the kernel, or wait until it gets sorted out ...16:18
Sam-I-Amthis box originally started out with -7416:19
russellbhttp://openvswitch.org/pipermail/discuss/2016-March/020513.html16:19
Sam-I-Am(which is pretty old.. january?) but ovs still didnt work. not sure if its the compiler thing or something else.16:19
Sam-I-Amhmm16:21
Sam-I-Amso this is probably a red herring16:21
Sam-I-Amtime to revert kernel and see what happens16:21
*** yamamoto has joined #openstack-neutron-ovn16:24
*** palexster has joined #openstack-neutron-ovn16:24
*** pcaruana has quit IRC16:25
*** brad_behle has quit IRC16:25
*** yamamoto has quit IRC16:28
*** brad_behle has joined #openstack-neutron-ovn16:29
*** aginwala has joined #openstack-neutron-ovn16:32
*** Mic22 has joined #openstack-neutron-ovn16:41
Sam-I-Amrussellb: seems like its this package in particular - linux-libc-dev:amd64 ?16:43
russellbhrm?16:43
Sam-I-Amthe thing that broke ovs builds on ubuntu16:43
russellbno, should be the kernel16:43
Sam-I-Aminteresting. these other boxen have < .83 kernel and still dont build16:44
Sam-I-Ambut they do have .83 of that package16:44
russellbrebooted to run the older kernel?16:44
Sam-I-Amyeah16:45
Sam-I-Aminterestingly, most of these failures were on kernels prior to .8316:45
Sam-I-Amthe only diff between working and not-working seems to be that package16:45
russellbthis is kernel code though, it doesn't use glibc16:46
russellbwait, linux-libc16:46
russellbyes16:47
russellbyou're right16:47
russellbthat's kernel headers16:47
Sam-I-Amwell, now one of my labs has that package16:47
russellbum16:47
russellbit should actually be linux-headers16:48
Sam-I-Ambut is running kernel 3.13.0-7716:48
russellb(ノಠ益ಠ)ノ彡┻━┻16:48
Sam-I-AmTitle: openvswitch-datapath-dkms 2.5.90-2.11810+gitd1174d01: openvswitch kernel module failed to build16:48
russellbhonesly i'm hoping that one of the ovs kernel devs takes notice and fixes16:48
Sam-I-Amwell, there we go16:48
Sam-I-Amsooooo16:48
Sam-I-Amthis box has kernel -7716:49
russellbmaybe you should use centos16:49
russellb:-p16:49
Sam-I-Aminterestingly, the only reason this works is because the module built once already16:49
Sam-I-Amso ovs is just re-using the existing conntrack module16:49
* russellb nods16:49
*** aginwala has quit IRC16:49
Sam-I-Amthat explains the problem in CI16:50
Sam-I-Amand i bet vagrant works because they havent pushed a new image, or i havent updated the image.16:50
russellbsounds right16:50
*** aginwala has joined #openstack-neutron-ovn16:50
Sam-I-Amso yeah, this is a pretty serious problem16:50
*** aginwala has quit IRC16:54
*** aginwala has joined #openstack-neutron-ovn16:55
*** gangil has joined #openstack-neutron-ovn16:56
*** gangil has joined #openstack-neutron-ovn16:56
*** aginwala has quit IRC17:00
*** aginwala has joined #openstack-neutron-ovn17:03
*** aginwala has quit IRC17:03
*** aginwala has joined #openstack-neutron-ovn17:04
*** lathiat has quit IRC17:18
*** lathiat has joined #openstack-neutron-ovn17:20
*** a_ta has quit IRC17:24
*** a_ta has joined #openstack-neutron-ovn17:24
*** a_ta has quit IRC17:28
*** yamamoto has joined #openstack-neutron-ovn17:29
*** aginwala has quit IRC17:29
*** aginwala has joined #openstack-neutron-ovn17:32
*** yamamoto has quit IRC17:35
*** yamamoto has joined #openstack-neutron-ovn17:36
*** yamamoto has quit IRC17:37
*** aginwala has quit IRC17:38
*** aginwala has joined #openstack-neutron-ovn17:38
*** aginwala has quit IRC17:39
*** aginwala has joined #openstack-neutron-ovn17:40
*** s3wong has joined #openstack-neutron-ovn17:57
*** yamamoto has joined #openstack-neutron-ovn17:59
*** salv-orlando has joined #openstack-neutron-ovn18:02
*** aginwala has quit IRC18:10
*** a_ta has joined #openstack-neutron-ovn18:10
*** yamamoto has quit IRC18:13
*** aginwala has joined #openstack-neutron-ovn18:17
Sam-I-Amrussellb: upgrading to 4.4 fixed it18:26
russellbcool, yep, that'll do it18:26
Sam-I-Amso happens we'll need 4.4 to make the nats anyway, so might as well get this going now18:28
russellbfor the moment anyway18:30
russellbnat support will get backported to the ovs in-tree module as well18:30
*** asti has quit IRC18:31
*** asti has joined #openstack-neutron-ovn18:32
*** aginwala has quit IRC18:34
*** azbiswas has joined #openstack-neutron-ovn18:48
*** aginwala has joined #openstack-neutron-ovn18:49
*** aginwala has quit IRC18:50
*** aginwala has joined #openstack-neutron-ovn18:50
*** aginwala has quit IRC18:56
*** a_ta has quit IRC19:12
*** a_ta has joined #openstack-neutron-ovn19:12
*** yamamoto has joined #openstack-neutron-ovn19:14
*** a_ta has quit IRC19:17
*** a_ta has joined #openstack-neutron-ovn19:18
*** yamamoto has quit IRC19:21
*** aginwala has joined #openstack-neutron-ovn19:29
russellbSam-I-Am: build fix -> https://patchwork.ozlabs.org/patch/600294/19:48
Sam-I-Amrussellb: cool19:52
*** fzdarsky is now known as fzdarsky|afk19:53
*** tongli has joined #openstack-neutron-ovn20:18
*** fzdarsky|afk has quit IRC20:19
*** tongli has quit IRC20:20
*** pcaruana has joined #openstack-neutron-ovn20:21
*** aginwala has quit IRC20:29
*** aginwala has joined #openstack-neutron-ovn20:33
*** regXboi has joined #openstack-neutron-ovn20:40
*** jckasper has quit IRC20:44
*** jckasper has joined #openstack-neutron-ovn21:03
*** jckasper has quit IRC21:07
*** jckasper has joined #openstack-neutron-ovn21:08
*** aginwala has quit IRC21:09
*** a_ta has quit IRC21:15
*** a_ta has joined #openstack-neutron-ovn21:16
*** a_ta has quit IRC21:20
*** regXboi has quit IRC21:20
*** aginwala has joined #openstack-neutron-ovn21:25
*** jckasper has quit IRC21:25
*** jckasper has joined #openstack-neutron-ovn21:26
*** banix has quit IRC21:27
*** jckasper has quit IRC21:29
*** jckasper has joined #openstack-neutron-ovn21:29
*** asti has quit IRC21:37
*** aginwala has quit IRC21:45
*** aginwala has joined #openstack-neutron-ovn21:52
*** jckasper has quit IRC21:52
*** jckasper has joined #openstack-neutron-ovn21:53
*** jckasper has quit IRC21:53
*** jckasper has joined #openstack-neutron-ovn21:53
*** rtheis has quit IRC21:56
*** pcaruana has quit IRC22:06
*** aginwala has quit IRC22:07
*** aginwala has joined #openstack-neutron-ovn22:10
*** aginwala has quit IRC22:20
*** aginwala has joined #openstack-neutron-ovn22:22
openstackgerritChandra Sekhar Vejendla proposed openstack/networking-ovn: WIP: Floating IP support  https://review.openstack.org/29554722:24
*** brad_behle has quit IRC22:26
Sam-I-Ammoo?22:57
*** aginwala has quit IRC22:58
*** openstackgerrit has quit IRC23:03
*** openstackgerrit_ is now known as openstackgerrit23:03
*** openstackgerrit has quit IRC23:03
*** openstackgerrit_ has joined #openstack-neutron-ovn23:04
*** openstackgerrit_ is now known as openstackgerrit23:04
*** openstackgerrit_ has joined #openstack-neutron-ovn23:09
*** gangil has quit IRC23:10
*** gangil has joined #openstack-neutron-ovn23:21
*** gangil has joined #openstack-neutron-ovn23:21
*** aginwala has joined #openstack-neutron-ovn23:46
*** aginwala has quit IRC23:54
russellbSam-I-Am: sup23:57
Sam-I-Amrussellb: yo23:57
Sam-I-Amwhat brings you here this late?23:57
« Sunday, 2016-03-20 Index Tuesday, 2016-03-22 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!