Saturday, 2013-11-23

« Friday, 2013-11-22 Index Sunday, 2013-11-24 »
SpamapShttp://docs.openstack.org/network-admin/admin/content/app_demo_routers_with_private_networks.html hmm that looks more like what I recall actually00:02
*** edmund has quit IRC00:07
*** cd-undercloud has joined #tripleo00:12
cd-undercloud************** overcloud complete status=1 ************00:12
*** cd-undercloud has quit IRC00:12
SpamapSok, tripleo-cd stopped00:13
SpamapShm, machine times are 10 minutes apart, that is annoying00:17
SpamapSwasn't somebody going to do ntp? :-P00:17
SpamapS2013-11-23 00:11:34,112.112 2779 TRACE nova.network.api [instance: 7e7bebf9-f1e5-40ed-9c0a-131c803eff5d] RuntimeError: maximum recursion depth exceeded while calling a Python object00:19
SpamapSthat can't be good....00:19
*** ccrouch has quit IRC00:24
*** vipul is now known as vipul-away00:25
*** vipul-away is now known as vipul00:25
SpamapSaha, another race00:40
SpamapSdoing the same nova boot again worked fine00:41
SpamapSso we have to wait for something else before trying to spin up instances00:41
SpamapShmm00:42
SpamapSthe tenant can't see et-net00:42
SpamapSext-net00:43
*** boris-42 has joined #tripleo01:02
*** boris-42 has quit IRC01:12
*** cwolferh has quit IRC01:33
SpamapSok I think this is just the async nature of neutron, but the error returned by nova-compute is crap01:38
SpamapSlooks like we might be running into https://bugs.launchpad.net/neutron/+bug/124425501:56
uvirtbotLaunchpad bug 1244255 in neutron "binding_failed because of l2 agent assumed down" [High,Confirmed]01:56
*** ccrouch has joined #tripleo02:11
*** sdague has joined #tripleo02:12
*** ccrouch has quit IRC02:16
*** michchap has quit IRC02:37
*** michchap has joined #tripleo02:39
*** toci-bot has joined #tripleo03:08
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_CDwkhdt/03:08
*** toci-bot has quit IRC03:08
*** sdake_ has joined #tripleo03:29
*** rongze has joined #tripleo03:50
*** blamar has quit IRC04:35
lifelessSpamapS: interesting - in a 'may you live in interesting times' wy04:55
lifelessway04:55
*** rongze has quit IRC05:15
*** rongze has joined #tripleo05:16
*** CaptTofu has quit IRC05:24
*** CaptTofu has joined #tripleo05:25
SpamapSlifeless: yeah exactly05:31
SpamapSlifeless: reading through neutron's code it is just crawling with races. :-P05:31
SpamapSlifeless: I restarted tripleo-cd. I have added a wait loop that will delay things while the L2 agents come up.05:41
*** rongze has quit IRC05:54
*** rongze has joined #tripleo06:06
*** tzumainn has quit IRC06:08
*** akuznetsov has joined #tripleo06:13
*** cd-undercloud has joined #tripleo06:27
cd-undercloud************** overcloud complete status=1 ************06:27
*** cd-undercloud has quit IRC06:27
*** dmojoryder has joined #tripleo06:33
mordredSpamapS: consider yourself free to fix all the races06:45
SpamapSmordred: also the error reporting is... confusing07:06
SpamapSits like.. log everything.. but nothing.07:06
SpamapSargh.. I hate waiting.... goooooo fasttturrrr07:15
*** vkozhukalov has joined #tripleo07:17
*** pleia2 has quit IRC07:37
*** pleia2 has joined #tripleo07:38
lifelessSpamapS:  This use case is an extension of the above Flat Network use cases. In addition to being able to see one or more shared networks via the OpenStack Networking API, tenants can also have access to private per-tenant networks (only visible to tenant users).07:43
lifelessSpamapS: bah07:43
lifelesshttp://docs.openstack.org/network-admin/admin/content/use_cases_mixed.html07:43
lifelessSpamapS: I think thats us07:43
lifelessSpamapS: shared net07:43
lifelessmm, no07:43
lifelessdammit07:44
lifelessSpamapS: you were right, these docs are super confusing07:45
SpamapSlifeless: if the agent-list spin wait works I think I may see if we can do that with waitconditions..07:47
SpamapSlifeless: I suspect it is possible we are just barely beating the l2 agent on notcompute to active status... we may just need  pre-configure.d to check more carefully07:48
SpamapSthough I'd think booting an instance would only need l2 agent on compute to attach a vif07:48
*** akuznetsov has quit IRC07:48
SpamapS2013-11-23 03:21:14.504 4640 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-eafaa671-ca12-4000-9b91-d91c96353537 None None] Failed to set-up gre tunnel port to 10.10.16.17107:49
SpamapSok.. so like.. now what? :-P07:49
SpamapSoh.. I'm jumping the gun07:49
SpamapSstill syncing dbs07:49
*** cd-undercloud has joined #tripleo07:51
cd-undercloud************** overcloud complete status=1 ************07:51
*** cd-undercloud has quit IRC07:51
SpamapSinteresting07:54
*** akuznetsov has joined #tripleo07:54
SpamapSok so trying to do a 'neutron agent-list' right before 'nova boot' results in an auth policy failure...07:55
SpamapSbut 10 seconds later it works07:55
SpamapSand then further.. compute's l2 agent seems to just sit there and wait a long time before trying to reestablish the GRE tunnel07:56
* SpamapS doesn't need to talk to teddy.. just needs to blather on in IRC07:56
lifeless:>07:57
SpamapS| 3fb9f6cf-b545-4a34-a490-dda834973d1e | Open vSwitch agent | overcloud-novacompute0-ubrjpv4jz64a | xxx   | True           |07:58
lifelessjust think of the lovely bug report you'll be able to give our Neutron friends07:58
SpamapSI wonder if it doesn't try again to establish the tunnel until we try to attach something07:59
SpamapSlifeless: I've got 2 already ready to report in my notes.. looks like a third now.07:59
SpamapSmake that 4... neutron agent-update has ridiculous help08:02
SpamapShttps://bugs.launchpad.net/python-neutronclient/+bug/125424108:04
uvirtbotLaunchpad bug 1254241 in python-neutronclient "agent-update online help tells nothing about what the command actually does" [Undecided,New]08:04
*** rongze_ has joined #tripleo08:04
*** rongze has quit IRC08:08
*** rongze has joined #tripleo08:09
*** rongze_ has quit IRC08:13
SpamapS2013-11-23T03:51:02Z|00184|bridge|WARN|could not configure network device gre-10.10.16.174 (Invalid argument)08:15
SpamapShrm08:15
*** toci-bot has joined #tripleo08:16
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_OVwEdNf/08:16
*** toci-bot has quit IRC08:16
*** ccrouch has joined #tripleo08:16
*** ccrouch has joined #tripleo08:17
SpamapSauto br-tun08:19
SpamapSiface br-tun inet dhcp08:19
SpamapSthaaaat doesn't seem right..08:20
SpamapScat /sys/class/net/br-tun/carrier08:20
SpamapS108:20
*** ccrouch has quit IRC08:21
*** rongze has quit IRC08:30
*** akuznetsov has quit IRC08:41
SpamapSahhhhhh09:10
SpamapSduplicate agents09:10
SpamapSanother bug09:10
SpamapScrikey09:10
sdake_hey spamaps09:26
sdake_up late09:26
SpamapSsdake_: yeah09:26
SpamapSsdake_: too much tea :-P09:26
sdake_me too09:26
sdake_kid has chess tourney so i'm in a hotel about 1.5 hr drive from my house09:26
SpamapStrying to figure out why I got two l2 agent records for the same host :-P09:27
SpamapSsdake_: hotels are where I usually sleep the best actually09:27
sdake_neutron - nobody knows how it works ;)09:27
SpamapSsdake_: yeah, I think it is time for a bit of a come-to-jesus moment for Neutron. We all have to get out and push or its never going to get up the hill. :-P09:28
sdake_watching some wacky channel called "RT"09:29
sdake_its like news on steroids09:29
sdake_would be nice to have on cox cable09:29
SpamapS"TODAY ALL THE PEOPLE SUCKED!"09:29
SpamapSoh not like, roid rage..09:29
sdake_nah just pretty edgy09:29
sdake_for example they just compared the amsterdam policy of giving 5 beers and tobacco to alcoholics to clean up the parks to QE-infinity09:30
sdake_its kind of like a legit onion09:31
sdake_hard to explain09:31
SpamapSnice09:31
sdake_just found out my power supply for my kncminer is insufficient because they just cranked up amp usage on the hardware09:31
sdake_so now have to ship this power supply back groan09:32
SpamapSI hate returning stuff09:33
sdake_"The conclusion from the 2008 crash was we need better mathmatical models to convince people they should go deeper into debt"09:33
sdake_lol09:33
SpamapSI hate buying stuff too, but then its like "Gah I hated buying you now I have to find your box and send you back!" .. double hate09:33
sdake_ya shipping back sucks09:34
sdake_200 bucks though09:34
sdake_almost not worth the trouble09:34
SpamapShttps://bugs.launchpad.net/tripleo/+bug/125424609:39
uvirtbotLaunchpad bug 1254246 in tripleo "somehow getting duplicate openvswitch agents for the same host" [Critical,In progress]09:39
SpamapSlifeless: ^^ not a mountain of analysis, but that _does_ seem to be the thing breaking nova booting and the subsequent commands after.09:40
sdake_well i guess im going to ptfo09:40
sdake_night :)09:40
SpamapSsdake_: sleep well!09:40
SpamapSI should crash soon too09:40
openstackgerritClint Byrum proposed a change to openstack/tripleo-incubator: Make ext-net shared so users can use floatingips  https://review.openstack.org/5809009:52
*** toci-bot has joined #tripleo10:06
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_BW3Jtfa/10:06
*** toci-bot has quit IRC10:06
*** cd-undercloud has joined #tripleo10:34
cd-undercloud************** overcloud complete status=1 ************10:34
*** cd-undercloud has quit IRC10:34
*** cd-undercloud has joined #tripleo11:20
cd-undercloud************** overcloud complete status=1 ************11:20
*** cd-undercloud has quit IRC11:20
*** akuznetsov has joined #tripleo11:47
*** akuznetsov has quit IRC11:55
*** boris-42 has joined #tripleo12:04
*** cd-undercloud has joined #tripleo12:05
cd-undercloud************** overcloud complete status=1 ************12:05
*** cd-undercloud has quit IRC12:05
*** akuznetsov has joined #tripleo12:08
*** boris-42 has quit IRC12:12
*** akuznetsov has quit IRC12:13
*** cd-undercloud has joined #tripleo13:07
cd-undercloud************** overcloud complete status=1 ************13:07
*** cd-undercloud has quit IRC13:07
*** rongze has joined #tripleo13:14
*** rongze_ has joined #tripleo13:49
*** cd-undercloud has joined #tripleo13:50
cd-undercloud************** overcloud complete status=1 ************13:50
*** cd-undercloud has quit IRC13:50
*** rongze has quit IRC13:51
*** toci-bot has joined #tripleo14:08
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_IiaiJjD/14:08
*** toci-bot has quit IRC14:08
*** rongze_ has quit IRC14:15
*** rongze has joined #tripleo14:17
*** rongze_ has joined #tripleo14:26
*** rongze has quit IRC14:30
*** rongze has joined #tripleo14:32
*** rongze_ has quit IRC14:33
*** rongze_ has joined #tripleo14:34
*** cd-undercloud has joined #tripleo14:36
cd-undercloud************** overcloud complete status=0 ************14:36
*** cd-undercloud has quit IRC14:36
*** rongze has quit IRC14:37
*** rongze_ has quit IRC14:42
*** rongze has joined #tripleo14:43
*** rongze_ has joined #tripleo14:44
*** rongze has quit IRC14:47
*** CaptTofu has quit IRC14:49
*** CaptTofu has joined #tripleo14:50
*** victor_lowther has joined #tripleo14:55
*** ccrouch has joined #tripleo14:57
*** rongze_ has quit IRC15:05
*** rongze has joined #tripleo15:07
*** cd-undercloud has joined #tripleo15:28
cd-undercloud************** overcloud complete status=1 ************15:28
*** cd-undercloud has quit IRC15:28
*** rongze_ has joined #tripleo15:49
*** rongze has quit IRC15:53
*** toci-bot has joined #tripleo16:04
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_Spqe83k/16:04
*** toci-bot has quit IRC16:04
*** vkozhukalov has quit IRC16:06
*** edmund has joined #tripleo16:15
*** cd-undercloud has joined #tripleo16:16
cd-undercloud************** overcloud complete status=1 ************16:16
*** cd-undercloud has quit IRC16:16
*** ccrouch has quit IRC16:22
*** rongze_ has quit IRC16:31
*** boris-42 has joined #tripleo16:51
*** vkozhukalov has joined #tripleo16:51
*** akuznetsov has joined #tripleo16:55
*** cd-undercloud has joined #tripleo17:00
cd-undercloud************** overcloud complete status=1 ************17:00
*** cd-undercloud has quit IRC17:00
*** akuznetsov has quit IRC17:14
*** MarkAtwood has joined #tripleo17:16
*** akuznetsov has joined #tripleo17:22
*** rongze has joined #tripleo17:32
*** rongze has quit IRC17:40
*** cd-undercloud has joined #tripleo18:01
cd-undercloud************** overcloud complete status=1 ************18:01
*** cd-undercloud has quit IRC18:01
lifelessSpamapS: hmm, host name change could cause it?18:02
lifelessah no, same hostname18:03
*** jergerber has quit IRC18:18
*** CaptTofu has quit IRC18:28
*** CaptTofu has joined #tripleo18:29
*** rongze has joined #tripleo18:36
*** rongze has quit IRC18:44
*** cd-undercloud has joined #tripleo18:48
cd-undercloud************** overcloud complete status=1 ************18:48
*** cd-undercloud has quit IRC18:48
lifelessSpamapS: I'm wondering if backing out the ml2 change would 'fix' things19:12
lifelessSpamapS: but19:12
lifelessSpamapS: I don't know19:12
SpamapSlifeless: I suspect ml2 as well, reading through the code.. lots of "lets make it easier to write lots of drivers" .. not a lot of "let's make our agents more robust"19:15
*** boris-42 has quit IRC19:24
*** cd-undercloud has joined #tripleo19:35
cd-undercloud************** overcloud complete status=1 ************19:35
*** cd-undercloud has quit IRC19:35
*** rongze has joined #tripleo19:41
*** rongze has quit IRC19:46
*** cd-undercloud has joined #tripleo20:22
cd-undercloud************** overcloud complete status=1 ************20:22
*** cd-undercloud has quit IRC20:22
SpamapSUnable to find network with name 'ext-net'20:26
SpamapSlifeless: https://review.openstack.org/#/c/58090/ <-- that addresses the last two fails20:26
SpamapSlifeless: unless you didn't mean to give tenants access to ext-net20:26
lifelessthats wrong20:27
lifelessfloating ip networks shouldn't be shared20:27
SpamapSthey're not visible to users20:27
SpamapShow else would they be able to use them?20:27
lifelessfloatingip-create20:27
SpamapSon a net they can't see: fails20:28
lifelesssounds like a regression20:28
SpamapSperhaps there is an update to policy we're missing?20:28
lifelessperhaps we updated to policy and that broke it?20:28
lifelessbut you're specifically not meant to have direct access to ext-net.20:30
SpamapSok20:30
lifelessbecause if you do20:30
lifelessyou'll have to manually specify a nic on boot20:30
lifelessand life will suck20:30
* SpamapS has started just reading the entire openstack administrators guide because these things are just too confusing.20:30
lifelesswhich your patch demonstrates perfectly20:31
SpamapSyes indeed it does20:31
SpamapStrying to find where in the documentation the hand-off between administrator and user is for floating ips.20:33
lifelesshttp://docs.openstack.org/network-admin/admin/content/demo_per_tenant_router_network_config.html20:35
lifelessneutron net-create Ext-Net --provider:network_type local --router:external true20:35
lifelessneutron subnet-create Ext-Net 30.0.0.0/24 --disable-dhcp20:35
lifeless...20:35
lifeless$ neutron --os-tenant-name TenantA --os-username UserA --os-password password \20:36
lifeless  --os-auth-url=http://localhost:5000/v2.0 floatingip-create Ext-Net20:36
lifelessThis is what we do AFAICT20:36
lifelesswhat did you see happen differently ?20:36
SpamapSfloatingip-create as the tenant says ext-net doesn't exist20:36
SpamapSI see where if your router has a route to the network, you should be able to allocate a floating ip on it.20:36
SpamapShttp://docs.openstack.org/network-admin/admin/content/l3_workflow.html says as much20:37
SpamapShm20:37
SpamapSrouter doesn't exist on default-net either20:37
lifelessit specifically says ext-net does not exist?20:39
SpamapS12:26 < SpamapS> Unable to find network with name 'ext-net'20:39
SpamapSI was specifically pasting the error that caused status=120:40
SpamapSok default-router does have ports on default-net and ext-net20:40
lifelessinteresting20:40
lifelessok, I need to bounce - gl, but the /commands we run/ are correct: we haven't changed them AFAIK.20:40
lifelessso it's a server issue20:41
SpamapSagreed20:41
SpamapSI have to run too.. but will backburner it whilst shopping for lumber.20:41
*** rongze has joined #tripleo20:41
SpamapSI've stopped tripleo-cd20:43
SpamapSovercloud is useful for figuring out wtf on this20:44
* SpamapS goes off to do "Saturday"20:44
*** rongze has quit IRC20:46
*** MarkAtwood has quit IRC21:04
*** CaptTofu has quit IRC21:38
*** CaptTofu has joined #tripleo21:38
*** sdake_ has quit IRC21:40
*** sdake_ has joined #tripleo21:40
*** rongze has joined #tripleo21:42
*** rongze has quit IRC21:46
*** vkozhukalov has quit IRC21:47
*** victor_lowther has quit IRC22:08
*** toci-bot has joined #tripleo22:08
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_UBCWFKt/22:08
*** toci-bot has quit IRC22:08
*** csd has joined #tripleo22:15
*** rongze has joined #tripleo22:43
*** rongze has quit IRC22:47
*** CaptTofu has quit IRC22:51
*** CaptTofu has joined #tripleo22:51
*** sdake_ has quit IRC23:12
*** sdake_ has joined #tripleo23:13
*** sdake_ has quit IRC23:13
*** sdake_ has joined #tripleo23:13
*** CaptTofu has quit IRC23:21
*** CaptTofu has joined #tripleo23:21
*** rongze has joined #tripleo23:43
*** rongze has quit IRC23:48
« Friday, 2013-11-22 Index Sunday, 2013-11-24 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!