| *** yosh has quit IRC | 00:00 | |
| *** miclorb_ has quit IRC | 00:01 | |
| *** yosh has joined #openstack | 00:01 | |
| *** miclorb has joined #openstack | 00:02 | |
| *** RobertLaptop has joined #openstack | 00:11 | |
| *** adjohn has joined #openstack | 00:37 | |
| *** anticw_ is now known as anticw | 00:41 | |
| *** aimon has joined #openstack | 00:46 | |
| *** aimon_ has joined #openstack | 00:48 | |
| *** aimon has quit IRC | 00:48 | |
| *** aimon_ is now known as aimon | 00:48 | |
| RobertLaptop | Question do Storage nodes needs to be Storage Servers or Just drives on a few servers? I am looking into setting up a small cluster only about 10 to 20 servers that will grow overtime. | 00:57 |
|---|---|---|
| notmyname | RobertLaptop: the storage nodes need to run server processes. you could make storage nodes out of "real" servers or even a controller head unit connected to JBOD devices | 01:01 |
| notmyname | but storage nodes need to run some combination of account, container, and object servers plus their associated background housekeeping processes | 01:02 |
| RobertLaptop | That is good to know. | 01:02 |
| *** aimon has quit IRC | 01:10 | |
| *** aimon has joined #openstack | 01:10 | |
| *** aimon has joined #openstack | 01:12 | |
| *** aimon_ has joined #openstack | 01:13 | |
| *** aimon has quit IRC | 01:13 | |
| *** aimon_ is now known as aimon | 01:13 | |
| *** aimon has quit IRC | 01:15 | |
| *** aimon has joined #openstack | 01:15 | |
| *** miclorb has quit IRC | 01:17 | |
| *** aimon has quit IRC | 01:17 | |
| *** aimon has joined #openstack | 01:17 | |
| *** aimon has joined #openstack | 01:17 | |
| *** aimon_ has joined #openstack | 01:19 | |
| *** aimon_ has joined #openstack | 01:19 | |
| *** aimon has quit IRC | 01:19 | |
| *** aimon_ is now known as aimon | 01:19 | |
| *** miclorb has joined #openstack | 01:24 | |
| *** aimon has quit IRC | 01:26 | |
| *** miclorb has quit IRC | 01:27 | |
| *** sebastianstadil has joined #openstack | 01:48 | |
| *** vvuksan has quit IRC | 01:53 | |
| winston-d | Any chance creiht is here? | 02:11 |
| *** winston-d has left #openstack | 02:12 | |
| *** winston-d has joined #openstack | 02:12 | |
| *** zul has quit IRC | 02:20 | |
| *** pvo has joined #openstack | 02:24 | |
| *** pvo has quit IRC | 02:24 | |
| *** Camerlengo has quit IRC | 02:25 | |
| *** Gh0sty has quit IRC | 02:36 | |
| creiht | Failbaitr: there shouldn't be that much overhead | 02:38 |
| creiht | winston-d: howdy :) | 02:41 |
| winston-d | creiht: hi there! | 02:42 |
| winston-d | creiht: I saw your response about Rackspace cloudfile API | 02:42 |
| creiht | Failbaitr: That could be due to lots of things (network latencies, failed nodes, misconfigured nodes) | 02:42 |
| winston-d | creiht: I'm reading its doc right now. and will try PHP binding \ | 02:42 |
| creiht | Failbaitr: If you look at the logs on the proxy server, and find one of the transactions that takes a while, the log line will have a transaction id that you can grep all the storage node logs with | 02:43 |
| creiht | This will at least tell if you if a failed node, or something like that is causing an issue | 02:43 |
| creiht | winston-d: cool... feel free to let me know if you run into any issues | 02:43 |
| winston-d | creiht: sure. and finally i got my SATA disk. will re-run swiftbench when i have time | 02:44 |
| creiht | cool | 02:45 |
| *** miclorb_ has joined #openstack | 02:55 | |
| openstackhudson | Project nova build #506: SUCCESS in 1 min 32 sec: http://hudson.openstack.org/job/nova/506/ | 03:07 |
| openstackhudson | Tarmac: Fixes describe_security_groups by forcing it to return a list instead of a generator. | 03:07 |
| openf|y | be back later | 03:13 |
| *** openf|y has left #openstack | 03:13 | |
| openstackhudson | Project nova build #507: SUCCESS in 1 min 34 sec: http://hudson.openstack.org/job/nova/507/ | 03:17 |
| openstackhudson | Tarmac: Fixes a typo in the auth checking for DescribeAvailabilityZones. | 03:17 |
| *** littleidea has joined #openstack | 03:30 | |
| *** Ryan_Lane is now known as Ryan_Lane|food | 03:30 | |
| *** pvo has joined #openstack | 03:44 | |
| *** stewart_ is now known as stewart | 03:47 | |
| *** pvo has quit IRC | 03:52 | |
| *** e1mer has joined #openstack | 03:54 | |
| *** e1mer has joined #openstack | 03:54 | |
| *** joearnold has joined #openstack | 04:04 | |
| *** littleidea has quit IRC | 04:27 | |
| *** littleidea has joined #openstack | 04:28 | |
| *** alekibango has joined #openstack | 04:35 | |
| *** Gh0sty has joined #openstack | 04:55 | |
| *** Gh0sty has quit IRC | 05:00 | |
| *** westmaas has quit IRC | 05:18 | |
| *** Gh0sty has joined #openstack | 05:19 | |
| *** dweimer has quit IRC | 05:29 | |
| *** dweimer has joined #openstack | 05:33 | |
| *** f4m8_ is now known as f4m8 | 05:46 | |
| *** miclor___ has joined #openstack | 05:53 | |
| *** miclorb_ has quit IRC | 05:55 | |
| uvirtbot | New bug: #718548 in nova "use_nova_chains fails due to a missing flag" [Undecided,New] https://launchpad.net/bugs/718548 | 05:56 |
| *** arun_ has quit IRC | 05:58 | |
| *** joearnold has quit IRC | 06:00 | |
| *** miclor___ has quit IRC | 06:00 | |
| *** arun_ has joined #openstack | 06:12 | |
| *** arun_ has joined #openstack | 06:12 | |
| *** e1mer has quit IRC | 06:16 | |
| *** kpepple has joined #openstack | 06:42 | |
| *** hadrian has quit IRC | 06:43 | |
| *** kpepple has quit IRC | 06:44 | |
| *** kpepple has joined #openstack | 06:45 | |
| *** guigui has joined #openstack | 06:47 | |
| *** Ryan_Lane|food has quit IRC | 07:22 | |
| *** ramkrsna has joined #openstack | 07:23 | |
| *** Ryan_Lane has joined #openstack | 07:26 | |
| *** reldan has joined #openstack | 07:31 | |
| *** Ryan_Lane has quit IRC | 07:40 | |
| *** RJD22 is now known as RJD22|away | 07:45 | |
| *** fabiand_ has joined #openstack | 07:51 | |
| *** CloudChris has joined #openstack | 07:54 | |
| *** rcc has joined #openstack | 07:55 | |
| *** thimble has joined #openstack | 07:58 | |
| Failbaitr | creiht: ok, thnx, I was hoping there shouldn't be that much overhead | 07:58 |
| soren | vishy: I'm here now. | 08:03 |
| *** DigitalFlux has joined #openstack | 08:04 | |
| *** Nacx has joined #openstack | 08:05 | |
| DigitalFlux | Hi Guys | 08:12 |
| DigitalFlux | If i need to launch servers using the OpenStack API (using the cloudservers command), do i have to upload the image into glance ? | 08:13 |
| DigitalFlux | I can't seem to get Glance up by any means :( | 08:13 |
| *** allsystemsarego has joined #openstack | 08:14 | |
| *** allsystemsarego has joined #openstack | 08:14 | |
| vishy | soren: o hai! | 08:15 |
| vishy | soren: hudson is having intermittent failures which seem to be related to old versions of nova.sqlite sitting around | 08:16 |
| soren | vishy: Sounds odd. run_tests.sh should be removing that. | 08:17 |
| vishy | soren: does the debian build package run tests as well? | 08:17 |
| vishy | without using run_tests.sh? | 08:18 |
| soren | *grumble* | 08:18 |
| soren | It used to. | 08:18 |
| vishy | I saw the same error on my build box | 08:18 |
| soren | It got disabled because one of the tests were failing. | 08:18 |
| vishy | when i did buildpackage (which runs the tests for me) | 08:18 |
| vishy | but removing nova.sqlite fixed it | 08:18 |
| *** drico_ has joined #openstack | 08:18 | |
| *** eikke has joined #openstack | 08:18 | |
| vishy | otherwise we have some weird condition which is causing random failures | 08:19 |
| soren | vishy: Do you have an example of the failure somewhere? | 08:19 |
| vishy | sure: | 08:19 |
| vishy | soren: https://code.launchpad.net/~devcamcar/nova/lp712782/+merge/49531 | 08:20 |
| *** befreax has joined #openstack | 08:20 | |
| soren | vishy: I don't think that's a leftover nova.sqlite. | 08:20 |
| vishy | soren: it seemed to start with the sorting security groups patch | 08:20 |
| soren | vishy: I think it's because test_authorize_revoke_security_group_cidr doesn't clean up after itself. | 08:21 |
| soren | ...if it fails. | 08:21 |
| soren | We have a lot of those :( | 08:21 |
| vishy | so it just randomly fails sometimes? | 08:21 |
| soren | No no, I mean. | 08:21 |
| soren | Er.. | 08:21 |
| soren | Hang on, let me stare at it for little bit. | 08:21 |
| vishy | for history | 08:21 |
| vishy | the patch after the secgroup sorting patch failed | 08:22 |
| vishy | with that error | 08:22 |
| vishy | and when i ran the tests on trunk, they failed as well | 08:22 |
| vishy | so i made a patch that seemed to fix | 08:22 |
| vishy | which was to use list(sorted()) I thought it was failing because sorted returns a generator | 08:23 |
| vishy | which merged fine and i thought the issue was fixed | 08:23 |
| vishy | but after a couple other patches it started happening again | 08:23 |
| soren | It doesn't really make sense to me that a leftover nova.sqlite would cause that first error. | 08:24 |
| vishy | the other failures after the first are definitely cascading errors | 08:24 |
| soren | (RevokeSecurityBlah not being able to find a particular rule) | 08:24 |
| vishy | it is possible that the rule already exists but is deleted=1 | 08:24 |
| vishy | so it is not added again | 08:25 |
| soren | ...and recreating the rule doesn't clear the deleted flag? I guess that's possible. | 08:25 |
| vishy | that is my initial guess | 08:25 |
| vishy | all i know is i tried to build twice | 08:25 |
| vishy | fail, fail | 08:25 |
| vishy | deleted nova.sqlite -> success | 08:25 |
| * soren wishes we could get rid of the deleted flag | 08:25 | |
| vishy | it is possible that it is a weird race condition | 08:26 |
| vishy | and i just got lucky | 08:26 |
| vishy | but... | 08:26 |
| *** eikke has quit IRC | 08:26 | |
| soren | vishy: I can't reproduce it :( | 08:34 |
| soren | vishy: Hah, but I triggered another bug. Awesome. | 08:37 |
| vishy | soren: hehe | 08:37 |
| vishy | soren: I'm sure it will pop up again :) | 08:38 |
| soren | brb | 08:38 |
| *** perra has joined #openstack | 08:39 | |
| *** perra has quit IRC | 08:44 | |
| *** MarcMorata has joined #openstack | 08:48 | |
| soren | back | 08:49 |
| *** reldan has quit IRC | 08:52 | |
| DigitalFlux | Can someone please supply me the DB schema of Glance ? | 08:52 |
| DigitalFlux | I just can't do any actions on it, and i can't seem to find something like nova-manage db sync to create the DB for me .. | 08:52 |
| *** HugoKuo_ has joined #openstack | 09:08 | |
| *** miclorb has joined #openstack | 09:09 | |
| *** HugoKuo has quit IRC | 09:12 | |
| *** berendt has joined #openstack | 09:20 | |
| *** drico_ has quit IRC | 09:27 | |
| *** littleidea has quit IRC | 09:33 | |
| * soren sighs | 09:34 | |
| *** reldan has joined #openstack | 09:34 | |
| uvirtbot | New bug: #718621 in nova "introduce new FLAG for setting default DNSMASQ interface in nova-dhcpbridge" [Undecided,New] https://launchpad.net/bugs/718621 | 09:41 |
| *** adjohn has quit IRC | 09:43 | |
| *** perra has joined #openstack | 09:45 | |
| *** perra has quit IRC | 09:48 | |
| DigitalFlux | soren: So the plan is to migrate all of the API to OpenStack's (Rackspace) and deprecate EC2 API ? | 10:11 |
| DigitalFlux | soren: where all of the VMs will have persistant storage in the future ? | 10:11 |
| *** calavera has joined #openstack | 10:17 | |
| *** miclorb has quit IRC | 10:21 | |
| *** adjohn has joined #openstack | 10:35 | |
| soren | DigitalFlux: The EC2 API isn't going away. | 10:43 |
| soren | DigitalFlux: But at some point, I doubt people are going to generally want to use it. | 10:43 |
| soren | DigitalFlux: We can't extend the EC2 API. It's not ours to extend, so if we add clever new features, we may only be able to add them to the OpenStack API. | 10:44 |
| *** asksol__ has joined #openstack | 10:44 | |
| DigitalFlux | soeren: That's great | 10:55 |
| DigitalFlux | soren: | 10:55 |
| *** berendt has quit IRC | 10:55 | |
| *** thimble has quit IRC | 10:55 | |
| DigitalFlux | soren: The point is, Most of the focus that i saw was biasing towards the EC2 API | 10:55 |
| DigitalFlux | soren: the docs on the wiki .. | 10:55 |
| DigitalFlux | soren: I am actually having problems right now using python-novatools to launch an instance | 10:56 |
| *** CloudChris has quit IRC | 10:56 | |
| *** CloudChris has joined #openstack | 10:57 | |
| ttx | DigitalFlux: I managed to use them, but they had some limits. Like your project must be named "openstack". | 10:58 |
| openstackhudson | Project nova build #508: SUCCESS in 1 min 38 sec: http://hudson.openstack.org/job/nova/508/ | 11:02 |
| openstackhudson | Tarmac: Modified S3ImageService to return the format defined in BaseService to allow EC2 API's DescribeImages to work against Glance. | 11:02 |
| *** rchavik has joined #openstack | 11:05 | |
| ttx | soren: was looking into bug 695866 -- how would you actually do a separate binary package for adminclient.py ? Looks like we'd need to specialcase it in python-nova.install, which looks costly to me | 11:06 |
| uvirtbot | Launchpad bug 695866 in nova "Admin client should be packaged separately" [Medium,Confirmed] https://launchpad.net/bugs/695866 | 11:06 |
| *** zul has joined #openstack | 11:07 | |
| ttx | soren: or should it be separated from nova itself ? | 11:07 |
| DigitalFlux | ttx: Yeah, dealt with that, but still i can't boot an instance, throws a "list index out of range (HTTP 500)" | 11:09 |
| DigitalFlux | ttx: soren: either it's going into nova or packaged standalone, i would like it support more features :-) | 11:10 |
| ttx | DigitalFlux: fwiw I managed to get it to work by following http://wiki.openstack.org/XenServerDevelopment and dropping the xenserver sepcific stuff | 11:10 |
| *** reldan has quit IRC | 11:10 | |
| DigitalFlux | Like replacing the need for nova-manage | 11:10 |
| DigitalFlux | ttx: that's what i'm following :-) | 11:10 |
| * ttx lunches | 11:11 | |
| alekibango | DigitalFlux: some wiki page describing your way with python-novatools would be sweet | 11:12 |
| DigitalFlux | alekibango: Indeed, as soon as i get it up first ;) | 11:19 |
| *** RJD22|away is now known as RJD22 | 11:29 | |
| *** Seoman has joined #openstack | 11:44 | |
| *** MarcMorata has quit IRC | 11:45 | |
| *** MarcMorata has joined #openstack | 11:45 | |
| uvirtbot | New bug: #718675 in nova "Can't create db with PostgreSQL backend" [Undecided,New] https://launchpad.net/bugs/718675 | 11:46 |
| uvirtbot | New bug: #718674 in nova "Can't create db with PostgreSQL backend" [Undecided,New] https://launchpad.net/bugs/718674 | 11:47 |
| *** xEU has joined #openstack | 11:48 | |
| *** reldan has joined #openstack | 12:01 | |
| *** herki has quit IRC | 12:04 | |
| *** herki has joined #openstack | 12:05 | |
| *** adjohn has quit IRC | 12:10 | |
| *** Abd4llA has joined #openstack | 12:10 | |
| *** fabiand_ has quit IRC | 12:13 | |
| *** vdo has joined #openstack | 12:13 | |
| vdo | hi | 12:13 |
| vdo | how does openstack load the wmi module for hyperv? | 12:14 |
| vdo | is not win32 only? | 12:15 |
| soren | It is. | 12:20 |
| soren | So nova-compute runs on Windows in that situation. | 12:20 |
| *** thimble has joined #openstack | 12:20 | |
| vdo | ok thanks... | 12:21 |
| *** vdo has left #openstack | 12:21 | |
| *** h0cin has joined #openstack | 12:22 | |
| *** fabiand_ has joined #openstack | 12:24 | |
| *** CloudChris has quit IRC | 12:27 | |
| *** CloudChris has joined #openstack | 12:28 | |
| *** ramkrsna has quit IRC | 12:32 | |
| *** ramkrsna has joined #openstack | 12:32 | |
| *** reldan has quit IRC | 12:33 | |
| *** berendt has joined #openstack | 12:38 | |
| *** ctennis has quit IRC | 12:42 | |
| *** befreax has quit IRC | 12:46 | |
| *** ctennis has joined #openstack | 12:58 | |
| *** ctennis has joined #openstack | 12:58 | |
| *** adjohn has joined #openstack | 13:00 | |
| *** westmaas has joined #openstack | 13:14 | |
| xEU | hi | 13:15 |
| xEU | i'm trying elasticfox w/ openstack ; nova-api returns http 400 ( credentials seems ok on logfile). Do you know if there's some specific settings on elasticfox to configure ? | 13:17 |
| soren | Which version? | 13:17 |
| soren | Of Openstack. | 13:17 |
| xEU | apt-add-repository ppa:nova-core/release | 13:18 |
| xEU | 2011.1~bzr645-0ubuntu0ppa1~maverick1 | 13:18 |
| soren | Ah, right, I think ElasticFox tries to fiddle with some VPC stuff that we don't have in Nova. | 13:18 |
| soren | DescribeVpcs and DescribeSubnets, specifically. | 13:18 |
| xEU | on tcpdump i'd see http post, whereas euca2ools use http get | 13:19 |
| xEU | don't know if that matters at all | 13:19 |
| soren | That shoulnd't be a problem. | 13:19 |
| soren | Well, it matters, but I believe we handle it correctly. | 13:19 |
| xEU | here 'Images tab' is in error ; DescribeSecurityGroups | 13:20 |
| xEU | well, everything is in error :( | 13:20 |
| RobertLaptop | Question soren I think yesterday you said that nova had a S3 module built in. It that layer documented? I am asking because I am wonder how fault tolerant that layer is? | 13:27 |
| *** hadrian has joined #openstack | 13:30 | |
| ttx | soren: saw my question at 11:06 UTC ? | 13:41 |
| soren | RobertLaptop: Not at all fault tolerant. | 13:47 |
| soren | ttx: Ah, yes. | 13:48 |
| soren | ttx: I was busy fixing Plymouth right then. | 13:48 |
| soren | ttx: I'd just let python-nova pick it up, and then delete it afterwards. | 13:48 |
| ttx | soren: or move it where somewhere nova-adminclient can pick it up. ok | 13:51 |
| soren | ttx: I'd just point nova-adminclient at it, and let both of them pick it up and then yank it out of python-nova afterwards. | 13:51 |
| ttx | soren: ok, I'll try to whip up something. | 13:52 |
| soren | cool. | 13:53 |
| * soren stares at eventlet a bit more. | 13:53 | |
| *** dendro-afk is now known as dendrobates | 13:53 | |
| iammartian | vishy: this is from friday - nova.sh runs fine, i'm just a dummy | 13:53 |
| * iammartian logs in to openstack wiki to clarify some running instructions | 13:54 | |
| *** ppetraki has joined #openstack | 13:57 | |
| iammartian | when running compute from nova.sh, is it normal for nova-volume to fail? I only have local storage configured per nova.sh | 13:58 |
| iammartian | the error is: 2011-02-14 07:51:27,320 CRITICAL nova.root [-] volume group nova-volumes doesn't exist | 13:58 |
| *** dprince has joined #openstack | 13:59 | |
| openstackhudson | Project nova-tarmac build #61,222: FAILURE in 3.7 sec: http://hudson.openstack.org/job/nova-tarmac/61222/ | 14:06 |
| annegentle | heya soren and Ryan_Lane, saw the whine about package installs and doc. :) I was trying to doc installs on production systems i.e. LTS - but let me know how packages work in production and I'll update. | 14:08 |
| openstackhudson | Project dashboard-tarmac build #3,610: FAILURE in 1.7 sec: http://hudson.openstack.org/job/dashboard-tarmac/3610/ | 14:10 |
| openstackhudson | Project nova-tarmac build #61,223: STILL FAILING in 1.7 sec: http://hudson.openstack.org/job/nova-tarmac/61223/ | 14:10 |
| *** Edwin_ has joined #openstack | 14:12 | |
| jaypipes | *yawn* | 14:12 |
| *** EdwinGrubbs has quit IRC | 14:13 | |
| dabo | jaypipes: bored already? :) | 14:14 |
| jaypipes | dabo: hehe, no, just getting ready for the day :) | 14:15 |
| openstackhudson | Project dashboard-tarmac build #3,611: STILL FAILING in 1.6 sec: http://hudson.openstack.org/job/dashboard-tarmac/3611/ | 14:15 |
| openstackhudson | Project nova-tarmac build #61,224: STILL FAILING in 1.5 sec: http://hudson.openstack.org/job/nova-tarmac/61224/ | 14:15 |
| *** Edwin_ has quit IRC | 14:20 | |
| openstackhudson | Project dashboard-tarmac build #3,612: STILL FAILING in 1.6 sec: http://hudson.openstack.org/job/dashboard-tarmac/3612/ | 14:20 |
| openstackhudson | Project nova-tarmac build #61,225: STILL FAILING in 1.4 sec: http://hudson.openstack.org/job/nova-tarmac/61225/ | 14:20 |
| jaypipes | soren: what's Plymouth? | 14:25 |
| soren | jaypipes: That thing that makes your boot look pretty on Ubuntu. | 14:25 |
| openstackhudson | Project dashboard-tarmac build #3,613: STILL FAILING in 1.6 sec: http://hudson.openstack.org/job/dashboard-tarmac/3613/ | 14:25 |
| openstackhudson | Project nova-tarmac build #61,226: STILL FAILING in 1.6 sec: http://hudson.openstack.org/job/nova-tarmac/61226/ | 14:25 |
| jaypipes | soren: I see. interesting... | 14:26 |
| jaypipes | soren: btw, when I first read that, I saw "makes your booty look pretty on Ubuntu", which was somewhat disturbing until I re-read it. | 14:27 |
| soren | jaypipes: I was working on fixing a bug in the cloud images this weekend, and I introduced a regression. | 14:27 |
| soren | That's what I meant to say :) | 14:27 |
| jaypipes | :) | 14:27 |
| soren | I'm on my way out the door. | 14:28 |
| soren | Should I disable those tarmac jobs for onw? | 14:28 |
| soren | now? | 14:28 |
| soren | There seems to be some problem and I don't have the time to work out what it is. | 14:28 |
| * soren does so | 14:28 | |
| soren | Anyone with Hudson privs can turn them back on. | 14:29 |
| soren | If it's urgent, give me a call, I can do it remotely. | 14:29 |
| *** adjohn has quit IRC | 14:29 | |
| *** rchavik has quit IRC | 14:31 | |
| jaypipes | soren: k, ttyl | 14:31 |
| *** gondoi has joined #openstack | 14:32 | |
| ttx | jaypipes/mtaylor: are you making progress over the bexar translations issue ? I'm trying to set an ETA for 2011.1.1 | 14:36 |
| *** berendt has quit IRC | 14:37 | |
| jaypipes | ttx: will be fixed today. | 14:37 |
| jaypipes | ttx: Friday was, ehm, pretty distracting for a number of threads, I mean reasons... | 14:38 |
| ttx | jaypipes: right :) | 14:38 |
| *** ramkrsna has quit IRC | 14:46 | |
| *** f4m8 is now known as f4m8_ | 14:46 | |
| *** imsplitbit has quit IRC | 14:49 | |
| *** imsplitbit has joined #openstack | 14:49 | |
| *** xEU has quit IRC | 14:50 | |
| *** iammartian has quit IRC | 14:52 | |
| *** freeflyi1g has joined #openstack | 14:52 | |
| *** freeflying has quit IRC | 14:53 | |
| annegentle | ttx or jaypipes: I'm trying to merge my changes back to trunk of openstack-manuals, but I'm unsure of the right process. Either of you got a minute? | 14:57 |
| jaypipes | annegentle: sure, what's the issue you are running into? | 14:57 |
| *** littleidea has joined #openstack | 15:03 | |
| *** littleidea has quit IRC | 15:04 | |
| *** pvo has joined #openstack | 15:07 | |
| *** pvo has quit IRC | 15:07 | |
| *** pvo has joined #openstack | 15:07 | |
| *** ChanServ sets mode: +v pvo | 15:07 | |
| *** hub_cap has joined #openstack | 15:15 | |
| *** benlake has joined #openstack | 15:19 | |
| *** lvaughn_ has quit IRC | 15:20 | |
| *** lvaughn has joined #openstack | 15:20 | |
| RichiH | are there any reference setups for openstack and/or specific suggestions on how to ensure high i/o capability? | 15:20 |
| annegentle | yay all fixed up, thanks jaypipes | 15:23 |
| jaypipes | annegentle: np | 15:25 |
| *** hub_cap has joined #openstack | 15:25 | |
| *** Ryan_Lane has joined #openstack | 15:28 | |
| *** mustfeed has joined #openstack | 15:28 | |
| mustfeed | hi | 15:28 |
| annegentle | RichiH: not sure about high i/o capability specifically, but Ryan_Lane has a reference setup for Mediawiki - plus Rackspace and NASA. You're thinking of high i/o for storage or computing? | 15:31 |
| mustfeed | can someone help me with the Nova deployment tool (Single Server Installation)? I got an Error when i run the 'python deploy.py all' | 15:32 |
| *** vvuksan has joined #openstack | 15:32 | |
| annegentle | mustfeed: really I'd point you to the nova.sh script at http://wiki.openstack.org/NovaInstall/DevInstallScript | 15:33 |
| Ryan_Lane | I have a setup for jobs | 15:34 |
| Ryan_Lane | Err | 15:34 |
| annegentle | mustfeed: oh sorry you want massive deploy? | 15:34 |
| Ryan_Lane | Stupid phone client | 15:34 |
| Ryan_Lane | I have a setup for nova | 15:34 |
| annegentle | mustfeed: sorry I typed too fast (for once) :) | 15:34 |
| Ryan_Lane | Nelson has a setup for swift | 15:34 |
| RichiH | annegentle: lots of disk i/o | 15:36 |
| RichiH | i.e. database workloads | 15:36 |
| RichiH | which can't fit into memory | 15:36 |
| Ryan_Lane | I don't think swift or nova is appropriate for that | 15:37 |
| wedjat | hi guys, it appears that this link is broken : http://docs.openstack.org/openstack-object-storage/admin/os-objectstore-book.pdf | 15:37 |
| annegentle | wedjat: ok, on it | 15:37 |
| wedjat | it appears on this page : http://openstack.org/projects/storage/ | 15:37 |
| wedjat | annegentle: :) | 15:37 |
| *** mustfeed has quit IRC | 15:40 | |
| *** littleidea has joined #openstack | 15:42 | |
| *** lvaughn_ has joined #openstack | 15:42 | |
| *** mustfeed has joined #openstack | 15:43 | |
| *** DigitalFlux has quit IRC | 15:45 | |
| *** lvaughn has quit IRC | 15:46 | |
| pvo | are tests broken in trunk right now? | 15:50 |
| pvo | or just me? | 15:50 |
| *** calavera has quit IRC | 15:50 | |
| *** abecc has joined #openstack | 15:50 | |
| *** dragondm has joined #openstack | 15:52 | |
| *** deshantm has quit IRC | 15:52 | |
| RobertLaptop | pvo I am pretty certain yes. "soren> There seems to be some problem and I don't have the time to work out what it is." | 15:54 |
| jaypipes | RichiH: I don't think Nova is appropriate for that either... | 15:55 |
| *** rnirmal has joined #openstack | 15:56 | |
| jaypipes | pvo: the test suite needs to clear the nova.sqlite database used during tests before running tests... it's producing false positives in the test runs, AFAICT. | 15:56 |
| *** littleidea has quit IRC | 15:58 | |
| *** Ryan_Lane has quit IRC | 16:01 | |
| *** Ryan_Lane has joined #openstack | 16:03 | |
| *** deshantm has joined #openstack | 16:04 | |
| *** guigui has quit IRC | 16:07 | |
| *** Ryan_Lane has quit IRC | 16:08 | |
| *** asksol has quit IRC | 16:08 | |
| *** decramy_ is now known as decramy | 16:10 | |
| jaypipes | mtaylor: heya. after cherry-picking r656 and r657 from lp:~mordred/nova/use-distutils-extra into a local branch of bexar, I'm able to successfully build the message catalogs: http://paste.openstack.org/show/693/. I know you said there were a couple warnings in the output you were looking into, but does that output look normal to you? Anything I should change? | 16:10 |
| jaypipes | mtaylor: this is in relation to https://bugs.launchpad.net/nova/+bug/714679 of course... | 16:10 |
| uvirtbot | Launchpad bug 714679 in nova "Missing translations in Bexar release tarball" [High,In progress] | 16:11 |
| *** kashyapc has joined #openstack | 16:15 | |
| creiht | RobertLaptop: swift provides scalable object storage outside of nova | 16:22 |
| *** littleidea has joined #openstack | 16:24 | |
| RobertLaptop | Yea. I was debating on if I needed to implement swift or not but I need my data to survive hardware failure. | 16:28 |
| *** Nacx has quit IRC | 16:28 | |
| *** zuez- has joined #openstack | 16:30 | |
| dsockwell | apologies if someone answered when I asked before, but does openbox expose a VNC terminal for guest instances where available? | 16:30 |
| dsockwell | openstack | 16:30 |
| *** kashyapc has quit IRC | 16:31 | |
| dsockwell | i've always grumbled when projects have cutesy names, but then i can't remember the generic ones :( | 16:31 |
| RichiH | jaypipes: what do you plan on selling to customers who need to run a db, then? | 16:35 |
| pvo | dsockwell: depends on the hypervisor | 16:35 |
| dsockwell | let's assume i'm using one that provides a vnc service | 16:36 |
| pvo | dsockwell: there is a ajax terminal feature | 16:36 |
| uvirtbot | New bug: #718847 in nova "change logging message if no compute notes available" [Undecided,New] https://launchpad.net/bugs/718847 | 16:36 |
| dsockwell | ok, will that interface with a virtual framebuffer as well as serial output/ | 16:36 |
| *** blueadept has joined #openstack | 16:36 | |
| RobertLaptop | creiht, Yea. Swift looks to provide a lot more then I need right now. But if what is built into the compute part isn't going to protect my data then I need to implant swift. | 16:37 |
| pvo | dsockwell: I know it does framebuffer | 16:37 |
| pvo | looking for serial | 16:37 |
| dsockwell | all i know about the dashboard is what I saw on vimeo, and that was (apparently) xen pv with serial out | 16:37 |
| *** ccustine has joined #openstack | 16:38 | |
| dsockwell | at any rate, that's exactly what i needed to know, thanks pvo | 16:38 |
| pvo | dsockwell: cool | 16:38 |
| dsockwell | which hypervisor are you using, if I might ask? | 16:39 |
| jaypipes | RichiH: depends on what they need. plus, I don't personally plan on selling anything...I'm a developer. | 16:39 |
| pvo | dsockwell: XenServer 5.5 and moving to 5.6 | 16:39 |
| RichiH | jaypipes: you know what i mean ;) | 16:39 |
| pvo | 5.6fp1 | 16:39 |
| dsockwell | cool. i'm probably going to use it with qemu, i hope the tooling is similar | 16:40 |
| jaypipes | RichiH: nothing *wrong* with running a database server on a Nova guest VM, just think it's better to run database servers on bare metal and have them as shared resources for the VMs. | 16:40 |
| dsockwell | seeing as it's probably the same code i'm optimistic | 16:40 |
| RichiH | possible, but not ideal | 16:41 |
| jaypipes | RichiH: because, clearly, no matter the excellence of the virtualization layer, it's going to consume *some* level of resources from the host machine, and if you (or anyone) is serious about database performance, that's not acceptable, no? | 16:41 |
| dsockwell | if it was me i'd have a separate db machine/cluster sold as a value-add, and grudge subscribers i/o on their instances | 16:42 |
| pvo | jaypipes: I would agree but its a tradeoff between flexibility of virt and bare metal | 16:42 |
| pvo | I can both | 16:42 |
| *** littleidea has quit IRC | 16:42 | |
| jaypipes | pvo: I can both? :) | 16:43 |
| jaypipes | pvo, RichiH: like I said, nothing wrong with running a database server in a VM. Just be aware of the tradeoffs, that's all... | 16:44 |
| jaypipes | pvo, RichiH: and to be clear, RichiH's first question was about high-I/O environments. And if the need is truly for high I/O and database intensive stuff, I don't recommend *any* virtualization, not just specifically to do with Nova. | 16:45 |
| *** mustfeed has quit IRC | 16:47 | |
| *** littleidea has joined #openstack | 16:47 | |
| *** Nacx has joined #openstack | 16:48 | |
| *** littleidea has quit IRC | 16:49 | |
| annegentle | anyone feel like teaching me to fish? I want to understand the other options for the --null_kernel flag. The default is "nokernel" - how do I find the other option(s)? | 16:49 |
| *** littleidea has joined #openstack | 16:50 | |
| *** kashyapc has joined #openstack | 16:50 | |
| pvo | I can *see* both. :p | 16:54 |
| RichiH | jaypipes: obviously, i don't need insane workloads, else i would go bare metal, anyway. but i'd like to be able to provide a system that can do a lot of i/o | 16:54 |
| RichiH | as, in my experience, disk i/o is always the bottleneck | 16:55 |
| jaypipes | pvo: :) | 16:55 |
| RichiH | you can scale out everything else trivially | 16:55 |
| jaypipes | RichiH: then I would focus on making sure the hypervisor and underlying volume storage you choose has the best I/O performance as possible. soren, antonym, and pvo would be good to chat with about the hypervisor choice, perhaps justinsb and vishy or termie_ might be good choices to chat with about volumes/device drivers | 16:56 |
| jaypipes | RichiH: I would ask that, if you do any benchmarks or come up with any recommendations, that you share those with the mailing list and/or add some comments to our documentation? | 16:57 |
| *** MarcMorata has quit IRC | 16:57 | |
| *** littleidea has quit IRC | 16:59 | |
| pvo | RichiH: we looked at this, and found openvz was higher performing than most other hvs, which makes sense. | 16:59 |
| RobertLaptop | OpenStack doesn't support OpenVz correct? | 17:02 |
| RobertLaptop | or have I just found old data in the wiki? | 17:02 |
| *** zuez- has quit IRC | 17:02 | |
| jaypipes | rcc: nice work on the bug patch. | 17:03 |
| rcc | really silly of me reinventing the wheel, forgive my python noobiness ;-) | 17:04 |
| *** littleidea has joined #openstack | 17:05 | |
| *** troytoman-away is now known as troytoman | 17:05 | |
| *** troytoma_ has joined #openstack | 17:06 | |
| *** DanEllis has joined #openstack | 17:07 | |
| jaypipes | rcc: hey, no worries. I need people like dabo all the time to show me The Pythonic Way. ;) | 17:08 |
| rcc | hehe | 17:09 |
| *** hggdh has quit IRC | 17:09 | |
| *** troytoma_ has quit IRC | 17:09 | |
| *** littleidea has quit IRC | 17:11 | |
| dabo | jaypipes: You learn quickly, grasshopper. | 17:11 |
| pvo | RobertLaptop: not today. | 17:11 |
| jaypipes | hehe | 17:12 |
| *** hggdh has joined #openstack | 17:12 | |
| RobertLaptop | Yea I am sure at some point someone will add the library for it. | 17:12 |
| pvo | libvirt supports openvz, but from what I hear, its a little broken | 17:13 |
| *** littleidea has joined #openstack | 17:13 | |
| imsplitbit | more than broken | 17:14 |
| imsplitbit | I am making a blueprint now for the openvz driver I am working on | 17:15 |
| imsplitbit | sorry forgot to make the bp | 17:15 |
| imsplitbit | before playing with the driver code | 17:15 |
| uvirtbot | New bug: #718869 in nova "nova-compute: can't spawn new instances after restarting RabbitMQ" [Undecided,New] https://launchpad.net/bugs/718869 | 17:16 |
| *** allsystemsarego has quit IRC | 17:16 | |
| *** sandywalsh has quit IRC | 17:18 | |
| *** maplebed has joined #openstack | 17:19 | |
| jaypipes | imsplitbit: morning daniel. | 17:19 |
| imsplitbit | jaypipes: morning | 17:23 |
| imsplitbit | OpenVz driver blueprint created | 17:23 |
| jaypipes | coo | 17:26 |
| *** MarcMorata has joined #openstack | 17:26 | |
| *** troytoman is now known as troytoman-away | 17:34 | |
| *** lvaughn_ has quit IRC | 17:37 | |
| *** lvaughn has joined #openstack | 17:37 | |
| *** lvaughn_ has joined #openstack | 17:42 | |
| *** sandywalsh has joined #openstack | 17:43 | |
| *** lvaughn_ has quit IRC | 17:45 | |
| *** lvaughn_ has joined #openstack | 17:45 | |
| *** lvaughn has quit IRC | 17:46 | |
| jaypipes | mtaylor: ping | 17:46 |
| *** mray has joined #openstack | 17:49 | |
| dsockwell | awesome | 17:50 |
| tr3buchet | anyone from citrix know if andy is around? | 17:52 |
| *** johnpur has joined #openstack | 17:52 | |
| *** ChanServ sets mode: +v johnpur | 17:52 | |
| *** justinsb has joined #openstack | 17:53 | |
| jaypipes | eday: you'll like this one: http://thedailywtf.com/Articles/The-Marshal.aspx | 17:56 |
| jaypipes | tr3buchet: Andy who? termie? | 17:57 |
| mtaylor | morning jaypipes | 17:57 |
| tr3buchet | jaypipes: andy southgate (not aware of his irc handle) | 17:58 |
| *** joearnold has joined #openstack | 17:59 | |
| eday | jaypipes: haha | 18:00 |
| jaypipes | mtaylor: ah, there you are little buddy. | 18:02 |
| jaypipes | tr3buchet: ok, sorry, no I don't know him. and I don't see any Citrixers on the channel atm. | 18:02 |
| tr3buchet | thanks jaypipes. have a conflict i need to work it | 18:03 |
| jaypipes | mtaylor: so, here is my build output with i18n compiles: http://paste.openstack.org/show/693/ | 18:03 |
| tr3buchet | work out* | 18:03 |
| jaypipes | mtaylor: was wondering if there's anything unusual in there and if there's anything more I need to do in your opinion before I propose for merging as 2011.1.1 in Bexar? | 18:03 |
| jaypipes | mtaylor: I know you mentioned "a bunch of warnings" when building, and I see some warnings, but is there any I should be concerned with? | 18:04 |
| mtaylor | jaypipes: nope, that's the output I would expect. only thing that concerned me was the noise of that list of unknown files | 18:06 |
| mtaylor | jaypipes: so I _do_ think we should file a bug on distutils.extra | 18:06 |
| jaypipes | mtaylor: about that noise? | 18:06 |
| mtaylor | yes | 18:06 |
| uvirtbot | New bug: #718900 in nova "env http_proxy must be disabled when starting nova daemons by hand" [Undecided,New] https://launchpad.net/bugs/718900 | 18:06 |
| jaypipes | mtaylor: k. | 18:06 |
| *** arun_ has quit IRC | 18:06 | |
| jaypipes | ttx: around? I think I'm ready to do the i18n post-bexar-release. | 18:07 |
| *** lvaughn has joined #openstack | 18:07 | |
| *** MarcMorata has quit IRC | 18:07 | |
| jarrod | can multiple servers share in the roles? | 18:10 |
| *** lvaughn_ has quit IRC | 18:10 | |
| jarrod | like i want to run api, manage, objectstore on 2 machines | 18:10 |
| *** dirakx has quit IRC | 18:12 | |
| mtaylor | jaypipes: are you happy with how the new i18n works (other than the warnings stuff?) | 18:15 |
| mtaylor | jaypipes: also, I was thinking we should figure out how best to have tarmac deal with i18n updates/merges from launchpad | 18:16 |
| jaypipes | mtaylor: yes. | 18:16 |
| jaypipes | mtaylor: and yes. | 18:16 |
| jaypipes | mtaylor: shouldn't be too hard... bzr merge lp:~nova-core/nova/translations && python setup.py build && bzr commit -m "Merge translations" && bzr push lp:nova... | 18:17 |
| mtaylor | jaypipes: totally - I just mean when/how to trigger that activity | 18:18 |
| jaypipes | mtaylor: I think only for releases... | 18:19 |
| jaypipes | mtaylor: or tarballs for any freeze milestone? | 18:19 |
| mtaylor | jaypipes: hell, I dunno. :) | 18:20 |
| *** Nacx has quit IRC | 18:24 | |
| *** troytoman-away is now known as troytoman | 18:33 | |
| jarrod | real quick, i'm deploying to multiple servers, but i want to have 2 controllers in case one fails | 18:37 |
| jarrod | can i not specify something like nova_api=ubuntu1,ubuntu2 | 18:37 |
| RobertLaptop | Question has anyone written any documentation on running windows under OpenStack and Migrating from VMware? | 18:40 |
| justinsb | Have a question about i18n? Why did we put the _ call everywhere - why didn't we just push it down a layer (i.e. into the LOG.debug statements and into a new exception class)? | 18:41 |
| *** dfg has joined #openstack | 18:43 | |
| *** clauden has joined #openstack | 18:45 | |
| dabo | justinsb: The phrases to be translated need to be identified. The _() function serves that purpose. Otherwise, how would the phrases be identified? | 18:46 |
| kpepple | RobertLaptop: haven't seen windows docs ... probably best to ping JordanRinke or vishy | 18:47 |
| justinsb | dabo: The exception class would take an english string and apply the _() function internally | 18:47 |
| *** fabiand_ has quit IRC | 18:47 | |
| justinsb | dabo: So we'd need _ in only a few key places, rather than throughout the code | 18:47 |
| dabo | justinsb: but you need to think about the entire problem: how do you then identify and extract the strings that will need to be translated? | 18:48 |
| RobertLaptop | kpepple, I think you just pinged them :) | 18:49 |
| annegentle | RobertLaptop: the nova services wouldn't run on Windows, but you could run Windows VMs | 18:50 |
| justinsb | dabo: So e.g. LOG.debug(_("Volume info: %(volume_name)s => %(volume_attributes)s") % locals()) would be LOG.debug("Volume info: %(volume_name)s => %(volume_attributes)s", locals()). And debug would call _() on the passed string. | 18:50 |
| RobertLaptop | annegentle, Yea. Looking for any info on building Windows VM's? | 18:50 |
| annegentle | RobertLaptop: this page may help http://wiki.openstack.org/HypervInstall | 18:50 |
| annegentle | RobertLaptop: and JordanRinke has the most experience with it so he can answer questions on building Windows VMs | 18:51 |
| RobertLaptop | I was look more for KVM doc's | 18:51 |
| RobertLaptop | Not a Fan of Hyperv | 18:51 |
| dabo | justinsb: I understand your point. Now think deeper: at some point, a translation for that phrase needs to be retrieved and substituted if the user's language is not English, correct? | 18:51 |
| dabo | Where does that translation come from? | 18:51 |
| RobertLaptop | annegentle, I will keep an eye out for JordanRinke then. | 18:52 |
| vvuksan | i have nova-network complaining that the networks table is missing | 18:52 |
| vvuksan | and it's there | 18:52 |
| vvuksan | actually nova-dhcpbridge | 18:52 |
| vvuksan | i suspect it's trying to use sqlite | 18:52 |
| vvuksan | is there a way to tell what storage engine it's trying to use ? | 18:52 |
| justinsb | dabo: Right, totally agree. I don't see how my proposal changes anything though? _() still gets called on the same string, just by a different function | 18:52 |
| *** evtoews has joined #openstack | 18:53 | |
| dabo | justinsb: ok, look at it a different way. I want to create translations for nova. How do I get a list of strings that need to be translated? | 18:53 |
| justinsb | dabo: Ah. How do we get them today? With a regex? | 18:54 |
| justinsb | dabo: Because if that's the case, we could just have more regexes... | 18:56 |
| dabo | justinsb: no, with a utility called gettext (or xgettext). It scans the source code for strings inside of _() and writes them out to a .pot file | 18:56 |
| dabo | It's a unix standard, and there are Python wrappers for it. | 18:57 |
| dabo | justinsb: remember, not all strings should be translated; just the ones that will be displayed to the user | 18:57 |
| *** Ryan_Lane has joined #openstack | 18:58 | |
| dabo | justinsb: look at this page, in the section entitled 'Process Overview': http://wiki.laptop.org/go/Python_i18n | 18:58 |
| justinsb | dabo: I know.. We'd need to match LOG.xxx(""), and raise xxx("") and a bunch more. I guess that if the Unix gods made the mistake a long time ago there's not a lot we can do now | 18:58 |
| sirp_ | creiht: we briefly talked about Popen.communicate() a while back; I know it's recommended to avoid OS deadlocks; however it buffers everything in memory. Do you happen to know the best way to handle arbitrary steams with Popen w/o deadlock issues? | 18:58 |
| *** joearnold has quit IRC | 18:58 | |
| *** joearnold has joined #openstack | 18:58 | |
| jarrod | can I have nova-volumes lvm group on all compute nodes so that it uses the local disk for each compute instance? | 18:59 |
| justinsb | dabo: Thanks for that link - I'll check it out | 18:59 |
| vishy | vvuksan: do you have --dhcpbridge_flagfile set in your flagfile? | 18:59 |
| vvuksan | vishy: looking | 18:59 |
| vvuksan | vishy: | 19:00 |
| vvuksan | no | 19:00 |
| vvuksan | add --dhcpbridge_flagfile=/etc/nova/nova.conf ? | 19:00 |
| vishy | yes | 19:01 |
| vishy | if that is where your conf is | 19:01 |
| *** asksol__ has quit IRC | 19:01 | |
| vishy | (it passes the flagfile param in ENV) | 19:01 |
| vvuksan | k | 19:01 |
| *** asksol has joined #openstack | 19:02 | |
| vvuksan | vishy: it's looking promising :-) | 19:02 |
| justinsb | dabo: Looks like xgettext supports configurable keywords. So now I don't get it again... | 19:03 |
| *** thimble has quit IRC | 19:03 | |
| creiht | sirp_: I think the better way is in the Popen call set stdout to a subprocess.PIPE | 19:03 |
| creiht | and then read data from proc.stdout.read() | 19:03 |
| dabo | justinsb: Yes, and Python allows you to use any name for the instance reference, not just 'self'. But standards make your life so much easier... :) | 19:04 |
| vishy | i am quite concerned about the leaking fd issue soren mentioned | 19:05 |
| sirp_ | creiht: gotcha, that's what i've been doing. just a little worried by this line in the py docs: "Warning Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process." | 19:06 |
| creiht | hrm | 19:06 |
| justinsb | dabo: Do you know if you can get PyLint to peer inside the format strings to detect variable references? And do you know how to get it to stop warning about the underscore being undefined? Right now my life isn't easier :-) | 19:06 |
| creiht | sirp_: I thought that was only if you called .wait()? | 19:06 |
| dabo | justinsb: no, sorry. I've found PyLint to be more trouble than it's worth many times | 19:07 |
| creiht | sirp_: communicate has the opposite problem of everything being buffered in memory | 19:07 |
| sirp_ | creiht: jhmm, that's possible, might be misreading that | 19:07 |
| justinsb | dabo: Yeah, it's clear that nobody is running it given all the problems it finds in nova :-) | 19:08 |
| creiht | sirp_: hrm... but there is a comment above the .std* descriptions :/ | 19:08 |
| creiht | vishy: which leaking fd issue? | 19:08 |
| vishy | creiht: https://bitbucket.org/which_linden/eventlet/issue/77/subprocess-module- | 19:09 |
| dabo | justinsb: pylint used to be used, but the code was beginning to get filled with pylint instructions, which makes it pretty pointless. "PyLint's complaining? Let me add a comment to shut it up." | 19:09 |
| justinsb | dabo: We could say the same thing about _() pollution :-) | 19:10 |
| dabo | justinsb: One man's pollution is another man's explicit marking. | 19:10 |
| creiht | vishy: interesting | 19:11 |
| justinsb | dabo: Fair enough! | 19:11 |
| creiht | vishy: I guess we haven't seen that in swift since we don't mix monkey patching and green imports | 19:12 |
| creiht | It seems like you should just do one or the other | 19:12 |
| *** jc_smith has joined #openstack | 19:12 | |
| blamar | justinsb, pylint --additional-builtins=_ should work when running? | 19:15 |
| *** CloudChris has quit IRC | 19:15 | |
| vishy | yes not sure where it is being done exactly | 19:16 |
| justinsb | blamar: Thanks - that works wonderfully! | 19:17 |
| *** troytoman is now known as troytoman-away | 19:17 | |
| *** zul_ has joined #openstack | 19:18 | |
| *** zul has quit IRC | 19:18 | |
| *** zul_ is now known as zul | 19:21 | |
| *** zul has joined #openstack | 19:21 | |
| creiht | sirp_: http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/ | 19:21 |
| JordanRinke | pvo: you around? | 19:21 |
| JordanRinke | or anyone in here have experience with nova on Xen? | 19:21 |
| creiht | mtaylor: you around, I was just poked that the hudson stuff for swift is a bit borked at the moment? | 19:24 |
| dabo | JordanRinke: wassip? | 19:24 |
| mtaylor | creiht: oh yeah? | 19:24 |
| dabo | s/ip/ip | 19:24 |
| dabo | arrrgh | 19:24 |
| mtaylor | creiht: an I blame jaypipes ? | 19:24 |
| creiht | sure, as long as someone fixes it :) | 19:24 |
| dabo | mtaylor: you can always blame jaypipes | 19:24 |
| mtaylor | creiht: in what way is it broken? | 19:24 |
| jaypipes | nice. | 19:25 |
| JordanRinke | dabo: trying to get xen running as a compute node and having all kinds of problems | 19:25 |
| creiht | mtaylor: http://hudson.openstack.org/job/swift-tarmac/60747/console | 19:25 |
| mtaylor | creiht: ah! well piddle | 19:25 |
| dabo | JordanRinke: are you running compute in domU on the xenserver? | 19:25 |
| creiht | sirp_: http://stackoverflow.com/questions/1277866/python-subprocess-module-looping-over-stdout-of-child-process | 19:26 |
| creiht | ^- seems to indicate that if you don't do stdin, or stderr, then you are fine, but not sure if that is true | 19:26 |
| uvirtbot | creiht: Error: "-" is not a valid command. | 19:26 |
| creiht | heh | 19:26 |
| creiht | mtaylor: and while we are chatting, ppas? :) | 19:27 |
| JordanRinke | dabo: I have a ubuntu image running on domU with the XenAPI installed | 19:28 |
| mtaylor | creiht: yes! | 19:28 |
| JordanRinke | which is running compute | 19:28 |
| mtaylor | creiht: and also, the breakage is due to upgrading tarmac to fix one problem, which has uncovered another problem. :) | 19:28 |
| *** msassak has joined #openstack | 19:29 | |
| dabo | JordanRinke: well, that's good. What are the problems? | 19:29 |
| pvo | JordanRinke: ya | 19:30 |
| JordanRinke | pvo: Ed is helping me, was going to ask for an ozone dude with xen experience | 19:32 |
| *** dirakx has joined #openstack | 19:32 | |
| pvo | JordanRinke: cool | 19:32 |
| JordanRinke | dabo: I keep getting an error saying that no networks exist for br100, I set the flat_network_bridge=xenbr0 and it gave me an error about xenbr0 once | 19:33 |
| JordanRinke | and now it just keeps giving me the br100 error | 19:33 |
| sirp_ | creiht: yeah, looks like Martelli is saying that i can use stdin to write to the process as long as i don't read from stdout at the same time | 19:33 |
| JordanRinke | I am running api/network etc on another box, this is just a compute node | 19:33 |
| JordanRinke | i tried editing the dbase to add a network for xenbr0 and also edited the br100 reference from the original network to have xenbr0 | 19:34 |
| JordanRinke | ive pretty much got no idea what to try now | 19:34 |
| dabo | JordanRinke: Hmmm... I'm using an image that ant created for us, and we run the whole stack on that, not just compute | 19:34 |
| dabo | JordanRinke: I've pinged the ozone team to see if any of them can help | 19:35 |
| *** littleidea has quit IRC | 19:36 | |
| JordanRinke | dabo: yeah, just running compute seems to be the wrench in this | 19:37 |
| JordanRinke | because I need this environment to run Xen and Hyper-V so that makes it all kinds of interesting | 19:38 |
| dabo | JordanRinke: that's how it's supposed to be in practice, so you're not doing anything you're not supposed to. | 19:38 |
| mtaylor | creiht: should be fixed - will do debs in just a bit | 19:38 |
| *** zul_ has joined #openstack | 19:40 | |
| *** zul has quit IRC | 19:41 | |
| *** troytoman-away is now known as troytoman | 19:41 | |
| openstackhudson | Project swift build #197: SUCCESS in 27 sec: http://hudson.openstack.org/job/swift/197/ | 19:46 |
| openstackhudson | Tarmac: Added lock file configuration to sample rsync.conf. | 19:46 |
| soren | tarmac jobs re-enabled. They seem to be in good shape now. | 19:46 |
| creiht | mtaylor: cool, thanks! | 19:47 |
| *** nelson has joined #openstack | 19:49 | |
| openstackhudson | Yippie, build fixed! | 19:50 |
| openstackhudson | Project dashboard-tarmac build #3,614: FIXED in 2 sec: http://hudson.openstack.org/job/dashboard-tarmac/3614/ | 19:50 |
| *** littleidea has joined #openstack | 19:51 | |
| vishy | soren: did you figure out what was going wrong? | 19:51 |
| openstackhudson | Yippie, build fixed! | 19:52 |
| openstackhudson | Project nova-tarmac build #61,227: FIXED in 6 min 20 sec: http://hudson.openstack.org/job/nova-tarmac/61227/ | 19:52 |
| vishy | soren: failure is back: https://code.launchpad.net/~sleepsonthefloor/nova/716847/+merge/49689 | 19:52 |
| vishy | soren: perhaps it only happens if two branches merge in a row? | 19:54 |
| soren | vishy: I've solved it! | 19:55 |
| soren | vishy: The leaking fd thing. | 19:55 |
| soren | vishy: I thought I updated the bug? | 19:55 |
| soren | vishy: I got completely sidetracked. I'll take a look at the failing tests once I close two other problems. | 19:56 |
| openstackhudson | Project nova build #509: SUCCESS in 1 min 33 sec: http://hudson.openstack.org/job/nova/509/ | 19:57 |
| openstackhudson | Tarmac: Made kernel and ram disk be deleted in xen api upon instance termination | 19:57 |
| annegentle | I just have to echo "Yippie!" | 20:00 |
| soren | vishy: Ok, about the test failure.. I'm clueless as to what's going on. | 20:00 |
| soren | vishy: Looking at https://code.launchpad.net/~sleepsonthefloor/nova/716847/+merge/49689, that /tmp/tmpXXXXXX thing at the top says that it's really run from a tmpdir. There can't be any leftovers from previous runs. | 20:01 |
| *** troytoman is now known as troytoman-away | 20:01 | |
| soren | vishy: I've fixed the eventlet thing and pushed it into the ppa. The Nova side of things is a tiny patch, which I'll propose in a second. | 20:02 |
| sandywalsh | has anything changed recently on how database tables get created initially? I thought it just had to appear in register_models. | 20:03 |
| jk0 | sandywalsh: need to run db sync from nova-manage now | 20:03 |
| sandywalsh | I've done that | 20:03 |
| *** MarcMorata has joined #openstack | 20:04 | |
| soren | nova-manage db sync creates the tables based on the stuff in the migrate scripts. | 20:04 |
| jk0 | if it's something new, I believe there has to be a migration for it | 20:04 |
| sandywalsh | but my table doesn't get created | 20:04 |
| soren | Right, what jk0 said. | 20:04 |
| sandywalsh | k, looking, thx | 20:04 |
| *** littleidea has quit IRC | 20:04 | |
| *** h0cin has quit IRC | 20:04 | |
| *** sandywalsh_ has joined #openstack | 20:08 | |
| soren | vishy: There. https://code.launchpad.net/~soren/nova/eventlet-subprocess/+merge/49698 | 20:09 |
| soren | vishy: That, along with the patched eventlet in the PPA should fix this completely. | 20:10 |
| *** brd_from_italy has joined #openstack | 20:10 | |
| soren | Let me just say that debugging that was *no* fun at all. | 20:10 |
| jk0 | nice find | 20:11 |
| kpepple | soren: does this mean that any tests using subprocess need to change ? | 20:11 |
| *** sandywalsh has quit IRC | 20:11 | |
| *** masumotok has joined #openstack | 20:12 | |
| soren | kpepple: The change should be completely transparent. | 20:12 |
| *** troytoman-away is now known as troytoman | 20:12 | |
| soren | kpepple: Except, of course, from the fact that it fixes this bug. | 20:12 |
| soren | :) | 20:12 |
| *** CloudChris has joined #openstack | 20:12 | |
| kpepple | soren: thx | 20:12 |
| soren | kpepple: Sure. | 20:13 |
| *** littleidea has joined #openstack | 20:14 | |
| *** sandywalsh_ has quit IRC | 20:14 | |
| *** Ryan_Lane_ has joined #openstack | 20:15 | |
| *** Ryan_Lane has quit IRC | 20:16 | |
| *** Ryan_Lane_ is now known as Ryan_Lane | 20:17 | |
| henrichrubin | hi, i am following the wik for manual multinode install and I can't seem to launch an instance. http://paste.org/pastebin/view/28875 . maybe a bad image, ttylinux-uec-amd64-12.1_2.6.35-22_1.tar.gz? | 20:18 |
| kpepple | henrichrubin: it can't find your image | 20:19 |
| *** pvo has quit IRC | 20:19 | |
| kpepple | henrichrubin: what does euca-describe-images look like ? | 20:19 |
| henrichrubin | kpepple: http://paste.org/pastebin/view/28876 . | 20:20 |
| henrichrubin | also, to deregister an image, do i have to do that manually through the db? | 20:21 |
| *** jtran has joined #openstack | 20:21 | |
| RichiH | jaypipes: sure, i'll share results when/if i have any | 20:23 |
| RichiH | jaypipes: i won't create a test deployment now, but i want to get the main concerns out well before i start | 20:23 |
| kpepple | henrichrubin: shouldn't you be using ami-gaamnc3m (machine) not the ami-4vkunrid (kernel) ? | 20:24 |
| dprince | henrichrubin: I hit the same error 20 minute ago as well. | 20:24 |
| vvuksan | if 169.254.169.254 doesn't work | 20:24 |
| vvuksan | and iptables entry exists | 20:24 |
| vvuksan | what should i look at ? | 20:24 |
| vvuksan | if I sniff on the bridge interface I see arp requests but they receive no answers | 20:25 |
| jaypipes | RichiH: sure, no worries...was more talking about the future :) | 20:25 |
| masumotok | soren, vishy, termie: merge request again. I don't know you give me a review this time again, but if so, I appreciate if you give some advice for this proposal. | 20:26 |
| jaypipes | masumotok: live migration, round 2 :) | 20:26 |
| masumotok | jaypipes: hope happy round 2 :) | 20:27 |
| jaypipes | masumotok: :) | 20:27 |
| henrichrubin | kpepple: i tried am-gaamnc3m and it gave the same error | 20:27 |
| *** sandywalsh_ has joined #openstack | 20:28 | |
| *** MarcMorata has quit IRC | 20:29 | |
| blamar | vishy, jaypipes, re: #717467 would it make sense to use pip for deployment and not just development? or is that up to the end user to make sure when he updates his OS OpenStack doesn't fail? | 20:29 |
| *** lvaughn has quit IRC | 20:30 | |
| kpepple | henrichrubin: hmm ... | 20:30 |
| *** lvaughn has joined #openstack | 20:30 | |
| henrichrubin | dprince: did you figure it out? | 20:31 |
| henrichrubin | kpepple: maybe I should try w/ a different image? | 20:31 |
| dprince | henrichrubin: Not yet. I hit it while doing something else... | 20:32 |
| jaypipes | blamar: kinda. it's up to the packagers to make sure that debs and rpms contain dependencies that are correct, but putting dependencies on certain versions of libraries is generally not looked nicely upon if there is a workaround for it... | 20:32 |
| *** zul_ is now known as zul | 20:33 | |
| kpepple | henrichrubin: maybe it's something in your s3 implementation ? | 20:33 |
| *** hub_cap has quit IRC | 20:33 | |
| jaypipes | blamar: also, most things are deployed/installed via automation and or packages, not via pip. there's generally no guarantee that a system would have pip on it :) | 20:33 |
| dprince | henrichrubin: I think S3ImageService is broken on trunk. I'm not able to launch an instance using the ec2 controller. | 20:33 |
| blamar | jaypipes, right, I've seen projects *include* everything needed to have a virtual environment to always ensure version compatibility and I was curious if it was something worth looking in to | 20:34 |
| jaypipes | blamar: I think so far we've used the tools/pip-requires file for development and a general guide for those who do packaging, but it's not the absolute required list of tihngs that must be installed... for instance, rabbitmq isn't listed there ;) | 20:34 |
| jaypipes | blamar: ya, I've seen that too. I've also seen projects have a requirements/ directory with a dev, prod, and test pip-requires file in there.... | 20:35 |
| *** desai has joined #openstack | 20:36 | |
| blamar | jaypipes, seems to provide a lot of stability to the deployment, what are the downsides? | 20:36 |
| henrichrubin | kpepple: i an using ubuntu 10.04. i uninstalled euca2ools and reinstalled it. same thing. | 20:37 |
| jaypipes | blamar: because most people don't deploy using pip. :) | 20:38 |
| desai | so we have run into a problem where the nova-network failed with a traceback (and then failed routinely for a while, thanks to upstart restarting it when it disappeared) and we've ended up with a rabbitmq queue for the network service instance that is ~90k deep | 20:39 |
| jaypipes | blamar: most installs are done via puppet/chef or via yum/apt-get | 20:39 |
| desai | now that i've fixed the root cause, it isn't failing anymore, but it is only processing ~20 events a second | 20:39 |
| desai | is there a safe way to dump the contents of one of those component queues in rabbitmq, or is that a terrible idea? | 20:40 |
| jaypipes | blamar: that's why we use the pip-requires file mostly for devs who find it easy to install necessary development deps into a virtualenv. | 20:40 |
| *** lvaughn_ has joined #openstack | 20:40 | |
| blamar | jaypipes, yeah... using pip + virtualenv in deployment doesn't disallow you from deploying via yum/apt/pacman/emerge/etc or using chef/puppet? I'm not talking about deploying Openstack via pip | 20:40 |
| kpepple | henrichrubin: can use change your imagestore away from s3? | 20:40 |
| jaypipes | blamar: hope that makes sense. not saying it isn't useful to have dependencies in a pip-requires file, just saying that most production deploys don't use pip. | 20:40 |
| jaypipes | blamar: hmm, ok, perhaps I'm not understanding you correctly then... | 20:41 |
| blamar | jaypipes, hmm, my fault, lemme try and rephrase | 20:41 |
| *** jheiss has joined #openstack | 20:41 | |
| henrichrubin | kpepple: not sure how to do that. any wiki? | 20:42 |
| *** arun_ has joined #openstack | 20:42 | |
| *** troytoman is now known as troytoman-away | 20:43 | |
| blamar | jaypipes, OpenStack has an RPM/deb/etc would be deployed and all binaries/tests run in virtualenv which was populated by a pip-requires just like they do right now in dev | 20:44 |
| *** lvaughn has quit IRC | 20:44 | |
| blamar | jaypipes, having the python binaries (nova/bin/*) run out of a venv is the big key I guess | 20:45 |
| jaypipes | blamar: sure, that's fine. I was just asking that we not put sqlalchemy-migration==1.6.0 (or whatever the version was), and instead put sqlalchemy-migration with no specific version if there is a simple workaround. | 20:46 |
| kpepple | henrichrubin: looking | 20:46 |
| soren | jaypipes: +1 on that. | 20:46 |
| soren | jaypipes: We're in for a crap load of pain if we require very specific versions of stuff and still expect various distros to pick us up. | 20:47 |
| jaypipes | soren: right, I'm trying to minimize that as much as we can. | 20:47 |
| *** masumotok has quit IRC | 20:47 | |
| blamar | jaypipes, soren: this problems has been solved for us in a way we don't need to code for every minor version change of every dependency (no matter how trivial the fix might be on our part) | 20:48 |
| blamar | a la the virtual environment | 20:48 |
| blamar | as long as the target OS supports pip, we have compatibility for that OS, no waiting for packages to get up to date or anything, it actually expands our market | 20:49 |
| soren | Relying on distros for packaging is Good Thing[tm]. | 20:50 |
| kpepple | henrichrubin: I think it's just the image_service flag in nova.conf | 20:50 |
| annegentle | kpepple: yep, that's right, but what do you change it to from nova.image.s3.S3ImageService? | 20:51 |
| annegentle | kpepple: I need to figure out how to walk the source code :) | 20:51 |
| mtaylor | blamar: I don't believe the distros are going to be likely to allow packages with pip/venv inside of them ... unless I'm wrong soren? | 20:52 |
| blamar | mtaylor, ah, hadn't thought about that, why is that? | 20:52 |
| soren | Heck no. | 20:53 |
| kpepple | annegentle: I thought if you pulled that it defaulted to the objectstore | 20:53 |
| soren | Because it completely circumvents the work put into packaging things properly. | 20:53 |
| mtaylor | blamar: because the whole idea of the distro is that they have versions of things that have been added to the distro. having a package download and install random-stuff from elsewhere would circumvent the entire idea of the distro | 20:53 |
| annegentle | kpepple: hm. nova.image.s3.S3ImageService is the default for the --image_service nova.conf flag | 20:53 |
| evtoews | hi all, i'm very close to having a working nova installation. the last problem is that my instances can't contact the metadata service at 169.254.169.254 so they can't get a key and i can ssh into them. how do i make the metadata service accessible? | 20:53 |
| annegentle | kpepple: oh you're saying leave it blank? | 20:53 |
| mtaylor | blamar: however, if it makes you feel any better, the philosophical conflict between language devs and distro packagers is not limited to just openstack at the moment.. you should see the pain that is dealing with java in this context | 20:54 |
| henrichrubin | kpepple: i tried this in nova.conf.... --image_service=nova.image.glance.GlanceImageService | 20:56 |
| kpepple | annegentle: I usually do | 20:57 |
| henrichrubin | but i am getting a connection error...http://paste.org/pastebin/view/28878 | 20:57 |
| kpepple | henrichrubin: and what is glance backended with? | 20:58 |
| annegentle | kpepple: ok, then I would guess blank means "nova.image.s3.S3ImageService" since that's the default | 20:58 |
| vvuksan | evtoews: i have the same problem | 20:58 |
| annegentle | henrichrubin: kpepple: Glance has to be installed, see http://glance.openstack.org/installing.html | 20:58 |
| vvuksan | it worked on one of my machines but it doesn't work on consecutive | 20:58 |
| vvuksan | dunno what the magic sauce was :-/ | 20:59 |
| *** dendrobates is now known as dendro-afk | 20:59 | |
| evtoews | vvuksan: did you try the iptables rules mentioned in http://wiki.openstack.org/NovaInstall/MultipleServer ? | 20:59 |
| vvuksan | yeah I got the prerouting rule | 20:59 |
| evtoews | vvuksan: i tried that but no luck... | 21:00 |
| annegentle | henrichrubin: can you try nova.image.local.LocalImageService? | 21:00 |
| sandywalsh_ | shouldn't there be a Cactus migration table in trunk now? | 21:00 |
| sandywalsh_ | I only see austin and bexar | 21:00 |
| henrichrubin | annegentle: i installed glance first. i'll try local next. | 21:00 |
| annegentle | henrichrubin: ah, ok. | 21:01 |
| henrichrubin | annegentle: do i have to do anything special beside set the flag for glance? i am getting the connection refused error. | 21:02 |
| vvuksan | evtoews: try this | 21:03 |
| vvuksan | ip addr add 169.254.169.254/32 scope link dev eth0 | 21:03 |
| evtoews | vvuksan: k. 1 minute. | 21:03 |
| annegentle | henrichrubin: I'm just trying it today myself... but it looks like your nova-api service isn't running? | 21:05 |
| evtoews | vvuksan: is eth0 your public or private interface? for me it's my public interface and it's already tied to the bridge br100. | 21:05 |
| vvuksan | public | 21:06 |
| henrichrubin | annegentle: i tried with the local image service, but all my images are gone. when i try to uec-publish-tarball it doesn't work | 21:06 |
| *** dprince has quit IRC | 21:06 | |
| henrichrubin | annegentle: it is running. i can see the logs | 21:06 |
| annegentle | henrichrubin: ok and is rabbitmq running? | 21:07 |
| annegentle | henrichrubin: just walking through what we've seen before... | 21:08 |
| henrichrubin | annegentle: yes it is running too. how do i register an image using nova.image.local.LocalImageService | 21:09 |
| *** msassak has quit IRC | 21:09 | |
| jaypipes | soren: ok, maybe the eventlet patch wasn't that easy after all ;) | 21:09 |
| soren | jaypipes: No fun *at all*. | 21:10 |
| soren | jaypipes: Any attempts I made to add debugging fiddled with the timing so I couldn't trigger it. | 21:10 |
| jaypipes | suck. | 21:10 |
| evtoews | vvuksan: no luck. now i can ping 169.254.169.254 but my instance is still "waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id" | 21:11 |
| vvuksan | if you do iptables -t nat -L on the host machine | 21:12 |
| vvuksan | what's the 169.254 line ? | 21:12 |
| desai | also, if you add a -v to the iptables line (iptables -t nat -L -v) you can tell if the rule is getting hit or not | 21:13 |
| evtoews | vvuksan: DNAT tcp -- anywhere 169.254.169.254 tcp dpt:www to:67.98.36.184:8773 | 21:13 |
| vvuksan | evtoews: that looks good | 21:13 |
| annegentle | henrichrubin: should be uec-publish-tarball like you're doing... must be time to log a bug? | 21:13 |
| vvuksan | can you telnet to 67.98.36.184:8773 ? | 21:13 |
| vvuksan | that works ? | 21:14 |
| desai | vvuksan: i had a problem where there was an iptables rule on the compute node that was causing the traffic to get re-written too early (and end up on the wrong vlan); the only way to sort it out was to check if the iptables rules were getting hit or not (with -v) | 21:15 |
| evtoews | vvuksan: nope. could not resolve 67.98.36.184:8773/telnet: Name or service not known | 21:15 |
| vvuksan | evtoews: do telnet 67.98.36.184 8773 | 21:16 |
| jaypipes | mtaylor: hey bro, could you take a quick look at http://launchpadlibrarian.net/64352256/buildlog_ubuntu-natty-i386.glance_2011.2~bzr70-0ubuntu0ppa1~natty1_FAILEDTOBUILD.txt.gz? | 21:16 |
| jaypipes | mtaylor: looks like the builder box needs PasteDeploy and sqlalchemy-migrations installed on it... | 21:17 |
| * jaypipes notes that Nova must not have any test cases that actually use PasteDeploy or sqlalchemy-migrations... | 21:17 | |
| vvuksan | desai: I see | 21:18 |
| soren | jaypipes: I believe you're correct. | 21:18 |
| desai | any rabbitmq experts around? we have a queue that has built up a big backlog; i've fixed the underlying issue, but nova-network has been processing the backlog for over an hour at this point. is it safe to dump (and is there a way to) the contents of the queue? | 21:18 |
| *** miclorb_ has joined #openstack | 21:18 | |
| evtoews | vvuksan: yep. connected. | 21:18 |
| soren | jaypipes, mtaylor: I got it. | 21:18 |
| jaypipes | soren, mtaylor: or PM me access creds to the hudson box and I can do it myself, sorry to bother... | 21:18 |
| soren | jaypipes: It's not missing on the hudson box. | 21:18 |
| soren | jaypipes: It's missing as a build-dependency in the packaging. | 21:19 |
| jaypipes | soren: ? | 21:19 |
| jaypipes | soren: gotcha. thx | 21:19 |
| soren | jaypipes: python-paste and python-migrate, is it? | 21:19 |
| soren | jaypipes: Or is it python-pastedeploy? | 21:19 |
| soren | The difference is a bit of a mystery to me. | 21:20 |
| sandywalsh_ | Taking migration #3 (Cactus) | 21:20 |
| jaypipes | soren: python-paste, no? | 21:20 |
| jaypipes | soren: nope, python-pastedeploy | 21:20 |
| soren | jaypipes: python-pastedeploy, it seems, actually. | 21:20 |
| jaypipes | soren: well, there's both, but the package you want is python-pastedeploy, yes | 21:21 |
| soren | jaypipes: Right, that's what I meant :) | 21:21 |
| * soren moves that packaging code to its rightful home | 21:22 | |
| jaypipes | soren: while you're at it, Glance no longer has dependencies on: python-gflags, python-twisted-core, or python-twisted-bin | 21:23 |
| *** dubsquared has joined #openstack | 21:24 | |
| uvirtbot | New bug: #718994 in nova "rpc layer should use a threadpool or some other magic to add concurrency" [Undecided,New] https://launchpad.net/bugs/718994 | 21:24 |
| soren | jaypipes: Oh, lovely. | 21:24 |
| * soren fixes | 21:24 | |
| henrichrubin | annegentle: opened https://bugs.launchpad.net/nova/+bug/719002 . what is the recommended image_service? | 21:24 |
| uvirtbot | Launchpad bug 719002 in nova "local image service fails on uec-publish-tarball" [Undecided,New] | 21:24 |
| jaypipes | soren: I removed them from the pip-requires file. Why didn't you notice? ;P | 21:26 |
| uvirtbot | New bug: #718999 in nova "Expired Auth Token Deletion Failure" [Undecided,New] https://launchpad.net/bugs/718999 | 21:26 |
| * soren sobs | 21:28 | |
| Ryan_Lane | are volumes associated with projects, or users? | 21:28 |
| Ryan_Lane | it seems, from the output of describe_volume, that it's users | 21:29 |
| Ryan_Lane | but that seems kind of odd | 21:29 |
| uvirtbot | New bug: #719002 in nova "local image service fails on uec-publish-tarball" [Undecided,New] https://launchpad.net/bugs/719002 | 21:31 |
| jarrod | does the Volume Manager use lvm vgs available on the compute nodes? | 21:31 |
| dubsquared | jarred: doesn't have to, nova-compute is not a requirement of nova-volume | 21:33 |
| dubsquared | but it can | 21:33 |
| jarrod | ok, that what i want it to do. use nova-volumes group on the local compute nodes | 21:33 |
| jarrod | awesome | 21:33 |
| dubsquared | yeah, that should be good then | 21:33 |
| jarrod | :) | 21:33 |
| jarrod | im assuming i can specify that on the startup options for nova-volume | 21:34 |
| dubsquared | im working on getting this on the openstack wiki | 21:34 |
| jarrod | ok, i was wondering where the docs were | 21:34 |
| dubsquared | but this is what i got so far: http://www.dubsquared.com/?p=120 | 21:34 |
| jarrod | nova-volume has eluded me so far | 21:34 |
| jarrod | but nova-volume only runs on one node, and manages the volumes on all compute nodes? | 21:35 |
| jarrod | or do you have to run nova-volume on all nodes you want the storage | 21:35 |
| uvirtbot | New bug: #719004 in nova "nova-network crashes with bad data" [Undecided,New] https://launchpad.net/bugs/719004 | 21:35 |
| dubsquared | jarred: pretty good question, i have nova-volume on all my nova-compute nodes just to make double use of them | 21:37 |
| *** Abd4llA has quit IRC | 21:37 | |
| dubsquared | my guess would be needing nova-volume everywhere there is storage, but that is beyond my expertise atm | 21:37 |
| jarrod | ok thanks | 21:38 |
| dubsquared | you can spin up an entirely new node, and just have nova-volume on it…don't think nova-volume on another node would know it's there | 21:38 |
| jarrod | how do you tell the CC/api the location of all nova-volumes | 21:38 |
| henrichrubin | annegentle: there is no register function in LocalImageService or GlanceImageService class? then how does one register the images? | 21:38 |
| dubsquared | you tell nova-volumes where the CC/API is | 21:38 |
| dubsquared | via nova.conf | 21:38 |
| jarrod | k k | 21:39 |
| jarrod | i wish the deploymenttool installed multipled instances of nova-volume | 21:40 |
| jarrod | without me having to modify it ;) | 21:40 |
| dubsquared | i haven't used that yet, it works pretty well? | 21:41 |
| jarrod | ive had a few errors | 21:41 |
| jarrod | but it looks like a good start | 21:41 |
| *** Ryan_Lane has quit IRC | 21:41 | |
| dubsquared | you update the wiki with your findings :D | 21:41 |
| *** burris has quit IRC | 21:41 | |
| *** westmaas has quit IRC | 21:42 | |
| soren | jaypipes: There are still a couple of failures. I'll take care of them. | 21:42 |
| jaypipes | soren: thx mate | 21:43 |
| *** Ryan_Lane has joined #openstack | 21:45 | |
| *** GasbaKid has joined #openstack | 21:48 | |
| *** littleidea has quit IRC | 21:49 | |
| vvuksan | is virtio network interface used on Ubuntu hosts ? | 21:49 |
| vishy | no | 21:50 |
| vishy | but it should be | 21:50 |
| vvuksan | so 8139 ? | 21:50 |
| vishy | bug 702040 | 21:51 |
| uvirtbot | Launchpad bug 702040 in nova "Slow network performance since Virtio net drivers are always disabled" [Wishlist,Confirmed] https://launchpad.net/bugs/702040 | 21:51 |
| annegentle | henrichrubin: can be with euca-register, ala: http://nova.openstack.org/adminguide/single.node.install.html#step-13-send-commands-to-upload-sample-image-to-the-cloud. But I'm unfamiliar with all of the image stores options, still learning. | 21:51 |
| vvuksan | vishy: shouldn't at least e1000 be enabled :-) | 21:52 |
| vvuksan | if not virtio | 21:52 |
| vishy | :) | 21:52 |
| vishy | it uses whatever is the default in qemu/kvm right now | 21:52 |
| vvuksan | any idea how to set the default ? | 21:52 |
| vishy | well you can change libvirt.xml to use a diff driver | 21:53 |
| vvuksan | right | 21:53 |
| vvuksan | which I did :-) | 21:54 |
| vvuksan | but it's not the default | 21:54 |
| vishy | change the template | 21:54 |
| vishy | libvirt.xml.template | 21:54 |
| *** DanEllis has quit IRC | 21:54 | |
| vvuksan | i see | 21:55 |
| *** dubsquared has quit IRC | 21:55 | |
| vvuksan | vishy: my suggestion still stands. Why don't we default to e1000 | 21:56 |
| *** dendro-afk is now known as dendrobates | 21:56 | |
| vvuksan | i'll comment on the bug :-) | 21:57 |
| vishy | vvuksan: put it in libvirt and propose a merge | 21:57 |
| vishy | does it really make that much difference? | 21:57 |
| vvuksan | it does | 21:57 |
| vvuksan | 8139 is 100 Mbits | 21:57 |
| vishy | s/libvirt/libvirt.xml.template | 21:57 |
| vvuksan | e1000 is Gig | 21:57 |
| vishy | ah ok then | 21:58 |
| vishy | but have you actually tested performance? | 21:58 |
| vishy | i would be surprised if there is much difference | 21:58 |
| vvuksan | some people have tested it :-) | 22:00 |
| *** ctennis has quit IRC | 22:00 | |
| vvuksan | vishy: http://www.linux-kvm.org/page/Using_VirtIO_NIC | 22:00 |
| vvuksan | with 8139 you'll top off at < 100 Mbit | 22:01 |
| *** GasbaKid has quit IRC | 22:01 | |
| *** GasbaKid has joined #openstack | 22:01 | |
| Ryan_Lane | are volumes associated with users or projects? I'd like to get volumes, and then only display ones in a specific project, but I don't see any project association in describe_volumes... | 22:01 |
| Ryan_Lane | hmm. seems to be associated with a project, but it isn't returned as a value in the EC2 api | 22:04 |
| *** gaveen has joined #openstack | 22:05 | |
| *** gaveen has joined #openstack | 22:05 | |
| *** ctennis has joined #openstack | 22:13 | |
| *** ctennis has joined #openstack | 22:13 | |
| *** msassak has joined #openstack | 22:19 | |
| *** msassak has joined #openstack | 22:20 | |
| *** GasbaKid has quit IRC | 22:22 | |
| *** dirakx has quit IRC | 22:23 | |
| *** vvuksan has quit IRC | 22:24 | |
| *** ppetraki has quit IRC | 22:26 | |
| *** littleidea has joined #openstack | 22:28 | |
| *** gaveen has quit IRC | 22:28 | |
| soren | jaypipes: Glance succesfully built everywhere. | 22:33 |
| soren | brb | 22:33 |
| *** bluesteam has joined #openstack | 22:37 | |
| evtoews | vvuksan: i'm still thrashing around trying to get the metadata service to respond. i noticed that when i do "curl http://169.254.169.254:8773/" it returns a list of what looks like api versions (... 2008-02-01 2008-09-01 2009-04-04). so at least the service is running and appears to be working but i just need a way to properly connect to it. | 22:37 |
| *** MarcMorata has joined #openstack | 22:39 | |
| desai | evtoews: are you using vlans for your network or not? i had a problem where the traffic to 169.254.169.254 was hitting an iptables rule on the nova-compute server and going off to the bitbucket in the sky | 22:39 |
| desai | try to see if the rule is seeing any traffic on your nova-network server using iptables -t nat -L -v | 22:39 |
| desai | i found it by looking at the traffic on the nova-compute machine with tcpdump and looking at the iptables rule stats on the same machine | 22:40 |
| evtoews | desai: i'm using nova.network.manager.FlatManager | 22:41 |
| *** bluesteam has quit IRC | 22:41 | |
| desai | ok, then that exact thing probably isn't the issue | 22:41 |
| desai | are you running on multiple systems? | 22:41 |
| *** blueadept has quit IRC | 22:41 | |
| *** gaveen has joined #openstack | 22:41 | |
| evtoews | desai: just trying to get it running 100% on one system first. | 22:41 |
| *** bluesteam has joined #openstack | 22:41 | |
| desai | hm, if you run tcpdump when you're starting up an instances (you'll probably need to dump br100 by default, iirc) do you see the requests to 169.254.169.254? | 22:42 |
| *** bluesteam has quit IRC | 22:42 | |
| evtoews | desai: fwiw when i do a iptables -t nat -L -v i get " 0 0 DNAT tcp -- any any anywhere 169.254.169.254 tcp dpt:www to:67.98.36.184:8773 " | 22:43 |
| evtoews | desai: i'll give tcpdump a shot. | 22:43 |
| *** abecc has quit IRC | 22:43 | |
| desai | evtoews: that means that the request isn't getting to the nat rule at all | 22:44 |
| evtoews | desai: any troubleshooting tips? :) | 22:44 |
| desai | well, i haven't worked with flatmanager at all | 22:45 |
| desai | (luckly, we have the right gear to use vlanmanager off of the bat) | 22:45 |
| desai | so eventually, cloud-init times out in the default tiny images | 22:45 |
| desai | you should be able to ssh into the internal address | 22:45 |
| desai | right? | 22:45 |
| evtoews | desai: i can ssh to it but i can't actually login properly because the instance can't get the public key from the metadata service. | 22:47 |
| desai | hm, that's weird; when cloud-init was failing, i was still able to ssh in | 22:48 |
| desai | are you supplying your public key to ssh with -i? | 22:48 |
| *** johnpur has quit IRC | 22:49 | |
| evtoews | desai: when i run tcpdump the message i'm seeing most it "ARP, Request who-has 10.0.0.1 tell 10.0.0.3, length 28". the instance i just started has an ip of 10.0.0.3 | 22:49 |
| evtoews | desai: s/it/is/ | 22:49 |
| desai | do you have an interface on your machine with the address 10.0.0.1? | 22:51 |
| *** gaveen has quit IRC | 22:51 | |
| desai | and you presumably can ping 10.0.0.3, right? | 22:51 |
| evtoews | desai: yep. i've tried doing "ssh -i mykey.priv root@10.0.0.2" and "ssh -i mykey.priv ubuntu@10.0.0.2". 10.0.0.2 is the instance i started before 10.0.0.3 | 22:53 |
| desai | hm | 22:53 |
| evtoews | desai: strange. i can ping 10.0.0.2 but not 10.0.0.3. | 22:53 |
| desai | weird | 22:54 |
| *** imsplitbit has quit IRC | 22:54 | |
| desai | does the bridging setup look reasonable (brctl show)? | 22:54 |
| evtoews | desai: *no* interface on my machine with the address 10.0.0.1 | 22:54 |
| desai | that is interesting | 22:54 |
| desai | that sounds like a problem to me | 22:54 |
| *** rcc has quit IRC | 22:55 | |
| desai | does /var/log/nova/nova-network.log contain any errors? | 22:55 |
| desai | but, take that with a grain of salt, since i don't know flatmanager at all | 22:55 |
| desai | with vlanmanager, you end up with a brXXX interface with the gateway ip address for the vm network | 22:55 |
| desai | on the nova-network machine | 22:56 |
| *** gaveen has joined #openstack | 22:56 | |
| *** gaveen has joined #openstack | 22:56 | |
| desai | and then vnetXX interfaces are handed out to the kvm instances | 22:56 |
| evtoews | desai: brctl show looks okay. interfaces eth0 vnet0 vnet1 | 22:56 |
| desai | hm | 22:56 |
| *** MarcMorata has quit IRC | 22:57 | |
| desai | if you can ping 10.0.0.2, and you don't have a 10.0.0.1 interface on the system, how are you getting there? | 22:57 |
| desai | can you traceroute? | 22:57 |
| evtoews | desai: no errors in nova-network.log | 22:57 |
| desai | anything in the nova-compute.log? | 22:58 |
| desai | it sounds like the bridging setup is working properly | 22:58 |
| desai | do you have ip forwarding turned on? | 22:59 |
| evtoews | desai: traceroute 10.0.0.2 | 23:00 |
| evtoews | traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets | 23:00 |
| evtoews | send: Operation not permitted | 23:00 |
| *** Ryan_Lane has quit IRC | 23:00 | |
| evtoews | desai: i have not turn on ip forwarding explicity. | 23:03 |
| desai | try that | 23:03 |
| *** dendrobates is now known as dendro-afk | 23:03 | |
| evtoews | sysctl net.ipv4.ip_forward | 23:03 |
| evtoews | net.ipv4.ip_forward = 1 | 23:03 |
| *** adjohn has joined #openstack | 23:03 | |
| evtoews | desai: looks like it was already on. i thought that was it... | 23:03 |
| *** abecc has joined #openstack | 23:03 | |
| desai | hm, you got me then | 23:04 |
| evtoews | desai: fwiw, in nova-compute.log i see "instance instance-00000003: injecting key into image ami-bh81dyka" | 23:04 |
| evtoews | desai: just to be even more perplexing. | 23:05 |
| desai | yeah, i think that means it should be working | 23:05 |
| evtoews | desai: thanks for the help. maybe i'll try vlanmanager. i think our network should support it. i'll check. | 23:06 |
| jarrod | can i not run JUST nova-compute and nova-volume on my compute servers and have them register with the CC? | 23:07 |
| jarrod | or do I have to have all the packages running | 23:08 |
| desai | compute servers generally need to have nova-compute running | 23:09 |
| jarrod | and that is all? | 23:09 |
| jarrod | oh | 23:09 |
| jarrod | i forgot mysql-client | 23:09 |
| desai | running nova-network on compute nodes caused problems for me | 23:09 |
| jarrod | no wonder DOH | 23:09 |
| desai | has anyone seen errors like: ignoring error injecting data into image ami-otcm0xt9 (Mapped device was not found (we can only inject raw disk images): /dev/mapper/nbd15p1) | 23:10 |
| jarrod | modprobe nbd | 23:10 |
| jarrod | ? | 23:10 |
| desai | it is loaded | 23:10 |
| JordanRinke | haven't seen that before | 23:11 |
| soren | desai: Which os? Which kernel version? | 23:12 |
| desai | ubuntu maverick | 23:13 |
| desai | running bzr655 on the compute nodes | 23:13 |
| desai | this just started happening | 23:13 |
| desai | we had upgraded the central server to 669 because there were some bugfixes we needed (the api 400 lazyloading problem that was just fixed) | 23:14 |
| soren | Anything interesting in dmesg? | 23:14 |
| jarrod | why maverick | 23:14 |
| jarrod | 10.04 | 23:14 |
| desai | some auditing stuff from apparmon, but nothing else | 23:15 |
| JordanRinke | i had a problem which I believe vish fixed | 23:15 |
| *** msassak has quit IRC | 23:15 | |
| JordanRinke | where apparmor wasn't allowing access to ./local it allowed ./disk but not local | 23:15 |
| JordanRinke | and it gave me nbd / mapper errors | 23:15 |
| desai | hm | 23:15 |
| soren | That was ages ago, wasn't it? | 23:16 |
| JordanRinke | yeah | 23:16 |
| jarrod | hmm, i cannot figure out why nova-volume and nova-compute running on a compute machine with a proper nova.conf is not registering the services | 23:16 |
| JordanRinke | like I said, fixed and what not, just throwing stuff out there | 23:16 |
| desai | apparmor is showing the vm in enforce mode | 23:17 |
| desai | the kvm instance never actually gets off of the ground | 23:17 |
| desai | the console is empty | 23:17 |
| desai | console file, that is | 23:17 |
| JordanRinke | did it create the xml file? | 23:17 |
| * desai checks | 23:18 | |
| JordanRinke | you can try... virsh create libvirt.xml to see if libvirt can even start the instance | 23:18 |
| desai | virsh list shows the instance as running | 23:18 |
| desai | where is the xml file created? | 23:19 |
| soren | Which version of libvirt are you using? | 23:19 |
| desai | 0.8.3-1ubuntu14 | 23:19 |
| soren | ok | 23:19 |
| *** troytoman-away is now known as troytoman | 23:19 | |
| JordanRinke | /var/lib/nova/instances/instance-# provided you didn't change anything in the configs | 23:19 |
| soren | That should be fine, IIRC. | 23:19 |
| JordanRinke | if you go there, you can: virsh -d 5 create libvirt.xml | 23:20 |
| jarrod | it helps to have python-mysqldb installed | 23:20 |
| jarrod | JEEZ | 23:20 |
| JordanRinke | and that will give you extensive debugging info | 23:20 |
| jarrod | i feel dumb | 23:20 |
| *** dendro-afk is now known as dendrobates | 23:20 | |
| soren | Well... Most of the interesting libvirt stuff happens in the daemon. It won't show in that debug output. | 23:20 |
| *** gondoi has quit IRC | 23:21 | |
| desai | it complains about a uuid conflict anyway | 23:21 |
| soren | Then the VM should be funning. | 23:21 |
| soren | running, even :) | 23:21 |
| JordanRinke | it sounds like it is created and running | 23:21 |
| soren | What makes you say it's not running? | 23:21 |
| desai | it looks like it is | 23:21 |
| desai | there is a kvm process | 23:21 |
| desai | and it is doing stuff | 23:21 |
| desai | but nothing ever shows up in the console log | 23:21 |
| soren | What image are you using? | 23:22 |
| desai | and i'm not quite sure what to do to figure out what is going on | 23:22 |
| jarrod | how do i create an instance that uses an EBS-style volume? | 23:22 |
| desai | the minimal tty one | 23:22 |
| JordanRinke | desai: what image are you trying to spawn? | 23:25 |
| *** Gh0sty has quit IRC | 23:25 | |
| *** adjohn has quit IRC | 23:26 | |
| desai | ami-tty | 23:26 |
| desai | so, the kvm seems to be running, and is eating up 16% of a cpu | 23:27 |
| desai | but i dunno what it is actually doing | 23:27 |
| desai | and virsh shows no console available for domain | 23:27 |
| JordanRinke | isnt there a way to add vnc to the libvirt.xml to get a graphical console? | 23:29 |
| JordanRinke | I havent done it, if anyone has please correct me but I think it is something like | 23:30 |
| JordanRinke | <devices> | 23:30 |
| JordanRinke | <graphics type='vnc' port='5904' listen='192.168.1.5' passwd='password'> | 23:31 |
| JordanRinke | </devices> | 23:31 |
| JordanRinke | obviously changing the listen to an IP that you can hit locally and whatever for a password | 23:31 |
| JordanRinke | that is what I have in my notes, but I haven't actually tried that soo, use at your own risk | 23:32 |
| desai | can you do that on the fly? | 23:32 |
| JordanRinke | no, you will probably have to... | 23:32 |
| desai | also, i'm not going to be able to get vnc to that network anyway at the moment | 23:32 |
| *** eikke has joined #openstack | 23:32 | |
| JordanRinke | virsh destroy libvirt.xml | 23:32 |
| JordanRinke | and then | 23:32 |
| JordanRinke | virsh create libvirt.xml | 23:33 |
| JordanRinke | or the destroy command might take the domain/instance name (I don't recall of the top of my head) | 23:33 |
| jc_smith | Has anybody setup the openstack API to use multiple projects? | 23:34 |
| desai | let's see if going back to 655 helps | 23:34 |
| jc_smith | it seems to always just use 'openstack' as default, or you override it with --default-project, but never seems to use the one associated with the user | 23:35 |
| JordanRinke | jc_smith: from my experience that is the only option currently | 23:35 |
| JordanRinke | since a user can be associated to multiple projects, it can't really pick a project from a user, it has to be specified in some way | 23:36 |
| *** mray has quit IRC | 23:36 | |
| JordanRinke | and the only way I know of right now is that default-project flag | 23:36 |
| jarrod | i keep getting NoMoreAddresses | 23:37 |
| jarrod | what the heck | 23:37 |
| JordanRinke | jarrod: that can be caused by a number of things | 23:38 |
| JordanRinke | how did you setup your network? | 23:38 |
| jarrod | VlanManager | 23:38 |
| JordanRinke | nova-manage network create etc? | 23:38 |
| jarrod | yes | 23:38 |
| jc_smith | so, if I want to have my users partitioned from one another, so they can't see each others servers for example, how would I set that up? | 23:38 |
| desai | soren: is there a way to dump the rabbitmq queues when things fail badly? i spent a few hours waiting for a 90k deep queue to drain earlier | 23:38 |
| jarrod | i just added more too | 23:38 |
| JordanRinke | when that has happened to me when testing I have just hopped in the mysql database and dumped the fixed_ips, networks, and instances tables and ran the create network command again | 23:40 |
| JordanRinke | probably the worst solution possible though | 23:40 |
| jarrod | i did the killall dnsmasq; service nova-network restart and now its hanging in networking | 23:41 |
| jarrod | suckkkkk | 23:41 |
| soren | desai: I'm sure ther eis. | 23:41 |
| soren | desai: Let me know when you find out. | 23:41 |
| soren | desai: :D | 23:41 |
| desai | soren: would it be safe, you think? | 23:41 |
| desai | also, i figured out the other issue; it was a mixed version problem, so my bad (669 on server, 655 on client) | 23:42 |
| soren | desai: To clear the queue? | 23:42 |
| desai | yeah, would it be safe to clear the queue | 23:42 |
| soren | desai: I guess. This doesn't sound like a production system anyway :) | 23:42 |
| desai | well, we're hoping it will get there soon-ish | 23:42 |
| desai | ;) | 23:42 |
| desai | but yeah, running trunk won't be the plan then, once we find a version we're happy with | 23:42 |
| jarrod | maybe i should be using floating | 23:43 |
| jarrod | instead of fixed | 23:43 |
| tr3buchet | http://pastie.org/1564598 | 23:43 |
| tr3buchet | what am i doing wrong here? | 23:43 |
| tr3buchet | trying to run tests | 23:43 |
| *** desai has quit IRC | 23:45 | |
| jarrod | (nova.root): TRACE: Stderr: 'sudo: dnsmasq: command not found\n' <-- there is my problem | 23:47 |
| jarrod | or so i thought | 23:49 |
| jarrod | dangit | 23:49 |
| openstackhudson | Project nova build #510: SUCCESS in 1 min 32 sec: http://hudson.openstack.org/job/nova/510/ | 23:52 |
| openstackhudson | Tarmac: Adds missing flag that makes use_nova_chains work properly. | 23:52 |
| jc_smith | Is there a way currently to partition users from one another, so that users can only see their respective images/servers, or is this coming with the outcome of the multi-tenancy thread I saw in the mailing list? | 23:55 |
| kpepple | tr3buchet: how are you running tests ? | 23:55 |
| soren | vishy: *sigh* | 23:55 |
| tr3buchet | in venv on my dev box | 23:55 |
| tr3buchet | kpepple: seems to working on the test box though | 23:56 |
| soren | vishy: I updated eventlet on the Hudson box, and now I can't reproduce the failure anymore. It used to happen about 60% of the time. | 23:56 |
| soren | vishy: Lovely. Now it happened again. Gah. | 23:56 |
| kpepple | tr3buchet: with the ./run_tests.sh script ? | 23:56 |
| tr3buchet | kpepple yes | 23:57 |
| kpepple | tr3buchet: i think it's telling you that you don't have the sqlalchemy-migrate egg installed ... might want to ./run_tests.sh -f to rebuild venv | 23:58 |
| jarrod | instance 9: starting... <-- once I get that -- how can I debug to find out what is keeping it in "networking" ? | 23:58 |
| kpepple | tr3buchet: ooops. hold on ... i'm getting the same thing with ppa:nightly | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!