| termie | comstud: it's pretty old code | 00:00 |
|---|---|---|
| comstud | now i'm seeing a.. supposedly unrelated error in tests | 00:01 |
| comstud | getting a NoMoreAddresses for CloudTestCase.test_associate_disassociate_address | 00:01 |
| comstud | but | 00:01 |
| termie | that happens if you use the real queue, usually | 00:02 |
| comstud | only when I run the tests the whole way through | 00:02 |
| comstud | when i run the test individually, it's fine | 00:02 |
| comstud | but if i go back to trunk, the tests run the whole way through | 00:02 |
| termie | comstud: if you switch nova.tests.fake_flags to have fake_rabbit=False they don't | 00:03 |
| comstud | it's there | 00:03 |
| termie | fake_rabbit is True in trunk | 00:03 |
| termie | you need to flip the flag to test against a real rabbitmq | 00:03 |
| comstud | er | 00:03 |
| comstud | ok | 00:04 |
| comstud | i have fake_rabbit = True in both places here | 00:05 |
| comstud | 1 fails, 1 does not | 00:05 |
| termie | right | 00:05 |
| comstud | changing to false | 00:05 |
| termie | but i am telling you that in trunk it fails with False, so presumably if you made it act more like the real rabbit it might also fail | 00:05 |
| comstud | aha. | 00:05 |
| comstud | that is what i've done, essentially | 00:06 |
| comstud | fakerabbit's Backend.consume() is not an iterator like real rabbit | 00:06 |
| comstud | fakerabbot tries to pull out of a queue, call callback, and then raises StopIteration | 00:06 |
| comstud | which is completely wrong | 00:06 |
| termie | i would argue slightly wrong since an entire project has been able to successfully use it for a year ;) | 00:07 |
| comstud | haha | 00:07 |
| comstud | sure :) | 00:07 |
| comstud | now a different test just hangs with real rabbit | 00:08 |
| comstud | hrmph | 00:08 |
| termie | i don't get hangs with the real rabbit, only a couple failing tests | 00:08 |
| termie | (in trunk) | 00:08 |
| comstud | this is related to my changes with the Topic consumers | 00:08 |
| comstud | instead of calling fetch() every 0.1s | 00:09 |
| comstud | i'm just calling iterconsume() which iterates forever with limit=None | 00:09 |
| comstud | using that with 3 consumers at once | 00:10 |
| comstud | obviously, that was never used before | 00:10 |
| comstud | anywhere in the code | 00:10 |
| comstud | well | 00:11 |
| termie | which test is hanging? | 00:11 |
| comstud | test_ajax_console | 00:11 |
| termie | all those tests are kind of funny | 00:11 |
| comstud | i had it hang before, also, when i thought i'd fixed fakerabbit properly | 00:11 |
| termie | whenever i mess with sync/async bits those ones are the ones that complain | 00:11 |
| comstud | what's annoying is that i'm pretty positive the rpc.py code is correct | 00:12 |
| comstud | hehe | 00:12 |
| comstud | I need a test for the test. | 00:12 |
| termie | probably worth looking at what those tests are doing | 00:13 |
| comstud | most of this is completely foreign to me at this point | 00:13 |
| termie | hehe, that just means you'll be an expert once you fix it | 00:13 |
| comstud | going to push up the fakerabbit fixes here | 00:13 |
| comstud | yeah, really. | 00:13 |
| comstud | well, pushed what I think is the correct fakerabbit fixes into lp:~cbehrens/nova/rpc-improvements | 00:15 |
| comstud | i'll have to spend more time on the tests tomorrow | 00:15 |
| termie | i'll look it over and see whether anything catches my eye | 00:16 |
| comstud | thanks | 00:16 |
| comstud | taking a break for now | 00:17 |
| *** salv-orlando has left #openstack-dev | 00:22 | |
| *** galstrom has joined #openstack-dev | 00:26 | |
| *** galstrom has quit IRC | 00:28 | |
| *** galstrom has joined #openstack-dev | 00:29 | |
| *** galstrom has quit IRC | 00:29 | |
| *** Tv has quit IRC | 00:33 | |
| *** blamar__ has joined #openstack-dev | 00:49 | |
| *** dragondm has quit IRC | 01:03 | |
| *** exlt has quit IRC | 01:11 | |
| *** jdurgin has quit IRC | 01:12 | |
| *** dragondm has joined #openstack-dev | 01:12 | |
| *** Tv has joined #openstack-dev | 01:13 | |
| *** Tv has left #openstack-dev | 01:13 | |
| *** Tushar has quit IRC | 01:14 | |
| *** namaqua has joined #openstack-dev | 01:26 | |
| *** namaqua has quit IRC | 01:27 | |
| *** dragondm has quit IRC | 01:58 | |
| *** jkoelker has quit IRC | 01:58 | |
| *** blamar__ has quit IRC | 02:02 | |
| *** markvoelker has joined #openstack-dev | 02:27 | |
| vishy | comstud, termie: fixed the bug lp:~vishvananda/nova/rpc-improvements | 02:56 |
| vishy | consumers weren't ever getting removed | 02:56 |
| comstud | looking | 02:58 |
| comstud | ahh, good call | 02:59 |
| comstud | thanks for the assist. like the changes | 03:11 |
| comstud | have a test that's failing yet tho | 03:11 |
| comstud | GreenletExit exception | 03:12 |
| comstud | looks like you try to catch it tho | 03:12 |
| * comstud looks | 03:12 | |
| comstud | ah, not here | 03:15 |
| * comstud tries this | 03:15 | |
| comstud | fixes. just need to catch it for csetthread.wait().. when service shuts down | 03:23 |
| comstud | http://paste.openstack.org/show/1366/ | 03:25 |
| comstud | this one looks interesting | 03:25 |
| comstud | termie made reference to this, i think | 03:27 |
| comstud | ah | 03:43 |
| comstud | this report_interval stuff again, and tests relying on this | 03:43 |
| *** Zangetsue has joined #openstack-dev | 03:55 | |
| *** blamar__ has joined #openstack-dev | 03:55 | |
| *** blamar has quit IRC | 04:09 | |
| *** jaypipes has quit IRC | 04:09 | |
| *** _0x44 has quit IRC | 04:09 | |
| *** jaypipes has joined #openstack-dev | 04:11 | |
| *** blamar has joined #openstack-dev | 04:12 | |
| *** _0x44 has joined #openstack-dev | 04:12 | |
| comstud | time to learn mox i guess | 04:16 |
| vishy | comstud: yeah that is just because we changed how service loads the consumers but not the test | 05:01 |
| comstud | yeah | 05:02 |
| comstud | i'm struggling to modify the tests 'the right way' | 05:02 |
| comstud | i got something to work, but it's a hack | 05:03 |
| comstud | ok, i think i *just* figured it out | 05:04 |
| comstud | ok, well, side effect is fixing lp783720 | 05:11 |
| pyhole | LP Bug #783720 in OpenStack Compute (nova): "services don't listen for rpc calls when report_interval is 0" [Status: In Progress, Assignee: Chris Behrens] https://bugs.launchpad.net/bugs/783720 | 05:11 |
| uvirtbot | Launchpad bug 783720 in nova "services don't listen for rpc calls when report_interval is 0" [Undecided,In progress] | 05:11 |
| uvirtbot | Launchpad bug 783720 in nova "services don't listen for rpc calls when report_interval is 0" [Undecided,In progress] https://launchpad.net/bugs/783720 | 05:11 |
| comstud | good, all tests pass now | 05:13 |
| vishy | comstud: nice | 05:15 |
| vishy | comstud: hopefully that gives us a big performance boost :) | 05:16 |
| comstud | that's what I'm hoping | 05:16 |
| comstud | unfortunately h48 was split up into 4 zones today and doesn't ahve the compute nodes set up | 05:16 |
| comstud | so I can't test yet | 05:16 |
| comstud | hopefully tomorrow | 05:16 |
| comstud | ok, connection pool size is a flag now | 05:17 |
| vishy | and now termie can finish up his multicall code | 05:21 |
| vishy | and i can finish my no-db-messaging :) | 05:21 |
| *** blamar__ has quit IRC | 05:21 | |
| comstud | no idea what either of those are, but cool ;) | 05:25 |
| comstud | linked this branch to 771512, also, and updated comments there | 05:25 |
| comstud | lp771512 | 05:26 |
| pyhole | LP Bug #771512 in OpenStack Compute (nova): "Timeout from API with 50 Simultaneous Builds" [Status: In Progress, Assignee: Chris Behrens] https://bugs.launchpad.net/bugs/771512 | 05:26 |
| uvirtbot | Launchpad bug 771512 in nova "Timeout from API with 50 Simultaneous Builds" [High,In progress] | 05:26 |
| uvirtbot | Launchpad bug 771512 in nova "Timeout from API with 50 Simultaneous Builds" [High,In progress] https://launchpad.net/bugs/771512 | 05:26 |
| comstud | eventlet patch helps, but ultimately this is probably the real fix. | 05:26 |
| comstud | hopefully. | 05:26 |
| comstud | IMO, 'nova boot' still takes too long to get a response from the API... I need to investigate where the delay is | 05:29 |
| comstud | vishy: thanks again for the help with the tests | 05:35 |
| comstud | want to pound on this before merge prop.. hopefully tomorrow | 05:35 |
| comstud | i'm outtie for now | 05:35 |
| * comstud & poof | 05:35 | |
| soren | comstud: Are you really gone? | 05:37 |
| comstud | almost | 05:38 |
| comstud | :) | 05:38 |
| vishy | comstud: np.. see you tomorrow | 05:38 |
| comstud | soren: almost | 05:38 |
| soren | comstud: Just saw your comment above. | 05:38 |
| soren | comstud: ""eventlet patch helps, but ultimately this is probably the real fix." | 05:38 |
| comstud | yeah | 05:38 |
| soren | comstud: does that mean you don't need the patch applied? | 05:39 |
| comstud | i don't know yet | 05:39 |
| comstud | until I can stress test it | 05:39 |
| comstud | unfortunately the environment I was using is currently down | 05:39 |
| comstud | waiting on ant for that | 05:39 |
| comstud | i suspect that it'll eliminate the requirement for the eventlet patch | 05:40 |
| comstud | even though we agree the eventlet issue is certainly a bug | 05:40 |
| soren | comstud: So your new patch reduces the time for each connection? | 05:40 |
| soren | comstud: Oh, ok. | 05:40 |
| comstud | lp:~cbehrens/nova/rpc-improvements | 05:40 |
| comstud | implements connection pooling | 05:40 |
| comstud | which will put a cap on # of connections to rabbit | 05:40 |
| comstud | and will re-use connections | 05:41 |
| soren | comstud: I see, ok. | 05:41 |
| comstud | so all of the pounding of connect() should go away | 05:41 |
| soren | comstud: I'll build eventlet packages with the patch nonetheless. | 05:41 |
| comstud | alrighty | 05:41 |
| soren | comstud: Thanks. Have a good night :) | 05:41 |
| comstud | no problem:) you too. | 05:42 |
| comstud | or | 05:42 |
| comstud | have a good day | 05:42 |
| comstud | :) | 05:42 |
| * comstud & gone for reals | 05:42 | |
| *** Zangetsue has quit IRC | 05:57 | |
| *** Zangetsue has joined #openstack-dev | 06:08 | |
| *** zaitcev has quit IRC | 07:26 | |
| *** dysinger has joined #openstack-dev | 08:07 | |
| *** dysinger has quit IRC | 09:25 | |
| *** markvoelker has quit IRC | 10:56 | |
| *** markvoelker has joined #openstack-dev | 10:57 | |
| *** markvoelker has quit IRC | 11:26 | |
| *** markvoelker has joined #openstack-dev | 11:26 | |
| BK_man | comstud: | 11:29 |
| *** dysinger has joined #openstack-dev | 11:43 | |
| *** dprince has joined #openstack-dev | 12:44 | |
| *** mattray has joined #openstack-dev | 13:23 | |
| *** Zangetsue_ has joined #openstack-dev | 13:30 | |
| *** Zangetsue has quit IRC | 13:32 | |
| *** Zangetsue_ is now known as Zangetsue | 13:32 | |
| *** lorin1 has joined #openstack-dev | 13:36 | |
| *** mattray has quit IRC | 13:39 | |
| *** openstackjenkins has quit IRC | 13:42 | |
| *** rnirmal has joined #openstack-dev | 13:54 | |
| *** jmeredit has joined #openstack-dev | 14:10 | |
| *** bcwaldon has joined #openstack-dev | 14:12 | |
| *** Zangetsue has quit IRC | 14:14 | |
| *** jkoelker has joined #openstack-dev | 14:31 | |
| *** dragondm has joined #openstack-dev | 14:56 | |
| *** cp16net has joined #openstack-dev | 15:10 | |
| *** alex-meade has joined #openstack-dev | 15:21 | |
| bcwaldon | How long should it take for Hudson to pick up a MP that is set to "Approved" and merge it in? | 15:37 |
| jk0 | usually only a few minutes | 15:37 |
| jk0 | but I've seen it take much longer | 15:37 |
| bcwaldon | That's what I thought. I have one in glance that was approved an hour ago and hasn't been picked up yet | 15:37 |
| bcwaldon | https://code.launchpad.net/~rackspace-titan/glance/api-results-filtering/+merge/61056 | 15:38 |
| jk0 | ah, the glance one I'm not sure about | 15:38 |
| jk0 | but nova is usually pretty quick | 15:38 |
| bcwaldon | I'd assume it is the same system, but I have no idea | 15:38 |
| jk0 | same hudson, but something could be backed up | 15:39 |
| *** zaitcev has joined #openstack-dev | 15:40 | |
| soren | http://jenkins.openstack.org/ seems to be acting up. | 15:42 |
| soren | Should be on its way back | 15:44 |
| soren | bcwaldon: ^ This is why your mp wasn't picked up. | 15:44 |
| bcwaldon | gotcha | 15:44 |
| *** openstackjenkins has joined #openstack-dev | 15:44 | |
| soren | openstackjenkins: Welcome back. | 15:45 |
| openstackjenkins | soren did you mean me? Unknown command 'Welcome' | 15:45 |
| openstackjenkins | Use !jenkinshelp to get help! | 15:45 |
| comstud | yawns | 15:48 |
| *** alex-meade has quit IRC | 15:53 | |
| *** ameade has joined #openstack-dev | 15:54 | |
| openstackjenkins | Project swift build #262: SUCCESS in 27 sec: http://jenkins.openstack.org/job/swift/262/ | 16:02 |
| openstackjenkins | * Tarmac: Editing the install docs based on testing in the internal Swift training course this week. | 16:02 |
| openstackjenkins | * Tarmac: Adding container stats collector, unit tests, and refactoring some of the stats code. There will have to be changes to both the swift and rackswift conf files before this can be released. Please DO NOT approve this branch for merge until glange's stats stuff is all ready to go. gracias. | 16:02 |
| *** salv-orlando has joined #openstack-dev | 16:10 | |
| *** salv-orlando has left #openstack-dev | 16:11 | |
| *** salv-orlando has joined #openstack-dev | 16:12 | |
| *** blamar__ has joined #openstack-dev | 16:20 | |
| *** johnpur has joined #openstack-dev | 16:25 | |
| *** ChanServ sets mode: +v johnpur | 16:25 | |
| bcwaldon | soren: weird build error on glance, can you check it out? | 16:25 |
| *** agarwalla has joined #openstack-dev | 16:36 | |
| *** jdurgin has joined #openstack-dev | 16:58 | |
| *** salv-orlando has quit IRC | 17:05 | |
| *** hub_cap has joined #openstack-dev | 17:13 | |
| *** jmeredit has quit IRC | 17:18 | |
| *** hub_cap is now known as hub-cap | 17:32 | |
| *** hub-cap is now known as hub_cap | 17:36 | |
| *** blamar has quit IRC | 17:43 | |
| *** antonyy has joined #openstack-dev | 17:43 | |
| openstackjenkins | Project nova build #912: SUCCESS in 2 min 45 sec: http://jenkins.openstack.org/job/nova/912/ | 17:44 |
| openstackjenkins | Tarmac: Added missing xenhost plugin. This was causing warnings to pop up in the compute logs during periodic_task runs. It must have not been bzr add'd when this code was merged. | 17:44 |
| *** mgius has joined #openstack-dev | 17:45 | |
| *** hub-cap has joined #openstack-dev | 17:53 | |
| *** hub_cap has quit IRC | 17:55 | |
| *** hub-cap is now known as hub_cap | 17:55 | |
| *** markvoelker has quit IRC | 18:22 | |
| openstackjenkins | Project nova build #913: SUCCESS in 2 min 41 sec: http://jenkins.openstack.org/job/nova/913/ | 18:24 |
| openstackjenkins | Tarmac: Implements a basic mechanism for pushing notifications out to interested parties. The rationale for implementing notifications this way is that the responsibility for them shouldn't fall to Nova. As such, we simply will be pushing messages to a queue where another worker entirely can be written to push messages around to subscribers. | 18:24 |
| *** jmeredit has joined #openstack-dev | 18:26 | |
| *** blamar_ has joined #openstack-dev | 18:43 | |
| comstud | vishy around? | 18:52 |
| vishy | comstud: aye | 18:53 |
| comstud | running into a fun flags problem | 18:53 |
| comstud | we have FLAGS.host for hostname | 18:53 |
| vishy | is it the one ant reported? | 18:53 |
| comstud | yeah | 18:54 |
| comstud | but | 18:54 |
| comstud | i know the problem | 18:54 |
| comstud | i just don't know that it's solvable easily without renaming a flag | 18:54 |
| comstud | problem is --host is matching --host_state_interval | 18:54 |
| comstud | after --host_state_interval is defined in compute | 18:54 |
| *** mszilagyi has joined #openstack-dev | 18:54 | |
| vishy | weird, matching how? | 18:55 |
| comstud | the ParseNewFlags stuff in nova/flags.py is only passing in the new flags in compute to match | 18:55 |
| comstud | 'host=' is not in the list | 18:55 |
| comstud | so when it sees --host option, it's matching the new --host_state_interval | 18:55 |
| comstud | i think because gflags allows you to shortcut flag names? | 18:55 |
| johan___ | oh, i hate that | 18:56 |
| johan___ | getopt has similar behavior | 18:57 |
| vishy | eew | 18:58 |
| vishy | wow just noticed there is a typo in gflags | 19:00 |
| vishy | er nova/flags.py | 19:00 |
| *** jmeredit has quit IRC | 19:01 | |
| vishy | comstud: yes looks like it uses getopt underneath | 19:04 |
| comstud | yeah | 19:05 |
| comstud | so | 19:05 |
| comstud | short term fix is to rename host_state_interval | 19:05 |
| comstud | or easy fix | 19:05 |
| comstud | or rename 'host' to 'hostname' | 19:05 |
| comstud | but that would potentially break configs, so | 19:06 |
| comstud | so probably the former.. although it'd be nice to change this behavior in general | 19:06 |
| vishy | comstud: | 19:09 |
| *** jmeredit has joined #openstack-dev | 19:09 | |
| vishy | does this fix it: http://pastie.org/1923091 ? | 19:09 |
| comstud | trying | 19:10 |
| comstud | seems so, off hand | 19:11 |
| comstud | ah | 19:12 |
| comstud | nevermind, no it doesn't | 19:12 |
| vishy | darn :) | 19:12 |
| comstud | hehe :) | 19:12 |
| vishy | i was hoping reparsing would work | 19:12 |
| comstud | atm, nova/flags.py is like a foreign language to me | 19:13 |
| comstud | very easy to reproduce | 19:15 |
| comstud | bin/nova-compute --host=foo --help | 19:15 |
| vishy | i'll get termie on it | 19:16 |
| vishy | - except gflags.UnrecognizedFlagError: | 19:20 |
| vishy | + except (gflags.UnrecognizedFlagError, gflags.IllegalFlagValue): | 19:20 |
| vishy | will stop the error at least... | 19:20 |
| comstud | ahh | 19:21 |
| comstud | another thing is patching getopt.long_has_args | 19:21 |
| comstud | but that's kinda ugly | 19:21 |
| comstud | maybe | 19:21 |
| * comstud looks at that | 19:22 | |
| termie | HEYO | 19:23 |
| vishy | termie: so due to getopt allowing short matching | 19:23 |
| vishy | --host is matching --host_state_interval | 19:23 |
| vishy | which is causing issues... | 19:25 |
| termie | that is only theoretically because we aren't passing in the full set of flags when we are "ParseNewFlags"ing? | 19:25 |
| vishy | that was my guess | 19:25 |
| comstud | yes | 19:25 |
| comstud | that's correct | 19:25 |
| termie | k, taking a look, seems relatively easy to fix | 19:27 |
| vishy | termie: my pastie above was a first attempt at reparsing if any flag was dirty (includes a fix for misnamed __is_dirty), but the first parse actually fails because the types were different | 19:29 |
| vishy | so i thought about catching gflags.IllegalFlagValue as well as UnrecognizedFlagError, but it seems a little hacky | 19:29 |
| termie | i don't think that matters | 19:29 |
| vishy | so --host_state_interval is an int | 19:29 |
| vishy | so bin/nova-compute --host=foo actually crashes | 19:30 |
| termie | i don't think that stuff is going to apply | 19:30 |
| termie | it won't get the wrong flag anymore | 19:30 |
| comstud | if we want to disallow partial name matching: | 19:31 |
| comstud | http://paste.openstack.org/show/1368/ | 19:31 |
| comstud | that seems to work | 19:31 |
| termie | http://pastie.org/1923197 | 19:31 |
| vishy | termie: k | 19:31 |
| termie | you guys sound like you are trying to fix a different problem | 19:32 |
| *** dprince has quit IRC | 19:32 | |
| termie | the issue is that we are only checking the new flags when we reparse, not all the flags | 19:32 |
| comstud | yeah, that's the real issue. | 19:32 |
| termie | that patch makes it check all the flags, it will no longer match host_state_interval because host will be in there | 19:32 |
| termie | anyway, try that patch? | 19:32 |
| comstud | trying | 19:32 |
| termie | and/or tell me how you are testing | 19:32 |
| termie | seems to fix it for me | 19:33 |
| termie | ./bin/nova-compute --host=foo --help crashes before it | 19:33 |
| termie | but not after | 19:33 |
| comstud | seems to | 19:34 |
| comstud | looks good to me | 19:35 |
| vishy | yes | 19:36 |
| vishy | def ClearDirty(self): | 19:36 |
| vishy | - self.__dict__['__is_dirty'] = [] | 19:36 |
| vishy | + self.__dict__['__dirty'] = [] | 19:36 |
| comstud | yeah, got that in here too | 19:36 |
| vishy | that fix cuts out about half of the reparses | 19:36 |
| termie | i'll add that in and shoot a merge prop, or should i just send my changes? | 19:36 |
| comstud | go for it | 19:36 |
| comstud | you get the bug #? | 19:36 |
| termie | nope | 19:37 |
| jk0 | lp784743 | 19:37 |
| termie | vish is the whole patch from http://pastie.org/1923091 necessary? | 19:37 |
| pyhole | LP Bug #784743 in OpenStack Compute (nova): "IllegalFlagValue: flag --host_state_interval" [Status: New, Assignee: None] https://bugs.launchpad.net/bugs/784743 | 19:37 |
| uvirtbot | Launchpad bug 784743 in nova "IllegalFlagValue: flag --host_state_interval" [Undecided,New] | 19:37 |
| uvirtbot | Launchpad bug 784743 in nova "IllegalFlagValue: flag --host_state_interval" [Undecided,New] https://launchpad.net/bugs/784743 | 19:37 |
| comstud | there ya go | 19:37 |
| termie | vishy: or just the bottom part? | 19:37 |
| comstud | just the bottom part is what I have | 19:38 |
| termie | vishy: i don't understand the purpose of the top part | 19:38 |
| comstud | it was another attempt at fixing it | 19:39 |
| comstud | that didn't work | 19:39 |
| vishy | termie: no just the is_dirty part | 19:39 |
| comstud | :) | 19:39 |
| termie | k, running tests then pushing | 19:39 |
| vishy | the otehr stuff was just me trying to get it to reparse if it matched part of a dirty flag | 19:39 |
| vishy | i guess we should probably add a test for this as well | 19:44 |
| termie | https://code.launchpad.net/~termie/nova/flags_reparse/+merge/61468 | 19:53 |
| *** jmeredit has quit IRC | 20:02 | |
| *** jmeredit has joined #openstack-dev | 20:03 | |
| vishy | termie: http://pastie.org/1923368 | 20:04 |
| *** dprince has joined #openstack-dev | 20:05 | |
| termie | pushedd | 20:17 |
| termie | upgedating differator | 20:17 |
| comstud | thnx | 20:22 |
| termie | mtaylor: mist01 (that random ip address) is back up now | 20:23 |
| vishy | comstud: we added a test | 20:23 |
| termie | mtaylor: and if you could give me access to look at the job configs, my user on jenkins is termie | 20:26 |
| *** Tv has joined #openstack-dev | 20:26 | |
| termie | mtaylor: and access info to the servers | 20:28 |
| *** dprince has quit IRC | 20:30 | |
| comstud | i'm not understanding how that test passes, though | 20:39 |
| comstud | particularly this line: | 20:39 |
| comstud | 101 self.assert_(self.global_FLAGS.duplicate_answer_long, 60) | 20:39 |
| ironcamel | vishy: you still there? | 20:39 |
| ironcamel | vishy: i think you may be able to help me. i keep getting cloud-setup: failed 15/30: up 28.84. request failed | 20:39 |
| ironcamel | in the cosole.log file, of a vm instance | 20:40 |
| ironcamel | cloud-setup: failed 15/30: up 28.84. request failed | 20:40 |
| ironcamel | wget: server returned error: HTTP/1.1 500 Internal Server Error | 20:40 |
| comstud | ah | 20:40 |
| ironcamel | know where cloud-setup script is? | 20:40 |
| vishy | comstud: the first parse actually matches the long version | 20:40 |
| *** johnpur has quit IRC | 20:40 | |
| comstud | yeah | 20:41 |
| comstud | got it | 20:41 |
| vishy | comstud: but the reparse fixes it | 20:41 |
| comstud | would have been more obvious if the test checked for '60' | 20:41 |
| comstud | for me | 20:41 |
| comstud | :) | 20:41 |
| comstud | but yeah, okay | 20:41 |
| comstud | got it. | 20:41 |
| vishy | comstud: doesn't it? | 20:41 |
| comstud | the string 60, i mean | 20:42 |
| comstud | since the flag is a string | 20:42 |
| vishy | oh that line is a typo | 20:42 |
| vishy | it is supposed to be the string | 20:43 |
| vishy | it is passing because it is assert_ instead of assertEqual | 20:43 |
| comstud | aha. | 20:43 |
| comstud | duh | 20:44 |
| comstud | updated comment | 20:45 |
| vishy | ironcamul: yes that means that metadata is failing | 20:46 |
| vishy | ironcamel: ^^ | 20:46 |
| ironcamel | vishy: any suggestions on how to track this down? | 20:46 |
| vishy | ironcamel: you should check nova-api and see if there is an error message, it is returning 500 | 20:46 |
| vishy | if not you might need to add a log.exception somewere | 20:47 |
| ironcamel | nova-api is fine | 20:47 |
| ironcamel | we are adding support for using arbitrary glance servers | 20:47 |
| ironcamel | by passing an imageRef when you create a server | 20:47 |
| ironcamel | we got it to the point that the server is created with the image from the external image server | 20:48 |
| ironcamel | but networking doesn't seem to be set up | 20:48 |
| ironcamel | creating instances via the default glance server works fine | 20:48 |
| ironcamel | starting DHCP forEthernet interface eth0 [ ^[[1;32mOK^[[0;39m ] | 20:48 |
| ironcamel | cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id | 20:48 |
| ironcamel | wget: server returned error: HTTP/1.1 500 Internal Server Error | 20:48 |
| ironcamel | ^^^ is what is in console.log of the vm | 20:49 |
| uvirtbot | ironcamel: Error: "^^" is not a valid command. | 20:49 |
| *** agarwalla has quit IRC | 20:49 | |
| vishy | ironcamel: i understand | 20:50 |
| vishy | nova-api is what responds to metadata | 20:50 |
| vishy | if you don't see error message | 20:51 |
| vishy | then try adding the following | 20:51 |
| vishy | to nova/api/ec2/metadatarequesthandler.py | 20:51 |
| vishy | on line 74 | 20:51 |
| vishy | try: | 20:51 |
| ironcamel | ec2? that seems strange, since i am working mostly inside the openstack api | 20:53 |
| ironcamel | but i will take your word for it :) | 20:53 |
| vishy | ironcamel: http://pastie.org/1923677 | 20:53 |
| vishy | you should get a log in nova-api with that patch in with the exception that is causing metadata to fail | 20:54 |
| ironcamel | vishy: i will try that. thanks! | 20:54 |
| vishy | (you will have to restart nova-api after patching) | 20:55 |
| *** heckj has joined #openstack-dev | 20:57 | |
| ironcamel | vishy: your patch was right on target. shows us exactly what is wrong. | 21:02 |
| vishy | cool | 21:09 |
| vishy | we should probably have that patch in there actually | 21:09 |
| *** lorin1 has quit IRC | 21:10 | |
| *** markvoelker has joined #openstack-dev | 21:11 | |
| ironcamel | vishy: ok, we will keep that in there | 21:12 |
| vishy | i'm going to propose it | 21:14 |
| comstud | interesting merge failure on that flags fix | 21:15 |
| termie | yeah | 21:15 |
| termie | we were just talking about it | 21:15 |
| *** bcwaldon has quit IRC | 21:21 | |
| ironcamel | vishy: do you have an opinion about this patch http://pastie.org/1923818 | 21:24 |
| *** jmeredit has quit IRC | 21:26 | |
| ironcamel | the reason for that patch is that in osapi, we want to support image_id's that are not ints. we are going to support hrefs. | 21:27 |
| *** hub_cap has quit IRC | 21:33 | |
| *** salv-orlando has joined #openstack-dev | 21:35 | |
| vishy | ironcamel: hmm | 21:37 |
| throughnothing | its not an ideal patch, but i dont think its very ideal for all servers to talk to the ec2 api even if they were never created using ec2? | 21:38 |
| vishy | ironcamel: well metadata has to be set somehow | 21:38 |
| throughnothing | yeah | 21:39 |
| vishy | ironcamel: seems like that needs to be changed inside of image_ec2_id | 21:39 |
| vishy | throughnothing: images will be switched to uuids and/or url + uuid anyway | 21:39 |
| vishy | throughnothing: so there will need to be some translation in ec2 to make it work | 21:40 |
| vishy | ultimately ec2 will just have to provide some mapping | 21:41 |
| vishy | to ec2 style ids | 21:41 |
| throughnothing | ok | 21:42 |
| throughnothing | vishy: so is this a little better: http://pastie.org/1923923 | 21:43 |
| throughnothing | essentially the same logic just moved to that function | 21:43 |
| vishy | throughnothing: I'm not sure i like the ec2 metadata service ever returning anything but an ec2_id | 21:45 |
| throughnothing | currently what happens if you pass in an osapi/glance id ? | 21:46 |
| vishy | it converts it | 21:46 |
| vishy | if glance is not using ids | 21:46 |
| vishy | then we need to maintain a mapping in ec2_api of glance ids to ec2 ids | 21:46 |
| vishy | or else ec2_api is broken | 21:47 |
| ironcamel | vishy: mapping how? in the db? | 21:48 |
| vishy | if this is just for an internal hack and you never want to support ec2 than the above is fine | 21:48 |
| vishy | ironcamel: yes we'll need to have a db mapping | 21:48 |
| ironcamel | nope, this is a global hack for everyone to enjoy | 21:48 |
| vishy | ironcamel: this is something that has to be done anyway, because we are switching to UUIDs for vms and volumes as well | 21:49 |
| vishy | there is even a blueprint about it :) | 21:49 |
| vishy | https://blueprints.launchpad.net/nova/+spec/ec2-id-compatibilty | 21:49 |
| *** jmeredit has joined #openstack-dev | 21:49 | |
| ironcamel | vishy: i don't think this breaks the ec2 api. the patch behaves as before it the image_id is an int | 21:49 |
| ironcamel | if it is an osapi style href, then it would behave differently. right? | 21:50 |
| vishy | well sure, but basically it means that images aren't compatible accross apis | 21:50 |
| vishy | my guess is that euca-describe-images would be broken if you had one of these images in | 21:51 |
| vishy | unless you are only changing the image_id in the instance record | 21:51 |
| ironcamel | we are changing it in the db as well | 21:51 |
| vishy | but you aren't changing it in glance? | 21:51 |
| vishy | ironcamel: is this for improvements to snapshotting? | 21:52 |
| ironcamel | in glance? our changes allow glance to support both id's and hrefs | 21:52 |
| vishy | ironcamel: right, so if i request all images from glance, will some come back with hrefs? | 21:52 |
| ironcamel | vishy: no, this is to support images from arbitrary glance srevers | 21:52 |
| ironcamel | vishy: yes | 21:52 |
| throughnothing | our changes are trying to allow you to create a server using the osapi using an HREF to a glance server that is not the default configured glance server | 21:52 |
| jk0 | vishy: mind taking another quick peak at https://code.launchpad.net/~johannes.erdfelt/nova/lp784134/+merge/61282 ? | 21:52 |
| vishy | throughnothing: ah i see | 21:53 |
| vishy | so it only changes it in the instance record | 21:53 |
| vishy | hmm perhaps we should make metadata just return a dummy image id... | 21:53 |
| vishy | i'm concerned that this will break some things but I don't really see another method until we have a mapping | 21:53 |
| throughnothing | that actually sounds fine to me | 21:53 |
| throughnothing | returning a dummy id | 21:54 |
| vishy | maybe have it return ami-FFFFFFFF ? | 21:54 |
| throughnothing | but what is the condition on which we return that? | 21:54 |
| vishy | actually it is small | 21:54 |
| vishy | ami-00000000 | 21:54 |
| vishy | is maybe better | 21:54 |
| vishy | i guess if it can't parse it | 21:54 |
| throughnothing | ok, so the same ValueError logic should be ok for now | 21:55 |
| ironcamel | that could work | 21:55 |
| vishy | yeah, put a todo in there that it can come out when we have mapping in place | 21:55 |
| vishy | so if you launch from an external server through the ec2 api it will create a mapping | 21:55 |
| throughnothing | so the only diff here is rather than returning the raw 'image_id' if it cant parse, then we return something at least formatted like an ec2id | 21:56 |
| vishy | throughnothing: i think that is the most compatible solution | 21:56 |
| throughnothing | vishy: sounds good to me, thanks for your help! | 21:56 |
| vishy | it means bundle-image will probably be broken | 21:56 |
| ironcamel | sorry, what does bundle-image do? | 21:57 |
| *** bcwaldon has joined #openstack-dev | 22:01 | |
| vishy | it is the way of creating a new image from a running instance | 22:01 |
| vishy | it usually checks its parent ami's | 22:02 |
| throughnothing | ah | 22:02 |
| throughnothing | so it'll only be broken for instances using the new href support | 22:02 |
| vishy | ironcamel: does your code handle ami style images? | 22:02 |
| ironcamel | vishy: we are testing on ami style images | 22:03 |
| ironcamel | on libvirt | 22:03 |
| vishy | how does it handle the separate kernel and ramdisk from another server? | 22:04 |
| ironcamel | and it seems to be working | 22:04 |
| vishy | are ramdisk_id and kernel_id still forced to be ints? | 22:04 |
| ironcamel | that is a good question | 22:04 |
| ironcamel | i think we left them as ints | 22:05 |
| vishy | and if so, is there some logic somewhere to make sure they come from the other server? | 22:05 |
| throughnothing | vishy: kernel and ramdisks are ints | 22:05 |
| ironcamel | because the glance_client instance is created based on the imageRef | 22:05 |
| throughnothing | they are assumed (and for now required) to be on the same glance server | 22:05 |
| vishy | ah ok | 22:05 |
| vishy | cool, sounds like a neat patch | 22:05 |
| ironcamel | and it knows how to handle ramdisk ids | 22:05 |
| throughnothing | vishy: its finally working today with that last ec2 patch :) | 22:06 |
| ironcamel | vishy: we hope so. since you feel that way, i think you should get to review it :) | 22:06 |
| vishy | yay! | 22:06 |
| vishy | :p | 22:06 |
| *** salv-orlando has quit IRC | 22:14 | |
| *** ameade has quit IRC | 22:15 | |
| *** jmeredit has quit IRC | 22:23 | |
| *** blamar__ has quit IRC | 22:40 | |
| *** markvoelker has left #openstack-dev | 22:42 | |
| *** jkoelker has quit IRC | 23:01 | |
| *** cp16net has quit IRC | 23:15 | |
| *** rnirmal has quit IRC | 23:21 | |
| *** antonyy has quit IRC | 23:22 | |
| *** mszilagyi has quit IRC | 23:24 | |
| *** westmaas_ has joined #openstack-dev | 23:27 | |
| *** cweidenk1ller has joined #openstack-dev | 23:28 | |
| *** blamar_ has quit IRC | 23:32 | |
| *** cweidenkeller has quit IRC | 23:32 | |
| *** westmaas has quit IRC | 23:32 | |
| *** Tv has quit IRC | 23:33 | |
| *** blamar_ has joined #openstack-dev | 23:40 | |
| *** heckj has quit IRC | 23:47 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!