Wednesday, 2015-04-29

*** peristeri has quit IRC00:00
*** tsekiyama has quit IRC00:18
*** david-lyle has quit IRC00:32
*** spzala has joined #openstack-glance01:20
*** david-lyle has joined #openstack-glance01:20
*** annashen has joined #openstack-glance01:39
*** annashen has quit IRC01:42
*** dims_ is now known as dims01:52
*** harlowja is now known as harlowja_away02:03
*** david-lyle has quit IRC02:30
*** david-lyle has joined #openstack-glance02:31
*** ayoung has joined #openstack-glance02:36
*** david-lyle has quit IRC02:36
openstackgerritYongfeng Du proposed openstack/glance-specs: Clarify the "add" operation behavior  https://review.openstack.org/17849103:11
*** david-lyle has joined #openstack-glance03:32
*** changbl has joined #openstack-glance03:33
*** wayne__ has quit IRC03:43
*** sudipto has joined #openstack-glance03:43
*** sudipto has quit IRC05:04
*** nlevinki has joined #openstack-glance05:25
*** ajayaa has joined #openstack-glance05:38
*** annashen has joined #openstack-glance05:45
*** abhishekk has joined #openstack-glance05:48
*** tshefi has joined #openstack-glance05:56
*** annashen has quit IRC05:57
*** sgotliv has quit IRC06:01
openstackgerritOpenStack Proposal Bot proposed openstack/glance: Imported Translations from Transifex  https://review.openstack.org/17663306:01
*** Longgeek has joined #openstack-glance06:03
*** sudipto has joined #openstack-glance06:06
*** Longgeek has quit IRC06:08
*** pcaruana has joined #openstack-glance06:09
*** groen692 has joined #openstack-glance06:10
*** zul has joined #openstack-glance06:17
*** pkoniszewski has joined #openstack-glance06:19
*** zul has quit IRC06:24
*** zul has joined #openstack-glance06:25
*** sudipto has quit IRC06:45
*** sudipto has joined #openstack-glance06:50
*** Longgeek has joined #openstack-glance06:56
*** Longgeek has quit IRC07:02
*** Longgeek has joined #openstack-glance07:06
*** aarefiev has quit IRC07:24
*** bfic has joined #openstack-glance07:25
*** zul has quit IRC07:32
*** zul has joined #openstack-glance07:32
*** sgotliv has joined #openstack-glance07:34
*** aarefiev has joined #openstack-glance07:40
*** sgotliv has quit IRC07:57
*** jistr has joined #openstack-glance07:58
*** Longgeek has quit IRC08:05
*** zul has quit IRC08:09
*** dims has quit IRC08:10
*** zul has joined #openstack-glance08:25
openstackgerritKamil Rykowski proposed openstack/python-glanceclient: Extend unittests coverage for v2 tasks module  https://review.openstack.org/16543608:29
openstackgerritKamil Rykowski proposed openstack/python-glanceclient: Use assertIn instead of assertTrue in tests  https://review.openstack.org/16537908:30
*** sgotliv has joined #openstack-glance08:53
*** bfic has quit IRC08:54
*** annashen has joined #openstack-glance08:57
*** MattMan has quit IRC09:00
*** MattMan has joined #openstack-glance09:01
*** annashen has quit IRC09:02
*** bfic has joined #openstack-glance09:04
*** bfic has quit IRC09:04
*** e0ne has joined #openstack-glance09:06
*** Longgeek has joined #openstack-glance09:06
*** e0ne is now known as e0ne_09:06
*** Longgeek has quit IRC09:12
*** Longgeek has joined #openstack-glance09:13
*** e0ne_ is now known as e0ne09:14
*** zul has quit IRC09:27
*** krykowski has joined #openstack-glance09:35
*** sudipto has quit IRC09:36
openstackgerritKamil Rykowski proposed openstack/glance: Image member name validation  https://review.openstack.org/17541509:43
*** aswadr has joined #openstack-glance09:44
*** zul has joined #openstack-glance09:51
*** dims has joined #openstack-glance09:51
*** sudipto has joined #openstack-glance09:52
*** sgotliv has quit IRC09:55
*** ajayaa has quit IRC09:57
*** bfic has joined #openstack-glance10:02
*** bfic has quit IRC10:02
*** e0ne is now known as e0ne_10:03
*** e0ne_ is now known as e0ne10:05
*** mtreinish has quit IRC10:07
*** ajayaa has joined #openstack-glance10:10
*** sgotliv has joined #openstack-glance10:13
*** zul has quit IRC10:19
*** arif-ali has quit IRC10:20
*** bfic has joined #openstack-glance10:56
*** bfic has quit IRC10:57
*** bfic has joined #openstack-glance10:58
*** krykowski has quit IRC11:03
*** krykowski_ has joined #openstack-glance11:04
*** dims has quit IRC11:18
*** sudipto has quit IRC11:20
*** e0ne is now known as e0ne_11:30
*** ajayaa has quit IRC11:30
*** aix has joined #openstack-glance11:35
*** sudipto has joined #openstack-glance11:35
*** krykowski_ has quit IRC11:38
*** bfic has quit IRC11:40
*** arif-ali has joined #openstack-glance11:40
*** cdelatte has joined #openstack-glance11:42
*** delattec has joined #openstack-glance11:42
*** ajayaa has joined #openstack-glance11:42
*** delatte has joined #openstack-glance11:44
*** krykowski has joined #openstack-glance11:45
*** cdelatte has quit IRC11:47
*** delattec has quit IRC11:47
*** cdelatte has joined #openstack-glance11:47
*** delatte has quit IRC11:52
*** cdelatte has quit IRC11:52
*** haomaiwa_ has quit IRC11:52
*** zul has joined #openstack-glance11:53
*** dims has joined #openstack-glance11:56
*** bfic has joined #openstack-glance12:01
*** krykowski has quit IRC12:03
*** krykowski has joined #openstack-glance12:03
*** openstackgerrit has quit IRC12:07
*** openstackgerrit has joined #openstack-glance12:07
*** krykowski has quit IRC12:08
openstackgerritMerged openstack/glance_store: Drop use of 'oslo' namespace package  https://review.openstack.org/17833612:10
*** sudipto has quit IRC12:10
*** bfic has quit IRC12:26
*** e0ne_ is now known as e0ne12:26
*** EmilienM is now known as EmilienM|afk12:41
*** ajayaa has quit IRC12:41
*** raildo has joined #openstack-glance12:41
*** lifeless has quit IRC12:43
*** rodrigods has joined #openstack-glance12:43
*** chlong has joined #openstack-glance12:44
*** krykowski has joined #openstack-glance12:45
*** krykowski has quit IRC12:52
*** e0ne is now known as e0ne_12:54
*** bfic has joined #openstack-glance12:59
*** pdb_ has quit IRC13:00
*** pdb_ has joined #openstack-glance13:00
*** annegentle has joined #openstack-glance13:03
*** julim has joined #openstack-glance13:05
*** spzala has quit IRC13:05
*** krykowski has joined #openstack-glance13:06
*** e0ne_ is now known as e0ne13:06
*** julim has quit IRC13:06
*** julim has joined #openstack-glance13:08
openstackgerritCindy Pallares proposed openstack/glance-specs: Complete refactor to glance-store's API  https://review.openstack.org/12655013:13
*** julim has quit IRC13:18
*** peristeri has joined #openstack-glance13:25
*** zeddii has quit IRC13:27
*** ajayaa has joined #openstack-glance13:34
*** zeddii has joined #openstack-glance13:36
*** EmilienM|afk is now known as EmilienM13:37
*** stpierre has joined #openstack-glance13:37
*** e0ne is now known as e0ne_13:38
*** lifeless has joined #openstack-glance13:38
*** e0ne_ is now known as e0ne13:39
*** abhishekk has quit IRC13:42
*** mtreinish has joined #openstack-glance13:54
*** zz_jgrimm is now known as jgrimm14:01
*** mtreinish has quit IRC14:05
*** bfic has quit IRC14:06
*** ericpeterson has joined #openstack-glance14:08
*** cbits has joined #openstack-glance14:08
*** cbits has left #openstack-glance14:09
*** sigmavirus24_awa is now known as sigmavirus2414:09
*** haomaiwang has joined #openstack-glance14:10
*** pkoniszewski has quit IRC14:11
*** annegentle has quit IRC14:12
*** annegentle has joined #openstack-glance14:17
*** tshefi has quit IRC14:21
*** asrangne has joined #openstack-glance14:26
*** aswadr has quit IRC14:28
*** julim has joined #openstack-glance14:31
openstackgerritInessa Vasilevskaya proposed openstack/glance: Fix HTTP 500 on NotAuthenticated in registry  https://review.openstack.org/17869714:34
*** dims has quit IRC14:42
*** bkopilov has quit IRC14:42
*** dims has joined #openstack-glance14:42
*** vijendar has joined #openstack-glance14:42
*** mtreinish has joined #openstack-glance14:45
*** bkopilov has joined #openstack-glance14:45
*** marcusvrn_ has joined #openstack-glance14:49
*** mtreinish has quit IRC14:49
*** nlevinki has quit IRC14:50
*** ajayaa has quit IRC14:52
*** pkoniszewski has joined #openstack-glance14:53
*** peristeri has quit IRC15:04
*** ajayaa has joined #openstack-glance15:05
*** peristeri has joined #openstack-glance15:33
*** julim has quit IRC15:43
*** mtreinish has joined #openstack-glance15:45
*** groen692 has quit IRC15:51
*** annegentle has quit IRC15:53
*** asrangne has quit IRC15:53
*** sudipto has joined #openstack-glance16:11
*** jistr has quit IRC16:11
*** zul has quit IRC16:15
*** julim has joined #openstack-glance16:17
*** chlong has quit IRC16:19
*** pcaruana has quit IRC16:22
*** sudipto has quit IRC16:44
*** annegentle has joined #openstack-glance16:54
*** marcusvrn_ is now known as marcusvrn16:55
*** annegentle has quit IRC17:01
*** pkoniszewski has quit IRC17:01
*** SpamapS has joined #openstack-glance17:01
SpamapSHello everybody. I'm poking at some glance v2 stuff and wondering if anybody can point me to where glance v2 is tested in the gate.17:02
*** annegentle has joined #openstack-glance17:03
*** greghaynes has joined #openstack-glance17:05
*** aix has quit IRC17:10
*** sgotliv has quit IRC17:13
*** marcusvrn has quit IRC17:19
SpamapSmtreinish: so I'm looking at tempests's v2 API tests and I'm a bit confused as to how this differs from the way RAX's glance v2 works..17:23
SpamapSmtreinish: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/services/image/v2/json/image_client.py#n13617:23
SpamapSmtreinish: It's my understanding RAX forbids that upload17:24
greghaynesSpamapS: I have a rax test setup I can give you some specifics17:24
* greghaynes makes an image17:24
mtreinishSpamapS: that could be, I've never tried running it against rax17:24
SpamapSgreghaynes: crazy idea, run just those tempest tests against rax?17:24
SpamapSthe task code is in trunk, so we should probably have tests for it if we're not going to kick it out.17:25
greghayneshah, I dont have any kind of tempest setup, not sure how much work that is to get going?17:25
SpamapSgreghaynes: you _might_ be able to convince mtreinish to help you get one up quickly. :)17:25
mtreinishgreghaynes: http://docs.openstack.org/developer/tempest/configuration.html#tempest-configuration is a start17:25
SpamapSfaster than I can get my RAX account activated surely. :)17:25
greghaynesyes, that requires a phone call17:25
SpamapSphone?17:25
SpamapSwhats that?17:25
mtreinishbut you don't need to do much to setup just running the glance api tests serially17:25
* SpamapS is just a caveman17:25
greghaynesSpamapS: exactly17:25
greghaynesmtreinish: awesome17:26
mtreinishgreghaynes, SpamapS: I'd do it myself but my free credit has like a week left on it17:26
SpamapSnoooooo17:26
SpamapSshould be infinite for openstack-devs ;)17:26
mtreinishSpamapS: yes, I agree fully :)17:26
greghaynesI still have to battle with getting the rax expense report to go through :(17:27
mtreinishgreghaynes: to run it you should just need to put the keystone endpoint, a set of credentials to run the tests with, and probably set the catalog name for glance (because I doubt they use the default tempest value)17:28
mtreinishoh, and probably a random http address to use as an "image" for the tests17:29
SpamapSgreghaynes: while you're at it, if you can try and run the tests from here against rax, I can help you too: https://review.openstack.org/17877417:32
greghaynesok, ill mess with that now17:32
greghaynesheres what I get from glanceclient http://paste.ubuntu.com/10942747/17:32
SpamapSgreghaynes: 'tox -efunctional test_image' with OS_xxx set properly should do it17:32
SpamapSgreghaynes: v1 should obviously fail... :)17:32
mtreinishgreghaynes: heh, so they're rate limiting you?17:32
greghayneshrm, I wonder if that X-Auth-Token is a problem in a paste ;17:32
greghaynesmtreinish: they say that17:33
SpamapSquota for images?17:33
SpamapSgreghaynes: yeah you should invalidate that token17:33
greghaynesI think its not actually a rate limit though17:33
greghaynesthey say that persistently17:33
greghayneswhich might be how they "disable" support for that, setting a rate of 0 somewhere17:34
mtreinishgreghaynes: heh, that makes it super clear :)17:34
SpamapSgreghaynes: you may have to ask for image upload?17:35
mtreinishthat's like neutron 503s when you try to create a tenant network in a vlan provider network config17:35
SpamapSmtreinish: no, neutron knows: you're doing it wrong17:35
SpamapS503 is just the way to tell you that you doing it wrong, broke it's brain17:35
SpamapSits17:35
greghaynesSpamapS: I did, and they confirmed I have it. I think that means I can task-create17:35
SpamapSgreghaynes: ok, so this sort of proves it doesn't it? :)17:36
SpamapSgreghaynes: can you try the shade tests?17:36
mtreinishSpamapS: heh, I know: service unavailable == you're doing it wrong17:36
SpamapSWhat I'm seeing is that when I run that against devstack, it fails because of a missing field, import_from_format17:36
greghaynesSpamapS: is there a patch I need to make it hit a remote cloud?17:37
SpamapSmtreinish: It's the REST equivalent of "I'm not even going to respond to that."17:37
SpamapSgreghaynes: no it respects OS_*17:37
greghaynesoh, neat17:37
SpamapSI think...17:37
greghaynesI could see how that would work via the os-cloud-cfg magic17:38
mtreinishSpamapS: or you could just give a 4xx response with a real error message17:38
SpamapSmtreinish: politeness is overrated.17:38
mtreinishSpamapS: definitely :)17:38
SpamapSmtreinish: http://cdn.meme.am/instances/500x/61733074.jpg17:40
mtreinishSpamapS: lol17:41
greghaynespython dep installing for tests is slow...17:41
greghaynesyouall must depend on either lxml or pyopenssl17:42
SpamapSyep17:42
SpamapSboth17:42
SpamapStakes my hpcloud instance 8 minutes to build the venv17:42
mtreinishgreghaynes: if you're using any of the python-*clients both get pulled in17:43
greghaynesawesome!17:43
SpamapSso if I get around the import_from_format issue by adding that field in shade, the next problem I have is that glance-api wants to fetch from http:// and I'm just giving it a swift container name.17:43
greghaynesSpamapS: ok, tests passed with OS_* set to rax.. I suspect it didnt respect them17:46
greghaynesactually, your tests are mostly fake clients?17:46
SpamapSgreghaynes: -e functional17:47
SpamapSgreghaynes: 'tox -efunctional test_image'17:47
SpamapSgreghaynes: I don't care about the unit tests. :)17:47
greghaynesok, well, dep installing again then17:48
SpamapSgreghaynes: :(17:48
greghayneshehe17:48
greghaynessays passed...17:50
greghaynesim not sure it did anything17:50
greghaynesah, the arg messed it up17:51
*** e0ne has quit IRC17:57
SpamapSgreghaynes: the arg should limit to just test_image18:00
*** harlowja_away is now known as harlowja18:00
SpamapSgreghaynes: oh, maybe it has to be tox -efunctional ?18:00
SpamapSgreghaynes: ?18:03
mtreinishSpamapS: sigh I'm trying to run tempest against rax, it's exploding on the date format in the tokens...18:03
mtreinishiterop ftw18:03
*** krykowski has quit IRC18:04
*** dhellmann has quit IRC18:05
mtreinishugh, there is are microseconds in the response...18:09
SpamapSmtreinish: microseconds are microuseful18:12
* harlowja didn't do it, lol18:15
mtreinishSpamapS, greghaynes: http://paste.openstack.org/show/212486/ so I get the 413 on the image create18:15
greghaynessounds right18:16
*** harlowja has quit IRC18:16
*** harlowja has joined #openstack-glance18:16
mtreinishSpamapS: heh, especially for an expiration time, you need as much precision as possible :)18:16
SpamapSalright, so perhaps whats needed is a devstack-gate that has that configured, and tests that use tasks/imports18:16
*** harlowja has quit IRC18:17
mtreinishSpamapS: configure devstack to 413 on these requests? I would assume the tasks/imports stuff would just work if we added tests for them18:18
SpamapSmtreinish: well I'm kind of the mindset that since the point of the tasks is that some people don't want glance-api to be in the data-plane .... testing that tasks work when 413 happens seems like the right way to go?18:20
*** harlowja has joined #openstack-glance18:20
SpamapSmtreinish: but isolating it into the two things is fine too :-P18:20
mtreinishSpamapS: oh, I don't care one way or the other, I was just confused on exactly you were proposing. I do feel that if we want glance to behave that way it should be the glance default18:23
mtreinishbecause our job config permutation count is high enough as it is :)18:23
*** harlowja has quit IRC18:29
*** harlowja has joined #openstack-glance18:29
*** e0ne has joined #openstack-glance18:31
*** e0ne is now known as e0ne_18:31
SpamapSmtreinish: good point18:31
*** e0ne_ is now known as e0ne18:32
*** e0ne is now known as e0ne_18:32
*** e0ne_ is now known as e0ne18:32
*** e0ne has quit IRC18:32
*** annegentle has quit IRC18:36
*** e0ne has joined #openstack-glance18:48
*** annegentle has joined #openstack-glance18:49
*** annegentle has quit IRC18:52
*** annegentle has joined #openstack-glance18:53
*** annashen has joined #openstack-glance18:55
*** annashen has quit IRC19:01
*** annashen has joined #openstack-glance19:02
*** ericpeterson has quit IRC19:06
*** dhellmann has joined #openstack-glance19:06
*** annashen has quit IRC19:06
*** jaypipes has quit IRC19:06
openstackgerritOpenStack Proposal Bot proposed openstack/glance: Updated from global requirements  https://review.openstack.org/17682719:16
openstackgerritOpenStack Proposal Bot proposed openstack/python-glanceclient: Updated from global requirements  https://review.openstack.org/15558119:23
*** jaypipes has joined #openstack-glance19:23
*** ajayaa has quit IRC19:23
*** david-lyle has quit IRC19:44
*** spzala has joined #openstack-glance19:44
*** david-lyle has joined #openstack-glance19:44
*** ajayaa has joined #openstack-glance19:48
*** ericpeterson has joined #openstack-glance19:53
*** arif-ali has quit IRC19:56
harlowjais there going to be a session on taskflow + glance in the summit, just wondering?20:11
* harlowja should probably attend that one if so :)20:13
*** annegentle has quit IRC20:15
*** annegentle has joined #openstack-glance20:23
*** e0ne has quit IRC20:26
SpamapSharlowja: is this thing on?20:29
* SpamapS taps20:29
harlowjahollaa20:29
harlowjaSpamapS sup dawg20:31
lifelessSpamapS: HOWWWDEEEEEEE20:34
SpamapSyou are both wonderful, but I'm more wondering where the regulars are. ;)20:35
SpamapSI have glancey questiones20:35
*** e0ne has joined #openstack-glance20:36
lifelessSpamapS: don't ask to ask :)20:38
SpamapSlifeless: ^^^ quite a while ago, asking if anybody here knows if glance v2 is tested in the gate20:38
SpamapSlifeless: and the ensuing discovery has been... interesting. :)20:38
SpamapSbasically the glance v2 in the gate is not hte glance v2 in RAX20:38
lifelesshave you asked treinish or sdague ?20:38
SpamapSlifeless: treinish wrote the v2 tests :)20:39
SpamapSand yes20:39
lifelessgreat20:39
lifelessI'm surprised they didn't know20:39
SpamapSthe deal is, those test the one where glance is still accepting PUT's of images.20:39
SpamapSwhich RAX returns 413 for20:39
SpamapSyou have to instead create an import task in RAX20:39
SpamapSand that code exists in upstream20:39
SpamapSbut is zomg different20:39
SpamapSand currently I'm making an attempt to see if I can get it to work at all20:40
lifelessmay the force be with you :)20:41
*** e0ne has quit IRC20:44
*** mtreinish_ has joined #openstack-glance20:45
kragnizSpamapS: nikhil_k_ is the person to talk to about tasks20:47
kragniz(also rax)20:47
SpamapSkragniz: indeed I do see some reference to that name in the commits20:50
*** mtreinish has quit IRC20:51
*** mtreinish_ is now known as mtreinish20:51
SpamapScurrently getting 500's now for missing fields20:52
kragniz:/20:54
kragnizI'm afraid I don't know much about tasks20:55
SpamapSand when I add them, even more fun..20:55
SpamapSDBError: (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '), NULL, NULL, 'queued', 0, NULL, 0, 0, '19572f9748f24bab8c3ed76592f49151', 0)' at line 1") 'INSERT INTO images (created_at, updated_at, deleted_at, deleted, id, name, disk_format, container_format, size, virtual_size, status, is_public, checksum, min_disk, min_ram, owner,20:55
*** belmoreira has joined #openstack-glance20:55
SpamapSkragniz: it's ok, it's just that we have to stop merging API's without proper tests20:56
*** raildo has quit IRC20:58
kragnizoh dear...21:01
kragnizSpamapS: I agree with that21:01
*** peristeri has quit IRC21:09
sigmavirus24SpamapS: flaper87 can help with tasks too21:10
SpamapSCool21:11
SpamapSI'm still nailing down why it might be so different when talking to RAX vs. a devstack.21:12
*** ajayaa has quit IRC21:13
sigmavirus24Well then you definitely want nikhil_k_ for that21:15
sigmavirus24He's on our public cloud team that manages our glance deployment sort of21:15
sigmavirus24Also rosmaita21:15
sigmavirus24And Hemmanth but I don't see him around21:15
sigmavirus24Oh jcook may be able to help you too21:15
*** arif-ali has joined #openstack-glance21:19
*** annashen has joined #openstack-glance21:21
SpamapSso my biggest concern is that backtraces during tasks are not leading to the tasks marked as failed21:21
SpamapSthey just sit as 'processing' forever21:21
nikhil_k_sigmavirus24: SpamapS : hi21:22
SpamapSnikhil_k_: greetings21:23
nikhil_k_:-)21:23
nikhil_k_how can I help?21:23
nikhil_k_trying to scroll back21:23
nikhil_k_and read fast21:23
SpamapSnikhil_k_: So we have some logic in the shade library which works well for uploading images to RAX21:23
SpamapSnikhil_k_: Simpler to summarize here.21:23
SpamapSnikhil_k_: I'm trying to write functional tests to test this logic against devstack.21:23
SpamapSand that does not work. :-/21:24
nikhil_k_huh21:24
nikhil_k_uploading as in POST/PUT or tasks?21:24
nikhil_k_I see tasks mentioned above21:25
SpamapSnikhil_k_: http://git.openstack.org/cgit/openstack-infra/shade/tree/shade/__init__.py#n1146 is the logic21:25
SpamapSnikhil_k_: uploading as in tasks21:25
nikhil_k_ohk21:25
nikhil_k_yep, that's not upstream yet21:25
nikhil_k_issue with glance_store not completely compatible21:25
nikhil_k_and we need to copy a bunch of swift driver onto the tasks script for doing something that can be 2 liner21:26
SpamapSok, so that is RAX-private still then?21:26
SpamapSas in, that API hasn't stabilized yet?21:27
nikhil_k_I try to avoid useing that word, prefer "Rack" the other one is our stock symbol. Also anything I say here isn't related to stocks ;-)21:27
nikhil_k_But yes21:27
nikhil_k_The API is all good21:27
SpamapSExcept there's no testing of the API.. so... ;)21:27
nikhil_k_tasks allow some elements ni the body request21:28
nikhil_k_that can be deployer specific21:28
nikhil_k_if you are seeing that error then something's really wrong21:28
SpamapSYeah that's entirely fine, and as long as the task is rejected with a 4xx because you didn't have the right deployer-specific data that's good.21:28
SpamapSCurrently I get 500, or an immortal processing task.21:28
*** Longgeek has quit IRC21:29
nikhil_k_Oh I see you are worried about the DB error specifically21:29
SpamapSnikhil_k_: so I'm going to back away. If you say it doesn't work upstream, I'm not going to try and brute force around that.21:29
nikhil_k_Please do not21:29
SpamapSnikhil_k_: the DB error is at the end of a long chain of other fields that were required in devstack that weren't required for an upload to the rackspace cloud.21:30
nikhil_k_Upstream is still fragile. Hopefully I get enough time this cycle to fix this.21:30
nikhil_k_SpamapS: yep, the import script in master isn't nice yet21:30
*** julim has quit IRC21:30
nikhil_k_Basically, the validation at API level is minimal by design21:30
nikhil_k_because, these are supposed to be asynchronous21:31
SpamapSAnd that would be fine.. if they produced errors.. or were retried..21:31
nikhil_k_So, we need to tighten valiadtion in the import script for it to work nice in devstack :-) All bugs welcome!21:31
nikhil_k_Agree21:31
SpamapSnikhil_k_: that brings up another point. The async bits are realy poorly documented. It took me 2 hours of digging to figure out that glance-api just executes things immediately.21:31
SpamapSI was looking everywhere for an async worker.21:32
nikhil_k_SpamapS: even that's configurable. Depending on the type of executor implmented :)21:32
SpamapSsee 'executor implemented' ... ???21:32
nikhil_k_ohk, yes the default one is just an eventlet thread >.<21:32
nikhil_k_err21:32
nikhil_k_"executor deployed"21:32
SpamapSI have the result of git clone glance. I see no executor other than the one that gets pulled into glance-api in very mysterious ways.21:32
nikhil_k_heh21:33
nikhil_k_taskflow is pulled in21:33
*** annashen has quit IRC21:33
jcooksigmavirus24, SpamapS: hmm?21:34
sigmavirus24jcook: nevermind. nikhil_k_ helped SpamapS out21:34
sigmavirus24sorry for the ping21:34
jcooksigmavirus24: ha, procrastination ftw21:34
sigmavirus24^5 jcook21:34
jcooknikhil_k_, sigmavirus24, SpamapS: glad I could be of help ;)21:35
nikhil_k_we are too :)21:35
jcooknikhil_k_: >_>:21:35
nikhil_k_SpamapS: So, how can I really help?21:35
jcook>_>; *21:35
nikhil_k_:)21:35
nikhil_k_jcook: :P21:35
greghaynesnikhil_k_: Is there a patch stack upstream for the swift import stuff?21:36
SpamapSSo, I'll say again, I'm going to back away from this. I don't actually think the task API is the right way to go, I think interoperability trumps scalability here, and it's worth the effort to make a scalable glance upload endpoint so end-users don't have to understand how to write to all the backends....21:36
greghaynesnikhil_k_: something we could pull and test against that might resemble what rax is doing21:36
*** harlowja is now known as harlowja_away21:36
SpamapSThat, combined with the fact that this doesn't work upstream yet, and isn't tested even minimally, makes it a huge time sink that I can't afford.21:36
jcookthose statements perked my interest21:37
nikhil_k_greghaynes: I can share stuff. It's no biggie as I had shared in a mini-summit anyways :)21:37
jcook"interoperability trumps scalability"21:37
*** harlowja_away is now known as harlowja21:37
greghaynesnikhil_k_: yes please21:37
nikhil_k_SpamapS: The reason tasks were implemented is because glance cannot handle objects as well as something like swift21:38
SpamapSjcook: Indeed. We're already bending over backward to have nodepool be able to upload to hpcloud (which uses glance v1) and rackspace (which uses v2+tasks). When it lands upstream, we might have _three_ ways we have to upload... :-P21:38
mtreinishif someone could take a look at https://bugs.launchpad.net/glance/+bug/1450209 glance is spewing garbage in it's log files on every gate run21:38
openstackLaunchpad bug 1450209 in Glance "Glance API logs a never ending stream of 404 tracebacks from glance-registry " [Undecided,New]21:38
jcookSpamapS: not disagreeing about statement, just curious what we're talking about since scalability is on my radar21:39
*** stpierre has quit IRC21:39
greghaynesnot to mention the only way were figuring out how to make it work with these clouds is lots of info were getting directly from the cloud operators - making this work against an arbitraty cloud would be almost hopeless21:39
SpamapSWhereas if you can just take the swift proxy that does all this heavy PUT lifting, and put it in front of glance at the right spot, I think you'd be able to just let users PUT to /images/foo21:39
jcookSpamapS: oooh, now you are talking about a problem I think I am looking to tackle21:40
SpamapSnikhil_k_: No disrespect is meant. I completely understand why a real, heavily loaded cloud, might choose the tasks interface.21:40
SpamapSnikhil_k_: I'm more concerned for the ecosystem which will need to deal with this problem, and may not all be able to store their images in swift, at which point, the interop matrix gets _really_ tricky.21:40
jcookSpamapS: I have a cross-project summit session proposed that might interest you21:40
SpamapSjcook: yes?21:41
jcookSpamapS: what is the problem you are trying to solve? upload reliability, upload performance, download reliability, download performance, or something else21:41
SpamapSjcook: so, the only problem I want to solve is that the tasks systme that Rackspace has chosen to facilitate uploads to their clouds has a hidden interoperability problem in it as it moves upstream.21:42
jcookSpamapS: ah, perhaps not then. I have to admin that my knowledge of tasks is limited. For us, upload reliability is more important than performance right now.21:43
SpamapSjcook: well are you experiencing unreliable uploads because glance-api wont' scale?21:44
jcookSpamapS: I need to spend a little time looking at this subject before I can talk to it21:44
SpamapSThat seems like it could be a real problem.21:44
sigmavirus24mtreinish: I'll take a look at what's landed recently21:44
jcookSpamapS: I'd say unreliable in that Glance doesn't do retries well21:44
SpamapSLikely one nikhil_k_ and the Rackspace team spent quite a bit of time looking at before choosing the path they took.21:44
mtreinishsigmavirus24: thanks, I don't know when that started tbh, I just noticed it a couple of days go21:44
* mtreinish checks logstash21:45
jcookSpamapS: moving an immutable object should not result in 409 on full try or fail midstream and not retry bit that failed21:45
SpamapSjcook: indeed.21:45
sigmavirus24 mtreinish which jobs are causing this? Jobs from master or kilo?21:45
mtreinishsigmavirus24: I've seen it on master, I haven't checked kilo21:46
mtreinishsigmavirus24: looks like just on master: http://goo.gl/7hol8Y21:49
SpamapSjcook: I mean, another option which we could do is just require that all clouds have a swift API and let the image API just be a catalog of pointers to swift objects.21:49
SpamapSjcook: if that swift API happens to write to CEPH, or gluster, or sheepdog, or NFS so be it.21:50
SpamapSjcook: then your image upload reliability would be as good as swift's.21:50
jcookjcook: that seems like an implementation that could work in the context of what I had in mind21:51
SpamapSjcook: Lets make sure to connect at the summit and discuss this. :)21:51
SpamapSnikhil_k_: you too. :)21:51
*** annegentle has quit IRC21:51
jcookSpamapS: https://etherpad.openstack.org/p/liberty-caching21:52
jcookLine item 15: https://docs.google.com/spreadsheets/d/1vCTZBJKCMZ2xBhglnuK3ciKo3E8UMFo5S5lmIAYMCSE/edit#gid=82750341821:52
jcookSpamapS: that session is mostly about downloads and caching. However, if you look at reference architecture, it is very related to what you are saying here21:52
SpamapSjcook: didn't you hear? CEPH solved that. Just give everything 4x 10Gbe and sacrifice a goat to the RADOS gods.21:53
jcookSpamapS: Upload Recovery proposal is a bit terse atm, but here: https://etherpad.openstack.org/p/liberty-glance-summit-topics21:53
jcookSpamapS: lol21:53
*** ericpeterson has quit IRC21:54
SpamapSjcook: I took issue with your statement about LRU btw ;)21:55
jcookSpamapS: did you write it >_>;21:56
* jcook gives SpamapS a virtual hug.21:56
sigmavirus24mtreinish: this is interesting. I'm not seeing much that should/could be causing this21:57
*** ericpeterson has joined #openstack-glance21:57
SpamapShttp://en.wikipedia.org/wiki/Cache_algorithms#Examples hah fascinating read21:57
jcookSpamapS: oh, I see your comment. I should probably clarify that it doesn't use the column that is basically hit count21:57
SpamapSI had never heard of "the clairvoyant algorithm"21:58
SpamapSjcook: that would be LFU21:58
jcookSpamapS: I think we need something that considers both21:59
SpamapSjcook: really I think glance would benefit from one of the LRU+LFU approaches.22:00
jcookSpamapS: LRU makes sense in processor cache22:00
jcookSpamapS: +122:00
lifelesscaching what data?22:00
SpamapSjcook: LFU, IIRC, does better with higher-cost misses.22:00
jcooklifeless: images in glance22:01
jcooklifeless: goal is to improve build times22:01
lifelessjcook: on hypervisors?22:01
SpamapShonestly if you want fast builds though, CEPH, like not even joking, is the way to go.22:01
jcooklifeless: was talking about in Glance22:01
lifelessjcook: yes, I'm confused and seeking to resolve the confusion22:01
SpamapSworth the extra overhead of a dedicated storage network.22:01
jcookSpamapS: I should probably look into CEPH22:02
lifelessSpamapS: sheepdog should be equivalent22:02
jcookdownload times from glance cache are actually pretty quick22:02
lifelesssame basic principle - scaled spindles, COW22:02
SpamapSlifeless: I don't know how sheepdog works, but probably worth glancing at (no pun intended)22:02
jcookat least from my tests on rax prod22:02
lifelessjcook: so the cache is in addition to the hypervisor image cache22:02
lifelessjcook: its holding a copy of the compressed image in a glance server?22:02
jcooklifeless: yeah, glance has cache on api nodes right now22:03
jcooklifeless: it uses LRU (which actually had a bug that made it MRU until recently fixed)22:03
SpamapSdoohhhhhh22:03
openstackgerritKent Wang proposed openstack/python-glanceclient: Support add multiple images/tenants in members  https://review.openstack.org/17885722:03
lifelessjcook: so, why isn't that done via e.g. squid or varnish ?22:03
SpamapSIIRC MRU is only advisable when you have completely random access patterns22:03
lifelessjcook: e.g. dedicated memory caching tools22:03
jcooklifeless: ding ding ding, we have a winner22:03
jcooklifeless: +122:04
lifelessSpamapS: also sequential apparently..22:04
SpamapSlifeless: right because you'd at least get some hits.22:04
lifelessjcook: I mean, I can advise based on the data we have in the squid project about caching algorithms...22:04
SpamapSlifeless: broken clock problem :)22:04
lifelessjcook: but honestly just using squid would be better :)22:04
SpamapSvarnish is so hip tho22:04
lifelesssame same22:04
jcooklifeless: we are actively testing that, we have some interesting results show that glance can be ever so slightly faster but still want to move to squid22:05
SpamapSsquid == Zoolander, Varnish == Hansel22:05
lifelessjcook: if you want me to have a look over your squid conf, let me know22:05
lifelessSpamapS: oh I like22:05
SpamapShttp://2.bp.blogspot.com/-cDs50EboctY/VQCCNT6qzDI/AAAAAAAACvc/STbSETJD3rI/s1600/tumblr_nl08iwtDZf1qlvwnco2_500.gif22:05
jcooklifeless: ultimately, though I want small chunks cached on determistically selected node with random order parallel downloads22:05
lifelessSpamapS: I'll get practicing my blue velvet22:05
jcooklifeless: I should have common sync up with you22:06
SpamapSlifeless: you'll need Magnum if they rewrite varnish in Rust...22:06
lifelessSpamapS: whatshisname would never22:06
sigmavirus24mtreinish: are you still around?22:06
SpamapStrue true22:07
lifelessjcook: small chunks for scatter gather reads?22:07
lifelessjcook: I suspect you'd be better off with very optimised linear on single nodes, for everything but extreme scales - and there just redundant copies22:07
jcooklifeless: use case, build n thousand instances simultaneously off uncachedi mage22:07
lifelessjcook: multicast!22:07
lifelessjcook: I wrote a spec for that for glance22:08
lifelessspecish I should say - etherpad, before in-gerrit specs22:08
SpamapSa few past-lives ago I used multicast-rsync to deploy images. :)22:08
SpamapSworks quite well even at medium-scale of 40 nodes.22:09
jcooklifeless: maybe, I think deterministically caching in selected cache (either glance or squid vs using peer caching) and random ordering your downloads means object moves from swift on average 1 time when builds are less than segments, and chunks can be pulled in parallel from glance without wasting bandwidth or space22:09
mtreinishsigmavirus24: yep22:10
jcooklifeless: I am intrigued by your multicast option though, I should look at this22:10
* SpamapS needs to step away from the super fun caching talk for a bit. :)22:10
mtreinishsigmavirus24: sorry, yeah it looks like it's been logging that for >= 10days it goes back our full logstash history22:10
sigmavirus24mtreinish: I'm not seeing what's caused these to suddenly start popping up so much22:10
lifelessjcook: oh look https://blueprints.launchpad.net/glance/+spec/multicast-image-transfer22:10
sigmavirus24mtreinish: ah that makes more sense22:10
lifelessand the etherpad https://etherpad.openstack.org/p/glance-multicast-image-transfer22:11
lifelessI think we ended up saying that frisbee looked super interesting22:11
lifelessin terms of working, being fairly modern22:11
sigmavirus24mtreinish: most of the changes in the apparently affected areas are from 201422:11
jcooklifeless: I'll have to read through that and think about the trade offs of the different solutions. Seems to be similar use case22:12
*** vijendar has quit IRC22:12
lifelessjcook: cool22:12
sigmavirus24I'll have to dig into this more tomorrow22:13
lifelessjcook: I'm happy to be a sounding board!22:13
lifelessjcook: I'm very keen on using the right tool for a given problem :>22:13
jcooklifeless: ^_^22:14
jcooklifeless: will you be at the summit?22:14
harlowjaI'm back, had food, SpamapS nikhil_k_ poke me if u need me22:14
lifelessjcook: I shall22:14
harlowja*actually was back a little while ago, but just remembered about this talk/chat, lol22:14
jcooklifeless, SpamapS: we should sync up at summit on the caching stuff / quick build times22:15
SpamapSjcook: yes indeed22:15
mtreinishsigmavirus24: hmm, ok22:15
SpamapSjcook: you could also stick your nova-compute cache on AFS.. ;)22:16
jcookSpamapS: Andrew File System22:18
*** belmoreira has quit IRC22:18
jcook?22:18
lifelessyes, SpamapS is getting indoctrinated by infra atm22:18
mtreinishsigmavirus24: I mean it could have been going on that long, people normally don't look at the log files unless the tests fail22:18
mtreinishbut I think someone would have noticed it long before if it was introduced last year22:19
jcookyeah DFS is definitely a possibility for the HV cache. Reduces chance of cache miss22:19
sigmavirus24mtreinish: yeah that's what I'm thinking22:19
sigmavirus24so i'm going to look elsewhere22:19
sigmavirus24and try to replicate this22:19
SpamapSjcook: realistically putting it on AFS just means you're slave to AFS's caching :)22:22
SpamapSwhich is overly generic and likely won't be as optimized as a multicast solution.22:22
jcookSpamapS: yeah, I'd be curious how fast it is vs pulling from Glance / squid / multicat / w/e22:23
jcookSpamapS: Glance is actually pretty quick depending how close it is to HV, cache misses are a real problem though22:24
nikhil_k_It prolly depends on the kind of glance node22:28
nikhil_k_for a generic solution like this glance supporting bit torrent would help22:28
*** ericpeterson has quit IRC22:44
*** bkopilov has quit IRC23:10
*** bkopilov has joined #openstack-glance23:25
*** ericpeterson has joined #openstack-glance23:45
*** dims has quit IRC23:50
*** chlong has joined #openstack-glance23:50
*** ericpeterson has quit IRC23:51
*** annashen has joined #openstack-glance23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!