Thursday, 2014-01-30

*** csd has joined #openstack-swift00:00
pelusetorgomatic:  FYI with /info on policies the output for when un-named policies are used is not all that great.  Exmaples:  (a) nothing defined you will see one un-named policy (implied 0) (b) >1 unnamed you will only see one blank policy as there's nothing unique when the dict item is added.  Think its worth making these cases prettier?00:16
*** Midnightmyth has joined #openstack-swift00:22
peluseI could call them <unnamed-1>, <unnamed-2>, etc....00:22
*** matsuhashi has joined #openstack-swift00:26
portantenotmyname, *all*: I have been away from swift work for the last few days, mostly, is the wiki page still good for review priorities00:27
portante?00:27
gholttorgomatic: glange: Thanks guys on the ssync review; hoping to do a release within a week or so and didn't want to lag another month on ssync testing. :)00:27
*** _bluev has quit IRC00:29
peluseportante:  you know my input :)00:30
gholtDamn, I figured I'd do a trade since Sam approved my change I'd get one of his. Apparently he has a lot pending... let's see then.00:30
pelusegholt: any interest in looking at the patch to ssync to support policies? https://review.openstack.org/#/c/65347/00:31
gholtI can look that over, unfortunately my spin up time on ec is pretty high, but that's okay, I can at least see if anything pops out at me.00:32
pelusegholt:  thanks.  there's no EC in this one, its just support for multiple object rings via storage policies00:32
torgomaticgholt: the DLO patch is probably the most pressing, but also the largest, which is probably why it's been sitting so long :)00:37
gholtpeluse: Do you expect an upgrade to policies to essentialy be: shutdown all background jobs (replicators, updaters, etc.) then upgrade main servers (proxy, object, container, account) then turn back on the jobs? Or can a default policy be assumed when an server is talking to a new server during an upgrade?00:39
gholttorgomatic: Yeah, that one is begging for completion.00:39
pelusegholt: if no new policies are defined in between when the proxy and the other services are upgraded everything will work (which is the likely case).  I guess you could screw yourself up if for some reason you updated the proxy, defined policy 1 before upgrading the rest and then tried to use it00:42
pelusegholt:  I actually just added a todo item today to write up the end user docs for this and we'll be sure to cover the proper upgrade steps...00:43
gholtWell on this, and I may be reading it wrong, won't the ssync_receiver blow up if it receives a request from the sender that doesn't have a policy?00:44
pelusegholt:  so if the proxy is done first it will always add the proper default before anything is sent to the backend.  So another posible screw up in the upgrade would be to do the services first...00:45
gholtWith hundreds of hosts, everything will be done in every order you can imagine. :)00:46
pelusegholt:  good point!00:46
gholtWhich is fine if it blows up, obviously not fine if it could interpret something incorrectly, best if it can not blow up and interpret a sane default. Heh00:47
pelusegholt:  thanks, I'll add a note to the TODO item on the docs to brainstorm the upgrade proceedures which may result in some other general code changes as well as specific instructions00:48
gholtI'm making the assumption there's a concept of a default policy that corresponds what was running prior to the upgrade for all the existing data.00:48
swifterdarrellgholt: (cc pandemicsyn) portante: you guys seen this bug?  https://bugs.launchpad.net/swift/+bug/127431200:48
swifterdarrellgholt: (cc pandemicsyn) portante: early-quorum-related00:48
gholtswifterdarrell: Hadn't seen it, but I could imagine it's an issue00:50
pelusegholt:  correct.  there's 2 concepts here, a 'policy 0' which is used for access to containers that exist already and have no policy and a 'default' which is used to create new containers when no policy is specified (allowing for something other than repl for new containers)00:50
swifterdarrellgholt: cool; that 2nd suggested solution seem okay (I like it better than the first one I thought of)00:50
swifterdarrellgholt: (sorry that was supposed to end in a ?) ;)00:51
pelusegholt:  BTW 'policy 0' and the default can indeed be one and the same but they don't have to be...00:52
gholtswifterdarrell: Yeah, that or just make a subclass that everybody uses which overrides waitall. Same same00:52
*** Midnightmyth has quit IRC00:52
pelusegotta run... catch you guys later00:53
gholtpeluse: Gotcha, 0 and default are distinct concepts but may be identical in reality. Cool00:53
swifterdarrellgholt: *nod*  I haven't looked yet, but my hunch is that every user actually wants the new functionality (i.e. every caller of waitall() really wants wait-some-time-then-give-up-with-appropriate-draining)00:55
swifterdarrellgholt: obvs, if that's not true, I'd leave some way to get ahold of the old behavior00:55
swifterdarrellgholt: that's even assuming I get some time to implement the fix! heh00:55
gholtWhat's the side effect of the bug? Timeouts raised in weird places causing odd things to explode?00:57
gholtAlso, I think GreenAsyncPile was redbo's brainchild. :)00:58
swifterdarrellgholt: it doesn't actually hurt anything, best I can tell; the symptom I observed at least once was 3rd backend request ("laggard") on an obj PUT finished up after 1.5s with flying colors;  The client got the 201 after just over 0.5s (the post_quorum_timeout).00:58
swifterdarrellgholt: then... TWO MINUTES LATER, the log has a line for that same (uniquely-named) object with Timeout(10s)00:59
swifterdarrellgholt: lol?00:59
gholtOh gotcha.00:59
swifterdarrellgholt: to be fair, the GreenAsyncPile is sweet, it's just not getting drained in some cases now00:59
swifterdarrellgholt: if the laggard really did take > 10s the timeout would be legit and should get logged00:59
swifterdarrellgholt: as for the 2m delay in logging, I got nothin'00:59
* swifterdarrell shrugs00:59
gholtLeaving stuff sitting for all that extra time == bad (for memory, general resources) and the error report it makes is wrong.01:00
swifterdarrellyup01:00
swifterdarrellbut it's not causing any new req failures that weren't already gonna fail01:00
swifterdarrellno data loss, etc.01:00
swifterdarrellno lying to the client01:00
swifterdarrelljust confusion for the operator ;)01:00
gholtGood, nobody likes emergency fixes. :)01:00
swifterdarrellgholt: ya, I consider this one as a "sometime in the next release"01:01
portanteswifterdarrell: I have not seen that bug, but did read your report01:04
*** csd has quit IRC01:08
*** krtaylor has joined #openstack-swift01:19
*** zackf has quit IRC01:24
*** krtaylor has quit IRC01:26
*** nosnos has joined #openstack-swift01:38
*** zaitcev has quit IRC01:48
*** zaitcev has joined #openstack-swift01:48
*** alpha_ori has quit IRC01:55
*** alpha_ori has joined #openstack-swift01:58
*** otherjon_ has quit IRC02:13
openstackgerritSamuel Merritt proposed a change to openstack/swift: Move all DLO functionality to middleware  https://review.openstack.org/6332602:40
*** pheadron1 has joined #openstack-swift03:10
*** krtaylor has joined #openstack-swift03:12
*** pheadron has quit IRC03:13
openstackgerritA change was merged to openstack/swift: Config option to lower the timeout for recoverable object GETs.  https://review.openstack.org/6950903:27
*** matsuhashi has quit IRC03:29
*** nshaikh has joined #openstack-swift03:34
*** chandankumar_ has joined #openstack-swift03:36
*** chandankumar_ has quit IRC03:40
*** chandankumar_ has joined #openstack-swift03:41
*** leizhang has joined #openstack-swift03:44
*** chandankumar_ has quit IRC03:46
*** chandankumar_ has joined #openstack-swift03:47
*** chandankumar_ has quit IRC03:47
*** chandankumar__ has joined #openstack-swift03:47
*** chandankumar__ has left #openstack-swift03:49
*** leizhang has quit IRC04:01
*** matsuhashi has joined #openstack-swift04:02
*** saurabh_ has joined #openstack-swift04:23
*** pheadron has joined #openstack-swift04:25
*** pheadron1 has quit IRC04:28
*** pheadron1 has joined #openstack-swift04:30
madhurinotmyname: ping??04:33
*** pheadron has quit IRC04:33
*** gvernik has quit IRC04:42
openstackgerritA change was merged to openstack/swift: Some functional tests for object versioning  https://review.openstack.org/6295404:44
*** matsuhashi has quit IRC05:14
*** matsuhas_ has joined #openstack-swift05:17
*** ppai has joined #openstack-swift05:26
*** pheadron has joined #openstack-swift05:27
*** pheadron1 has quit IRC05:30
*** hurricanerix has quit IRC05:37
*** gyee has quit IRC05:43
*** zaitcev has quit IRC05:45
*** zackf has joined #openstack-swift05:51
*** zackf has quit IRC05:56
*** psharma has joined #openstack-swift06:00
*** madhuri has quit IRC06:03
*** pheadron1 has joined #openstack-swift06:14
*** pheadron has quit IRC06:17
Anjuhii all,06:34
AnjuIs swift s3 compatible ?06:34
koolhead17Anju: did you check the swift doc by any chance :)06:41
Anjukoolhead17:  hii. yes   :)  :)06:41
koolhead17you would not have asked then :P06:42
koolhead17Anju: http://docs.openstack.org/grizzly/openstack-object-storage/admin/content//configuring-openstack-object-storage-with-s3_api.html << See if it helps cheers06:42
Anjukoolhead17:  so by adding the middlware only06:42
Anjuwe can interact from s3 api to swift /06:42
koolhead17ya06:42
koolhead17see the doc :P06:43
koolhead17good luck06:43
Anjukoolhead17:  thanks06:43
*** acoles has quit IRC08:14
*** odyssey4me has joined #openstack-swift08:17
*** acoles has joined #openstack-swift08:20
*** fbo_away is now known as fbo08:22
*** mlipchuk has joined #openstack-swift08:34
*** nosnos has quit IRC08:39
*** nosnos has joined #openstack-swift08:39
*** nacim has joined #openstack-swift08:46
*** madhuri has joined #openstack-swift08:46
madhuriHello could anyone tell me what does fast POST method means?08:47
*** _bluev has joined #openstack-swift08:53
*** matsuhas_ has quit IRC09:33
*** ppai has quit IRC09:37
*** matsuhashi has joined #openstack-swift09:39
*** xga has joined #openstack-swift09:42
*** nosnos has quit IRC09:46
*** nosnos has joined #openstack-swift09:46
*** nosnos has quit IRC09:47
*** Manish_ has joined #openstack-swift09:48
Manish_Hi..09:48
*** matsuhashi has quit IRC09:50
*** ppai has joined #openstack-swift09:54
*** Manish_ has quit IRC09:55
*** acoles_ has joined #openstack-swift10:26
*** acoles_ has quit IRC10:44
*** Midnightmyth has joined #openstack-swift10:48
*** xga has quit IRC10:51
*** ppai has quit IRC11:07
*** xga has joined #openstack-swift11:08
openstackgerritA change was merged to openstack/swift: Skip delete_at_update for replication requests  https://review.openstack.org/6852911:11
*** ppai has joined #openstack-swift11:20
*** pheadron1 has quit IRC11:39
*** xga has quit IRC11:40
*** Midnightmyth has quit IRC11:43
*** pheadron has joined #openstack-swift11:54
*** xga has joined #openstack-swift11:56
*** ppai has quit IRC12:06
*** nacim has quit IRC12:10
*** ppai has joined #openstack-swift12:24
*** ppai has quit IRC12:42
*** nacim has joined #openstack-swift12:43
*** Midnightmyth has joined #openstack-swift12:55
*** foexle has joined #openstack-swift13:00
*** haomaiwa_ has quit IRC13:23
*** judd7_ has joined #openstack-swift13:28
*** haomaiwang has joined #openstack-swift13:30
*** judd7 has quit IRC13:31
*** haomaiwa_ has joined #openstack-swift13:33
*** haomaiwang has quit IRC13:35
*** xga_ has joined #openstack-swift14:01
*** xga has quit IRC14:01
*** psharma has quit IRC14:02
*** mjseger has quit IRC14:26
*** Bamberrouse has joined #openstack-swift14:33
*** Bamberrouse has left #openstack-swift14:33
*** mlipchuk has quit IRC14:51
*** nshaikh has left #openstack-swift14:58
*** byeager has joined #openstack-swift15:00
*** bada has joined #openstack-swift15:06
*** xga has joined #openstack-swift15:25
*** xga_ has quit IRC15:26
*** xga has quit IRC15:37
*** pberis has joined #openstack-swift15:40
*** hurricanerix has joined #openstack-swift15:41
*** pberis has quit IRC15:41
*** xga has joined #openstack-swift15:41
*** pberis has joined #openstack-swift15:42
*** hurricanerix has quit IRC15:42
*** hurricanerix has joined #openstack-swift15:43
*** zackf has joined #openstack-swift15:45
*** pheadron1 has joined #openstack-swift15:45
*** pheadron has quit IRC15:48
*** tongli has joined #openstack-swift15:50
*** xga has quit IRC15:56
*** xga has joined #openstack-swift15:58
*** pheadron has joined #openstack-swift16:10
*** pheadron1 has quit IRC16:13
*** xga has quit IRC16:18
*** xga has joined #openstack-swift16:26
*** k00lkidz has joined #openstack-swift16:28
*** zul has quit IRC16:29
*** xga has quit IRC16:30
k00lkidzSwift is try to put objects into an account that doesn't exist. The auth that shows up in the logs doesn't match up with the auth on all the current account when doing a swift list.16:31
*** zul has joined #openstack-swift16:32
*** xga has joined #openstack-swift16:32
k00lkidzAnyone got a remedy?16:32
*** rook][ has joined #openstack-swift16:33
_bluevk00lkidz: do you have account_autocreate set in your proxy-server.conf  ?16:36
k00lkidzyes16:38
_bluevk00lkidz: tempauth or keystone ?16:44
k00lkidzswauth16:44
_bluevif you run "swauth-list -A http://127.0.0.1/auth/ -K <<supersecret>> <<account_name>>"  run the proxy server, does it give the account URL you expect ?16:45
k00lkidzthe problem is none of the account url when I run the command you listed matches the one in the logs16:47
*** xga has quit IRC16:49
*** xga has joined #openstack-swift16:49
*** gyee has joined #openstack-swift16:51
*** byeager has quit IRC17:01
*** byeager has joined #openstack-swift17:01
k00lkidz_bluev: do you have a clue on what happen?17:03
*** byeager_ has joined #openstack-swift17:04
*** byeager has quit IRC17:05
k00lkidzgholt: I am experiencing PUT errors in Swift using Swauth. Swift is trying to PUT objects into an account that doesn't exist.  When I did a swauth-list on all current accounts, none of the account urls match up with the errored account URL in the logs.17:06
k00lkidzgholt: have you heard of such error? solution?17:07
_bluevk00lkidz: my guess is that the user is forcing the storageURL and us not using the one returned from authentication with swauth17:15
*** byeager_ has quit IRC17:18
*** byeager has joined #openstack-swift17:19
k00lkidz_bluev: do you have a good way to trace back to the source? I didn't see any connection info.17:21
*** byeager has quit IRC17:23
*** _bluev has quit IRC17:26
*** zackf1 has joined #openstack-swift17:34
k00lkidz_bluev: the source is from another storage server according to the logs. How/why does the storage server use/know of the account? Is there a way to tell the storage server to not use the non-existent account?17:34
*** zaitcev has joined #openstack-swift17:35
*** xga has quit IRC17:36
*** xga has joined #openstack-swift17:36
*** zackf has quit IRC17:37
*** csd has joined #openstack-swift17:44
*** gvernik_ has joined #openstack-swift17:45
*** gvernik_ has quit IRC17:50
*** gvernik has joined #openstack-swift17:50
*** xga has quit IRC17:50
*** xga has joined #openstack-swift17:51
*** shri has joined #openstack-swift17:55
*** krtaylor has quit IRC17:56
*** byeager has joined #openstack-swift18:08
*** byeager has quit IRC18:09
*** byeager has joined #openstack-swift18:09
*** k00lkidz has quit IRC18:10
*** Midnightmyth has quit IRC18:10
*** Midnightmyth has joined #openstack-swift18:11
*** byeager has quit IRC18:12
*** byeager has joined #openstack-swift18:12
*** byeager has quit IRC18:17
*** xga has quit IRC18:17
*** zackf has joined #openstack-swift18:21
*** zackf1 has quit IRC18:21
shrinotmyname, dfg: Just wanted to bring this to your attention: https://bugs.launchpad.net/swift/+bug/127463418:25
shriwould be great if we fix this is 1.1218:25
notmynamehello world. just opened my IRC client for the first time in 2 days18:26
notmynameshri: 1.12 has already been released18:27
shrioh.. :-)18:27
notmynameshri: it's something that could be pretty easily added. it's always 1.0 right now18:27
notmynameshri: you should propose a patch to add it :-)18:28
shriyou guys will be way faster in doing it18:28
*** nacim has quit IRC18:30
*** judd7_ has quit IRC18:32
*** judd7 has joined #openstack-swift18:32
*** fbo is now known as fbo_away18:33
*** gyee has quit IRC18:34
*** byeager has joined #openstack-swift18:36
*** odyssey4me has quit IRC18:53
*** _bluev has joined #openstack-swift19:10
*** markd has joined #openstack-swift19:11
*** byeager_ has joined #openstack-swift19:25
*** byeager has quit IRC19:26
*** gvernik has quit IRC19:34
*** k00lkidz has joined #openstack-swift19:37
*** csd has quit IRC19:38
*** gvernik has joined #openstack-swift19:40
*** shri1 has joined #openstack-swift19:41
*** fbo_away is now known as fbo19:42
*** shri has quit IRC19:44
*** csd has joined #openstack-swift19:45
*** gyee has joined #openstack-swift19:47
k00lkidzgholt: We currently have a Swift cluster using Swauth.  The storage servers, for some reason, are using an nonexistent account url to PUT objects for replication. Why causes this and how do we stop them?19:49
*** finite has joined #openstack-swift19:50
*** fbo is now known as fbo_away19:52
*** pberis has quit IRC19:57
rook][hello all, anyone have a sec to answer an ACL question?   I have two users in an account and want one of the read only to a container.  Is that possible?20:13
rook][swift stat thecontainer   Account: AUTH_838dec975ad948e1aab29c203f5dfb48 Container: thecontainer   Objects: 0     Bytes: 0  Read ACL: cosbenchro Write ACL: cosbench   Sync To:   Sync Key:  Accept-Ranges: bytes X-Timestamp: 1391112176.91371 Content-Type: text/plain; charset=utf-820:15
rook][BUT, the ro account can write in the container.....20:15
rook][TIA20:15
*** _bluev has quit IRC20:29
rook][also tried to put '-' in the ACL:   Write ACL: -cosbenchro20:30
gholtk00lkidz: No idea what all that means exactly. Any pastes of actual log lines?20:32
gholtI mean, they can have lots of [redacted] in them, but knowing as close to the actual error text helps.20:32
k00lkidzgholt: do you prefer pastebin or directly in irc?20:33
k00lkidzgholt: the logs20:34
k00lkidzgholt: logs: https://gist.github.com/anonymous/871815020:35
k00lkidzgholt: here is a better log example: https://gist.github.com/anonymous/871820320:39
k00lkidzgholt: I put some comments before each logs20:40
rook][new ACLs same behavior20:40
rook][ Read ACL: cosbench:cosbenchro Write ACL: cosbench:cosbench20:40
k00lkidzgholt: the storage server is trying to replicate using an invalid account url causing PUT errors from what I can tell20:40
gholtkoolhead17: I see, I don't think that has anything to do with Swauth.20:41
gholtkoolhead17: Just looks like maybe you had an account at one time and now no longer do? And now there are containers that are trying to report to their account and cannot.20:41
gholtkoolhead17: This looks strinkingly similar to a report some one else sent my way the other day about missing accounts. The container names were similar to the one you posted of 11360081988543355245520:42
koolhead17gholt: k00lkidz ^^20:42
gholtOh, sorry, too many people that are kool in here. ;)20:43
k00lkidzgholt: that also person was me20:43
k00lkidzgholt: :)20:43
gholtOkay, so now why do you think it's Swauth?20:43
k00lkidzgholt: I am not sure this is right, but I thought the storage server reference their account url from swauth since it seems to be the account management component.20:45
k00lkidzgholt: I can't locate the containers for the auth url since the partition doesn't exist.20:47
gholtk00lkidz: None of what you posted has auth involved, tbh. It's all backend chatter and they don't use auth to talk to one another.20:49
gholtLast time we talked we were pretty sure the cluster lost the account dbs somehow, though we couldn't determine how, and that the best course of action was to recreate the account hash db, let things run, and see what turns up in that account.20:50
k00lkidzgholt: since I don't know the name of the account, how do I recreate the account hash db?20:51
*** _bluev has joined #openstack-swift20:51
gholtYou can do a PUT directly to the account server with the AUTH_xxx is talking about. You'll probably also have to include an x-timestamp header, though x-timestamp: 1 will probably work fine.20:52
*** s00perk00lkidz has joined #openstack-swift20:55
k00lkidzgholt: can you give me an example? I have never try putting an object directly20:59
gholtk00lkidz: Oh sure, if I look at your first gist https://gist.github.com/anonymous/8718150 it would be something like: curl -XPUT -Hx-timestamp:1 http://<ip_of_server_that_logged_that_line>:<account_server_port>/sde/367796/AUTH_ad4a8b7b-fdfd-4401-aad5-082907b6bf3a21:09
k00lkidzgholt: Just tried, didn't work.21:17
k00lkidzgholt: still getting the same logs.21:17
k00lkidzgholt: clarification, the curl went through, but it didn't fixed the issue.21:17
gholtOkay, well I'm not sure how much I can help from here.21:21
k00lkidzgholt: do you know of anyone else that can help?21:22
*** bsdkurt has quit IRC21:22
gholtWell, I mean, if I was there running that cluster I could help. :) But it's very hard for me to try to help over irc through someone else's hands and eyes. Sorry. :/ I'm not sure if somebody in here else is better at this than me.21:23
*** bsdkurt has joined #openstack-swift21:24
*** hurricanerix has quit IRC21:28
*** ozone227 has joined #openstack-swift21:42
*** foexle has quit IRC21:52
*** krtaylor has joined #openstack-swift21:56
openstackgerritJon Snitow proposed a change to openstack/swift: Privileged acct ACL header, new ACL syntax, TempAuth impl.  https://review.openstack.org/6322722:07
openstackgerritJon Snitow proposed a change to openstack/swift: Non-blocking but useful improvements to Account ACLs  https://review.openstack.org/6932122:07
*** pheadron1 has joined #openstack-swift22:08
*** pheadron has quit IRC22:10
*** Trixboxer has quit IRC22:11
*** pheadron1 is now known as pheadron22:11
*** finite has quit IRC22:12
*** s00perk00lkidz has quit IRC22:16
openstackgerritSamuel Merritt proposed a change to openstack/swift: Add Storage Policy Support to the Updater  https://review.openstack.org/6473722:21
*** ozone227 has quit IRC22:31
*** k00lkidz has quit IRC22:33
*** rook][ has quit IRC22:33
openstackgerritPeter Portante proposed a change to openstack/swift: In-process swift server for functional tests  https://review.openstack.org/6610822:43
*** zackf is now known as zackf122:48
*** zackf has joined #openstack-swift23:11
*** zackf1 has quit IRC23:13
*** zackf is now known as zakcf23:29
*** zakcf is now known as zackf23:29
*** byeager_ has quit IRC23:30
*** zackf1 has joined #openstack-swift23:52
*** shri has joined #openstack-swift23:52
*** zackf has quit IRC23:53
*** shri1 has quit IRC23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!