| *** kyles_ne has quit IRC | 00:00 | |
| *** kyles_ne has joined #openstack-swift | 00:01 | |
| *** kyles_ne has quit IRC | 00:05 | |
| *** dmsimard is now known as dmsimard_away | 00:05 | |
| *** r-daneel__ has quit IRC | 00:24 | |
| *** dmorita has joined #openstack-swift | 00:26 | |
| *** mwstorer has quit IRC | 00:27 | |
| *** openstackgerrit has quit IRC | 00:31 | |
| *** openstackgerrit has joined #openstack-swift | 00:33 | |
| *** echevemaster has quit IRC | 00:45 | |
| *** kyles_ne has joined #openstack-swift | 00:51 | |
| *** NM has joined #openstack-swift | 00:56 | |
| *** mkollaro1 has quit IRC | 01:04 | |
| *** kyles_ne has quit IRC | 01:16 | |
| *** echevemaster has joined #openstack-swift | 01:16 | |
| *** kyles_ne has joined #openstack-swift | 01:16 | |
| *** addnull has joined #openstack-swift | 01:17 | |
| *** kyles_ne has quit IRC | 01:21 | |
| *** peluse_ has quit IRC | 01:48 | |
| *** kevinbenton has quit IRC | 01:48 | |
| *** echevemaster has quit IRC | 01:48 | |
| *** cebruns has quit IRC | 01:48 | |
| *** acoles has quit IRC | 01:48 | |
| *** smart_developer has quit IRC | 01:48 | |
| *** fbo has quit IRC | 01:48 | |
| *** infotection has quit IRC | 01:48 | |
| *** mitz has quit IRC | 01:48 | |
| *** ondergetekende has quit IRC | 01:48 | |
| *** nottrobin has quit IRC | 01:48 | |
| *** goodes has quit IRC | 01:48 | |
| *** zhiyan has quit IRC | 01:48 | |
| *** xianghui has quit IRC | 01:48 | |
| *** akp has quit IRC | 01:48 | |
| *** astellwag has quit IRC | 01:48 | |
| *** theanalyst has quit IRC | 01:48 | |
| *** otoolee has quit IRC | 01:48 | |
| *** ujjain has quit IRC | 01:48 | |
| *** peluse_ has joined #openstack-swift | 01:53 | |
| *** kevinbenton has joined #openstack-swift | 01:53 | |
| *** echevemaster has joined #openstack-swift | 01:54 | |
| *** cebruns has joined #openstack-swift | 01:54 | |
| *** acoles has joined #openstack-swift | 01:54 | |
| *** smart_developer has joined #openstack-swift | 01:54 | |
| *** fbo has joined #openstack-swift | 01:54 | |
| *** infotection has joined #openstack-swift | 01:54 | |
| *** mitz has joined #openstack-swift | 01:54 | |
| *** ondergetekende has joined #openstack-swift | 01:54 | |
| *** nottrobin has joined #openstack-swift | 01:54 | |
| *** goodes has joined #openstack-swift | 01:54 | |
| *** zhiyan has joined #openstack-swift | 01:54 | |
| *** xianghui has joined #openstack-swift | 01:54 | |
| *** akp has joined #openstack-swift | 01:54 | |
| *** astellwag has joined #openstack-swift | 01:54 | |
| *** theanalyst has joined #openstack-swift | 01:54 | |
| *** otoolee has joined #openstack-swift | 01:54 | |
| *** ujjain has joined #openstack-swift | 01:54 | |
| *** sendak.freenode.net sets mode: +v acoles | 01:54 | |
| *** mitz has quit IRC | 01:54 | |
| *** mitz has joined #openstack-swift | 01:55 | |
| *** NM has quit IRC | 02:23 | |
| *** bill_az has quit IRC | 02:32 | |
| smart_developer | In Swift, what exactly does 'ZBF' mean? | 02:39 |
|---|---|---|
| smart_developer | (for example, but not limited to: a 'ZBF' process). | 02:39 |
| Alex_Gaynor | "Zero byte file" I believe | 02:39 |
| *** NM has joined #openstack-swift | 02:40 | |
| smart_developer | Thanks - then what about zero_byte_fps ? | 02:40 |
| smart_developer | I think ZBF might stand for zero_byte_fps ? | 02:41 |
| smart_developer | I'm trying to understand what zero_byte_fps and ZBF both mean when they are mentioned here: | 02:43 |
| smart_developer | "The auditor already uses fork(), so we decided to use the parallel | 02:43 |
| smart_developer | model from the replicator. Concurrency is set by the concurrency | 02:43 |
| smart_developer | parameter in the auditor stanza, which sets the number of parallel | 02:43 |
| smart_developer | checksum auditors. The actual number of parallel auditing processes | 02:43 |
| smart_developer | is concurrency + 1 if zero_byte_fps is non-zero. | 02:43 |
| smart_developer | 02:43 | |
| smart_developer | Only one ZBF process is forked, and a new ZBF process is forked as | 02:43 |
| smart_developer | soon as the current ZBF process finishes. Thus the last process | 02:43 |
| smart_developer | running will always be a ZBF process." | 02:43 |
| zaitcev | zero byte files per second | 02:44 |
| zaitcev | duuuuude | 02:44 |
| smart_developer | Why would be Swift be creating zero byte files, or multiple zero-byte-files if what you're saying is correct? (that zero_byte_fps = zero byte files per second). | 02:45 |
| *** vu has joined #openstack-swift | 02:48 | |
| zaitcev | it's a separate fps for the zbf auditor. Sometimes you want to prevent it from trashing your nodes, since ZBF has no sleep interval | 02:48 |
| zaitcev | As for where those files come from, it's a mystery, as far as I am concerned anyway. | 02:49 |
| smart_developer | How would it "trash my nodes" ? | 02:50 |
| zaitcev | A legend upon which Swift was founded spaketh that XFS creates those when its fsck-like code detects corruption. | 02:50 |
| smart_developer | zaitcev : and what do you mean that a zero byte file "has no sleep interval"? | 02:51 |
| smart_developer | zaitcev : So Swift puts a zero byte file wherever it sees a corrupted file ? | 02:51 |
| zaitcev | But recently I mentioned it to an old XFS developer, who said that it's a stupid myth that just won't die. And in reality the only way to generate zbfs is to crash with open files. Then, the journal replays will create those. | 02:51 |
| smart_developer | So is the zbf auditor's job to (1) create a zero byte file where a corrupted file exists, or (2) to remove a zero byte file whenever it sees one ? | 02:53 |
| zaitcev | No, its job is to detect them and delete them, so that suffix hashes get mismatched and then replicator recovers them. | 02:54 |
| zaitcev | Auditor by itself never screws with objects, I think. | 02:54 |
| zaitcev | Well, it might quarantine, I need to look | 02:54 |
| smart_developer | I think I read that the auditor places corrupted files in quarantine. | 02:55 |
| zaitcev | And by trash I meant thrash | 02:55 |
| smart_developer | So, why would the auditor place a zero byte file in quarantine, instead of just deleting it? | 02:55 |
| *** haomaiwang has quit IRC | 03:18 | |
| *** haomaiwa_ has joined #openstack-swift | 03:19 | |
| *** NM has quit IRC | 03:22 | |
| portante | smart_developer: so first, an object is represented on disk as a directory with a <timestamp>.data file in it | 03:31 |
| portante | with proper replication to ensure data robustness, there are 1 or more other copies of this setup on another node. | 03:32 |
| portante | if during file creation of that .data file the system fails before the data is written to disk for the file, but the directory contents were written to disk, (I believe) a zero byte file will result when the disk's journal is replayed | 03:33 |
| *** haomai___ has joined #openstack-swift | 03:34 | |
| *** haomaiwa_ has quit IRC | 03:38 | |
| portante | when the auditor detects a zero byte file, it quarantines it (potentially allowing for some kind of post-trauma analysis) allowing the replicator to detect the missing object on that node, restoring it with a copy from one of the other nodes | 03:38 |
| *** haomai___ has quit IRC | 03:41 | |
| *** haomaiwang has joined #openstack-swift | 03:41 | |
| openstackgerrit | Billy Olsen proposed a change to openstack/swift: Fix getaddrinfo if dnspython is installed. https://review.openstack.org/116618 | 03:43 |
| *** haomaiw__ has joined #openstack-swift | 03:57 | |
| *** haomaiwang has quit IRC | 03:59 | |
| *** echevemaster has quit IRC | 04:00 | |
| *** judd7 has quit IRC | 04:47 | |
| goodes | good morning | 05:12 |
| goodes | is anyone familiar with Sahara and list_endpoints? | 05:14 |
| *** kopparam has joined #openstack-swift | 05:18 | |
| mattoliverau | goodes: morning! ..I'm not sorry | 05:19 |
| goodes | mattoliverau: g'day - too me a few secs to parse your response :-S | 05:21 |
| mattoliverau | goodes: lol, sorry, I wanted to greet you, but not give your hopes up on answering your questions about sahara :P | 05:22 |
| goodes | This is not specific to Sahara, I am wondering/guessing if the whole goal of list_endpoints is to allow 'clients' to directly request objects from object servers and reduce latency by not going via the proxy | 05:22 |
| goodes | does that sound plausible? | 05:23 |
| goodes | I can wait till the US wakes up | 05:24 |
| mattoliverau | goodes: oh i see.. so long as you could go directly to the end point (i.e if there is a seperate network your servers were talking on) with an internal client. | 05:25 |
| mattoliverau | if you were using your proxy as a gateway into your swift cluster you couldn't by pass.. and you'd loose all the proxy protection the proxy gives you. But the list_endpoint could be used like that, or for diagnosis. | 05:27 |
| goodes | the documentation says 'This middleware makes it possible to integrate swift with software that relies on data locality information to avoid network overhead, such as Hadoop.' | 05:27 |
| goodes | but that would not help within the network if you are going through the proxy - it would only make sense if you are using it to determine which REGION to run the job | 05:28 |
| mattoliverau | So long as its accessible you could talk directly to them, like the replicators do etc. | 05:28 |
| mattoliverau | So sure.. | 05:28 |
| mattoliverau | It doesn't solve your problem but there is a concurrent read patch in review which can be used to "hopefully" speed up latency of the proxy. | 05:29 |
| goodes | or ... if you were soo sensitive to latency that you directly requested it from the object server - which it kind of evil and bad | 05:30 |
| mattoliverau | lol, yeah.. the concurrent read helps with the latter a little, send out requests to primary nodes all at once and repond as soon as you get 1 valid reponse. | 05:31 |
| zaitcev | Personally I consider that Hadoop thing a terrible kludge. | 05:31 |
| *** haomaiw__ has quit IRC | 05:31 | |
| goodes | agree, but BG is an important driver of object, so it can't be ignored | 05:32 |
| *** haomaiwang has joined #openstack-swift | 05:32 | |
| goodes | oops BG == BD == big data | 05:32 |
| goodes | the concurrent read is doing parallel reads to object servers? | 05:33 |
| *** haomai___ has joined #openstack-swift | 05:33 | |
| goodes | what is it using? | 05:33 |
| mattoliverau | greenthreads | 05:36 |
| *** haomaiwang has quit IRC | 05:37 | |
| *** ppai has joined #openstack-swift | 05:38 | |
| goodes | zaitcev: agree kludge - rather then doing it right trying to make it work like HDFS | 05:42 |
| goodes | mattoliverau: thanks for the help, getting of the train, now I need to work :-S. | 05:46 |
| mattoliverau | goodes: nps, have a good day :) | 05:46 |
| *** kopparam has quit IRC | 05:57 | |
| *** bkopilov has quit IRC | 06:03 | |
| *** zaitcev has quit IRC | 06:14 | |
| *** k4n0 has joined #openstack-swift | 06:26 | |
| *** nshaikh has joined #openstack-swift | 06:28 | |
| openstackgerrit | Christian Schwede proposed a change to openstack/swift: Fix bug in swift-ring-builder list_parts https://review.openstack.org/121893 | 06:31 |
| *** mahatic has joined #openstack-swift | 06:41 | |
| *** geaaru has joined #openstack-swift | 06:53 | |
| mattoliverau | I'm calling it a night. Night all, see talk to some of you at the swift meeting. | 07:17 |
| *** foexle has joined #openstack-swift | 07:27 | |
| *** bkopilov has joined #openstack-swift | 07:28 | |
| *** jamiehannaford has joined #openstack-swift | 07:57 | |
| *** Dafna has joined #openstack-swift | 08:29 | |
| *** nshaikh has quit IRC | 09:08 | |
| *** addnull has quit IRC | 09:37 | |
| *** mkollaro has joined #openstack-swift | 09:56 | |
| *** aix has joined #openstack-swift | 09:57 | |
| *** addnull has joined #openstack-swift | 10:27 | |
| *** addnull has quit IRC | 11:12 | |
| *** dmorita has quit IRC | 11:12 | |
| *** addnull has joined #openstack-swift | 11:17 | |
| *** judd7 has joined #openstack-swift | 11:21 | |
| *** NM has joined #openstack-swift | 11:30 | |
| *** occupant has quit IRC | 11:39 | |
| *** NM has quit IRC | 11:39 | |
| *** occupant has joined #openstack-swift | 11:44 | |
| *** dmsimard_away is now known as dmsimard | 11:48 | |
| *** NM has joined #openstack-swift | 11:48 | |
| *** NM has quit IRC | 12:00 | |
| *** judd7 has quit IRC | 12:08 | |
| *** jamiehannaford has quit IRC | 12:08 | |
| *** hhuang has joined #openstack-swift | 12:09 | |
| *** jamiehannaford has joined #openstack-swift | 12:09 | |
| *** ttrumm has joined #openstack-swift | 12:13 | |
| *** k4n0 has quit IRC | 12:14 | |
| *** addnull has quit IRC | 12:28 | |
| *** sandywalsh has quit IRC | 12:37 | |
| *** NM has joined #openstack-swift | 12:48 | |
| *** dencaval has quit IRC | 12:49 | |
| *** kopparam has joined #openstack-swift | 12:55 | |
| *** aix has quit IRC | 13:00 | |
| *** kopparam has quit IRC | 13:03 | |
| *** kopparam has joined #openstack-swift | 13:04 | |
| *** miqui has joined #openstack-swift | 13:06 | |
| *** kopparam has quit IRC | 13:08 | |
| *** jamiehannaford has quit IRC | 13:08 | |
| *** jamiehannaford has joined #openstack-swift | 13:11 | |
| *** echevemaster has joined #openstack-swift | 13:17 | |
| *** judd7 has joined #openstack-swift | 13:19 | |
| *** tdasilva has joined #openstack-swift | 13:23 | |
| openstackgerrit | Lorcan Browne proposed a change to openstack/swift: Add "--no-overlap" option to swift-dispersion populate https://review.openstack.org/118411 | 13:24 |
| *** aix has joined #openstack-swift | 13:27 | |
| *** kopparam has joined #openstack-swift | 13:34 | |
| *** bsdkurt has quit IRC | 13:36 | |
| *** bsdkurt has joined #openstack-swift | 13:37 | |
| *** kopparam has quit IRC | 13:38 | |
| *** judd7 has quit IRC | 13:39 | |
| *** hhuang has quit IRC | 13:40 | |
| *** judd7 has joined #openstack-swift | 13:46 | |
| openstackgerrit | Alistair Coles proposed a change to openstack/swift: Early checks for data migration config and headers https://review.openstack.org/121831 | 13:48 |
| tdasilva | acoles: do I need to do anything to make it easier for Gil to squash that patch? wondering if I need to separate the new driver into its own patch??? | 13:49 |
| tdasilva | not sure what the process is like... | 13:49 |
| acoles | tdasilva: good morning! yes, it would be helpful if you separated out your driver into another review that is dependent on your current one with the interface refactoring | 13:57 |
| *** kopparam has joined #openstack-swift | 13:57 | |
| *** nshaikh has joined #openstack-swift | 13:57 | |
| acoles | tdasilva: just want to check - are you happy for your refactoring to be squashed into gvernik's patch? | 13:58 |
| tdasilva | acoles: absolutely! :-) he could squash the whole thing if he wanted, no problem | 13:59 |
| tdasilva | acoles: but I can separate the patches too if you think it makes sense to have the new driver as a new future patch | 13:59 |
| *** hhuang has joined #openstack-swift | 14:00 | |
| acoles | tdasilva: great. perhaps we should keep your driver separate for easier review. idk, it might make reviewing easier | 14:01 |
| tdasilva | true true | 14:01 |
| tdasilva | i will do that | 14:01 |
| acoles | tdasilva: btw gvernik told me out of band that he is happy with those refactorings | 14:03 |
| tdasilva | acoles: yeah, he mentioned that in a comment too, which is cool...I think mattoliverau also took a look and liked it | 14:04 |
| openstackgerrit | Christian Schwede proposed a change to openstack/swift: Limit partition movement when rebalancing https://review.openstack.org/121422 | 14:04 |
| *** bkopilov has quit IRC | 14:05 | |
| *** ppai has quit IRC | 14:17 | |
| *** judd7 has quit IRC | 14:21 | |
| *** mahatic has quit IRC | 14:24 | |
| *** cschwede has joined #openstack-swift | 14:34 | |
| *** goodes has quit IRC | 14:36 | |
| *** jdaggett has quit IRC | 14:36 | |
| *** zhiyan has quit IRC | 14:36 | |
| *** serverascode has quit IRC | 14:36 | |
| *** Alex_Gaynor has quit IRC | 14:36 | |
| *** mahatic has joined #openstack-swift | 14:41 | |
| openstackgerrit | Lorcan Browne proposed a change to openstack/swift: Add "--no-overlap" option to swift-dispersion populate https://review.openstack.org/118411 | 14:45 |
| *** Alex_Gaynor has joined #openstack-swift | 14:48 | |
| *** zhiyan has joined #openstack-swift | 14:48 | |
| *** ttrumm_ has joined #openstack-swift | 14:51 | |
| *** goodes_ has joined #openstack-swift | 14:53 | |
| *** jdaggett has joined #openstack-swift | 14:53 | |
| *** serverascode has joined #openstack-swift | 14:53 | |
| *** nshaikh has left #openstack-swift | 14:54 | |
| *** ttrumm has quit IRC | 14:54 | |
| *** judd7 has joined #openstack-swift | 14:59 | |
| *** foexle has quit IRC | 15:04 | |
| *** zaitcev has joined #openstack-swift | 15:05 | |
| *** ChanServ sets mode: +v zaitcev | 15:05 | |
| *** kopparam has quit IRC | 15:06 | |
| *** judd7 has quit IRC | 15:07 | |
| *** judd7 has joined #openstack-swift | 15:08 | |
| *** ttrumm_ has quit IRC | 15:12 | |
| *** tongli has joined #openstack-swift | 15:14 | |
| *** kevinc has joined #openstack-swift | 15:15 | |
| *** AbyssOne is now known as JelleB | 15:19 | |
| *** LarsN has joined #openstack-swift | 15:19 | |
| *** jasondotstar has joined #openstack-swift | 15:21 | |
| *** aix has quit IRC | 15:28 | |
| *** judd7 has quit IRC | 15:29 | |
| *** aix has joined #openstack-swift | 15:40 | |
| *** Victor__ has quit IRC | 15:47 | |
| *** cschwede has quit IRC | 15:49 | |
| *** jamiehannaford has quit IRC | 15:54 | |
| *** chuck_ has joined #openstack-swift | 15:58 | |
| *** judd7 has joined #openstack-swift | 15:59 | |
| *** jamiehannaford has joined #openstack-swift | 16:01 | |
| *** dmsimard is now known as dmsimard_away | 16:10 | |
| *** kyles_ne has joined #openstack-swift | 16:13 | |
| *** hhuang has quit IRC | 16:16 | |
| openstackgerrit | Alan Erwin proposed a change to openstack/swift: Adding object partition check https://review.openstack.org/122194 | 16:20 |
| *** vu has quit IRC | 16:28 | |
| *** mwstorer has joined #openstack-swift | 16:29 | |
| *** chuck_ has quit IRC | 16:50 | |
| notmyname | hello, everything | 16:58 |
| notmyname | I wasn't online at all yesterday, so I'm catching up this morning | 16:58 |
| notmyname | what did I miss? | 16:58 |
| *** tongli has quit IRC | 17:00 | |
| *** occupant has quit IRC | 17:00 | |
| *** mkollaro has quit IRC | 17:00 | |
| *** briancline has quit IRC | 17:00 | |
| *** fifieldt_ has quit IRC | 17:00 | |
| *** wer has quit IRC | 17:00 | |
| *** tongli has joined #openstack-swift | 17:00 | |
| *** occupant has joined #openstack-swift | 17:00 | |
| *** mkollaro has joined #openstack-swift | 17:00 | |
| *** briancline has joined #openstack-swift | 17:00 | |
| *** fifieldt_ has joined #openstack-swift | 17:00 | |
| *** wer has joined #openstack-swift | 17:00 | |
| *** echevemaster has quit IRC | 17:00 | |
| *** yuan has quit IRC | 17:00 | |
| *** LarsN has quit IRC | 17:00 | |
| *** byeager_away has quit IRC | 17:00 | |
| notmyname | swift team meeting in 2 hours in #openstack-meeting | 17:01 |
| *** gyee has joined #openstack-swift | 17:02 | |
| *** LarsN has joined #openstack-swift | 17:02 | |
| *** echevemaster has joined #openstack-swift | 17:02 | |
| *** yuan has joined #openstack-swift | 17:02 | |
| *** byeager_away has joined #openstack-swift | 17:02 | |
| *** Dafna has quit IRC | 17:02 | |
| *** Dafna has joined #openstack-swift | 17:02 | |
| *** tkay has joined #openstack-swift | 17:21 | |
| *** geaaru has quit IRC | 17:22 | |
| *** tsg has joined #openstack-swift | 17:29 | |
| smart_developer | If allow_versions is not mentioned in the container server config, does that mean it is set to True, or False, by default? | 17:39 |
| smart_developer | It is the one that specifies whether Object Versioning is enabled or not. | 17:39 |
| smart_developer | http://docs.openstack.org/developer/swift/overview_object_versioning.html | 17:39 |
| smart_developer | more info ^ | 17:40 |
| notmyname | smart_developer: what's in the sample config file? | 17:42 |
| smart_developer | https://github.com/openstack/swift/blob/master/etc/container-server.conf-sample | 17:45 |
| smart_developer | Does this accurately reflect what all the default values are, for the Swift configurations? | 17:45 |
| smart_developer | I thought it was just a "sample" file. | 17:46 |
| smart_developer | Same with the ones for proxy-server.conf-sample, account-server.conf-sample, and object-server.conf-sample. | 17:46 |
| notmyname | smart_developer: yes. the sample config files accurately reflect the defaults. if a value is commented out, it's the default if not specified. if a values isn't commented out, you are required to provide your own | 17:46 |
| notmyname | we've been pretty good with that over time, but just recently mattoliverau went through and audited the sample configs to ensure they are accurate (and fixed a couple of things) | 17:47 |
| acoles | smart_developer: iirc you do need to explicitly set allow_versions = true | 17:48 |
| notmyname | eg in the container server conf, pipeline (line 61) is required to be specified, but eg node_timeout (line 72) isn't and the default is 3 | 17:49 |
| notmyname | smart_developer: so the default value for allow_versions is false, and if you want to enable it you should explicitly set it to true | 17:49 |
| *** Tyger has joined #openstack-swift | 17:51 | |
| smart_developer | Ah, I see. Thank you! | 17:57 |
| openstackgerrit | Thiago da Silva proposed a change to openstack/swift: Moving data migration handling to base driver class https://review.openstack.org/121728 | 18:00 |
| zaitcev | I tried to review a trivial patch by dfg... oops my tree is not passing tests... hack-hack-hack... the day is over, patch is not reviewed. drop dead, raise to alarm, join meetings. patch is still not reviewed | 18:07 |
| *** jamiehannaford has quit IRC | 18:10 | |
| *** jamiehannaford has joined #openstack-swift | 18:11 | |
| dfg | clayg: you there? | 18:18 |
| openstackgerrit | Christian Schwede proposed a change to openstack/swift: Limit partition movement when adding a new tier https://review.openstack.org/121422 | 18:22 |
| openstackgerrit | Christian Schwede proposed a change to openstack/swift: Return correct number of changed partitions https://review.openstack.org/120713 | 18:22 |
| clayg | dfg: ohia | 18:23 |
| clayg | dfg: you should tell me to look at your sqlite error limiting branch | 18:23 |
| openstackgerrit | Christian Schwede proposed a change to openstack/swift: Limit partition movement when adding a new tier https://review.openstack.org/121422 | 18:23 |
| dfg | clayg: when are you going to look at my error limiting thing? geez | 18:24 |
| dfg | clayg: was going to talk to you about that auditor patch | 18:24 |
| clayg | dfg: i know, i'm a totally slacker - thanks for reminding me | 18:26 |
| clayg | dfg: the think the quarantines all your disks? i didn't recognize that guys names - new cat? | 18:26 |
| dfg | clayg: we actually did see this happen. it was kinda weird. | 18:26 |
| clayg | dfg: that's crazy, any idea how? | 18:27 |
| dfg | clayg: ya he's a new guy Alan. | 18:27 |
| clayg | HI ALAN! WELCOME TO THE SHIT! | 18:27 |
| dfg | clayg: no not really. file system error? | 18:27 |
| dfg | although that doesn't make a ton of sense... | 18:27 |
| clayg | dfg: yeah.... maybe... but it seems more likely to look like gibberish than like a valid but incorrect partition? | 18:28 |
| clayg | dfg: did you have any luck tracking down the other copies? all 3 valid and in place? this one just got stuck in the wrong spot? | 18:28 |
| clayg | jesus... how'd you even find it? | 18:28 |
| dfg | a customer was getting a 403. the object server though it was a md5 collision- where it checks the metadata path vrs incoming path | 18:29 |
| dfg | but it wasn't. it was just this weird problem. | 18:29 |
| clayg | ok... well maybe that's besides the point. you saw it. i totally agree the auditor should be the guy to detect it. i guess the only question is if it should quarantine? seems to me it'd be more helpful just to move it to the correct part... | 18:30 |
| * clayg mind just blew up | 18:30 | |
| clayg | how did the proxy *find* it if it was in the wrong partition? | 18:30 |
| clayg | it was in the right partition.... but the path in metadata... that path shouldn't include the part?! | 18:30 |
| dfg | so like- customer req was to /a/c/o but when it got tot he object server somebody else's shit was sitting where /a/c/o should have been. | 18:31 |
| dfg | that the object that was there didn't belong there at all. it should have been in a whole different partition | 18:32 |
| clayg | holy wtf - thanks dhadas | 18:32 |
| dfg | we're not crazy worried about this fix. this was mostly just a first bug fix for the new guy to learn the system type of thing. but still- it did happen. | 18:34 |
| clayg | that's weird and scary | 18:34 |
| dfg | ya. but its also quite rare | 18:34 |
| torgomatic | solar flares? | 18:34 |
| dfg | something like that- hopefully | 18:35 |
| clayg | i still don't quite understand... was it right hash in the wrong part - or the object was in the wrong hashdir (which was in the right part) | 18:35 |
| clayg | dfg: alright well new guy simple bug fix or not - don't see why it's more helpful to throw it out than move it to the right place | 18:36 |
| clayg | dfg: the container replicator does a part check - and it just replicates the db's where it goes | 18:36 |
| dfg | so because its so off the wall i don't know about the move to the right partition thing. since at the point where this happens we're kinda in crazy territory. idk | 18:36 |
| clayg | dfg: the auditor could move the damn thing into the right place and replication would sweep it up like any other handoff | 18:36 |
| dfg | clayg: thats basically what i did manually | 18:37 |
| dfg | i guess moving it to the right place would be ok. | 18:37 |
| dfg | i mean in this case that would have fixed the problem. i just don't know if that will always be the case and should be just quarantine it because its not safe to assume anything at this point | 18:38 |
| clayg | dfg: i can think of a couple of ways a proxy might put all three copies in the wrong hashdir but no good idea how only one replica could be become in the wrong place | 18:39 |
| dfg | oh ya- i didn't answer that | 18:39 |
| clayg | dfg: ok, well let's get alan licensed up - thanks for all the info | 18:40 |
| clayg | dfg: having not seen it myself i'm inclined to trust your experience with the issue - so think about it and comment on the patch | 18:41 |
| dfg | clayg: ok. i should have written down all the details when this happened... i'll comment on the issue though | 18:43 |
| *** tristanC has joined #openstack-swift | 18:43 | |
| *** elambert has joined #openstack-swift | 18:47 | |
| *** kyles_ne has quit IRC | 18:48 | |
| *** kyles_ne has joined #openstack-swift | 18:49 | |
| notmyname | zaitcev: thanks for checking on swiftclient status | 18:49 |
| zaitcev | notmyname: purely because of stuff like this - http://autoqa.fedoraproject.org/results/864711-autotest/virt08.qa/upgradepath/results/python-swiftclient-2.html | 18:49 |
| zaitcev | notmyname: If you cut a release, I'll spin it for Fedora Rawhide and then the qa bot will permit me to backport fixes into F20 and F21. | 18:51 |
| *** kevinc has quit IRC | 18:51 | |
| *** gvernik has joined #openstack-swift | 18:51 | |
| *** kyles_ne has quit IRC | 18:53 | |
| *** aix has quit IRC | 18:55 | |
| *** kota_ has joined #openstack-swift | 18:56 | |
| *** shri has joined #openstack-swift | 18:57 | |
| notmyname | zaitcev: ah ok. after the meeting I've got to travel for the afternoon, so i'll probably cut it this evening | 18:57 |
| *** Kbee has joined #openstack-swift | 18:59 | |
| *** cutforth has joined #openstack-swift | 19:00 | |
| notmyname | meeting time in #openstack-meeting | 19:00 |
| *** goodes_ is now known as goodes | 19:02 | |
| *** gyee has quit IRC | 19:02 | |
| smart_developer | What is container-sync, a middleware ? | 19:14 |
| smart_developer | And by default is it enabled? | 19:14 |
| clayg | smart_developer: I don't think it's enabled by default - very few middleware's are | 19:17 |
| notmyname | http://docs.openstack.org/developer/swift/overview_container_sync.html | 19:17 |
| clayg | and it's not just a middleware = there's a background deamon | 19:17 |
| clayg | is the meeting like now or did I miss it? | 19:18 |
| clayg | torgomatic: are you feeling better today? | 19:18 |
| torgomatic | clayg: nope | 19:18 |
| clayg | torgomatic: sorry to hear that bro | 19:18 |
| notmyname | clayg: meeting's now | 19:18 |
| torgomatic | clayg: thanks; I guess this is what happens when preschool starts up is the kids all swap germs that they found over the summer | 19:19 |
| *** kyles_ne has joined #openstack-swift | 19:20 | |
| notmyname | torgomatic: sharing is caring | 19:21 |
| openstackgerrit | Keshava Bharadwaj proposed a change to openstack/swift: Provides proper error handling on builder unpickle https://review.openstack.org/122225 | 19:23 |
| *** mwstorer has quit IRC | 19:30 | |
| *** kevinc has joined #openstack-swift | 19:35 | |
| openstackgerrit | Christian Schwede proposed a change to openstack/swift: Limit partition movement when adding a new tier https://review.openstack.org/121422 | 19:38 |
| *** tongli has quit IRC | 19:38 | |
| * notmyname has to travel for the rest of the afternoon | 19:46 | |
| notmyname | I'll probably be back online tonight | 19:47 |
| mattoliverau | notmyname: happy travels | 19:47 |
| * mattoliverau goes back to bed, see y'all in a few hours. | 19:47 | |
| *** cutforth has quit IRC | 19:49 | |
| *** kota_ has quit IRC | 19:49 | |
| *** aix has joined #openstack-swift | 19:53 | |
| *** vu has joined #openstack-swift | 19:53 | |
| *** gvernik has quit IRC | 19:57 | |
| *** vu has quit IRC | 19:59 | |
| *** tsg has quit IRC | 20:00 | |
| *** tsg has joined #openstack-swift | 20:00 | |
| *** kyles_ne has quit IRC | 20:00 | |
| *** kyles_ne has joined #openstack-swift | 20:00 | |
| *** Kbee has quit IRC | 20:18 | |
| *** aix has quit IRC | 20:23 | |
| *** jasondotstar has quit IRC | 20:26 | |
| *** dencaval has joined #openstack-swift | 20:26 | |
| *** jamiehannaford has quit IRC | 20:27 | |
| dencaval | Hey guys, I'm curious. Does swift-proxy land the data on disk before sending to the storage-nodes? | 20:27 |
| torgomatic | dencaval: no, it's streamed out | 20:27 |
| dencaval | torgomatic hmm, thank you | 20:31 |
| smart_developer | clayg : I'm not sure if I understood you when you said "and it's not just a middleware = there's a background deamon" in regards to my question about container-sync. | 20:32 |
| smart_developer | Could you clarify ? | 20:33 |
| *** gyee has joined #openstack-swift | 20:35 | |
| clayg | the middleware is there for authorization mainly, the guy pushing the data to the remote cluster is a background daemon container-sync (more like the container-replicator) | 20:37 |
| smart_developer | clayg : Ok, so if container-sync is technically a middleware, then, why is it not specified in container-server.conf as [filter:container-sync], but rather just as [container-sync] ? | 20:42 |
| clayg | smart_developer: that's the daeon part, like [container-replicator] it's daemon has a configuration in the contain-server.conf | 20:47 |
| *** tdasilva has quit IRC | 20:47 | |
| clayg | smart_developer: there *is* a middleware component for authorization and that's configured in the proxy-server.conf | 20:47 |
| *** mwstorer has joined #openstack-swift | 20:49 | |
| *** miqui has quit IRC | 20:53 | |
| smart_developer | clayg : So without having the [filter:container-sync] portion inside proxy-server.conf, there is no point in having just [container-sync] inside container-server.conf ? | 20:56 |
| torgomatic | smart_developer: do you (A) have two or more distinct Swift clusters, and (B) want to keep a small subset of containers synchronized between them? | 21:09 |
| *** judd7 has quit IRC | 21:19 | |
| torgomatic | if you don't meet both those conditions, then container sync is of no use to you and should be disabled | 21:23 |
| clayg | torgomatic breaks it down for ya | 21:31 |
| smart_developer | I see. | 22:06 |
| smart_developer | Thank you very much. | 22:06 |
| *** ZBhatti has joined #openstack-swift | 22:10 | |
| *** NM has quit IRC | 22:13 | |
| *** smart_developer has left #openstack-swift | 22:16 | |
| *** smart_developer has joined #openstack-swift | 22:16 | |
| *** W9PDS has joined #openstack-swift | 22:17 | |
| mattoliverau | Morning.. again | 22:21 |
| *** mahatic has quit IRC | 22:24 | |
| *** W9PDS has quit IRC | 22:27 | |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Reject overly-taxing ranged-GET requests https://review.openstack.org/117579 | 22:48 |
| *** tgohad has joined #openstack-swift | 23:03 | |
| *** tsg has quit IRC | 23:03 | |
| smart_developer | Should /var/log/swift.log (Log output of any of the Swift services - proxy, account, container, and object) have owner:group to be root:root, swift:swift, or something else ? | 23:05 |
| smart_developer | If you're using rsyslog/logrotate to set up the log. | 23:05 |
| mattoliverau | smart_developer: if your using syslog then it should be owned by that user, so it can write, and in a linux world should have the adm group (at least in a debian world), so readonly admins can read logs. | 23:07 |
| *** kevinc has quit IRC | 23:39 | |
| *** ZBhatti has quit IRC | 23:44 | |
| *** mwstorer has quit IRC | 23:53 | |
| *** NM has joined #openstack-swift | 23:57 | |
| *** tsg has joined #openstack-swift | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!