| *** ppai has joined #openstack-swift | 00:04 | |
| *** zynisch_o7 has joined #openstack-swift | 00:05 | |
| *** ppai has quit IRC | 00:20 | |
| *** foexle has joined #openstack-swift | 00:26 | |
| *** barra204 has joined #openstack-swift | 00:26 | |
| *** barra204_ has joined #openstack-swift | 00:27 | |
| *** barra204_ is now known as shakamunyi | 00:28 | |
| *** zynisch_o7 has quit IRC | 00:33 | |
| *** foexle has quit IRC | 00:35 | |
| *** ir2ivps8 has quit IRC | 00:37 | |
| *** annegentle has joined #openstack-swift | 00:39 | |
| *** barra204_ has joined #openstack-swift | 00:40 | |
| *** barra204__ has joined #openstack-swift | 00:40 | |
| *** barra204 has quit IRC | 00:40 | |
| *** shakamunyi has quit IRC | 00:41 | |
| *** ppai has joined #openstack-swift | 00:43 | |
| *** vjujjuri has quit IRC | 00:44 | |
| *** barra204_ has quit IRC | 00:46 | |
| *** barra204__ has quit IRC | 00:46 | |
| *** zhill has quit IRC | 00:52 | |
| *** annegentle has quit IRC | 00:57 | |
| *** ppai has quit IRC | 01:03 | |
| openstackgerrit | Takashi Kajinami proposed openstack/swift: Add process name checking into swift-init https://review.openstack.org/116203 | 01:05 |
|---|---|---|
| *** km has quit IRC | 01:06 | |
| *** km has joined #openstack-swift | 01:07 | |
| *** minwoob has quit IRC | 01:28 | |
| *** thurloat is now known as thurloat_isgone | 01:37 | |
| *** pberis has joined #openstack-swift | 02:21 | |
| *** pberis has quit IRC | 02:21 | |
| *** barra204 has joined #openstack-swift | 02:25 | |
| *** barra204_ has joined #openstack-swift | 02:25 | |
| *** bkopilov_wfh has quit IRC | 02:27 | |
| *** barra204_ has quit IRC | 02:32 | |
| *** barra204 has quit IRC | 02:32 | |
| *** km has quit IRC | 02:57 | |
| *** wbhuber has quit IRC | 02:58 | |
| *** barra204 has joined #openstack-swift | 03:08 | |
| *** shakamunyi has joined #openstack-swift | 03:08 | |
| *** zz_petertr7 is now known as petertr7 | 03:41 | |
| *** sanchitmalhotra has joined #openstack-swift | 03:50 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 03:52 | |
| *** proteusguy has joined #openstack-swift | 03:52 | |
| *** chlong has quit IRC | 03:54 | |
| *** sanchitmalhotra has quit IRC | 03:54 | |
| *** chlong has joined #openstack-swift | 04:06 | |
| *** sanchitmalhotra has joined #openstack-swift | 04:14 | |
| *** sanchitmalhotra1 has quit IRC | 04:15 | |
| *** bkopilov_wfh has joined #openstack-swift | 04:16 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 04:23 | |
| *** sanchitmalhotra has quit IRC | 04:25 | |
| *** proteusguy has quit IRC | 04:25 | |
| *** annegentle has joined #openstack-swift | 04:29 | |
| *** zhill has joined #openstack-swift | 04:32 | |
| *** annegentle has quit IRC | 04:35 | |
| *** proteusguy has joined #openstack-swift | 04:37 | |
| *** sanchitmalhotra has joined #openstack-swift | 04:46 | |
| *** sanchitmalhotra1 has quit IRC | 04:48 | |
| *** shakamunyi has quit IRC | 04:50 | |
| *** barra204 has quit IRC | 04:50 | |
| *** chlong has quit IRC | 04:52 | |
| *** chlong has joined #openstack-swift | 05:04 | |
| openstackgerrit | Darrell Bishop proposed openstack/swift: Allow one object-server per disk deployment https://review.openstack.org/184189 | 05:20 |
| *** zul has joined #openstack-swift | 05:23 | |
| *** annegentle has joined #openstack-swift | 05:31 | |
| *** annegentle has quit IRC | 05:36 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:42 | |
| *** sanchitmalhotra has quit IRC | 05:43 | |
| *** sanchitmalhotra has joined #openstack-swift | 05:43 | |
| *** annegentle has joined #openstack-swift | 05:45 | |
| *** annegentle has quit IRC | 05:50 | |
| *** vjujjuri_ has joined #openstack-swift | 05:54 | |
| *** vjujjuri_ is now known as vjujjuri | 05:58 | |
| *** harlowja has joined #openstack-swift | 05:58 | |
| *** harlowja_ has joined #openstack-swift | 06:02 | |
| *** harlowja has quit IRC | 06:04 | |
| *** proteusguy has quit IRC | 06:16 | |
| *** harlowja_ has quit IRC | 06:34 | |
| *** petertr7 is now known as zz_petertr7 | 06:34 | |
| *** foexle has joined #openstack-swift | 06:46 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: Always decode command-line arguments as UTF-8 https://review.openstack.org/176945 | 06:47 |
| *** foexle has quit IRC | 06:50 | |
| *** annegentle has joined #openstack-swift | 06:51 | |
| *** annegentle has quit IRC | 06:56 | |
| *** foexle has joined #openstack-swift | 06:58 | |
| *** foexle has quit IRC | 06:58 | |
| *** vjujjuri has quit IRC | 06:59 | |
| *** zhill has quit IRC | 07:08 | |
| *** jistr has joined #openstack-swift | 07:18 | |
| *** annegentle has joined #openstack-swift | 07:52 | |
| *** mitz has joined #openstack-swift | 07:52 | |
| *** geaaru has joined #openstack-swift | 07:52 | |
| *** annegentle has quit IRC | 07:57 | |
| *** mitz has quit IRC | 07:58 | |
| *** mitz has joined #openstack-swift | 08:05 | |
| *** proteusguy has joined #openstack-swift | 08:33 | |
| *** chlong has quit IRC | 08:45 | |
| *** annegentle has joined #openstack-swift | 08:53 | |
| *** aix has joined #openstack-swift | 08:58 | |
| *** annegentle has quit IRC | 08:58 | |
| *** SkyRocknRoll has quit IRC | 09:14 | |
| *** SkyRocknRoll has joined #openstack-swift | 09:15 | |
| *** annegentle has joined #openstack-swift | 09:53 | |
| *** annegentle has quit IRC | 09:59 | |
| *** geaaru has quit IRC | 10:13 | |
| *** aix has quit IRC | 10:52 | |
| *** annegentle has joined #openstack-swift | 10:54 | |
| *** annegentle has quit IRC | 11:00 | |
| *** aix has joined #openstack-swift | 11:04 | |
| *** SkyRocknRoll has quit IRC | 11:10 | |
| *** SkyRocknRoll has joined #openstack-swift | 11:25 | |
| *** haomaiwa_ has quit IRC | 11:37 | |
| *** jistr is now known as jistr|class | 11:40 | |
| *** haomaiwang has joined #openstack-swift | 11:42 | |
| *** annegentle has joined #openstack-swift | 11:55 | |
| *** annegentle has quit IRC | 12:00 | |
| *** annegentle has joined #openstack-swift | 12:18 | |
| *** fthiagogv has joined #openstack-swift | 12:30 | |
| *** bkopilov_wfh has quit IRC | 12:31 | |
| *** kei_yama has quit IRC | 12:37 | |
| *** jistr|class is now known as jistr | 12:59 | |
| *** silor has joined #openstack-swift | 13:21 | |
| *** annegentle has quit IRC | 13:25 | |
| *** silor has quit IRC | 13:28 | |
| *** wbhuber has joined #openstack-swift | 13:32 | |
| *** dmorita has joined #openstack-swift | 13:32 | |
| *** silor has joined #openstack-swift | 13:33 | |
| *** annegentle has joined #openstack-swift | 13:34 | |
| *** annegentle has quit IRC | 13:37 | |
| *** zul has quit IRC | 13:43 | |
| *** annegentle has joined #openstack-swift | 13:50 | |
| *** annegentle has quit IRC | 13:56 | |
| *** admin6 has joined #openstack-swift | 14:09 | |
| *** bkopilov_wfh has joined #openstack-swift | 14:13 | |
| admin6 | Hi all, I’m trying to test erasure code with swift 2.3. I’ve declared a new storage policy (with index 3) but I’m wandering how to create the related ring, as one of the swift-ring-builder argument is the numer of replicas? | 14:26 |
| admin6 | shoud I declare only one replica? eg: swift-ring-builder object-3.builder 18 1 1 | 14:26 |
| *** minwoob has joined #openstack-swift | 14:27 | |
| glange | hang around all day and you might get an answer to that -- a lot of swift people are at the Openstack conference and are on Pacific time | 14:30 |
| glange | (I don't know the answer, myself) | 14:31 |
| swifterdarrell | admin6: iirc, the replica count needs to be equal to data + parity | 14:31 |
| admin6 | glange: :-) thanks… haven’t realize the openstack summit is currently on at vancouver | 14:32 |
| admin6 | swifterarrell: ok So I’ll as my EC is (10+4) I must declare swift-ring-builder object-3.builder 18 14 1 | 14:33 |
| tdasilva | admin6: here's an example of a EC storage policy: https://github.com/openstack/swift/blob/master/doc/saio/bin/remakerings | 14:33 |
| tdasilva | object-2 is the EC policy | 14:34 |
| *** zul has joined #openstack-swift | 14:34 | |
| admin6 | Is object2 on a (4+2) EC storage policy ? | 14:35 |
| tdasilva | yes | 14:36 |
| admin6 | ok. Thanks | 14:36 |
| tdasilva | https://github.com/openstack/swift/blob/master/doc/saio/swift/swift.conf | 14:36 |
| *** thurloat_isgone is now known as thurloat | 14:40 | |
| *** silor has quit IRC | 14:52 | |
| *** jkugel has joined #openstack-swift | 14:53 | |
| *** marzif has joined #openstack-swift | 15:01 | |
| *** zul has quit IRC | 15:10 | |
| *** zynisch_o7 has joined #openstack-swift | 15:10 | |
| *** zynisch_o7 has quit IRC | 15:15 | |
| *** vjujjuri has joined #openstack-swift | 15:19 | |
| *** nadeem has joined #openstack-swift | 15:21 | |
| *** nadeem has quit IRC | 15:22 | |
| *** jamielennox|away is now known as jamielennox | 15:25 | |
| *** Fin1te has joined #openstack-swift | 15:26 | |
| *** vjujjuri has quit IRC | 15:26 | |
| *** zz_petertr7 is now known as petertr7 | 15:27 | |
| *** nadeem has joined #openstack-swift | 15:27 | |
| *** petertr7 is now known as zz_petertr7 | 15:38 | |
| *** vjujjuri has joined #openstack-swift | 15:47 | |
| *** zz_petertr7 is now known as petertr7 | 15:48 | |
| *** vjujjuri_ has joined #openstack-swift | 15:52 | |
| *** dmorita has quit IRC | 15:52 | |
| *** vjujjuri has quit IRC | 15:55 | |
| *** vjujjuri_ is now known as vjujjuri | 15:55 | |
| *** Fin1te has quit IRC | 15:56 | |
| *** jistr has quit IRC | 15:57 | |
| *** jamielennox is now known as jamielennox|away | 15:57 | |
| openstackgerrit | Pete Zaitcev proposed openstack/swift: Allow SAIO to answer is_local_device() better https://review.openstack.org/183395 | 16:00 |
| *** dmorita has joined #openstack-swift | 16:04 | |
| openstackgerrit | Darrell Bishop proposed openstack/swift: Allow one object-server per disk deployment https://review.openstack.org/184189 | 16:05 |
| *** zynisch_o7 has joined #openstack-swift | 16:08 | |
| *** admin6 has left #openstack-swift | 16:09 | |
| *** kajinamit has joined #openstack-swift | 16:10 | |
| *** dmorita has quit IRC | 16:15 | |
| *** petertr7 is now known as zz_petertr7 | 16:18 | |
| *** dmorita has joined #openstack-swift | 16:20 | |
| *** SkyRocknRoll has quit IRC | 16:29 | |
| *** kajinamit has quit IRC | 16:35 | |
| *** dmorita has quit IRC | 16:39 | |
| *** dmorita has joined #openstack-swift | 16:42 | |
| *** silor has joined #openstack-swift | 16:47 | |
| openstackgerrit | Darrell Bishop proposed openstack/swift: Allow one object-server per disk deployment https://review.openstack.org/184189 | 16:48 |
| openstackgerrit | Darrell Bishop proposed openstack/swift: Allow one object-server per disk deployment https://review.openstack.org/184189 | 16:50 |
| *** zynisch_o7 has quit IRC | 16:50 | |
| *** zynisch_o7 has joined #openstack-swift | 16:53 | |
| *** zynisch_o7 has quit IRC | 16:58 | |
| *** fthiagogv_ has joined #openstack-swift | 17:00 | |
| *** fthiagogv has quit IRC | 17:03 | |
| *** dmorita has quit IRC | 17:10 | |
| *** dmorita has joined #openstack-swift | 17:12 | |
| *** thumpba has joined #openstack-swift | 17:12 | |
| *** thumpba_ has joined #openstack-swift | 17:14 | |
| *** dmorita has quit IRC | 17:16 | |
| *** thumpba has quit IRC | 17:17 | |
| *** zynisch_o7 has joined #openstack-swift | 17:22 | |
| *** dmorita has joined #openstack-swift | 17:27 | |
| *** silor has quit IRC | 17:33 | |
| *** silor has joined #openstack-swift | 17:34 | |
| *** zhill has joined #openstack-swift | 17:38 | |
| *** geaaru has joined #openstack-swift | 17:47 | |
| *** aix has quit IRC | 17:51 | |
| *** geaaru has quit IRC | 17:52 | |
| *** david-lyle has joined #openstack-swift | 17:53 | |
| *** Fin1te has joined #openstack-swift | 17:59 | |
| *** dmorita has quit IRC | 18:00 | |
| *** david-lyle has quit IRC | 18:01 | |
| *** david-lyle has joined #openstack-swift | 18:07 | |
| *** zz_petertr7 is now known as petertr7 | 18:11 | |
| *** jamielennox|away is now known as jamielennox | 18:12 | |
| *** zul has joined #openstack-swift | 18:13 | |
| *** dmorita has joined #openstack-swift | 18:17 | |
| *** proteusguy has quit IRC | 18:19 | |
| *** ppai has joined #openstack-swift | 18:20 | |
| *** silor has quit IRC | 18:21 | |
| *** dmorita has quit IRC | 18:28 | |
| *** dmorita has joined #openstack-swift | 18:29 | |
| *** fthiagogv_ has quit IRC | 18:48 | |
| *** barra204_ has joined #openstack-swift | 18:50 | |
| *** barra204 has joined #openstack-swift | 18:50 | |
| *** dmorita has quit IRC | 18:51 | |
| *** silor has joined #openstack-swift | 18:54 | |
| *** silor1 has joined #openstack-swift | 18:56 | |
| *** david-lyle has quit IRC | 18:57 | |
| *** zul has quit IRC | 18:57 | |
| *** silor has quit IRC | 18:58 | |
| *** ppai has quit IRC | 18:59 | |
| *** david-lyle has joined #openstack-swift | 19:01 | |
| *** dmorita has joined #openstack-swift | 19:01 | |
| *** jamielennox is now known as jamielennox|away | 19:02 | |
| *** Fin1te has quit IRC | 19:04 | |
| *** barra204__ has joined #openstack-swift | 19:05 | |
| *** barra204_ has quit IRC | 19:05 | |
| *** openstackgerrit has quit IRC | 19:06 | |
| *** openstackgerrit has joined #openstack-swift | 19:06 | |
| *** silor1 has quit IRC | 19:06 | |
| *** barra204 has quit IRC | 19:06 | |
| *** barra204 has joined #openstack-swift | 19:06 | |
| *** logan2 has quit IRC | 19:07 | |
| *** logan2 has joined #openstack-swift | 19:08 | |
| *** jamielennox|away is now known as jamielennox | 19:12 | |
| *** jamielennox is now known as jamielennox|away | 19:13 | |
| *** silor has joined #openstack-swift | 19:16 | |
| *** Fin1te has joined #openstack-swift | 19:20 | |
| *** barra204__ has quit IRC | 19:22 | |
| *** barra204 has quit IRC | 19:22 | |
| *** harlowja has joined #openstack-swift | 19:27 | |
| *** csd_ has joined #openstack-swift | 19:29 | |
| *** zynisch__ has joined #openstack-swift | 19:29 | |
| *** zynisch_o7 has quit IRC | 19:30 | |
| *** barra204 has joined #openstack-swift | 19:35 | |
| *** barra204__ has joined #openstack-swift | 19:35 | |
| *** barra204 has quit IRC | 19:37 | |
| *** barra204__ has quit IRC | 19:37 | |
| *** csd_ has quit IRC | 19:37 | |
| *** barra204 has joined #openstack-swift | 19:37 | |
| *** barra204__ has joined #openstack-swift | 19:37 | |
| *** silor has quit IRC | 19:42 | |
| *** dmorita has quit IRC | 19:44 | |
| *** harlowja has quit IRC | 19:46 | |
| *** petertr7 is now known as zz_petertr7 | 19:48 | |
| *** david-lyle has quit IRC | 19:52 | |
| *** zhill has quit IRC | 19:57 | |
| *** Fin1te has quit IRC | 19:59 | |
| *** vinsh has joined #openstack-swift | 20:07 | |
| *** thurloat is now known as thurloat_isgone | 20:10 | |
| *** nadeem_ has joined #openstack-swift | 20:12 | |
| *** nadeem has quit IRC | 20:15 | |
| clayg | swifterdarrell: i suppose we should try to think of replica as number of devices to which the part will be assigned | 20:17 |
| *** zz_petertr7 is now known as petertr7 | 20:22 | |
| *** vinsh has quit IRC | 20:26 | |
| *** Fin1te has joined #openstack-swift | 20:29 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift: add object post and delete methods to BaseObjectController https://review.openstack.org/184322 | 20:30 |
| MooingLemur | is my undestanding correct that in 2.3.0, a revert action in object-reconstructor will fail if the primary node owns another fragment of the same object? | 20:32 |
| MooingLemur | and by owns, I mean it just happens to be sitting there on disk, it doesn't really own it | 20:32 |
| MooingLemur | then the reverter host throws an exception because it won't then try a handoff node? | 20:33 |
| MooingLemur | due to https://bugs.launchpad.net/swift/+bug/1452619 | 20:33 |
| openstack | Launchpad bug 1452619 in OpenStack Object Storage (swift) "object-reconstructor crash in ssync_sender" [Undecided,Confirmed] | 20:33 |
| MooingLemur | in which case, if two devices have each other's fragments of the same object, it will never right itself? | 20:34 |
| clayg | MooingLemur: there's that bug, and another one too I think - all the stuff we've found so far is in launchpad | 20:35 |
| clayg | MooingLemur: that one has a patch on it that should apply fairly cleanly if you have a cluster in this state and the ability to test it out and provide feedback | 20:35 |
| glange | it's only beta, right? | 20:35 |
| MooingLemur | clayg: I'm just making sure I understand the current state of things with the bugs :) | 20:35 |
| clayg | acoles_away: and I will get a chance to talk about it more either tonight or tomorrow - but definately in the design sessions on Thursday and Friday | 20:36 |
| MooingLemur | yeah, I think I got my cluster into this state by bringing a device weight down to 0, then rebalancing with min_part_hours set to 0, and it moved way too much at once. | 20:36 |
| clayg | MooingLemur: yeah there was a simple KeyError when reverting a handoff frag while the primary node was down - which was easy enough to fix, but it lead us into the current rabbit hole with the 409s and multiple fragments on a node | 20:37 |
| clayg | MooingLemur: you're in a great spot to apply the attached patched - it should unwedge things quickly :D | 20:38 |
| clayg | MooingLemur: you know... if it works :P | 20:38 |
| MooingLemur | I'll apply the patch associated with that bug | 20:38 |
| clayg | MooingLemur: cool! acoles_away's change has a set of dependencies - I think the patch file I attached will also work and be a bit simpler to apply | 20:39 |
| clayg | MooingLemur: GOOD LUCK! | 20:39 |
| MooingLemur | I'm not sure I fully understand in what cases multiple fragments would be allowed on a single node | 20:39 |
| MooingLemur | does this patch allow that where it wasn't allowed before? | 20:40 |
| clayg | MooingLemur: yes | 20:40 |
| clayg | it introduces the case where a primary node will hold it's primary fragment even if it hasn't yet reverted a handoff fragment | 20:40 |
| MooingLemur | primaries must eat, even if they're not hungry | 20:40 |
| clayg | the only reason a primary node would be holding a non-primary fragment is when the consistency processes get a series of ring changes behind (either via pretend_min_part_passed or if they just weren't runing for a long time while the rings were getting changed and pushed out) | 20:41 |
| clayg | MooingLemur: YES! | 20:42 |
| clayg | MooingLemur: do you job lazy primary | 20:42 |
| MooingLemur | so just this patch? https://launchpadlibrarian.net/206446594/primaries-must-eat-primary-index-frags.patch | 20:42 |
| MooingLemur | or should I grab another larger one? | 20:43 |
| *** Fin1te has quit IRC | 20:43 | |
| MooingLemur | clayg: thanks :D | 20:44 |
| MooingLemur | https://bpaste.net/show/5f9947a26f2d :P | 20:47 |
| clayg | gentoo-portage-what-now!? | 20:48 |
| clayg | MooingLemur: yeah anyone that's one I'm currently got the higest hopes for | 20:48 |
| *** dmorita has joined #openstack-swift | 20:53 | |
| *** david-lyle has joined #openstack-swift | 20:55 | |
| *** dmorita has quit IRC | 20:57 | |
| *** david-lyle has quit IRC | 20:58 | |
| *** david-lyle has joined #openstack-swift | 21:01 | |
| *** dmorita has joined #openstack-swift | 21:02 | |
| openstackgerrit | Bill Huber proposed openstack/swift: Generate log warning messages when PUTs are smaller than segment size https://review.openstack.org/184325 | 21:06 |
| *** zul has joined #openstack-swift | 21:07 | |
| *** harlowja has joined #openstack-swift | 21:08 | |
| *** silor has joined #openstack-swift | 21:12 | |
| MooingLemur | clayg: This was happening a lot before the patch, but it's still happening now, too: https://bpaste.net/show/ff8f0ab11ef4 | 21:16 |
| MooingLemur | only on this one host now, though | 21:16 |
| *** vinsh has joined #openstack-swift | 21:17 | |
| clayg | MooingLemur: that looks like a different problem -> https://bugs.launchpad.net/swift/+bug/1452431 | 21:20 |
| openstack | Launchpad bug 1452431 in OpenStack Object Storage (swift) "some parts replicas assigned to duplicate devices in the ring" [Undecided,New] | 21:20 |
| clayg | MooingLemur: try running that script on your ring and let me know if it says we should cry | 21:21 |
| clayg | if it says SUCCESS then we can be happy doged that bullet | 21:21 |
| MooingLemur | https://bpaste.net/show/cc7529c5bf4e | 21:22 |
| MooingLemur | it's a 9+3 policy | 21:22 |
| *** barra204__ has quit IRC | 21:23 | |
| *** barra204_ has joined #openstack-swift | 21:23 | |
| *** barra204__ has joined #openstack-swift | 21:23 | |
| MooingLemur | ring looks like this: https://bpaste.net/show/3ec9250efa88 | 21:24 |
| *** barra204 has quit IRC | 21:24 | |
| clayg | MooingLemur: yeah sorry bro - try throwing something at torgomatic - i still don't have a really good plan for that | 21:24 |
| clayg | MooingLemur: well... you can *try* setting overload to like 100 million bajillion % and rebalance a bunch of times with pretend-min-part hours | 21:25 |
| clayg | MooingLemur: if you *create* the ring with a really big overload the ring tends to not get himself into this problem | 21:26 |
| MooingLemur | the hosts will eventually have roughly the same amount of space each eventually | 21:28 |
| *** zul has quit IRC | 21:33 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: Centralize header parsing https://review.openstack.org/184336 | 21:36 |
| *** dmorita has quit IRC | 21:37 | |
| *** petertr7 is now known as zz_petertr7 | 21:37 | |
| *** zz_petertr7 is now known as petertr7 | 21:38 | |
| MooingLemur | clayg: so my understanding of this problem is that it'll pretty much go away if you have sufficient devices and balanceable for optimal dispersion | 21:39 |
| *** Fin1te has joined #openstack-swift | 21:40 | |
| *** david-lyle has quit IRC | 21:41 | |
| *** zynisch__ has quit IRC | 21:41 | |
| *** dmorita has joined #openstack-swift | 21:44 | |
| *** ppai has joined #openstack-swift | 21:44 | |
| *** harlowja has quit IRC | 21:46 | |
| clayg | MooingLemur: yeah... well for some reason it seems "sticky" - the ring balancing won't immediately fix itself as soon as you give it enough overload or fix the dispersion issues | 21:49 |
| MooingLemur | yeah, I actually bumped the weight of a few zero-weight devices up, and one of them is stuck receiving no partitions | 21:49 |
| *** david-lyle has joined #openstack-swift | 21:49 | |
| MooingLemur | but now the partition count is happy | 21:50 |
| MooingLemur | so I'm gonna sync this ring out :) | 21:50 |
| MooingLemur | the locations of all the fragments are all fubared anyway | 21:51 |
| MooingLemur | I'm paying a visit to this set of machines this weekend anyway to replace some bad disks, and I was gonna swap the varied-size disks around a bit to make each host have about the same amount of storage | 21:53 |
| *** jkugel has quit IRC | 21:56 | |
| *** barra204 has joined #openstack-swift | 21:57 | |
| *** barra204__ has quit IRC | 21:57 | |
| *** barra204_ has quit IRC | 21:58 | |
| *** barra204_ has joined #openstack-swift | 21:58 | |
| MooingLemur | clayg: thanks for the help :) There's one more exception I'm seeing in my logs over and over again that I'm unsure about. I think that's the only outstanding thing. https://bpaste.net/show/504f137034a2 | 22:00 |
| MooingLemur | and I'm seeing tons of lines that say "Early disconnect" as well, probably related? | 22:01 |
| *** vinsh has quit IRC | 22:01 | |
| *** zul has joined #openstack-swift | 22:02 | |
| MooingLemur | oh, here's the other end of those: https://bpaste.net/show/423691e8615e | 22:03 |
| *** barra204_ is now known as shakamunyi | 22:04 | |
| openstackgerrit | Bill Huber proposed openstack/swift: Generate log warning messages when PUTs are smaller than segment size https://review.openstack.org/184340 | 22:07 |
| *** vinsh has joined #openstack-swift | 22:09 | |
| *** zynisch_o7 has joined #openstack-swift | 22:11 | |
| *** vinsh has quit IRC | 22:14 | |
| *** dmorita has quit IRC | 22:16 | |
| *** Fin1te has quit IRC | 22:18 | |
| *** david-lyle has quit IRC | 22:19 | |
| *** zul has quit IRC | 22:19 | |
| *** david-lyle has joined #openstack-swift | 22:21 | |
| *** vinsh has joined #openstack-swift | 22:23 | |
| *** zynisch_o7 has quit IRC | 22:26 | |
| *** david-lyle has quit IRC | 22:27 | |
| *** Fin1te has joined #openstack-swift | 22:31 | |
| *** ppai has quit IRC | 22:31 | |
| *** dmorita has joined #openstack-swift | 22:35 | |
| *** zynisch_o7 has joined #openstack-swift | 22:36 | |
| *** dmorita has quit IRC | 22:39 | |
| *** dmorita has joined #openstack-swift | 22:40 | |
| *** vinsh has quit IRC | 22:44 | |
| openstackgerrit | Bill Huber proposed openstack/swift: Generate log warning messages when PUTs are smaller than segment size https://review.openstack.org/184325 | 22:45 |
| *** barra204 has quit IRC | 22:46 | |
| *** shakamunyi has quit IRC | 22:46 | |
| *** wbhuber has quit IRC | 22:46 | |
| *** vinsh has joined #openstack-swift | 22:54 | |
| *** Fin1te has quit IRC | 22:57 | |
| *** asettle has joined #openstack-swift | 23:03 | |
| *** km has joined #openstack-swift | 23:03 | |
| swifterdarrell | clayg: torgomatic: ping? | 23:04 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Update container on fast-POST https://review.openstack.org/135380 | 23:05 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Add POST capability to ssync for .meta files https://review.openstack.org/138498 | 23:05 |
| *** Guest38176 is now known as mgagne | 23:05 | |
| *** mgagne has joined #openstack-swift | 23:05 | |
| *** vinsh has quit IRC | 23:07 | |
| swifterdarrell | clayg: torgomatic: could the proxy's "post quorum timeout" be hitting after 10 frags get PUT (in a 10+4 EC policy), resulting in Object PUT returning 503 for [201, 201, 201, 201, 201, 201, 201, 201, 201, 201] when some object-server backends go on to also succeed? | 23:08 |
| swifterdarrell | clayg: torgomatic: I'm looking at a PUT req that failed as above but all 14 object-server PUTs succeeded; I do have an aggressive "Proxy server post quorum timeout" of 0.08s... | 23:09 |
| *** dmorita has quit IRC | 23:13 | |
| swifterdarrell | clayg: torgomatic: I bumped that tunable up from 0.08 to 2 and still got some failures (this is a quite-overloaded 5-VM-in-one-laptop "cluster" w/default swift CLI thread concurrency levels), but a value of 10s pretty much eliminated the spurious failures | 23:14 |
| *** nadeem_ has quit IRC | 23:17 | |
| *** chlong has joined #openstack-swift | 23:19 | |
| *** chlong_ has joined #openstack-swift | 23:20 | |
| *** kei_yama has joined #openstack-swift | 23:21 | |
| *** chlong has quit IRC | 23:31 | |
| swifterdarrell | clayg: torgomatic: (and anyone else who loves & knows EC??), in ECObjectController._store_object, I see that policy.quorum (11 for my 10+4 EC policy) is used for "phase 1", but self._get_put_responses() apparently is allowed to succeed with but min_resp=2 responses (wat?), then the policy.quorum (11 for me) is used in self.best_response(...quorum_size=min_conns), allowing, I guess, < 11 responses to get counted because "2" wa | 23:32 |
| swifterdarrell | s satisfied within post_quorum_timeout, but only got, say 10 responses, not 11... | 23:32 |
| swifterdarrell | clayg: torgomatic: ah... this must be the same problem addressed here https://review.openstack.org/#/c/180795/ but I don't see mention of proxy post_quorum_timeout being a contributing factor | 23:34 |
| *** david-lyle has joined #openstack-swift | 23:35 | |
| *** dmorita has joined #openstack-swift | 23:36 | |
| occupant | hm, so why do XFS parititons sometimes unmount instantly and other times they take like 1-2 minutes each? | 23:37 |
| *** Fin1te has joined #openstack-swift | 23:45 | |
| *** zynisch_o7 has quit IRC | 23:50 | |
| *** silor has quit IRC | 23:51 | |
| *** zul has joined #openstack-swift | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!