| *** patchbot has quit IRC | 00:10 | |
| *** patchbot has joined #openstack-swift | 00:11 | |
| *** frankie64 has quit IRC | 00:12 | |
| *** gyee has quit IRC | 00:15 | |
| *** rcernin has quit IRC | 00:29 | |
| *** mikecmpbll has quit IRC | 00:30 | |
| *** awkwardpenguin has quit IRC | 01:26 | |
| *** awkwardpenguin has joined #openstack-swift | 01:27 | |
| *** paladox has quit IRC | 01:36 | |
| *** awkwardpenguin has quit IRC | 01:38 | |
| *** awkwardpenguin has joined #openstack-swift | 01:39 | |
| *** paladox has joined #openstack-swift | 01:43 | |
| openstackgerrit | Merged openstack/swift master: Enabling direct_client users to overwrite X-Timestamp https://review.openstack.org/603261 | 01:52 |
|---|---|---|
| *** rcernin has joined #openstack-swift | 02:01 | |
| *** two_tired has joined #openstack-swift | 02:05 | |
| *** itlinux_ has joined #openstack-swift | 02:28 | |
| *** itlinux has quit IRC | 02:31 | |
| openstackgerrit | Merged openstack/swift master: Refactor obj.server.ObjectController.PUT https://review.openstack.org/602205 | 02:33 |
| openstackgerrit | Merged openstack/swift master: Add validation for sorting_method values https://review.openstack.org/469269 | 02:34 |
| *** two_tired has quit IRC | 02:46 | |
| *** awkwardpenguin has quit IRC | 02:52 | |
| *** awkwardpenguin has joined #openstack-swift | 02:59 | |
| openstackgerrit | Merged openstack/swift master: Tighten status code assertions for object POST https://review.openstack.org/609468 | 03:03 |
| *** psachin has joined #openstack-swift | 03:07 | |
| *** awkwardpenguin has quit IRC | 03:25 | |
| *** awkwardpenguin has joined #openstack-swift | 03:26 | |
| *** awkwardpenguin has quit IRC | 04:43 | |
| *** awkwardpenguin has joined #openstack-swift | 04:59 | |
| *** awkwardpenguin has quit IRC | 05:05 | |
| *** hoonetorg has quit IRC | 05:09 | |
| *** awkwardpenguin has joined #openstack-swift | 05:33 | |
| *** awkwardpenguin has quit IRC | 05:38 | |
| *** pcaruana has joined #openstack-swift | 06:01 | |
| *** e0ne has joined #openstack-swift | 06:02 | |
| *** e0ne has quit IRC | 06:29 | |
| *** fatema_ has joined #openstack-swift | 06:32 | |
| *** fatema_ has quit IRC | 06:34 | |
| *** rcernin has quit IRC | 07:01 | |
| *** awkwardpenguin has joined #openstack-swift | 07:02 | |
| *** awkwardpenguin has quit IRC | 07:07 | |
| *** awkwardpenguin has joined #openstack-swift | 07:08 | |
| *** awkwardpenguin has quit IRC | 07:13 | |
| *** awkwardpenguin has joined #openstack-swift | 07:46 | |
| *** awkwardpenguin has quit IRC | 07:46 | |
| openstackgerrit | Merged openstack/swift master: Allow EC to ignore invalid request ETag https://review.openstack.org/602467 | 08:07 |
| openstackgerrit | Merged openstack/swift master: Pass empty etag override values through encrypter https://review.openstack.org/346106 | 08:07 |
| *** mvkr has quit IRC | 08:38 | |
| *** mikecmpbll has joined #openstack-swift | 08:43 | |
| *** e0ne has joined #openstack-swift | 09:05 | |
| *** mvkr has joined #openstack-swift | 09:09 | |
| *** jistr is now known as jistr|call | 09:12 | |
| *** gkadam has quit IRC | 09:25 | |
| *** e0ne has quit IRC | 09:27 | |
| *** e0ne has joined #openstack-swift | 09:43 | |
| *** e0ne has quit IRC | 10:35 | |
| *** e0ne has joined #openstack-swift | 10:38 | |
| DHE | I don't know if this is a swift thing or an eventlet thing (or something else), has SO_REUSEADDR been considered for services? | 10:41 |
| *** jistr|call is now known as jistr | 11:00 | |
| *** e0ne has quit IRC | 11:04 | |
| *** e0ne has joined #openstack-swift | 11:51 | |
| *** SkyRocknRoll has joined #openstack-swift | 11:54 | |
| DHE | so as a hack, SO_REUSEADDR is already in use so you can just launch, say, the object server multiple times to reap the benefits... | 12:23 |
| *** SkyRocknRoll has quit IRC | 12:42 | |
| *** SkyRocknRoll has joined #openstack-swift | 12:54 | |
| DHE | notmyname: you may recall last week (?) I was having trouble with the reconstructor on my one-machine 30-disk lab box. This seems to have resolved it. | 12:58 |
| *** e0ne has quit IRC | 13:11 | |
| *** e0ne has joined #openstack-swift | 13:21 | |
| *** psachin has quit IRC | 13:38 | |
| *** mvkr has quit IRC | 14:12 | |
| *** e0ne has quit IRC | 14:33 | |
| *** e0ne has joined #openstack-swift | 14:42 | |
| *** openstackgerrit has quit IRC | 14:58 | |
| *** openstackgerrit has joined #openstack-swift | 15:00 | |
| *** gyee has joined #openstack-swift | 15:08 | |
| *** e0ne has quit IRC | 15:09 | |
| *** e0ne has joined #openstack-swift | 15:20 | |
| *** e0ne has quit IRC | 15:20 | |
| *** openstackgerrit has quit IRC | 15:22 | |
| *** mvkr has joined #openstack-swift | 15:27 | |
| *** awkwardpenguin has joined #openstack-swift | 15:47 | |
| notmyname | good morning | 15:59 |
| *** mvkr has quit IRC | 16:16 | |
| timburke | DHE: i know there was https://review.openstack.org/#/c/137659/ a while ago... | 16:18 |
| patchbot | patch 137659 - swift - Support of the Linux socket option SO_REUSEPORT (ABANDONED) - 8 patch sets | 16:18 |
| timburke | and https://review.openstack.org/#/c/312791/ more recently | 16:19 |
| patchbot | patch 312791 - swift - Remove unneeded setting of SO_REUSEADDR. (MERGED) - 2 patch sets | 16:19 |
| timburke | https://review.openstack.org/#/c/463876/ might also be kinda relevant? | 16:20 |
| patchbot | patch 463876 - swift - Open-code eventlet.listen() (MERGED) - 2 patch sets | 16:20 |
| *** SkyRocknRoll has quit IRC | 16:28 | |
| *** mikecmpbll has quit IRC | 16:31 | |
| DHE | timburke: reuseaddr is different from reuseport | 16:46 |
| DHE | it may help with scaling some issues | 16:46 |
| DHE | *some scaling issues | 16:47 |
| DHE | I'm trying it with the object server and things are much better | 16:47 |
| DHE | but setting it is not enough. specific actions regarding socket creation are needed to take advantage. | 16:48 |
| timburke | bah -- sorry. i'm not actually super-familiar with socket options; just grabbing everything i kinda remember mentioning SO_REUSE* | 16:50 |
| timburke | i think we *do* have it set, though -- eventlet defaults it on for us: https://github.com/eventlet/eventlet/blob/af407c7/eventlet/convenience.py#L38 | 16:50 |
| timburke | so what are the other actions that would have to be taken? | 16:50 |
| DHE | the point of SO_REUSEADDR is that it allows multiple sockets to be created each bind()ing to and listen()ing to the same IP/port. | 16:51 |
| DHE | then the kernel will round robin incoming connections to each listen socket | 16:51 |
| DHE | rather than having one listen socket and one accept() queue that multiple processes fight over (since poll() will wake them all up when any incoming connection arrives) | 16:52 |
| DHE | but since the option is enabled, you can instead launch the object server with a low number of workers and just launch it multiple times to get the benefit | 16:52 |
| DHE | while I think all services could benefit, in my particular workload the object server is the one getting destroyed by load | 16:53 |
| DHE | does that make sense? | 16:54 |
| DHE | I guess the best description of "the action to be taken" is to fork() all your worker processes first, then have each one independently make its own listen socket with SO_REUSEADDR. | 16:57 |
| DHE | rather than the traditional wisedom of making a listen socket, then forking your worker processes | 16:57 |
| *** awkwardpenguin has quit IRC | 16:57 | |
| *** awkwardpenguin has joined #openstack-swift | 17:04 | |
| *** awkwardpenguin has quit IRC | 17:29 | |
| *** ianychoi has quit IRC | 17:41 | |
| *** alecuyer has quit IRC | 17:48 | |
| *** alecuyer has joined #openstack-swift | 17:49 | |
| *** mvkr has joined #openstack-swift | 18:00 | |
| *** openstackstatus has quit IRC | 18:28 | |
| timburke | so i'm thinking about what happens when we find a non-partition in the replicator... | 18:28 |
| timburke | in particular, over around https://github.com/openstack/swift/blob/2.19.0/swift/obj/replicator.py#L824-L852 | 18:28 |
| timburke | if just *clearly not a partition* and the int() fails... we mark every device in the ring as failed?? that doesn't seem right... | 18:29 |
| *** SkyRocknRoll has joined #openstack-swift | 18:29 | |
| *** openstackstatus has joined #openstack-swift | 18:30 | |
| *** ChanServ sets mode: +v openstackstatus | 18:30 | |
| timburke | but if it's *close enough* to being a valid partition... we get back an empty list from get_part_nodes... which ends up flagging the current node as a handoff, replication trivially succeeds, and we *delete* whatever it was that we found | 18:30 |
| timburke | ...which also doesn't seem great... | 18:31 |
| *** SkyRocknRoll has quit IRC | 18:34 | |
| timburke | so i guess my question is: what do we *want* to happen when you see a directory (or file!) named '1234' or '-12' or 'foo' when your part power says partitions should be in the range [0, 1023]? | 18:34 |
| DHE | 1024 could happen if something goes wonky during a part power increase... | 18:36 |
| notmyname | timburke: I thought we had a check to simply pass over "weird" stuff when it's found. IOW an admin may have dropped a dir in there for whatever reason and we shouldn't try to parse it nor try to delete it. just ignore it | 18:37 |
| notmyname | but in your case, you're talking about something that looks like a valid part number (naming-wise) but not valid according to the ring that's available | 18:37 |
| notmyname | ? | 18:37 |
| DHE | as an aside, what would happen if int() is given something like 0123. Are there two possible instances of the same partition? | 18:39 |
| timburke | notmyname: yup. and fwiw, looks like reconstructor is more aggressive about non-int directories: https://github.com/openstack/swift/blob/2.19.0/swift/obj/reconstructor.py#L1088-L1093 | 18:39 |
| notmyname | hmm.. that doesn't seem right | 18:40 |
| timburke | DHE: seems like... and yeah, i'm a little worried about part-power increases, too | 18:40 |
| DHE | 6 instances of the object-server (as launched from a shell) solves all my reconstructor random failures problems | 18:42 |
| timburke | also fwiw, we apparently have tests to demonstrate that the replicator will delete invalid partitions: https://github.com/openstack/swift/blob/2.19.0/test/unit/obj/test_replicator.py#L1429 | 18:42 |
| notmyname | timburke: a quick search didn't show the commit I was looking for where we ignore unexpected directories (I know it's somewhere though) | 18:48 |
| timburke | notmyname: looks like the reconstructor behavior goes back to the start of EC: https://github.com/openstack/swift/commit/647b66a2ce4c85c43dcca49776d35c5ebb9cf15e#diff-3664f9c054deedefd2a5843f6625585cR812 | 18:49 |
| notmyname | doesn't surprise me | 18:49 |
| *** mikecmpbll has joined #openstack-swift | 18:58 | |
| timburke | all i wanted to do was click +2/+A on https://review.openstack.org/#/c/402522 but now i find myself trapped by ambivalence about when it's ok for swift to just *delete* data :-( | 19:13 |
| patchbot | patch 402522 - swift - get_part_nodes should raise error on invalid part - 2 patch sets | 19:13 |
| *** SkyRocknRoll has joined #openstack-swift | 20:13 | |
| *** pcaruana has quit IRC | 20:40 | |
| *** SkyRocknRoll has quit IRC | 21:51 | |
| *** mikecmpbll has quit IRC | 22:13 | |
| *** openstackgerrit has joined #openstack-swift | 22:21 | |
| *** rcernin has joined #openstack-swift | 22:23 | |
| *** openstackstatus has quit IRC | 22:28 | |
| mattoliverau | morning | 22:29 |
| *** openstackstatus has joined #openstack-swift | 22:29 | |
| *** ChanServ sets mode: +v openstackstatus | 22:29 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Allow arbitrary UTF-8 strings as delimiters in container listings https://review.openstack.org/609843 | 22:35 |
| timburke | i should probably do ^^^ for account servers, too, but one step at a time | 22:36 |
| *** spotz has quit IRC | 23:17 | |
| *** mattoliverau has quit IRC | 23:18 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add slo_validator callback https://review.openstack.org/609860 | 23:52 |
| *** spotz has joined #openstack-swift | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!