| *** rcernin has quit IRC | 00:28 | |
| *** rcernin has joined #openstack-swift | 00:31 | |
| *** djhankb has quit IRC | 01:01 | |
| *** djhankb has joined #openstack-swift | 01:04 | |
| *** gyee has quit IRC | 02:54 | |
| *** rcernin has quit IRC | 03:53 | |
| *** rcernin has joined #openstack-swift | 04:15 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-swift | 04:33 | |
| *** manuvakery has joined #openstack-swift | 04:52 | |
| *** shanu_ has joined #openstack-swift | 06:27 | |
| *** rcernin has quit IRC | 06:28 | |
| *** rcernin has joined #openstack-swift | 06:45 | |
| *** rcernin has quit IRC | 06:45 | |
| *** rcernin has joined #openstack-swift | 06:46 | |
| *** manuvakery has quit IRC | 07:02 | |
| *** mikecmpbll has joined #openstack-swift | 07:54 | |
| *** rcernin has quit IRC | 09:10 | |
| shanu_ | I am developing an application which uses openstack swift in the backend which supports S3APIs as well. When I upload file using multi-part upload for two or more times using S3 access keys via cyberduck, it was noticed that multiple version of the file is created inside `*+segments` container under path `{filename}/{key}/`. When querying for versions via AWS cli using `aws s3api list-object-versions` the | 09:31 |
|---|---|---|
| shanu_ | object has "VersionId": "null". Same was verified from aws s3api head-object were there was no key `VersionId` in the headers. | 09:31 |
| shanu_ | Deleting the file manifest from cyberduck deletes the segments that was created lately(on 2nd upload) and the previous segments are left untouched. There is no way to determine and delete such segments of the object. The bucket is not version enabled. | 09:31 |
| shanu_ | I am looking for a way to determine and delete such segments of the object. | 09:31 |
| shanu_ | Also when performing similar uploads using SWIFT cli, uploading the same file again overrides the existing segments of the object. Upload was done with `swift upload --use-slo`. | 09:31 |
| *** mikecmpbll has quit IRC | 09:57 | |
| *** mikecmpbll has joined #openstack-swift | 09:59 | |
| *** abelur has quit IRC | 10:11 | |
| *** donnyd has quit IRC | 10:11 | |
| *** samueldmq has quit IRC | 10:11 | |
| *** fyx has quit IRC | 10:11 | |
| *** jrosser has quit IRC | 10:11 | |
| *** mattoliverau has quit IRC | 10:11 | |
| *** corvus has quit IRC | 10:11 | |
| *** corvus has joined #openstack-swift | 10:11 | |
| *** donnyd has joined #openstack-swift | 10:11 | |
| *** abelur has joined #openstack-swift | 10:11 | |
| *** jrosser has joined #openstack-swift | 10:11 | |
| *** samueldmq has joined #openstack-swift | 10:16 | |
| *** mattoliverau has joined #openstack-swift | 10:16 | |
| *** tepper.freenode.net sets mode: +v mattoliverau | 10:16 | |
| *** tkajinam has quit IRC | 10:22 | |
| *** rcernin has joined #openstack-swift | 11:06 | |
| *** rcernin has quit IRC | 11:20 | |
| *** mikecmpbll has quit IRC | 11:26 | |
| *** mikecmpbll has joined #openstack-swift | 11:32 | |
| *** manuvakery has joined #openstack-swift | 11:47 | |
| rledisez | timburke: clayg about p 337861, the goal is to have only one instance of the object server while having different replication processes for each disk. IMHO, the same reasons we have servers_per_port for client-network apply to the replication-network. That's why we use this patch. | 13:01 |
| patchbot | https://review.opendev.org/#/c/337861/ - swift - Permit to bind object-server on replication_port - 7 patch sets | 13:01 |
| clayg | rledisez: having the process isolation seems reasonable - on servers with lots of disks in the chassis how do you balances # of client processes to # of replication processes per disk? | 13:07 |
| *** rdejoux has joined #openstack-swift | 13:10 | |
| rledisez | clayg: right now we use the same number, I didn't take time to split that. we usually use 2 or 4 | 13:14 |
| clayg | it seems reasonable to me that we'd support it - is that patch ready to go? i don't hear you complaining about getting it merged every week 😁 | 13:15 |
| clayg | i haven't seen zaitcev in a few 😞 | 13:15 |
| rledisez | we've been running it in production for few years now, so I guess it's "okay" :) | 13:15 |
| clayg | shanu_: the swift command line client does a HEAD request on the object before overwriting it, if it's a SLO it will do a multipart-manifest=delete https://docs.openstack.org/swift/latest/overview_large_objects.html#deleting-a-large-object | 13:17 |
| clayg | I thought s3api emulation tries to do something similar, but I didn't see the code for it any bug so i'm not sure... | 13:17 |
| clayg | shanu_: but yeah it looks like s3api leaves orphan segments behind in the +segments container on overwrites pretty reliably | 13:20 |
| *** mikecmpbll has quit IRC | 13:59 | |
| *** mikecmpbll has joined #openstack-swift | 14:01 | |
| *** gmann is now known as gmann_afk | 14:05 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: add swift-manage-shard-ranges shrink command https://review.opendev.org/741721 | 14:37 |
| timburke | shanu_, clayg: yeah, we really need to get to addressing https://bugs.launchpad.net/swift/+bug/1813202 :-/ | 14:40 |
| openstack | Launchpad bug 1813202 in OpenStack Object Storage (swift) "s3api doesn't clean up multipart upload parts when MU is overwritten" [Undecided,New] | 14:40 |
| clayg | timburke: gah! I could not find that when I was looking for it! | 14:41 |
| timburke | it's currently rather annoying, though: we'd need to issue a delete that may take a long time to complete before we start reading any of the client bytes; as a result, the client may well time out before we've had a chance to start any backend PUTs | 14:42 |
| clayg | does your SLO async delete API address the bug? | 14:42 |
| timburke | not yet -- https://review.opendev.org/#/c/733026/ just makes that delete way faster -- there still need to be changes to s3api to do the HEAD-then-delete on top of it | 14:43 |
| patchbot | patch 733026 - swift - Add a new URL parameter to allow for async cleanup... - 4 patch sets | 14:43 |
| timburke | note that we could probably also address https://bugs.launchpad.net/swift/+bug/1691523 at the same time | 14:44 |
| openstack | Launchpad bug 1691523 in OpenStack Object Storage (swift) "Multi-delete does not remove SLO segments" [Undecided,New] | 14:44 |
| timburke | (though for some reason, i thought we'd fixed that already...) | 14:44 |
| timburke | oh! no, that's for the normal swift bulk-delete -- never mind, that'd be a completely unrelated change | 14:47 |
| timburke | s3api's delete-multiple will handle MPUs fine, though there was a bugfix not so long ago: https://github.com/openstack/swift/commit/b35fc41 | 14:48 |
| ormandj | timburke: oh man, that segments one explains a lot | 15:04 |
| *** mikecmpbll has quit IRC | 15:10 | |
| *** mikecmpbll has joined #openstack-swift | 15:17 | |
| timburke | ormandj, fwiw i wrote up a script that can clean up a single account at a time: http://paste.openstack.org/show/797781/ | 15:23 |
| timburke | i think we maybe saw some issues under high concurrency | 15:24 |
| timburke | ormandj, cwright: oh! i thought of something late yesterday about the switch to servers-per-port -- you may want to change one server over, update the ring for that one server, push the updated ring out to all nodes, then repeat for each server | 15:32 |
| timburke | moving slowly like that should help prevent availability issues | 15:32 |
| clayg | timburke: we ended up making sure that the first disk on a server used the same port as the old ring - that way even proxies were using old rings (as they do for 30 until they check the mtime) they can still connect (all servers can still route to ask disks) | 15:36 |
| *** mikecmpbll has quit IRC | 16:01 | |
| ormandj | timburke: we will play it safe and do that, no problem at all, it's all automated anyways | 17:04 |
| ormandj | thank you for the head's up, we've had enough surprises :p | 17:04 |
| ormandj | timburke: re: the cleanup, appreciate that too, but we will have to hope for the fix, because the manual workload + billing fun from doing that for thousands upon thousands of customers will not be pretty :) | 17:05 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Authors/ChangeLog for 2.26.0 https://review.opendev.org/750537 | 17:22 |
| *** gyee has joined #openstack-swift | 17:31 | |
| *** gmann_afk is now known as gmann | 17:42 | |
| *** djhankb has quit IRC | 17:49 | |
| *** djhankb has joined #openstack-swift | 17:49 | |
| *** manuvakery has quit IRC | 18:27 | |
| ormandj | so swift-dispersion-report got sorted in: https://bugs.launchpad.net/swift/+bug/1468374 | 18:36 |
| openstack | Launchpad bug 1468374 in OpenStack Object Storage (swift) "swift dispersion does not support keystone auth v3" [Undecided,Fix released] - Assigned to Falk Reimann (falk-reimann) | 18:36 |
| ormandj | however, we're seeing the same issue with swift-dispersion-populate | 18:36 |
| ormandj | sorry, wrong bug: https://bugs.launchpad.net/swift/+bug/1863680 | 18:37 |
| openstack | Launchpad bug 1863680 in OpenStack Object Storage (swift) "python3 swift-dispersion-report issue" [High,Fix released] - Assigned to Sam Morrison (sorrison) | 18:37 |
| ormandj | we see what looks to be the same problem with populate re: python3 | 18:37 |
| openstackgerrit | OpenStack Release Bot proposed openstack/python-swiftclient stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/751381 | 18:40 |
| openstackgerrit | OpenStack Release Bot proposed openstack/python-swiftclient master: Update master for stable/victoria https://review.opendev.org/751382 | 18:40 |
| openstackgerrit | OpenStack Release Bot proposed openstack/python-swiftclient master: Add Python3 wallaby unit tests https://review.opendev.org/751383 | 18:40 |
| ormandj | timburke: was able to patch it to work, will link a diff | 18:40 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: Fix swift-dispersion-populate https://review.opendev.org/751384 | 18:46 |
| timburke | ormandj, i'm guessing something like ^^^ ? | 18:46 |
| timburke | thanks for the report -- sorry i'd missed that somehow when i was fixing up the patch to -report | 18:47 |
| ormandj | timburke: https://bugs.launchpad.net/swift/+bug/1895346 | 18:52 |
| openstack | Launchpad bug 1895346 in OpenStack Object Storage (swift) "swift-dispersion-populate and python3" [Undecided,New] | 18:52 |
| ormandj | lol | 18:52 |
| ormandj | yeah you called it :p | 18:52 |
| ormandj | you can close the report obviously | 18:53 |
| ormandj | sorry hadn't seen your message when i hit submit | 18:53 |
| ormandj | timburke: in interesting news, we changed ring to update ports for server_per_port but didn't update the replication port | 18:56 |
| ormandj | however the replicator is trying to use the non-replication port | 18:56 |
| ormandj | and it's broken all replication :) | 18:56 |
| ormandj | well, i should be more clear - let me write this up and i'll update - too much to stream of conscious in here. sorry :) | 18:57 |
| ormandj | protip: no servers_per_port in replication configuration file. /ignoreme | 19:04 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: Fix swift-dispersion-populate https://review.opendev.org/751384 | 19:35 |
| timburke | that bug was great; i shouldn't have piggy-backed off the old one. updated the bug refs on ^ | 19:36 |
| openstackgerrit | Clay Gerrard proposed openstack/python-swiftclient master: basic connection close tracking infra https://review.opendev.org/751418 | 19:47 |
| clayg | timburke: ormandj: I had "review s3api quota patch" on my todo list for today - is that p 749382 ??? | 20:24 |
| patchbot | https://review.opendev.org/#/c/749382/ - swift - s3api: Make quota-exceeded errors more obvious - 2 patch sets | 20:24 |
| clayg | cause *that* looks like a big honkin' refactor tied up with a minor cosmetic improvement? | 20:25 |
| timburke | clayg, yeah, that's the patch. sorry, it *is* getting away from me a bit | 21:22 |
| timburke | if you'd rather, we could stick with something like the first patchset -- it's much more surgical, but at some point we need to admit that the look-up-table style of things there is insufficient for what we'd like to express | 21:22 |
| clayg | it's just akward - i *like* the refactoring - normally what'd I do is try to play with some and see if I can "keep going" make it look _even better_ (because the process of changing the code will increase my confidence in the test coverage) | 21:45 |
| clayg | but OTOH, it obviously fixes the bug - so I'd also just like to :shipit: | 21:45 |
| clayg | TENSION | 21:45 |
| clayg | do you have any sense of how well existing tests are exercising all of those status response transformations? | 21:46 |
| clayg | maybe i could just throw in a few name errors | 21:47 |
| clayg | idk man that "raise KeyError" tho 🤮 | 21:51 |
| *** rcernin has joined #openstack-swift | 22:02 | |
| *** rcernin has quit IRC | 22:03 | |
| *** rcernin has joined #openstack-swift | 22:03 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Make quota-exceeded errors more obvious https://review.opendev.org/749382 | 22:37 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Refactor how we handle expected errors https://review.opendev.org/751555 | 22:37 |
| *** gyee has quit IRC | 22:46 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: Clean up some requirements https://review.opendev.org/750475 | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!