| opendevreview | Merged openstack/pyeclib master: Test on CentOS 9 Stream https://review.opendev.org/c/openstack/pyeclib/+/888178 | 00:03 |
|---|---|---|
| paladox | Oh right timburke. So we have to get the list of segments that match the main filename and set the header on them? | 00:17 |
| opendevreview | Jianjian Huo proposed openstack/swift master: proxy-server: add a global memcache lock to reduce backend requests https://review.opendev.org/c/openstack/swift/+/890174 | 06:12 |
| edausq | paladox: about x-delete-at on large object, you can read that too: https://bugs.launchpad.net/python-swiftclient/+bug/1478830 | 07:56 |
| opendevreview | Alistair Coles proposed openstack/swift master: proxy-server: clarify variables in get_account_info() https://review.opendev.org/c/openstack/swift/+/890325 | 10:33 |
| opendevreview | Merged openstack/swift master: proxy: Bring back logging/metrics for get_*_info requests https://review.opendev.org/c/openstack/swift/+/884931 | 11:39 |
| opendevreview | Alistair Coles proposed openstack/swift master: proxy-server: handle multipart GET disconnect before read https://review.opendev.org/c/openstack/swift/+/890336 | 13:52 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: wsgi: bad request syntax response missing txn-id https://review.opendev.org/c/openstack/swift/+/887904 | 16:25 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: wsgi: bad request syntax response missing txn-id https://review.opendev.org/c/openstack/swift/+/887904 | 17:01 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: slo: Support GET/HEAD request with PartNumber https://review.opendev.org/c/openstack/swift/+/888954 | 17:08 |
| opendevreview | Tim Burke proposed openstack/swift master: py3: apply str_to_wsgi to includes in _get_shard_ranges https://review.opendev.org/c/openstack/swift/+/889113 | 17:29 |
| reid_g | Good afternoon. Anybody around. I'm wondering what this message means object-reconstructor: 10.41.67.44:6200/d30/16074 Unexpected response: ":ERROR: 0 '14.999893188476562 seconds: /srv/node/d30/.lock'" | 18:13 |
| opendevreview | Jianjian Huo proposed openstack/swift master: proxy: add new metrics to account/container_info cache for skip/miss https://review.opendev.org/c/openstack/swift/+/885798 | 18:13 |
| reid_g | I see this message occasionally. If I look at that device, the files are quite old. | 18:18 |
| reid_g | I don't see any attr or data in the files | 18:18 |
| timburke | reid_g, looks like another reconstructor already had the device locked; see https://github.com/openstack/swift/blob/master/swift/obj/diskfile.py#L1419-L1422 and https://github.com/openstack/swift/blob/2.31.1/etc/object-server.conf-sample#L190-L195 | 18:22 |
| timburke | it's not something i'd necessarily worry about -- if replication/reconstruction's going slow, though, despite your disks and network not seeming terribly busy, you might want to increase that config value | 18:25 |
| reid_g | Some are quite old '-rwxr-xr-x 1 swift swift 0 Oct 7 2021 /srv/node/d0/.lock' | 18:31 |
| reid_g | Should these be rm'd? | 18:31 |
| timburke | age of the lock file doesn't really matter much -- once it gets created, it'll hang around forever. the better question would be whether anyone's currently holding the lock, and if so, since when | 18:32 |
| timburke | only place they should get taken is in ssync_receiver, though -- so if you want to make sure they've been released, you could always restart your object servers | 18:34 |
| reid_g | Oh gotcha | 18:34 |
| reid_g | Still trying to figure out why I have so many handoffs that aren't being cleaned up | 18:34 |
| reid_g | They are only in EC rings. | 18:35 |
| reid_g | Also may have found a bug with containers. We have some container db files being quarantined 'due to no account replicas exist' | 18:37 |
| reid_g | I assume that this is because the account isn't found. | 18:38 |
| reid_g | The container db file get quarantined, put back in place by replicator, quarantined, put back. Fast forward I have 7500 quarantines of the same 110M db file. | 18:39 |
| timburke | might be worth shutting down the container replicators for a bit, long enough to get all the container-updaters to quarantine it. should definitely be safe to delete all the 1,000s of copies in quarantine, though the core question of how we got a container DB with no account for it might be worth investigating | 18:42 |
| timburke | i know OVH has seen that sort of a problem before, though: https://review.opendev.org/c/openstack/swift/+/704435 | 18:43 |
| timburke | we wound up getting the quarantine behavior in https://github.com/openstack/swift/commit/bcff1282 | 18:43 |
| timburke | https://bugs.launchpad.net/swift/+bug/1300850 and https://bugs.launchpad.net/swift/+bug/1514528 are probably worth reading through | 18:44 |
| timburke | on the EC handoffs, you might run with `handoffs_only = true` for a bit to really prioritize those handoffs. maybe switch to `replication_concurrency_per_device = 0`, too, and just rely on the partition-level locking. you'd probably want to go back to your old configs after a few days, though, to make sure durability isn't too impacted | 18:49 |
| reid_g | I had the handoffs_only=true for a week and I didn't notice any changes and reverted it. They all seemed to be blocked passing back to 1 server. | 18:52 |
| reid_g | That server would throw these messages "object-server: x.x.x.x/d8/26366 read failed in ssync.Receiver: missing_check start: invalid literal for int() with base 16: b''" | 18:52 |
| reid_g | ssync_receiver ^ | 18:52 |
| reid_g | the ssync_sender would have a timeout message | 18:52 |
| reid_g | I will check those other links about the containers | 18:53 |
| timburke | the timeout's weird, but probably explains the "invalid literal" error -- maybe max_objects_per_revert could help make sure we make progress? | 18:56 |
| reid_g | I will check that out. thanks! | 18:59 |
| timburke | so i forgot about a dentist appointment i've got today -- it'll be tough for me to get back in time for any of the meeting, much less the start. if someone else would like to chair, i'd appreciate it, but otherwise, i propose we skip one more week. sorry for not realizing that earlier :-/ | 19:03 |
| timburke | only thing i really wanted to call attention to was https://review.opendev.org/c/openstack/swift/+/889113 py3: apply str_to_wsgi to includes in _get_shard_ranges | 19:04 |
| zaitcev | Does anyone know anything about this: | 20:29 |
| zaitcev | [fedora@zaitcev-swift-6 swift-work]$ python setup.py develop | 20:29 |
| zaitcev | /usr/lib/python3.11/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer. | 20:30 |
| kota | good morning | 21:01 |
| mattoliver | Morning | 21:03 |
| kota | oic, skip one more week, i found it in the irc log. | 21:03 |
| kota | oh. just i found an e-mail to call for presentation of GTC 2024, which will be hold in-person! | 21:06 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: slo: Support GET/HEAD request with PartNumber https://review.opendev.org/c/openstack/swift/+/888954 | 21:08 |
| mattoliver | Ahh dentist appointment, good spot kota | 21:08 |
| kota | mattoliver: yeah, I'm planning to get back on my bed to expand morning rest ;) | 21:09 |
| mattoliver | Me too! | 21:09 |
| kota | yup, see you next time | 21:09 |
| mattoliver | Good to see you briefly kota 😀 | 21:09 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: Improve coverage for FakeSwift tests https://review.opendev.org/c/openstack/swift/+/889785 | 21:31 |
| opendevreview | Clay Gerrard proposed openstack/swift master: squash: make preference as explicit as possible https://review.opendev.org/c/openstack/swift/+/890369 | 21:52 |
| timburke | back -- sorry again for the late notice (and the third skipped meeting) | 21:56 |
| timburke | zaitcev, i feel like i've seen that warning on occasion -- haven't dug into it yet, though, sorry | 21:57 |
| zaitcev | timburke: okay. Didn't know if we cared. | 21:58 |
| timburke | eventually, surely ;-) | 21:59 |
| timburke | looks like https://peps.python.org/pep-0517/ has to do with the transition to pyproject.toml -- i should probably go read https://til.simonwillison.net/python/pyproject | 21:59 |
| timburke | maybe try swapping over something like pyeclib or python-swiftclient first | 22:00 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: Improve coverage for FakeSwift tests https://review.opendev.org/c/openstack/swift/+/889785 | 22:33 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: Improve coverage for FakeSwift tests https://review.opendev.org/c/openstack/swift/+/889785 | 22:35 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: fix head request on multipart-manifest raw format https://review.opendev.org/c/openstack/swift/+/890246 | 22:47 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!