| opendevreview | ASHWIN A NAIR proposed openstack/swift master: proxy-server exception logging shows replication_ip/port https://review.opendev.org/c/openstack/swift/+/860866 | 01:59 |
|---|---|---|
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: proxy-server exception logging shows replication_ip/port https://review.opendev.org/c/openstack/swift/+/860866 | 02:53 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: proxy-server exception logging shows replication_ip/port https://review.opendev.org/c/openstack/swift/+/860866 | 03:25 |
| opendevreview | Tim Burke proposed openstack/swift master: CI: Install pip from pinned upstream for py2 probe tests https://review.opendev.org/c/openstack/swift/+/862758 | 03:45 |
| opendevreview | Tim Burke proposed openstack/swift master: CI: Install pip from pinned upstream for py2 probe tests https://review.opendev.org/c/openstack/swift/+/862758 | 04:22 |
| opendevreview | Tim Burke proposed openstack/swift master: CI: Install pip from pinned upstream for py2 probe tests https://review.opendev.org/c/openstack/swift/+/862758 | 04:23 |
| opendevreview | Tim Burke proposed openstack/swift master: CI: Install pip from pinned upstream for py2 probe tests https://review.opendev.org/c/openstack/swift/+/862758 | 04:25 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Sharder: add more logs during container cleaving. https://review.opendev.org/c/openstack/swift/+/862775 | 05:21 |
| mcape | hello all... i've run into problem with sharding containers... i have about 100 of shards of 80m+ container, which grew to 220m+, and now those shards are should be sharded too, i thought | 11:49 |
| mcape | so i tried to shard the shards, and it went well, until it didn't.. early in the morning when load started to pick up, sharding process stalled, one last shard remained in "in progress" state for 2-3 hours | 11:50 |
| mcape | eventually sharding completed, there are no more "sharding in progress" entries in recon cache, but now there are great many messages in the logs | 11:51 |
| mcape | /srv/node/device19/containers/690/2d1/<bla-bla>/<bla>.db is able to shard -- refusing to replicate objects to peer <bla>:6001/device17; have shard ranges and will wait for cleaving | 11:52 |
| mcape | 15000 in a hour | 11:53 |
| mcape | looks like sharding process is stuck... i tried to understand source code but <ahem> no luck | 11:58 |
| mcape | now i also see logs like Failed to get shard ranges from AUTH_bla/sharding_test: Unexpected response: 404 Not Found (<html><h1>Not Found</h1><p>The resource could not be found.</p></html>) | 12:34 |
| mcape | and logs like "Failed to put shard ranges to 172.31.3.2:6001/device19: 404" | 12:35 |
| mcape | maybe rings are out of sync somehow | 12:35 |
| mcape | no, container rings are in sync | 12:41 |
| mcape_ | looks like bulk of the audit warnings and "Failed to get shard ranges from" logs are about some very-very old container from the ages when the cluster was in pre-production stage, because is called "sharding_test"! | 14:04 |
| mcape_ | i recall that i made that one back in 2018 | 14:04 |
| mcape_ | but i can't find it in its account.. weird | 14:05 |
| mcape_ | yes... but there are many logs like <db> is able to shard -- refusing to replicate objects to peer <peer>, which are all about real containers with objects :-( | 14:12 |
| mcape_ | so maybe there is a way to deal with them? | 14:13 |
| mcape_ | sorry for messy story | 14:13 |
| mcape_ | there are dozens of such containers that are 'refusing to replicate' - i checked the databases inside, there are no objects | 15:17 |
| mcape_ | so it looks like they are remnants of stalled sharding process? | 15:18 |
| opendevreview | Alistair Coles proposed openstack/swift master: proxy: fix NodeIter nodes_left decrementing https://review.opendev.org/c/openstack/swift/+/862825 | 15:40 |
| opendevreview | Alistair Coles proposed openstack/swift master: proxy: reduce repeated logic in NodeIter https://review.opendev.org/c/openstack/swift/+/862826 | 15:40 |
| opendevreview | Tim Burke proposed openstack/swift stable/zed: Fix Zed CI https://review.opendev.org/c/openstack/swift/+/861629 | 16:02 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: proxy-server exception logging shows replication_ip/port https://review.opendev.org/c/openstack/swift/+/860866 | 16:33 |
| opendevreview | Merged openstack/swift master: CI: Install pip from pinned upstream for py2 probe tests https://review.opendev.org/c/openstack/swift/+/862758 | 17:07 |
| opendevreview | Merged openstack/swift master: Stop locking in our SysLogHandler https://review.opendev.org/c/openstack/swift/+/840232 | 19:52 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: Prepare tracing by adding a WSGI mixin for middlewares https://review.opendev.org/c/openstack/swift/+/797811 | 23:21 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: trace: Add RequestTraceMiddleware with OpTel https://review.opendev.org/c/openstack/swift/+/857559 | 23:21 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: Prepare tracing by adding a WSGI mixin for middlewares https://review.opendev.org/c/openstack/swift/+/797811 | 23:24 |
| opendevreview | Tim Burke proposed openstack/swift master: wsgi: Start workers in parallel, rather than serially https://review.opendev.org/c/openstack/swift/+/862843 | 23:26 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: trace: Add RequestTraceMiddleware with OpTel https://review.opendev.org/c/openstack/swift/+/857559 | 23:30 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Sharder: add more logs during container cleaving. https://review.opendev.org/c/openstack/swift/+/862775 | 23:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!