| opendevreview | Matthew Oliver proposed openstack/swift master: ShardRange: track last state change timestamp https://review.opendev.org/c/openstack/swift/+/926036 | 03:18 |
|---|---|---|
| xrkr | Hello @timburke and all, I am facing an issue with openstack swift.. See the below error .. "container-server: err ERROR __call__ error with GET /d782/57274/AUTH_exampleusr/example-c1 : [Errno 28] FALLOCATE_RESERVE fail -9.80838 <= 1 . This is specifically coming onnly for one account and container under it". I listed out all the other locations using swift-get-nodes and found out the curl returns 500 error. I am not seeing this for any other acco | 10:00 |
| xrkr | Pls advise | 10:01 |
| opendevreview | Alistair Coles proposed openstack/swift master: statsd: deprecate log_ prefix for options https://review.opendev.org/c/openstack/swift/+/922518 | 11:14 |
| opendevreview | Alistair Coles proposed openstack/swift master: proxy_logging: deprecate [access_]log_statsd_valid_http_methods option https://review.opendev.org/c/openstack/swift/+/926131 | 11:14 |
| acoles | xrkr: that error suggests that the disk d782 has insufficient space to persist updates to the container DB. The reason you'd see it with a container GET request is that object updates have accumulated in a pending file, and the container-server attempts to commit pending object updates to the container DB before returning the response to the GET request (to make sure the object listing is up to date). Before making the DB | 11:26 |
| acoles | commit it is checking the disk space. https://github.com/openstack/swift/blob/fd7861313f113652df10bcd31ff5ce79c09007d8/swift/common/db.py#L842 | 11:26 |
| acoles | you need to free up some space on that disk | 11:26 |
| xrkr | Thanks @acoles for the response. But i have ample space in the mountpoint. Its almst 60G free space.. Also, IUse is 1% | 12:18 |
| acoles | what is the size of the container DB file for the AUTH_exampleusr/example-c1 container on d782? I think it's going to try to pre-allocate approx the same size as the existing DB | 12:22 |
| xrkr | coles: Just looked at the size of db in the container folder.. Its whopping 73G | 12:32 |
| xrkr | Is this the cause? | 12:55 |
| acoles | I think so. the pre-allocate is checking for ~73G frees pace and finding only ~60G https://github.com/openstack/swift/blob/fd7861313f113652df10bcd31ff5ce79c09007d8/swift/common/db.py#L930 | 13:03 |
| acoles | free space* | 13:04 |
| acoles | xrkr: what's the approx number of objects in the container? it's possible the container DB doesn't have a correspondingly huge number of rows but has grown over time as objects are PUT then DELETEd (new row each time). If so it may be possible to vacuum the DB, but TBH I'm not the best person to advise on that: maybe timburke or mattoliver , check back later to allow for their timezones | 13:07 |
| opendevreview | Alistair Coles proposed openstack/swift master: Ignore 404s from handoffs when choosing response code https://review.opendev.org/c/openstack/swift/+/925993 | 13:12 |
| xrkr | ok, It could be more than 120M objects in that container. I do know that the growth is exponential but not immediate. Therefore i am unsure why the issue triggered now.. | 13:16 |
| xrkr | Anyways i will wait for timburke or mattoliver to respond. Thanks coles | 13:17 |
| acoles | xrkr: ok that's a decent amount of objects. Is the container sharded? https://docs.openstack.org/swift/latest/overview_container_sharding.html | 13:20 |
| acoles | mattoliver: if you can remember this far back, you may be able to help me https://review.opendev.org/c/openstack/swift/+/925993/comment/05af889a_3be2b4d4/ w.r.t. the related change you worked on | 13:23 |
| patch-bot | patch 925993 - swift - Ignore 404s from handoffs when choosing response code - 3 patch sets | 13:23 |
| xrkr | timburke: mattoliver: Can you help here pls. Just to add, we have 3 site cluster and the db is spanned across 3 clusters. | 15:39 |
| DHE | I'm confused, because it looks like that feature defaults to OFF, both in code and default config files.. so why is that going off? | 15:46 |
| opendevreview | Merged openstack/swift master: Move utils/logs.py related tests out of test_utils https://review.opendev.org/c/openstack/swift/+/925495 | 19:10 |
| mattoliver | xrkr: I do have this patch in the works to add container and account vacuuming to swift: https://review.opendev.org/c/openstack/swift/+/916861 | 23:44 |
| patch-bot | patch 916861 - swift - db_auditor: add vacuum support - 10 patch sets | 23:44 |
| mattoliver | If we can check to see how bloated your db is pretty easily if you have sqlite3 installed on a node. | 23:44 |
| mattoliver | We can see how many "Free" pages are in the db (wasted space) with a simple SQLite command. | 23:45 |
| mattoliver | sqlite3 <path_to_db> | 23:46 |
| mattoliver | PRAGMA freelist_count | 23:46 |
| mattoliver | You can get the page size with: PRAGMA page_size | 23:47 |
| mattoliver | You can also get to total number of pages in use (freelist and used pages) with: PRAGMA page_count | 23:48 |
| mattoliver | If it is really bloated you can manually vacuum them by running: VACCUM | 23:49 |
| DHE | I'm mostly concerned that this happened when the pre-allocation behaviour seems to be non-default... | 23:49 |
| mattoliver | but you might want to run that on all primarys' and the others could still be replicated around. Once my patch lands. You can configure the auditor to check and vacuum for you when the freelist space get to a certain size or percentage. | 23:50 |
| mattoliver | DHE: yeah, that is a little promblematic, maybe fallocate settings should be defaulted to on. Swift had the history of adding features but keeping backwards compatible, so people opt-in.. but yeah really there are probably a bunch of settings that really shouldn't be opt-in, not anymore. | 23:54 |
| mattoliver | Let's add it to the next meeting to discuss. And then we can raise a bug/featrue requeast to change the default. | 23:55 |
| mattoliver | I think, looking at our defaults and re-evaluating was an action item from PTG | 23:55 |
| mattoliver | *PTG's past. .. and probably mine :opps: | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!