| openstackgerrit | Merged openstack/swift master: py36: Fix syslog fallback to UDP https://review.opendev.org/c/openstack/swift/+/780939 | 00:18 |
|---|---|---|
| seongsoocho | Can I ignore this error message in cleanup job of relinker? "Error cleaning up /srv/node/sdl/objects/261/ba1/105f1a630d8689c4354744a3a46fcba1/1579789059.24190.data: DiskFileNotExist()" | 00:28 |
| *** timburke_ has joined #openstack-swift | 00:32 | |
| *** ChanServ sets mode: +v timburke_ | 00:32 | |
| *** timburke has quit IRC | 00:34 | |
| *** timburke_ has quit IRC | 00:42 | |
| *** timburke has joined #openstack-swift | 01:03 | |
| *** ChanServ sets mode: +v timburke | 01:03 | |
| *** ianychoi has joined #openstack-swift | 01:24 | |
| *** ianychoi_ has quit IRC | 01:26 | |
| *** psachin has joined #openstack-swift | 02:27 | |
| zaitcev | Guys | 02:57 |
| zaitcev | Suppose I swiftched to "new" expirers on object nodes. | 02:57 |
| zaitcev | One of them has dequeue_from_legacy=true. It does all the work. | 02:58 |
| zaitcev | And | 02:59 |
| zaitcev | It's the only expirer that recon shows. | 02:59 |
| zaitcev | -> http://192.168.50.2:6000/recon/expirer/object: {'object_expiration_pass': 0.011844873428344727, 'expired_last_pass': 0} | 03:00 |
| zaitcev | -> http://192.168.50.1:6200/recon/expirer/object: {'object_expiration_pass': 9543.668754816055, 'expired_last_pass': 32321} | 03:00 |
| zaitcev | The 50.1 is the legacy, 50.2 is the new one. | 03:00 |
| zaitcev | Is it going to start working if I switch the new expirer on? | 03:00 |
| zaitcev | I mean, in the proxy. | 03:01 |
| *** priteau has quit IRC | 03:03 | |
| zaitcev | Actually, why am I even asking this. Let's just set expiring_objects_container_divisor and see what happens. | 03:10 |
| *** rcernin has quit IRC | 03:22 | |
| *** evrardjp has quit IRC | 03:33 | |
| *** evrardjp has joined #openstack-swift | 03:33 | |
| *** rcernin has joined #openstack-swift | 03:33 | |
| *** rcernin has quit IRC | 03:41 | |
| *** rcernin has joined #openstack-swift | 03:42 | |
| *** manuvakery1 has joined #openstack-swift | 04:38 | |
| *** m75abrams has joined #openstack-swift | 04:59 | |
| *** timburke has quit IRC | 05:28 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: swift-manage-shard-ranges: check for internal gaps https://review.opendev.org/c/openstack/swift/+/781013 | 05:49 |
| mattoliverau | seongsoocho: it probably means the object was reclaimed during between the relinking and cleanup. I mean you can go check to see if there is a .ts there or in the linked path to confirm if you want. | 05:57 |
| *** dsariel has joined #openstack-swift | 06:09 | |
| seongsoocho | mattoliverau: thanks . How can I get a linked path of original object ? | 06:13 |
| *** mathiasb has joined #openstack-swift | 07:43 | |
| *** dsariel has quit IRC | 08:00 | |
| *** dsariel has joined #openstack-swift | 08:01 | |
| *** rcernin has quit IRC | 08:11 | |
| *** rpittau|afk is now known as rpittau | 08:23 | |
| *** rcernin has joined #openstack-swift | 09:17 | |
| *** openstackgerrit has quit IRC | 09:33 | |
| *** rcernin_ has joined #openstack-swift | 09:48 | |
| *** rcernin has quit IRC | 09:55 | |
| *** dsariel has quit IRC | 09:55 | |
| *** mathiasb has quit IRC | 09:55 | |
| acoles | seongsoocho: mattoliverau it's a .data file so should not be reclaimed, but there are other conditions that trigger a DiskFileNotExist (corrupt metadata maybe?) | 10:40 |
| acoles | the error is reported for the new location (i.e. in the 'next_part_power' partition) | 10:40 |
| acoles | so (a) try an ls and swift-object-info on that reported path, see what you actually have there | 10:41 |
| acoles | (b) check the old location which will be /srv/node/sdl/objects/130/ba1/105f1a630d8689c4354744a3a46fcba1 , note objects in old partition N should have been linked new partitions 2N and 2N+1, so object in new partition 261 would have been in old partition 130 | 10:44 |
| acoles | seongsoocho: please let us know what you discover, we've been fixing a few bugs in relinker, but also found that sometimes it is reporting false positives | 10:47 |
| *** priteau has joined #openstack-swift | 11:26 | |
| *** rcernin_ has quit IRC | 11:32 | |
| *** takamatsu has quit IRC | 11:57 | |
| seongsoocho | acoles: I checked new partition 2N, and 2N+1, but there are no object .... | 11:59 |
| seongsoocho | for more information, I used train code of relinker for relink job and use recent code of relinker for cleanup... Will this be a problem? | 12:00 |
| acoles | that should be ok | 12:01 |
| acoles | is there anything in the old partition? | 12:01 |
| seongsoocho | yes. the object are in the old partition. | 12:02 |
| seongsoocho | https://pastebin.com/hWXzzVw0. this is my terminal log | 12:05 |
| seongsoocho | Ah... I checked other server which is have replica, the object is deleted in other server.. | 12:11 |
| acoles | seongsoocho: is it possible that the object was deleted since you started relinking, and longer than reclaim_age has passed? It could be that the old partition has a .data, then a DELETE wrote a .ts in the new partition, then when the relinker cleanup reached that object the .ts was reclaimed in the new partition. Is the .ts file on other replica's server newer than the .data in the old partition? | 12:58 |
| acoles | I *think* timburke recommends increasing reclaim_age during relinking??? | 12:59 |
| seongsoocho | acoles: yes.. the object was deleted since I started relinking.. and I increased relcaim_age after the relink (before the cleanup) | 13:01 |
| *** openstackgerrit has joined #openstack-swift | 14:29 | |
| openstackgerrit | Merged openstack/swift master: Fix systemd notify unit test on macos https://review.opendev.org/c/openstack/swift/+/780680 | 14:29 |
| *** psachin has quit IRC | 15:51 | |
| *** josephillips has joined #openstack-swift | 15:52 | |
| *** m75abrams has quit IRC | 15:54 | |
| *** rpittau is now known as rpittau|afk | 17:08 | |
| *** timburke has joined #openstack-swift | 17:15 | |
| *** ChanServ sets mode: +v timburke | 17:15 | |
| openstackgerrit | Merged openstack/swift master: relinker: use abs path index in part power replace https://review.opendev.org/c/openstack/swift/+/780026 | 17:15 |
| *** dosaboy_ is now known as dosaboy | 17:16 | |
| *** alecuyer has quit IRC | 17:21 | |
| *** alecuyer has joined #openstack-swift | 17:25 | |
| *** gyee has joined #openstack-swift | 17:29 | |
| timburke | zaitcev, so... any "new" expirers won't do anything: https://github.com/openstack/swift/blob/2.26.0/swift/obj/expirer.py#L309-L313 | 17:36 |
| timburke | because we never quite got the momentum to review & merge https://review.opendev.org/c/openstack/swift/+/517389 | 17:36 |
| zaitcev | Yeah, I added "dequeue_from_legacy=True" to all of them and they seem to work now. | 17:36 |
| zaitcev | [object_expiration_pass] low: 96, high: 99, avg: 97.6, total: 292, Failed: 0.0%, no_result: 0, reported: 3 | 17:37 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Add config option to return 429s on ratelimit https://review.opendev.org/c/openstack/swift/+/729093 | 18:00 |
| openstackgerrit | Alistair Coles proposed openstack/swift master: swift-manage-shard-ranges: add repair and analyze commands https://review.opendev.org/c/openstack/swift/+/765624 | 18:30 |
| openstackgerrit | Alistair Coles proposed openstack/swift master: manage-shard-ranges: add --dry-run option for compact and repair https://review.opendev.org/c/openstack/swift/+/781144 | 18:30 |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Fix shrinking making acceptors prematurely active https://review.opendev.org/c/openstack/swift/+/774002 | 18:30 |
| openstackgerrit | Alistair Coles proposed openstack/swift master: sharder: stall cleaving at shard range gaps https://review.opendev.org/c/openstack/swift/+/777585 | 18:30 |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: Allow tempurl times to have units https://review.opendev.org/c/openstack/python-swiftclient/+/758500 | 18:40 |
| *** gyee has quit IRC | 18:46 | |
| *** josephillips has quit IRC | 18:49 | |
| *** josephillips has joined #openstack-swift | 18:52 | |
| *** gyee has joined #openstack-swift | 18:53 | |
| *** rcernin_ has joined #openstack-swift | 19:26 | |
| *** rcernin_ has quit IRC | 19:32 | |
| openstackgerrit | Merged openstack/swift master: Make more use of get_partition_for_hash in unit tests https://review.opendev.org/c/openstack/swift/+/781093 | 19:35 |
| *** gmann is now known as gmann_afk | 19:41 | |
| openstackgerrit | Merged openstack/swift master: Add test coverage for audit_location_generator yield_hashes arg https://review.opendev.org/c/openstack/swift/+/780648 | 19:50 |
| *** manuvakery1 has quit IRC | 19:56 | |
| *** rcernin_ has joined #openstack-swift | 19:57 | |
| *** rcernin_ has quit IRC | 20:02 | |
| *** rcernin_ has joined #openstack-swift | 20:32 | |
| *** rcernin_ has quit IRC | 20:37 | |
| *** rcernin_ has joined #openstack-swift | 20:50 | |
| timburke | almost meeting time! | 20:54 |
| mattoliverau | morning | 20:54 |
| kota_ | good morning | 20:59 |
| acoles | good morning kota_ mattoliverau | 21:00 |
| kota_ | acoles, mattoliverau: o/ | 21:01 |
| *** rcernin_ has quit IRC | 21:04 | |
| *** rcernin_ has joined #openstack-swift | 21:09 | |
| *** rcernin_ has quit IRC | 21:15 | |
| *** rcernin_ has joined #openstack-swift | 21:41 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add --headless option for cors functional test runner https://review.opendev.org/c/openstack/swift/+/781084 | 22:09 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Use headless option for cors functional test runner https://review.opendev.org/c/openstack/swift/+/781084 | 22:12 |
| *** gmann_afk is now known as gmann | 22:39 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Use headless option for cors functional test runner https://review.opendev.org/c/openstack/swift/+/781084 | 22:57 |
| *** timburke_ has joined #openstack-swift | 23:04 | |
| *** ChanServ sets mode: +v timburke_ | 23:04 | |
| *** timburke has quit IRC | 23:06 | |
| *** timburke_ has quit IRC | 23:31 | |
| *** timburke_ has joined #openstack-swift | 23:31 | |
| *** ChanServ sets mode: +v timburke_ | 23:31 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: relinker: Add /recon/relink endpoint and drop progress stats https://review.opendev.org/c/openstack/swift/+/779655 | 23:49 |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: relinker: Add policy to relinker progress output https://review.opendev.org/c/openstack/swift/+/780440 | 23:49 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!