| *** bigdogstl has joined #openstack-swift | 00:03 | |
| *** bigdogstl has quit IRC | 00:09 | |
| timburke | thinking about https://review.openstack.org/#/c/365371/ -- is it worth pointing out (in an UpgradeImpact, probably) that old proxies may have issues during a rolling upgrade? | 00:23 |
|---|---|---|
| patchbot | patch 365371 - swift - Add support for data segments to SLO and Segmented... | 00:23 |
| timburke | i certainly don't think it's worth plumbing in a config option to disable the new feature until the upgrade completes (as we did for https://github.com/openstack/swift/commit/96a0e07) | 00:23 |
| timburke | the closer comparison would be https://github.com/openstack/swift/commit/25d5e68 which doesn't even have the UpgradeImpact... | 00:23 |
| *** bigdogstl has joined #openstack-swift | 00:26 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: DRY out tests until the stone bleeds https://review.openstack.org/534507 | 00:30 |
| notmyname | first pass of a 2.17.0 changelog done. | 00:41 |
| openstackgerrit | John Dickinson proposed openstack/swift master: WIP https://review.openstack.org/534528 | 00:41 |
| *** tovin07_ has joined #openstack-swift | 00:46 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add support for data segments to SLO and SegmentedIterable https://review.openstack.org/365371 | 00:48 |
| *** tovin07_ has quit IRC | 01:00 | |
| *** tovin07 has quit IRC | 01:01 | |
| *** bigdogstl has quit IRC | 01:08 | |
| timburke | clayg: *grumble, grumble* http://logs.openstack.org/75/530175/6/check/swift-tox-py27/8099136/job-output.txt.gz#_2018-01-16_23_57_23_968856 | 01:11 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Keep object-updater stats logging consistent https://review.openstack.org/534579 | 01:11 |
| clayg | timburke: lol, no body! :P | 01:12 |
| timburke | yup | 01:13 |
| clayg | I'll fully admit I only ran the unittests for the modules I though would be effected | 01:13 |
| clayg | that's gate! | 01:13 |
| clayg | thanks | 01:14 |
| clayg | - if resp.status_int // 100 != 2: | 01:15 |
| clayg | + if resp.status_int // 100 != 2 and resp.body: | 01:15 |
| clayg | timburke: ^ ??? | 01:15 |
| clayg | cheating? | 01:15 |
| *** tovin07 has joined #openstack-swift | 01:15 | |
| *** bigdogstl has joined #openstack-swift | 01:15 | |
| *** bigdogstl has quit IRC | 01:17 | |
| *** bigdogstl has joined #openstack-swift | 01:17 | |
| *** bigdogstl has quit IRC | 01:19 | |
| *** bigdogstl has joined #openstack-swift | 01:19 | |
| timburke | only thing vaguely risky is that `if resp.body and resp.status_int // 100 != 2` has very different behavior... | 01:20 |
| timburke | but you can totally have a 401 (or something) with an empty body (eg, because you issued a HEAD) | 01:21 |
| clayg | sure... short-circut boolean is a thing tho... you just have to now .body is a property with a potentially expensive side effect :\ | 01:21 |
| timburke | so it doesn't feel much like cheating | 01:21 |
| *** bigdogstl has quit IRC | 01:21 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: DRY out tests until the stone bleeds https://review.openstack.org/534507 | 01:22 |
| kota_ | good morning | 01:24 |
| clayg | kota_: timburke and i were working on the internal client 499 fix | 01:25 |
| clayg | patch patch 530175 | 01:25 |
| patchbot | https://review.openstack.org/#/c/530175/ - swift - Fix InternalClient to drain response body if the r... | 01:25 |
| kota_ | yeah, i briefly looked at the irc logs | 01:26 |
| kota_ | it looks like it got second +2? | 01:26 |
| timburke | kota_: おはよ | 01:26 |
| kota_ | oh, but it failed at py27 gate | 01:26 |
| kota_ | timburke: lol | 01:27 |
| kota_ | timburke: おはよう, basic "good morning" needs the last "う" | 01:27 |
| *** bigdogstl has joined #openstack-swift | 01:27 | |
| timburke | bah! i was close. sorry. gotta hold it a little longer, yeah? | 01:28 |
| kota_ | that shorter one looks much similar with baby words ;) | 01:29 |
| clayg | ^ rofl | 01:30 |
| timburke | that's about where i am :D | 01:30 |
| openstackgerrit | Merged openstack/swift master: Update http with https https://review.openstack.org/534119 | 01:30 |
| kota_ | does http://logs.openstack.org/75/530175/6/check/swift-tox-py27/8099136/job-output.txt.gz#_2018-01-16_23_57_23_968856 show us that we should fix a small minor point on the assertion, maybe? | 01:30 |
| clayg | kota_: tim & I were discussing maybe just bothering to try and append the body if there *is* no body (timburke gave the very good example of a HEAD request) | 01:31 |
| kota_ | the assertion said "Unexpected response: 503 Service Unavailable () != Unexpected response: 503 Service Unavailable " | 01:32 |
| clayg | yeah, it was kind of a nice save - I hadn't acctually considered that not all errors will have a body | 01:32 |
| kota_ | sounds good point, error on HEAD request will be no body? | 01:34 |
| clayg | yeah... I *think* so? | 01:34 |
| *** bigdogstl has quit IRC | 01:34 | |
| timburke | per https://tools.ietf.org/html/rfc7231#section-4.3.2 -- "The HEAD method is identical to GET except that the server MUST NOT send a message body in the response" | 01:35 |
| kota_ | but anyway, i'm thinking it's safe to iterate app.iter because it will raise StopIteration immediately... | 01:35 |
| timburke | though there's also a note: "A payload within a HEAD request message has no defined semantics; sending a payload body on a HEAD request might cause some existing implementations to reject the request." | 01:36 |
| timburke | oh wait -- that was about requests anyway | 01:36 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Send correct number of X-Delete-At-* headers https://review.openstack.org/524548 | 01:36 |
| *** m_kazuhiro has joined #openstack-swift | 01:39 | |
| kota_ | let me look at the logs. | 01:39 |
| kota_ | anyway, it looks like I have to update the patch for 1) fix the py27 error, 2) call close_if_possible for app iterator. | 01:40 |
| kota_ | will do, sorry, i have another work in this morning, I'll do it in this afternoon. | 01:41 |
| kota_ | ah... | 01:46 |
| *** tovin07_ has joined #openstack-swift | 01:46 | |
| clayg | well, let me just try fixing the and resp.body thing | 01:47 |
| clayg | then maybe this afternoon you can make any other needed changes or +2 | 01:47 |
| kota_ | if unexpected response requires the resp.body in the new, I think the body drain in the above won't work well because iterate app_iter doesn't set the resp.body IIRC | 01:48 |
| kota_ | so the resp.body will be empty, always. | 01:48 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Fix InternalClient to drain response body if the request fails https://review.openstack.org/530175 | 01:49 |
| kota_ | new patch set!? | 01:49 |
| clayg | i just added the diff for the and resp.body fix | 01:50 |
| clayg | we only add the body to the log line if there is a body to add | 01:50 |
| kota_ | oic, that was commented at the gerrit | 01:50 |
| clayg | should fix the test - everything else is mostly trying to solve the requirements outlined in the commit message | 01:50 |
| kota_ | i'll try to check my understand on the resp.body specification. | 01:51 |
| clayg | oh yeah... that's relevant https://github.com/openstack/swift/blob/master/swift/common/swob.py#L300 | 01:55 |
| *** SkyRocknRoll has joined #openstack-swift | 01:59 | |
| *** bigdogstl has joined #openstack-swift | 02:00 | |
| clayg | timburke: good thought about moving resp = None | 02:01 |
| *** bigdogstl has quit IRC | 02:06 | |
| m_kazuhiro | good morning | 02:12 |
| *** bigdogstl has joined #openstack-swift | 02:13 | |
| *** two_tired has joined #openstack-swift | 02:16 | |
| *** JimCheung has quit IRC | 02:38 | |
| *** gyee has quit IRC | 02:41 | |
| *** JimCheung has joined #openstack-swift | 02:57 | |
| *** JimCheung has quit IRC | 03:02 | |
| *** itlinux has joined #openstack-swift | 03:25 | |
| *** bigdogstl has quit IRC | 03:35 | |
| *** kei_yama has quit IRC | 03:43 | |
| *** kei_yama has joined #openstack-swift | 04:02 | |
| *** bigdogstl has joined #openstack-swift | 04:03 | |
| *** fishbone_ has quit IRC | 04:05 | |
| *** bigdogstl has quit IRC | 04:14 | |
| *** psachin has joined #openstack-swift | 04:31 | |
| *** psachin has quit IRC | 04:31 | |
| *** psachin has joined #openstack-swift | 04:33 | |
| *** bigdogstl has joined #openstack-swift | 04:34 | |
| *** m_kazuhiro_ has joined #openstack-swift | 04:34 | |
| *** m_kazuhiro has quit IRC | 04:37 | |
| *** bigdogstl has quit IRC | 04:40 | |
| *** bigdogstl has joined #openstack-swift | 04:43 | |
| *** m_kazuhiro_ has quit IRC | 04:46 | |
| *** m_kazuhiro has joined #openstack-swift | 04:47 | |
| *** two_tired has quit IRC | 04:47 | |
| *** bigdogstl has quit IRC | 04:47 | |
| *** links has joined #openstack-swift | 04:49 | |
| *** f0o has quit IRC | 04:52 | |
| *** f0o has joined #openstack-swift | 04:59 | |
| *** bigdogstl has joined #openstack-swift | 05:06 | |
| *** f0o has quit IRC | 05:11 | |
| *** bigdogstl has quit IRC | 05:12 | |
| kota_ | i got my misunderstanding at the internal client patch. | 05:13 |
| *** f0o has joined #openstack-swift | 05:13 | |
| *** bigdogstl has joined #openstack-swift | 05:14 | |
| openstackgerrit | Merged openstack/swift feature/deep: Add copy method to ShardRange https://review.openstack.org/531486 | 05:15 |
| openstackgerrit | Merged openstack/swift master: Add a note about the cost of COPY for setting metadata https://review.openstack.org/534389 | 05:15 |
| *** m_kazuhiro has quit IRC | 05:17 | |
| *** chsc has joined #openstack-swift | 05:19 | |
| *** bigdogstl has quit IRC | 05:20 | |
| *** chsc has quit IRC | 05:28 | |
| openstackgerrit | Merged openstack/swift master: Let recon-cron work with conf.d https://review.openstack.org/532696 | 05:30 |
| openstackgerrit | Merged openstack/swift master: expirer: unexpected responses don't warrant tracebacks https://review.openstack.org/496534 | 05:30 |
| openstackgerrit | Merged openstack/swift master: Add tests for X-Backend-Clean-Expiring-Object-Queue true https://review.openstack.org/533254 | 05:30 |
| openstackgerrit | Merged openstack/swift feature/deep: Define entries of shard range db row in one place https://review.openstack.org/531487 | 05:30 |
| *** chsc has joined #openstack-swift | 05:35 | |
| *** bigdogstl has joined #openstack-swift | 05:38 | |
| *** JimCheung has joined #openstack-swift | 05:39 | |
| *** chsc has quit IRC | 05:41 | |
| *** bigdogstl has quit IRC | 05:43 | |
| *** JimCheung has quit IRC | 05:43 | |
| *** bigdogstl has joined #openstack-swift | 05:50 | |
| *** bigdogstl has quit IRC | 06:01 | |
| mattoliverau | phew, /me was lost down a cloud HA bug rabbit hole that I hoped would only take me 1/2 a day so I could spend the arvo swifting.. and it's now it's 5pm.. so turned out more rabbit hole-y then I thought :( | 06:01 |
| mattoliverau | maybe I can get in a review or something tonight to help move things along. | 06:02 |
| kota_ | mattoliverau: take your time. a rabbit hole-y problem appears unexpectedly always. | 06:13 |
| mattoliverau | that's true | 06:14 |
| torgomatic | I notice in our gate jobs that there's this swift-tox-func-post-as-copy. Since post_as_copy is now a no-op, could we kill off that job and save some test cycles? | 06:16 |
| openstackgerrit | Samuel Merritt proposed openstack/swift master: Remove old post-as-copy leftovers from tests. https://review.openstack.org/534639 | 06:20 |
| torgomatic | let's see what happens :) | 06:20 |
| *** m_kazuhiro has joined #openstack-swift | 06:23 | |
| notmyname | torgomatic: yeah, I think that would be ok | 06:24 |
| mattoliverau | so long as none of our stable branches need it. It should be ok. Otherwise we'd could filter out all braches but it.. but I'd need to go look... and currently my brain feels fired | 06:28 |
| mattoliverau | *fried | 06:28 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift master: Fix InternalClient to drain response body if the request fails https://review.openstack.org/530175 | 06:38 |
| kota_ | timburke, clayg: just fix my typo clayg suggested at the old patch set. other changes all look good to me. | 06:39 |
| kota_ | mattoliverau, notmyname, torgomatic: good catch, and I'm realizing to wonder if we would ought to backport zuul migration patch to stable branch? | 06:46 |
| kota_ | i mean something around dc1c55c9a07c03fe85f4bcc52419a42d75ae30fa | 06:47 |
| kota_ | https://github.com/openstack/swift/commit/dc1c55c9a07c03fe85f4bcc52419a42d75ae30fa | 06:47 |
| kota_ | to enable zuul v3 job on stable branch. | 06:48 |
| *** bigdogstl has joined #openstack-swift | 07:03 | |
| *** bigdogstl has quit IRC | 07:12 | |
| *** threestrands_ has quit IRC | 07:20 | |
| *** armaan has joined #openstack-swift | 07:22 | |
| *** kei_yama_ has joined #openstack-swift | 07:36 | |
| *** kei_yama has quit IRC | 07:38 | |
| *** armaan has quit IRC | 07:38 | |
| *** m_kazuhiro has quit IRC | 07:38 | |
| *** hseipp has joined #openstack-swift | 07:42 | |
| *** pcaruana has joined #openstack-swift | 07:48 | |
| *** pcaruana has quit IRC | 07:55 | |
| *** pcaruana has joined #openstack-swift | 07:55 | |
| *** hoonetorg has quit IRC | 08:10 | |
| *** bigdogstl has joined #openstack-swift | 08:12 | |
| *** geaaru has joined #openstack-swift | 08:17 | |
| *** gkadam has joined #openstack-swift | 08:21 | |
| *** bigdogstl has quit IRC | 08:21 | |
| *** hoonetorg has joined #openstack-swift | 08:26 | |
| *** cshastri has joined #openstack-swift | 08:29 | |
| *** tesseract has joined #openstack-swift | 08:29 | |
| *** kei_yama_ has quit IRC | 08:39 | |
| *** silor has joined #openstack-swift | 08:43 | |
| *** mvk has joined #openstack-swift | 08:44 | |
| *** rcernin has quit IRC | 08:45 | |
| *** psachin` has joined #openstack-swift | 08:53 | |
| acoles | good morning | 08:54 |
| *** m_kazuhiro has joined #openstack-swift | 08:54 | |
| *** psachin has quit IRC | 08:54 | |
| *** cbartz has joined #openstack-swift | 09:00 | |
| *** tovin07 has quit IRC | 09:01 | |
| *** tovin07_ has quit IRC | 09:01 | |
| *** tovin07_ has joined #openstack-swift | 09:01 | |
| *** tovin07 has joined #openstack-swift | 09:01 | |
| *** StevenK has quit IRC | 09:02 | |
| openstackgerrit | guotao proposed openstack/swift master: Adjust the error response codes order https://review.openstack.org/534692 | 09:03 |
| *** psachin` has quit IRC | 09:09 | |
| *** psachin has joined #openstack-swift | 09:13 | |
| *** StevenK has joined #openstack-swift | 09:17 | |
| *** bigdogstl has joined #openstack-swift | 09:19 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift feature/s3api: As a first step, let's skip the test if not s3api in the middleware https://review.openstack.org/534703 | 09:27 |
| kota_ | acoles: morning | 09:28 |
| *** bigdogstl has quit IRC | 09:30 | |
| *** cbartz has quit IRC | 09:31 | |
| *** cbartz has joined #openstack-swift | 09:34 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: DRY out tests until the stone bleeds https://review.openstack.org/534507 | 09:37 |
| acoles | kota_: o/ | 09:40 |
| *** rcernin has joined #openstack-swift | 09:40 | |
| *** SkyRocknRoll has quit IRC | 09:45 | |
| *** SkyRocknRoll has joined #openstack-swift | 09:47 | |
| *** tovin07_ has quit IRC | 09:53 | |
| *** mvk has quit IRC | 09:55 | |
| *** bigdogstl has joined #openstack-swift | 09:57 | |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift master: Update object-expirer to use general task queue system https://review.openstack.org/517389 | 10:00 |
| *** bigdogstl has quit IRC | 10:06 | |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift master: Update object-expirer to use general task queue system https://review.openstack.org/517389 | 10:11 |
| *** m_kazuhiro has quit IRC | 10:22 | |
| acoles | timburke: thanks for moving those sharding patches along | 10:31 |
| *** bigdogstl has joined #openstack-swift | 10:33 | |
| *** bigdogstl has quit IRC | 10:38 | |
| *** mvk has joined #openstack-swift | 10:43 | |
| *** tovin07_ has joined #openstack-swift | 11:02 | |
| *** tovin07_ has left #openstack-swift | 11:02 | |
| *** bigdogstl has joined #openstack-swift | 11:04 | |
| *** bigdogstl has quit IRC | 11:14 | |
| *** bigdogstl has joined #openstack-swift | 11:45 | |
| *** bigdogstl has quit IRC | 11:52 | |
| *** JimCheung has joined #openstack-swift | 12:03 | |
| openstackgerrit | Merged openstack/swift master: Send correct number of X-Delete-At-* headers https://review.openstack.org/524548 | 12:06 |
| *** JimCheung has quit IRC | 12:08 | |
| *** bigdogstl has joined #openstack-swift | 12:13 | |
| *** rcernin has quit IRC | 12:14 | |
| *** bigdogstl has quit IRC | 12:29 | |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Add state and state_timestamp to ShardRange https://review.openstack.org/531488 | 12:31 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Set shard range state during sharding https://review.openstack.org/531489 | 12:31 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Don't attempt to merge root and shard listings in proxy https://review.openstack.org/532604 | 12:31 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Define singletons to represent min and max shard range bounds https://review.openstack.org/533200 | 12:31 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Simplify proxy-server getting sharded listings https://review.openstack.org/533201 | 12:31 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: only make single request for container listings https://review.openstack.org/512756 | 12:31 |
| *** bigdogstl has joined #openstack-swift | 12:31 | |
| *** bigdogstl has quit IRC | 12:42 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Preserve expiring object behaviour with old proxy-server https://review.openstack.org/534749 | 13:02 |
| acoles | timburke: torgomatic clayg: ^^ I am probably worrying too much but I think p 524548 breaks (very-)backwards compaatibility | 13:04 |
| patchbot | https://review.openstack.org/#/c/524548/ - swift - Send correct number of X-Delete-At-* headers (MERGED) | 13:04 |
| *** SkyRocknRoll has quit IRC | 13:28 | |
| *** bigdogstl has joined #openstack-swift | 13:29 | |
| *** silor has quit IRC | 13:32 | |
| *** bigdogstl has quit IRC | 13:40 | |
| notmyname | acoles: in what way? | 13:56 |
| acoles | notmyname: very old proxy servers don't send x-delete-at-container, so latest obj server won't create an expiring object record | 13:57 |
| acoles | very old ~ Havana ! | 13:58 |
| acoles | notmyname: p 534749 is an alternative | 13:58 |
| patchbot | https://review.openstack.org/#/c/534749/ - swift - Preserve expiring object behaviour with old proxy-... | 13:59 |
| *** armaan has joined #openstack-swift | 14:03 | |
| *** bigdogstl has joined #openstack-swift | 14:09 | |
| *** SkyRocknRoll has joined #openstack-swift | 14:10 | |
| *** bigdogstl has quit IRC | 14:19 | |
| *** arunman has joined #openstack-swift | 14:31 | |
| *** bigdogstl has joined #openstack-swift | 14:35 | |
| *** JimCheung has joined #openstack-swift | 14:40 | |
| openstackgerrit | Merged openstack/swift master: Minor cleanup in monitoring doc. https://review.openstack.org/534426 | 14:43 |
| *** JimCheung has quit IRC | 14:45 | |
| *** bigdogstl has quit IRC | 14:46 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Don't make async_pendings during object expiration https://review.openstack.org/532383 | 14:47 |
| openstackgerrit | Alistair Coles proposed openstack/swift master: check all nodes for async_pendings https://review.openstack.org/534774 | 14:47 |
| acoles | oops | 14:47 |
| *** ianychoi has quit IRC | 14:48 | |
| *** ianychoi has joined #openstack-swift | 14:52 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Don't make async_pendings during object expiration https://review.openstack.org/532383 | 14:57 |
| *** itlinux has quit IRC | 15:07 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: check all nodes for async_pendings https://review.openstack.org/534774 | 15:09 |
| *** bigdogstl has joined #openstack-swift | 15:10 | |
| *** links has quit IRC | 15:11 | |
| *** arunmani_ has joined #openstack-swift | 15:24 | |
| *** arunman has quit IRC | 15:25 | |
| *** bigdogstl has quit IRC | 15:32 | |
| *** bigdogstl has joined #openstack-swift | 15:40 | |
| *** bigdogstl has quit IRC | 15:45 | |
| *** arunmani_ has quit IRC | 15:48 | |
| *** arunmani_ has joined #openstack-swift | 15:48 | |
| *** arunmani_ has quit IRC | 15:53 | |
| *** itlinux has joined #openstack-swift | 15:57 | |
| *** silor has joined #openstack-swift | 16:00 | |
| *** two_tired has joined #openstack-swift | 16:02 | |
| *** itlinux_ has joined #openstack-swift | 16:04 | |
| *** itlinux has quit IRC | 16:06 | |
| *** two_tired has quit IRC | 16:08 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Fix intermittent unit test failure https://review.openstack.org/534825 | 16:10 |
| *** bigdogstl has joined #openstack-swift | 16:11 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: DRY out tests until the stone bleeds https://review.openstack.org/534507 | 16:12 |
| *** bigdogstl has quit IRC | 16:21 | |
| *** silor has quit IRC | 16:27 | |
| *** silor has joined #openstack-swift | 16:28 | |
| *** chsc has joined #openstack-swift | 16:37 | |
| *** chsc has joined #openstack-swift | 16:37 | |
| *** gyee has joined #openstack-swift | 16:39 | |
| *** arunmani_ has joined #openstack-swift | 16:46 | |
| *** pcaruana has quit IRC | 16:46 | |
| *** bigdogstl has joined #openstack-swift | 16:50 | |
| *** bigdogstl has quit IRC | 16:55 | |
| *** silor has quit IRC | 16:55 | |
| *** silor has joined #openstack-swift | 16:55 | |
| openstackgerrit | Merged openstack/swift feature/deep: annotate_failure: preserve tracebacks https://review.openstack.org/534454 | 16:56 |
| openstackgerrit | Merged openstack/swift feature/deep: Add more reasons to skip sharder probe tests https://review.openstack.org/531145 | 16:56 |
| *** armaan has quit IRC | 16:56 | |
| notmyname | acoles: after thinking on it during some meetings, I think it's probably something we should land (patch 534749 | 17:02 |
| acoles | notmyname: ok | 17:03 |
| *** cbartz has quit IRC | 17:07 | |
| *** tesseract has quit IRC | 17:21 | |
| *** hseipp has quit IRC | 17:26 | |
| *** bigdogstl has joined #openstack-swift | 17:27 | |
| *** arunmani_ has quit IRC | 17:29 | |
| *** ukaynar has joined #openstack-swift | 17:31 | |
| *** JimCheung has joined #openstack-swift | 17:31 | |
| *** bigdogstl has quit IRC | 17:36 | |
| *** JimCheung has quit IRC | 17:37 | |
| *** gkadam has quit IRC | 17:37 | |
| *** zaitcev_ has joined #openstack-swift | 17:42 | |
| *** ChanServ sets mode: +v zaitcev_ | 17:42 | |
| notmyname | acoles: got delayed by YAM (yet another meeting) | 17:43 |
| notmyname | so I think we should land patch 534749 (or something that is functionally equivalent) because a two-year-old swift cluster isn't that old and I don't want rolling upgrades to lose expiry writes during that process | 17:44 |
| patchbot | https://review.openstack.org/#/c/534749/ - swift - Preserve expiring object behaviour with old proxy-... | 17:44 |
| notmyname | but I think the only major concern is what happens when there's mismatched versions in the cluster (ie during an upgrade) | 17:45 |
| *** zaitcev has quit IRC | 17:46 | |
| acoles | notmyname: compatibility is with pre-Havana so c. 2013 | 17:48 |
| acoles | and yes concern is only when proxy is older than obj server | 17:48 |
| notmyname | oh | 17:48 |
| notmyname | I thought you said something about 2.7.0 somewhere | 17:49 |
| notmyname | I may have combined some memories of different conversations | 17:49 |
| acoles | notmyname: nope 1.9.0 | 17:49 |
| notmyname | oh! I probably saw the .9.0 part and just *assumed* it started with a 2 ;-) | 17:50 |
| acoles | lol | 17:50 |
| notmyname | so we're talking about during the upgrade process for four-year-old cluster. | 17:50 |
| notmyname | in that case... | 17:50 |
| notmyname | could we handle it in the release notes? | 17:50 |
| notmyname | what could we say? | 17:50 |
| notmyname | "if you upgrade to this version from pre-1.9.0, expiry won't work during the rolling upgrade" | 17:51 |
| notmyname | "...if you upgrade your storage nodes before the proxy nodes (which is what you should normally do anyway) | 17:51 |
| acoles | notmyname: fwiw timburke has +2 on 534749 | 17:52 |
| timburke | "... objects with expiration times set during the upgrade won't be scheduled for removal"? | 17:52 |
| timburke | it's not that it *won't work at all* | 17:53 |
| timburke | and really, it's not even "won't" it's "may not" | 17:53 |
| timburke | but it *does* mean that you don't have a nice, cleanly defined window wherein you're affected. you may be finding not-properly-removed data for quite some time after your upgrade | 17:55 |
| *** bigdogstl has joined #openstack-swift | 18:07 | |
| *** itlinux_ has quit IRC | 18:10 | |
| *** itlinux has joined #openstack-swift | 18:10 | |
| torgomatic | acoles: You're right; it does. I'd managed to convince myself that it wasn't important since Swift 1.9.0 was so old, but I could easily be wrong about that. | 18:12 |
| torgomatic | I added a meeting agenda item to talk about checkpoint releases. It'd be nice to have those every couple of years so that we don't have an unbounded amount of backwards compatibility to think about. | 18:14 |
| acoles | torgomatic: havana is old enough that i don't have the t-shirt ;) | 18:14 |
| *** ianychoi has quit IRC | 18:14 | |
| *** ianychoi has joined #openstack-swift | 18:15 | |
| openstackgerrit | Merged openstack/swift master: DRY out tests until the stone bleeds https://review.openstack.org/534507 | 18:15 |
| *** bigdogstl has quit IRC | 18:16 | |
| *** bigdogstl has joined #openstack-swift | 18:22 | |
| *** bigdogstl has quit IRC | 18:31 | |
| *** bigdogstl has joined #openstack-swift | 18:39 | |
| *** __david has joined #openstack-swift | 18:40 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift master: Don't make async_pendings during object expiration https://review.openstack.org/532383 | 18:40 |
| *** openstackstatus has quit IRC | 18:40 | |
| *** openstackstatus has joined #openstack-swift | 18:42 | |
| *** ChanServ sets mode: +v openstackstatus | 18:42 | |
| *** psachin has quit IRC | 18:52 | |
| *** geaaru has quit IRC | 18:58 | |
| *** mvk has quit IRC | 19:02 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: Revert "Add Constraints support" https://review.openstack.org/534881 | 19:03 |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: Revert "Add Constraints support" https://review.openstack.org/534881 | 19:14 |
| *** bigdogst_ has joined #openstack-swift | 19:18 | |
| timburke | ^^^ seems to fix the client gate (which no one really noticed was busted because no one really submits client patches) | 19:19 |
| *** armaan has joined #openstack-swift | 19:21 | |
| *** bigdogstl has quit IRC | 19:22 | |
| *** bigdogst_ has quit IRC | 19:26 | |
| *** bigdogstl has joined #openstack-swift | 19:31 | |
| *** bigdogstl has quit IRC | 19:36 | |
| *** silor has quit IRC | 19:43 | |
| *** bigdogstl has joined #openstack-swift | 19:48 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift master: Preserve expiring object behaviour with old proxy-server https://review.openstack.org/534749 | 19:48 |
| *** mvk has joined #openstack-swift | 19:50 | |
| timburke | notmyname: ok, i remember you saying that we had (some degree of) control over doc redirects -- how do i make http://docs.openstack.org/developer/swift/overview_object_versioning.html get me to https://docs.openstack.org/swift/latest/overview_object_versioning.html ? | 19:56 |
| notmyname | timburke: good question | 19:56 |
| notmyname | IIRC there's a top-level redirect definitions file we can use | 19:56 |
| notmyname | let me see if I can find some doc docs | 19:56 |
| timburke | doc doc doc doc | 19:57 |
| notmyname | https://docs.openstack.org/doc-contrib-guide/redirects.html | 19:57 |
| notmyname | maybe? still reading | 19:57 |
| *** joeljwright has joined #openstack-swift | 19:58 | |
| *** ChanServ sets mode: +v joeljwright | 19:58 | |
| notmyname | yeah, I think that's how to do it | 19:59 |
| *** bigdogstl has quit IRC | 19:59 | |
| *** cshastri has quit IRC | 20:06 | |
| *** gkadam has joined #openstack-swift | 20:10 | |
| *** ukaynar has quit IRC | 20:15 | |
| *** bigdogstl has joined #openstack-swift | 20:15 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift master: Remove old post-as-copy leftovers from tests. https://review.openstack.org/534639 | 20:17 |
| *** ukaynar has joined #openstack-swift | 20:19 | |
| *** bigdogstl has quit IRC | 20:28 | |
| *** geaaru has joined #openstack-swift | 20:30 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: internal_client: Don't retry when we expect the same reponse https://review.openstack.org/534399 | 20:37 |
| *** corvus is now known as jeblair | 20:37 | |
| *** jeblair is now known as corvus | 20:37 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add container-reconciler to source doc tree https://review.openstack.org/534943 | 20:39 |
| *** silor has joined #openstack-swift | 20:39 | |
| -openstackstatus- NOTICE: Zuul will be offline for a few minutes; existing changes will be re-enqueued; approvals during the downtime will need to be re-added. | 20:40 | |
| *** silor has quit IRC | 20:44 | |
| *** bigdogstl has joined #openstack-swift | 20:48 | |
| *** bigdogstl has quit IRC | 20:53 | |
| kota_ | good morning | 20:55 |
| *** ukaynar has quit IRC | 20:55 | |
| notmyname | hello kota_ | 20:55 |
| *** bigdogstl has joined #openstack-swift | 20:56 | |
| kota_ | hi notmyname | 20:56 |
| *** m_kazuhiro has joined #openstack-swift | 20:56 | |
| notmyname | clayg: acoles: tdasilva: cschwede: mattoliverau: kota_: m_kazuhiro: torgomatic: timburke: mathiasb: rledisez: just about meeting time courtesy ping | 20:59 |
| mattoliverau | Ta | 20:59 |
| mattoliverau | Morning | 20:59 |
| kota_ | o/ | 21:00 |
| m_kazuhiro | morning | 21:00 |
| notmyname | mattoliverau: I got scared this morning. couldn't find my hotel confirmation for next week. (fortunately, I did) | 21:00 |
| *** bigdogstl has quit IRC | 21:04 | |
| *** bigdogstl has joined #openstack-swift | 21:20 | |
| *** bigdogstl has quit IRC | 21:27 | |
| *** threestrands_ has joined #openstack-swift | 21:44 | |
| *** threestrands_ has quit IRC | 21:45 | |
| *** threestrands_ has joined #openstack-swift | 21:45 | |
| *** threestrands_ has quit IRC | 21:45 | |
| *** threestrands_ has joined #openstack-swift | 21:45 | |
| *** bigdogstl has joined #openstack-swift | 21:56 | |
| mattoliverau | Phew, otherwise backpackers or a tent in a park :p | 21:59 |
| joeljwright | notmyname: if you have a second, I was working on a review for 478611 and I realised I've lost my ability to +2 because I had to change my username and email after leaving Sohonet | 21:59 |
| openstackgerrit | Samuel Merritt proposed openstack/swift master: Improve object-updater's stats logging https://review.openstack.org/533207 | 21:59 |
| kota_ | mattoliverau: sounds like a summer vacation | 21:59 |
| kota_ | JK | 22:00 |
| *** m_kazuhiro has quit IRC | 22:01 | |
| *** rcernin has joined #openstack-swift | 22:02 | |
| clayg | I think the 412 precondition failed sort of makes sense if you think of x-if-delete-at sort of like x-if-match? | 22:04 |
| clayg | rledisez: I think getting off eventlet in the object server would be amazing | 22:04 |
| clayg | rledisez: all eyes on zaitcev_ | 22:04 |
| clayg | acctually how is the EC PUT update thing going? we fix that crazy headers non-sense? there was a couple of blockers that came out of tdasilva's call that he setup - but I never looped back around (busy) | 22:05 |
| timburke | clayg: sure? but it's not super useful, as the expirer -- 409 would be way more informative | 22:05 |
| zaitcev_ | It's ready for reviews, except 1 thing | 22:05 |
| clayg | it'd be great to have some traction on that going into the PTG - is anyone from redhat besides cschwede going to be there? | 22:05 |
| zaitcev_ | Oh, did I miss the meeting | 22:05 |
| joeljwright | zaitcev_: yeah, it just finished | 22:06 |
| *** bigdogstl has quit IRC | 22:08 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Reset state every iteration to avoid stale reference https://review.openstack.org/534957 | 22:09 |
| *** bigdogstl has joined #openstack-swift | 22:11 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: object-server can 409 in response to x-if-delete-at https://review.openstack.org/534959 | 22:16 |
| *** bigdogstl has quit IRC | 22:16 | |
| *** itlinux has quit IRC | 22:17 | |
| notmyname | joeljwright: go ahead and leave your +1 | 22:18 |
| joeljwright | sure, no problem, I was going to +1 while I test more anyway | 22:19 |
| joeljwright | notmyname: I just noticed all the other options had gone | 22:19 |
| joeljwright | :) | 22:19 |
| notmyname | joeljwright: FWIW your email is listed as a gmail one | 22:20 |
| notmyname | but /msg me a different one and I'll set it | 22:20 |
| notmyname | timburke: FWIW joeljwright is having issues with his gerrit id, and I'm getting errors when I try to re-add him. treat his +1 as a +2 for now until we can resolve it with infra | 22:29 |
| *** gkadam has quit IRC | 22:29 | |
| timburke | yeha, that was kinda gonna be my opinion regardless :-) | 22:29 |
| notmyname | heh, yeah :-) | 22:30 |
| joeljwright | timburke: notmyname: +1 (well, +2 really) | 22:30 |
| joeljwright | :) | 22:30 |
| notmyname | lol | 22:30 |
| acoles | goodnight | 22:32 |
| clayg | timburke: oh.. a mock.side_effect = [503, Exception] sorta situation *would* demonstrate the bug | 22:32 |
| joeljwright | good night peeps | 22:34 |
| joeljwright | notmyname: timburke: thanks for the help with my gerrit user | 22:34 |
| timburke | good night joeljwright | 22:34 |
| *** joeljwright has quit IRC | 22:34 | |
| *** ukaynar has joined #openstack-swift | 22:38 | |
| *** ukaynar has quit IRC | 22:39 | |
| *** kei_yama has joined #openstack-swift | 23:02 | |
| rcernin | ping zaitcev_ if we are going to sync the rings,builder with scp because md5sum is different across all nodes (we dont know which is correct). We chosen the first one. Would that cause any outage in Swift? How long would it take before it would decrease I/O with around 200GB data in Swift? | 23:02 |
| notmyname | it could cause an outage (eg if you push a ring that's wildly different than the one used to place data) | 23:03 |
| notmyname | why do you think it will decrease IO? | 23:03 |
| rcernin | notmyname: currently swift is throwing replicate errors because of the differend md5sum, as well we see cpu peaks on swift. mostly when it starts doing rsync. | 23:04 |
| *** bigdogstl has joined #openstack-swift | 23:07 | |
| notmyname | worst case, you push a wildy different ring and you lose availability of your data until replication moves everything back in place | 23:08 |
| notmyname | well, that's not truly worst case. you could break your cluster more than that if you push a ring with a different part power or the wrong IPs in it or something | 23:09 |
| notmyname | rcernin: are all of your rings different? | 23:09 |
| notmyname | ie all the md5sums | 23:09 |
| rcernin | notmyname: yes :( | 23:09 |
| notmyname | ah | 23:09 |
| notmyname | I'm not gonna lie. you could be in a better situation | 23:10 |
| notmyname | so what to do? | 23:11 |
| notmyname | some people prefer whiskey. I like tequila. | 23:11 |
| notmyname | ;-) | 23:11 |
| notmyname | rcernin: but in serousness, let's get you back on the happy path :-) | 23:12 |
| notmyname | how were you planning to chose which ring to push to the other nodes? | 23:13 |
| *** bigdogstl has quit IRC | 23:15 | |
| zaitcev_ | If only there was a tool that extracted the existing ring from *.ring.gz, then rcernin could use it to discern which builder file is the best match for the majority of the storage nodes. | 23:18 |
| zaitcev_ | Or heck, just re-construct an approximation of a builder file | 23:18 |
| rcernin | notmyname: it doesnt matter if they are all different. | 23:18 |
| zaitcev_ | that's why I said majority | 23:18 |
| notmyname | zaitcev_: IIRC the builder does a pretty decent job of that | 23:18 |
| rcernin | so the possibility one would loose the data. | 23:19 |
| zaitcev_ | well, you can construct the "least distance" metric and then find the closest builder | 23:19 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Recenter builder test expectation around random variance https://review.openstack.org/532978 | 23:19 |
| notmyname | rcernin: how many servers do you have the different rings on? | 23:19 |
| *** saint_ has joined #openstack-swift | 23:20 | |
| rcernin | zaitcev_: notmyname: recon --md5 output: http://paste.openstack.org/show/646637/ | 23:21 |
| notmyname | so only 3 servers, but all rings are out of sync | 23:24 |
| rcernin | yes to my understanding of that output | 23:24 |
| notmyname | were they built locally on the boxes? do you know how you got into this situation? | 23:24 |
| rcernin | 3 servers, 2 servers doesn't match the third server md5sum | 23:24 |
| rcernin | notmyname: most likely after replacing controller | 23:25 |
| rcernin | zaitcev_: ^^ | 23:26 |
| rcernin | notmyname: zaitcev_: https://bugzilla.redhat.com/show_bug.cgi?id=1310865 | 23:26 |
| openstack | bugzilla.redhat.com bug 1310865 in openstack-tripleo-heat-templates "Director might break Swift cluster when replacing / adding new nodes" [Unspecified,Closed: errata] - Assigned to cschwede | 23:26 |
| notmyname | is that the box where you built rings? (just guessing. there's not a "controller" component in swift, although it is used elsewhere in openstack projects) | 23:27 |
| rcernin | notmyname: its tripleo, its the bug I sent I believe. | 23:28 |
| rcernin | notmyname: with replacing the node, you break swift cluster. | 23:28 |
| notmyname | that sounds bad | 23:29 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Recenter builder test expectation around random variance https://review.openstack.org/532978 | 23:29 |
| rcernin | workaround was to copy the *.ring.gz and *.builder to new node manually | 23:30 |
| rcernin | how can we actually find the most closes builder? | 23:31 |
| zaitcev_ | The old TripleO used to re-build the rings from scratch every time. That's why it needed to control the random seed. | 23:31 |
| openstackgerrit | Tim Burke proposed openstack/swift master: slo: Send ETag header in 206 responses https://review.openstack.org/533321 | 23:32 |
| *** chsc has quit IRC | 23:32 | |
| zaitcev_ | rcernin: Since for your case *.builder used locally to produce *.ring.gz, you can just look at those on all 3 nodes by hand. Run "swift-ring-builder object.builder" with no other arguments, it shows you what's in the file. | 23:33 |
| notmyname | clayg: didn't you add a rebuild-builder-from-ring at some point? I didnt' see it in the ring builder usage text | 23:33 |
| clayg | write_builder | 23:34 |
| notmyname | thanks | 23:34 |
| notmyname | oh, there it is :-) | 23:34 |
| clayg | i totally want a script you can run on all the nodes to ls some datadirs and see what parts are there - then collect all those files and feed it into a script that builds a replica2part2dev table from what's on disk | 23:35 |
| clayg | so want that | 23:35 |
| clayg | it'd be a great ptg/hack-a-thon project | 23:35 |
| clayg | so much fun | 23:35 |
| notmyname | seems like rcernin really wants that too ;-) | 23:35 |
| clayg | it's an interesting problem because of over-replication | 23:36 |
| timburke | and does some weighting by how many bytes are there... | 23:36 |
| clayg | in your three replica ring you might have 4, 5, even maybe 14 different nodes that have *some* of a part | 23:36 |
| clayg | timburke: yeah counting up bytes used in each partition might be expensive - but you might get *something* out of suffix density (1 ls instead of a bazillian stats) | 23:37 |
| notmyname | rcernin: but you've only got 200GB of data. I'm guessing that's a small percentage of your available storage? | 23:37 |
| clayg | maybe something with blk_stat would help - idk | 23:37 |
| timburke | might need to be a two-pass sort of a thing, yeah | 23:38 |
| clayg | scoff @ 200GB - just but a copy of every obect on every disk - done | 23:38 |
| rcernin | notmyname: zaitcev_: the output of swift-ring-builder /etc/swift/object.builder is exactly same on all three nodes. | 23:39 |
| notmyname | rcernin: (first cd to some different directory away from /etc/swift) if you `swift-ring-builder object.builder write_ring` from one of those builder files, does the resulting .ring.gz match any of the ones you have? | 23:42 |
| zaitcev_ | Do we have the date in there somewhere? The last partition move time. | 23:47 |
| zaitcev_ | So, they can never match | 23:47 |
| rcernin | notmyname: swift-ring-builder object.builder generate the ring.gz files. Should it create it in . ? | 23:50 |
| rcernin | should I check the /etc/swift/*.ring.gz? | 23:50 |
| clayg | yeah i mean the whole md5 the rings thing came from a deployment pattern where you created *one* ring.gz and then shipped it out to the nodes - if you're deployment pattern is trying to have all the nodes deterministically create the same file on disk that's harder challenge isn't it. | 23:54 |
| notmyname | the hope is that one of the md5sums you have deployed right now actually matches the output of the write_ring for the builder you have | 23:54 |
| notmyname | so my idea was to create a .ring.gz somewhere other that /etc/swift/ (ie don't overwrite anything) and then compare the new one with the output of swift-recon from earlier | 23:54 |
| clayg | seems reasonable | 23:55 |
| rcernin | notmyname: thats great idea actually but how can you create *.ring.gz? | 23:55 |
| clayg | entirely possible all these files have the same replica2part2dev table and different md5's tho | 23:55 |
| notmyname | `cd /tmp && swift-ring-builder path/to/builder write_ring` | 23:56 |
| notmyname | then you'll have a new .ring.gz in /tmp | 23:56 |
| *** gyee has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!