| *** gyee has quit IRC | 01:44 | |
| zaitcev | I looked it up in my records and the transition to the 62xx port block started in 2014. | 02:15 |
|---|---|---|
| *** manuvakery has joined #openstack-swift | 02:31 | |
| *** rcernin has quit IRC | 03:29 | |
| *** rcernin has joined #openstack-swift | 03:41 | |
| *** rcernin has quit IRC | 03:58 | |
| *** rcernin has joined #openstack-swift | 04:15 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-swift | 04:33 | |
| *** manuvakery has quit IRC | 04:40 | |
| *** dsariel has joined #openstack-swift | 05:45 | |
| *** dsariel has quit IRC | 06:39 | |
| *** dsariel has joined #openstack-swift | 07:24 | |
| *** ccamel has joined #openstack-swift | 08:02 | |
| *** camelCaser has quit IRC | 08:03 | |
| *** dsariel has quit IRC | 08:53 | |
| *** dsariel has joined #openstack-swift | 09:07 | |
| *** rcernin has quit IRC | 10:33 | |
| *** tkajinam has quit IRC | 11:36 | |
| *** irclogbot_0 has quit IRC | 15:56 | |
| *** irclogbot_1 has joined #openstack-swift | 15:58 | |
| *** irclogbot_1 has quit IRC | 16:15 | |
| *** irclogbot_2 has joined #openstack-swift | 16:17 | |
| timburke | so i'm going to be out of town most of next week -- i propose we skip next week's meeting, unless someone else would like to chair it | 17:09 |
| openstackgerrit | Tim Burke proposed openstack/liberasurecode master: Be willing to write fragments with legacy crc https://review.opendev.org/738959 | 17:36 |
| *** gyee has joined #openstack-swift | 17:45 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add py3 probe tests on CentOS 8 https://review.opendev.org/690717 | 17:58 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: WIP: add swift-manage-shard-ranges shink command https://review.opendev.org/741721 | 18:18 |
| clayg | timburke: ^ please to be the "WTF clayg you're stupid" on this one if you have the bandwidth | 18:19 |
| clayg | i don't understand shard range timestamps - they have like 3 of them I think? | 18:39 |
| clayg | oh, or maybe 4! | 18:42 |
| timburke | yeah, i think the best source on understanding those is the ShardRange docstring: https://github.com/openstack/swift/blob/2.25.0/swift/common/utils.py#L4785-L4815 | 18:57 |
| timburke | it's still a little fuzzy -- like, i never quite remember why moving to shrinking requires en epoch change... | 19:01 |
| clayg | i also subtly mis-understood that sometimes a ShardRange can be created and only have _timestamp set such that if you update it you can change the .metadata_timestamp accessor property 🤯 | 19:04 |
| timburke | clayg, so i think i'm on-board with the approach -- there's probably some pathological cases where things get hairy (like if you've got ranges ''-'foo', 'bar'-'quux', 'quuux'-'' then you'll probably find it simpler to jiggle the bounds than try to shrink) but i think we've got pretty good odds on having a state that's workable with that shrinking behavior | 19:24 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: add swift-manage-shard-ranges shink command https://review.opendev.org/741721 | 19:44 |
| clayg | timburke: yeah i'm worried about not having ops tools to bounds jiggle, I feel like that'd be SUPER powerful | 19:45 |
| clayg | timburke: but I think the shrinking code I've got tested here will support all the known for sure already in prod cases | 19:45 |
| clayg | there's still some operator discretion about picking the "right" donor that I think we could codify to try and only pick shadowed overlaps that aren't receiving updates | 19:46 |
| clayg | I'm going to bounce off the cli tool and see if I can merge p 738114 | 19:47 |
| patchbot | https://review.opendev.org/#/c/738114/ - swift - Address a sharder/replicator race - 4 patch sets | 19:47 |
| timburke | 👍 | 19:48 |
| clayg | oh, no shoot - i just missed your comments; let try and grok that first | 19:48 |
| clayg | yeah... uhh I think you've already spotted some bugs but they should be pretty quick to hit with unittests | 19:51 |
| clayg | I'm onboard with a refactor for readability; as long as the tests don't have to change to much so I can see still that it's working the way I expect | 19:52 |
| clayg | ok, so at least find_"shinrking" and the --force are blockers, i don't really know how shrinking to the root works... | 20:02 |
| zaitcev | Suppose one has a Newton based cluster and wants to upgrade to Train. The method chosen, however, is to introduce new nodes with Train, rebalance everything, discard old Newton nodes. Is there a reason why this cannot work? | 20:17 |
| timburke | zaitcev, seems like it should work... my biggest concern would be the amount of data movement, but if they're willing to put up with it, ok... might want to make sure the train nodes are storage-only until you're ready to swap over proxies | 20:24 |
| zaitcev | timburke: I bridled at the idea at first, but the problem is jamming a new OS into old nodes. I'm just afraid if we're forgetting something that is not a possibility of the crc32 mismatch. | 20:26 |
| zaitcev | Life for example, our new expirer is not like the old expirer, they'll need "deque legacy" or what's that option | 20:27 |
| *** coreycb has joined #openstack-swift | 20:27 | |
| zaitcev | I can see the idea not to upgrade through 6 releases step-by-step may seem appealing. | 20:28 |
| timburke | so what all gets upgraded here? swift, of course -- you also say OS -- will python version change? will libec version change? | 20:35 |
| zaitcev | Yeah. It's basically a whole new cluster, just happens to have the same hash in swift.conf. They initially wanted to use container-sync to move the data. | 20:36 |
| timburke | everybody wants deque_from_legacy enabled, or to continue using a separate config file -- we still haven't landed p 517389 | 20:37 |
| patchbot | https://review.opendev.org/#/c/517389/ - swift - Add object-expirer new mode to execute tasks from ... - 46 patch sets | 20:37 |
| timburke | encryption enabled? | 20:38 |
| zaitcev | No | 20:38 |
| zaitcev | No EC either | 20:38 |
| timburke | nothing leaps out at me from `git log 2.10.0..2.23.1 --grep UpgradeImpact` | 20:42 |
| timburke | note that there are a fair few py3 fixes on stable/train that don't have a tag yet, though | 20:42 |
| zaitcev | Yeah, but weren't they rather peripheral anyway... I mean, tempurl, staticweb. | 20:44 |
| zaitcev | What really bothers me though, this is a huge cluster. The biggest I've ever seen, really. And, there's something not right with the performance there. Just adding a dozen of drives takes a week for rebalances to complete and move something like 100TB. | 20:47 |
| zaitcev | I thought container-sync would take a 3 years | 20:47 |
| timburke | is that with things like handoffs_only turned on? | 21:09 |
| timburke | yeah, i'd mainly worry about the speed of the transition :-/ | 21:09 |
| timburke | how crazy would it really be to take a node at a time, wipe the OS, do a fresh install on train, and keep the data drives across the transition? | 21:13 |
| openstackgerrit | Tim Burke proposed openstack/swift master: sharding: probe test to exercise manual shrinking https://review.opendev.org/744256 | 21:28 |
| timburke | clayg, so i think test_manage_shard_ranges_used_poorly_on_a_shard passes on ^^^ | 21:29 |
| timburke | test_manage_shard_ranges_used_poorly, not so much -- even after allowing to shrink into a CLEAVED shard | 21:29 |
| clayg | eek, that might be bad then 😬 | 21:40 |
| clayg | I see you left some some examples of things going on off the rails on p 741721 | 21:41 |
| patchbot | https://review.opendev.org/#/c/741721/ - swift - add swift-manage-shard-ranges shink command - 4 patch sets | 21:41 |
| clayg | so.. I'll start there on Monday morning | 21:42 |
| timburke | i still haven't actually tried out my suggested alternate find_shrinking_acceptors -- but i think the heavy preference for using overlaps will work in our favor? | 21:57 |
| timburke | i'm'a try it out | 21:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!