| opendevreview | Matthew Oliver proposed openstack/swift master: Prepare tracing by adding a WSGI mixin for middlewares https://review.opendev.org/c/openstack/swift/+/797811 | 06:43 |
|---|---|---|
| opendevreview | Matthew Oliver proposed openstack/swift master: trace: Add RequestTraceMiddleware https://review.opendev.org/c/openstack/swift/+/803879 | 06:43 |
| acoles | timburke_: sorry, I can't make the meeting today | 18:32 |
| timburke_ | no worries | 18:32 |
| *** timburke_ is now known as timburke | 20:27 | |
| timburke_ | almost meeting time! | 20:55 |
| *** timburke_ is now known as timburke | 20:55 | |
| kota | good morning | 20:55 |
| mattoliver | Morning | 20:59 |
| seongsoocho | good morning | 20:59 |
| timburke | #startmeeting swift | 21:01 |
| opendevmeet | Meeting started Wed Aug 25 21:01:30 2021 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:01 |
| opendevmeet | The meeting name has been set to 'swift' | 21:01 |
| timburke | who's here for the swift meeting? | 21:01 |
| mattoliver | o/ | 21:01 |
| seongsoocho | o/ | 21:01 |
| kota | o/ | 21:02 |
| timburke | first up | 21:03 |
| timburke | #topic elections | 21:03 |
| timburke | in case you missed it, the PTL and TC candidacy period ended | 21:03 |
| timburke | #link http://lists.openstack.org/pipermail/openstack-discuss/2021-August/024401.html | 21:04 |
| timburke | for the TC, there were four candidates for four slots. for the projects, only six had no candidate, which seems like an improvement over recent cycles | 21:04 |
| timburke | just an FYI -- shouldn't really impact us | 21:05 |
| timburke | #topic next meeting to be September 8 | 21:06 |
| timburke | i'm going to be out of town next week, and i'm guessing nobody wants to chair the meeting ;-) | 21:06 |
| mattoliver | happy to if there is anything important, but think its probably ok to skip.. does mean a sleep in ;) | 21:07 |
| timburke | #topic ring v2 format | 21:08 |
| timburke | so we've got a few different initiatives that want to get more or different data into rings | 21:08 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/761794 | 21:09 |
| timburke | ^^^ wants to allow wider device IDs so we can have more than 64k devices in a single ring | 21:09 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/790550 | 21:09 |
| mattoliver | nice problem to have :) (re: more devices) | 21:09 |
| timburke | ^^^ wants to add an extra replica, essentially, to track the set of previous primaries during a rebalance | 21:10 |
| timburke | ...but that has some complications with fractional replicas, which led to https://review.opendev.org/c/openstack/swift/+/803665 getting split out from it | 21:11 |
| timburke | between the two, we've started to feel out what we'd want from a new ring format, and there are a few different ideas for how we could serialize the data in a way that would lend itself toward further extensibility | 21:13 |
| mattoliver | i probably need to update that commit message. | 21:13 |
| timburke | a bunch of which mattoliver helpfully gathered into an etherpad: | 21:13 |
| timburke | #link https://etherpad.opendev.org/p/swift-ring-v2-serialization | 21:13 |
| timburke | so, if anyone else wants to think about how best to serialize (or more concretely, about what *else* we may want to serialize), join the discussion! i'd start with either that etherpad or the last patch i linked | 21:15 |
| mattoliver | note: current patchset of 803665 is implementing option 3 for part2dev.. not that it's the decision, but wanted to see how it shook out. | 21:17 |
| timburke | next up | 21:19 |
| timburke | #topic misplaced objects in sharded containers | 21:19 |
| timburke | i haven't been following this work that closely, but i figured i ought to see how it's going after seeing "DNM: shard policy migration: it's still not great" go by ;-) | 21:20 |
| mattoliver | yeah, to be honest I need to loop back to this and was my plan for today. | 21:21 |
| mattoliver | acoles: found an edge case where the patch up for review had an issue | 21:21 |
| mattoliver | and it has to do with the root -> shard migration code not quite being the same as how the reconcoler makes decision. | 21:22 |
| timburke | 👍 no worries, we can check back in later. interesting... do you feel like you guys are getting blocked? | 21:23 |
| mattoliver | So I think its more about getting those more aligned. But I'll now more once I read the DNM patch. | 21:23 |
| mattoliver | *know | 21:23 |
| timburke | cool | 21:23 |
| timburke | #topic open discussion | 21:24 |
| timburke | anything else we ought to bring up this week? | 21:24 |
| mattoliver | I've been playing with swift tagging and tracing requests, and it's starting to look pretty nice. It tracks and times every middleware and even follows down into the storage nodes. | 21:25 |
| mattoliver | so someone can add a trace_sig to the query string of a request and it gets traced. | 21:25 |
| mattoliver | And is using OpenTracing so a nice open standard. | 21:26 |
| mattoliver | I think I already have it as a topic at the PTG, so can show it off with pics there :) | 21:26 |
| seongsoocho | wow.. Is it different from transaction id? | 21:26 |
| mattoliver | yeah, its to get better detail into whats happening by being able to literally trace a request though the cluster and get more information then what you get with just logs. | 21:28 |
| mattoliver | so extends and works with txid :) | 21:28 |
| seongsoocho | cool ! | 21:29 |
| mattoliver | here is an example: https://drive.google.com/file/d/15eDwn3RF7DJhlNtgKB-dKdiQ-9oNF8TK/view?usp=sharing | 21:32 |
| mattoliver | its just a screen shot of a trace from jaeger trace. It's tracing an EC PUT through my VSAIO. | 21:32 |
| mattoliver | There are also logs (kv pairs) attached to different spans (the lines) that I'm pulling out. Like other timings and things like error limited nodes from the proxy.. but you cant see from that image, thats just giving you the overview. | 21:34 |
| timburke | very cool. the long delay between getting into the proxy app and getting to the object servers is definitely worth investigating | 21:35 |
| mattoliver | +1 | 21:35 |
| timburke | i wonder how much of that it is backend connection setup | 21:36 |
| mattoliver | just needs more instramenting to see what's happening. Starting to visualize might find targets for optimisations or even regressions | 21:36 |
| timburke | all right, i'm going to call it | 21:37 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:38 |
| mattoliver | thanks timburke enjoy your trip! | 21:38 |
| timburke | #endmeeting | 21:38 |
| opendevmeet | Meeting ended Wed Aug 25 21:38:55 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:38 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2021/swift.2021-08-25-21.01.html | 21:38 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2021/swift.2021-08-25-21.01.txt | 21:38 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2021/swift.2021-08-25-21.01.log.html | 21:38 |
| timburke | almost forgot that part ;-) | 21:39 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!