| opendevreview | Jianjian Huo proposed openstack/swift master: Memcached: emit memcache timing metrics when exceptions raised https://review.opendev.org/c/openstack/swift/+/880676 | 01:28 |
|---|---|---|
| opendevreview | Matthew Oliver proposed openstack/swift master: internal_client: Add iter_shard_ranges interface https://review.opendev.org/c/openstack/swift/+/877584 | 02:21 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Memcached: emit memcache timing metrics when exceptions raised https://review.opendev.org/c/openstack/swift/+/880676 | 03:36 |
| opendevreview | Alistair Coles proposed openstack/swift master: ECFragGetter: simplify iter_bytes_from_response_part https://review.opendev.org/c/openstack/swift/+/881512 | 09:33 |
| opendevreview | Alistair Coles proposed openstack/swift master: DNM: failing ECFragGetter timely timeout assertion https://review.opendev.org/c/openstack/swift/+/881513 | 09:33 |
| opendevreview | Alistair Coles proposed openstack/swift master: ECFragGetter: simplify iter_bytes_from_response_part https://review.opendev.org/c/openstack/swift/+/881512 | 10:34 |
| opendevreview | Alistair Coles proposed openstack/swift master: ECFragGetter: assume policy.fragment_size is non-zero https://review.opendev.org/c/openstack/swift/+/881547 | 10:34 |
| opendevreview | Alistair Coles proposed openstack/swift master: DNM: failing ECFragGetter timely timeout assertion https://review.opendev.org/c/openstack/swift/+/881513 | 11:01 |
| fvennetier | Hello. We have been bothered by PyCADF warnings in our swift logs. These came from ceilometermiddleware. A few weeks ago I made a pull request to fix the warning: https://review.opendev.org/c/openstack/ceilometermiddleware/+/877742 | 12:12 |
| fvennetier | Can one of you take a look at it? | 12:12 |
| opendevreview | Clay Gerrard proposed openstack/swift master: log timeout duration https://review.opendev.org/c/openstack/swift/+/881520 | 15:39 |
| opendevreview | Clay Gerrard proposed openstack/swift master: debug proxy ec pipeline client timeout test https://review.opendev.org/c/openstack/swift/+/881584 | 16:04 |
| timburke | fvennetier, i left a review, but note that i don't think there's any overlap between swift cores and ceilometer cores -- which is to say, i have no power there :P | 16:21 |
| opendevreview | Tim Burke proposed openstack/swift master: CI: Use [tox]requires to pin virtualenv https://review.opendev.org/c/openstack/swift/+/881597 | 17:56 |
| opendevreview | Tim Burke proposed openstack/swift master: CI: Use [tox]requires to pin virtualenv https://review.opendev.org/c/openstack/swift/+/881597 | 18:42 |
| opendevreview | Tim Burke proposed openstack/swift master: Support tox4 https://review.opendev.org/c/openstack/swift/+/881642 | 19:00 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Apr 26 21:00:11 2023 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift team meeting? | 21:00 |
| acoles | o/ | 21:01 |
| kota | o/ | 21:01 |
| mattoliver | o/ | 21:02 |
| timburke | sorry that it's been so long since i actually held a meeting | 21:03 |
| timburke | there were a couple items from whenever that last time was that i wanted to follow up on | 21:03 |
| timburke | #topic keepalive timeouts | 21:03 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/873744 | 21:03 |
| timburke | i finally got around to reworking that to be a plumbing-only patch! | 21:04 |
| timburke | thanks zaitcev for reviewing it | 21:04 |
| mattoliver | Oh cool! I'll take a look at it too then | 21:04 |
| timburke | i forget whether we (nvidia) have just recently started running with that and latest eventlet, or if that's going out next week, but it's looking good so far | 21:06 |
| timburke | #topic ssync and data with offsets | 21:06 |
| timburke | both patches have merged! | 21:06 |
| acoles | \o/ | 21:06 |
| timburke | but... they introduced a flakey probe test. keep an eye out for it in gate results | 21:07 |
| timburke | speaking of... | 21:07 |
| timburke | #topic gate issues | 21:07 |
| timburke | we recently had a busted lower-constraints job, after virtualenv dropped support for creating py2 envs | 21:08 |
| timburke | that's been fixed (and i've got a follow-up to fix it better) -- but it still impacts stable branches | 21:09 |
| timburke | once the follow-up lands, i'll propose some backports with the two patches squashed together | 21:09 |
| timburke | i've also seen some flakey tests lately | 21:10 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/2017024 | 21:10 |
| mattoliver | Kk, thanks for all that work timburke | 21:10 |
| timburke | was easy to reproduce locally, and not too bad to fix once i looked at some of the other tests in the file | 21:11 |
| timburke | see https://review.opendev.org/c/openstack/swift/+/881142 | 21:11 |
| timburke | but the probe test i mentioned... | 21:12 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/2017021 | 21:12 |
| timburke | i still haven't reproduced locally, and i haven't found any smoking guns in the gate job logs | 21:13 |
| timburke | if anyone has some cycles to spare on it, i'd appreciate any insights you can figure out -- this seems to be the leading cause of rechecks the past week or two | 21:14 |
| acoles | :/ | 21:14 |
| timburke | alternatively, we could consider removing the flakey test, but that doesn't seem great | 21:15 |
| acoles | subjectively, it does seem to be causing a lot of rechecks | 21:15 |
| timburke | lastly, i wanted to draw attention to a recent proxy error we've been dogpiling on | 21:16 |
| timburke | #topic ec frag iter errors | 21:16 |
| mattoliver | I'll keep an eye out for it, and if it happens to me I'll dig in, in the meantime will also add it to the bottom of my todo list and hope to get to it at some point. | 21:16 |
| timburke | indianwhocodes was investigating some differences in py2/py3 proxy behaviors, and started pulling at this "generator already executing" error | 21:17 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/880356 | 21:18 |
| timburke | (note that the error would happen under both py2 and py3, but the tracebacks got much noisier in py3, as it started adding more context about what other errors were in the process of being handled) | 21:19 |
| timburke | the more we thought about it, the weirder it seemed; eventually, we pieced together that one greenthread was trying to close out a generator that was currently executing (and blocked on IO) in another greenthread | 21:21 |
| timburke | this has led to a few different refactorings from clayg and acoles -- i'm really optimistic about where the EC GET code will wind up | 21:22 |
| timburke | two questions i've got about it, though: | 21:22 |
| timburke | 1. do we have an upstream bug about the error already? | 21:22 |
| timburke | and 2. do we have a fix yet? i think i heard that we've got a good idea of what needs to happen, but idk whether we've got a patch that could include a "closes-bug" | 21:24 |
| mattoliver | 1. not that I've seen. though I haven't looked. maybe indianwhocodes could write one? it'll be good educational experience. | 21:24 |
| timburke | maybe these will be better questions when indianwhocodes and clayg are around ;-) | 21:26 |
| kota | +1 | 21:26 |
| timburke | i can also bring it up out-of-band | 21:26 |
| mattoliver | yeah, sorry I haven't been following the work. | 21:27 |
| timburke | just wanted to (1) bring it to people's attention in case they wanted to help out or better understand it, and (2) call out the good work in digging deep on a complicated part of the proxy | 21:27 |
| mattoliver | +100 | 21:28 |
| timburke | that's all i've got | 21:28 |
| timburke | #topic open discussion | 21:28 |
| timburke | anything else we should bring up this week? | 21:28 |
| mattoliver | I've spend a bunch of time, and there will be a bunch more, adding unit tests to tracing. You might have seen some activitity | 21:29 |
| mattoliver | I'm basically adding some tracing asserts, ie what spans should be created to tests of middlewares that've i've actually gone in and instrumented a little (that have extra spans added) | 21:30 |
| timburke | whoo! i still need to give tracing a spin | 21:30 |
| mattoliver | It could be a never ending scope of how many and what type of unit tests, but just want to do something to get the code into a more upstream mergible state. | 21:31 |
| mattoliver | you should give it a whirl! (when you have time) | 21:31 |
| mattoliver | just yesterday I was adding tests to tempurl and saw clearing we run _get_hmac 4 times on HEADs. Then looking in the code, yup it does (and it's suppose to) but was obvious in the spans that were created, it was pretty cool to see it | 21:32 |
| mattoliver | *clearly | 21:33 |
| timburke | i remember that it requires py3 -- but can we merge it with the caveat that you should only configure it under py3 (but everything will still run fine under py2 as long as it's *not* configured)? | 21:33 |
| mattoliver | yeah, when we do, there will be an impact. | 21:33 |
| mattoliver | I'll double check, the concrete tracer implementations were defintely py3. | 21:34 |
| opendevreview | Shreeya Deshpande proposed openstack/swift master: Error logs changed for ChunkWriteTimeout https://review.opendev.org/c/openstack/swift/+/881648 | 21:35 |
| mattoliver | anyway, that's a good test, I can create a python2 venv and give it a whirl | 21:36 |
| mattoliver | thats all I have | 21:37 |
| timburke | oh, duh! i should just look at the zuul results! | 21:37 |
| mattoliver | oh yeah! lol | 21:37 |
| timburke | all right, i think i'll call it early then | 21:37 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:37 |
| timburke | #endmeeting | 21:37 |
| opendevmeet | Meeting ended Wed Apr 26 21:37:59 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:37 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2023/swift.2023-04-26-21.00.html | 21:37 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2023/swift.2023-04-26-21.00.txt | 21:37 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2023/swift.2023-04-26-21.00.log.html | 21:37 |
| opendevreview | Clay Gerrard proposed openstack/swift master: Log (Watchdog's) Timeouts with duration https://review.opendev.org/c/openstack/swift/+/881520 | 21:47 |
| opendevreview | Tim Burke proposed openstack/swift master: ring: Centralize device normalization https://review.opendev.org/c/openstack/swift/+/881671 | 22:36 |
| opendevreview | Merged openstack/swift master: CI: Use [tox]requires to pin virtualenv https://review.opendev.org/c/openstack/swift/+/881597 | 22:37 |
| opendevreview | Tim Burke proposed openstack/swift master: tests: Probe test configs should be in [1, 4], not [0, 3] https://review.opendev.org/c/openstack/swift/+/881672 | 22:39 |
| opendevreview | Tim Burke proposed openstack/swift stable/2023.1: CI: fix lower-constraints job https://review.opendev.org/c/openstack/swift/+/881673 | 22:45 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!