| opendevreview | Jianjian Huo proposed openstack/swift master: Sharder: add timing metrics for individual steps and total time spent. https://review.opendev.org/c/openstack/swift/+/882113 | 03:22 |
|---|---|---|
| opendevreview | Merged openstack/swift stable/zed: CI: fix lower-constraints job https://review.opendev.org/c/openstack/swift/+/881753 | 04:38 |
| opendevreview | Tim Burke proposed openstack/swift master: ring: Introduce a v2 ring format https://review.opendev.org/c/openstack/swift/+/834261 | 19:09 |
| opendevreview | Tim Burke proposed openstack/swift master: ring: Add a rebalance history in the ring https://review.opendev.org/c/openstack/swift/+/834621 | 19:09 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed May 3 21:00:35 2023 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| timburke | well, i'll go ahead and call out some things going on for the record, anyway ;-) | 21:03 |
| timburke | #topic gate issues | 21:03 |
| timburke | the remaining flakey probe test has been resolved | 21:03 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/2017021 | 21:04 |
| timburke | the issue was in how we were determining which reconstructors to run | 21:05 |
| timburke | in retrospect, it's amazing it *ever* passed (and probably a testament to how constrained the SAIO environment is; we should consider how we can get more virtual nodes in there) | 21:05 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/881672 | 21:06 |
| mattoliver | o/ | 21:06 |
| timburke | ^^^ the fix, if anyone wants more details | 21:06 |
| timburke | mattoliver, o/ | 21:06 |
| timburke | #topic EC "generator already executing" errors | 21:06 |
| mattoliver | Nice | 21:06 |
| timburke | looks like we have a fix in hand! | 21:07 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/880356 | 21:07 |
| timburke | good job indianwhocodes and clayg for running that down -- might still be good for us to write up an upstream bug we can point people at later if someone reports this for an older version of swift | 21:08 |
| mattoliver | +1 | 21:08 |
| timburke | fix seems to make sense to me; the test change needed to demonstrate it was pretty subtle and a reminder that it's hard to really be *sure* that tests demonstrate what we *think* they do | 21:09 |
| timburke | #topic infinite recursion error in tests | 21:09 |
| mattoliver | That topic around ominous | 21:10 |
| timburke | for a while now, i (and others) have noticed an infinite recursion error down in ssl handling when trying to run func tests in my SAIO | 21:11 |
| timburke | clayg and shreeya (who i don't think is on IRC yet) found a good way to avoid it, though! | 21:11 |
| mattoliver | oh cool, nice work Shreeya! we should get her on here | 21:12 |
| timburke | the core trouble was that requests-mock (required by some tests for s3token) incudes a pytest plugin which would cause requests to get imported before we get a chance to call eventlet.monkey_patch() | 21:13 |
| timburke | there was some debate about explicitly reloading ssl so it'd be properly monkey-patched, or pinning to an older requests-mock that didn't include the plugin... | 21:14 |
| timburke | then clayg noticed that we could just disable the thing via config options in tox.ini! | 21:15 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/882105 | 21:15 |
| mattoliver | oh cool | 21:15 |
| timburke | that's currently stacked on another change, though -- i think i'll swap the order if no one gets to it before me | 21:16 |
| timburke | last up | 21:16 |
| timburke | #topic reading py2 non-ascii metadata from py3 | 21:17 |
| timburke | #link https://launchpad.net/bugs/2012531 | 21:17 |
| timburke | we got a bug report about this more than a month ago -- it took longer than i wanted, but i think i've finally got a decent fix now | 21:18 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/878558 | 21:18 |
| mattoliver | could we squash it down? | 21:18 |
| mattoliver | oh yeah, that bug! | 21:18 |
| timburke | mattoliver, i was thinking keep 'em separate -- the pytest plugin fix will be really nice all on its own; i'm tired of uninstalling/reinstalling requests-mock depending on which tests i want to run :P | 21:19 |
| mattoliver | lol, fair enough! | 21:19 |
| mattoliver | ether way lets get a +2/+A on it ;) | 21:19 |
| timburke | the metadata fix is a little gross -- and feels brittle, like some future cpython change could come along and throw off our assumptions | 21:20 |
| mattoliver | I do have it on my todo list to review.. just got distracted with a downstream release this week. | 21:20 |
| timburke | so i might try to do a follow-up to add some third way of writing down metadata that wouldn't require this heuristic to determine whether (some) metadata was written down with wsgi strings or not | 21:22 |
| timburke | but i think we could have that as a separate piece of work, and try to get edausq unblocked before then | 21:23 |
| timburke | so if you've got time to review, mattoliver, that'd be great -- thanks! | 21:23 |
| timburke | that's all i've got by way of updates | 21:23 |
| timburke | #topic open discussion | 21:23 |
| timburke | anything else you'd like to bring up mattoliver? | 21:23 |
| jrosser | hello - i have a question which came up relating to swift deployed using openstack-ansible, and this https://opendev.org/openstack/swift/commit/3053c53ef7e9dd3efea10c535021786f7e6e92f2 | 21:25 |
| mattoliver | I'm still writing tracing unit tests that confirm the expected spans are present in the middlewares that I've instrumented. Just working through versioned writes. atm. So tedius and boring, but getting through it slowly | 21:25 |
| jrosser | for a use case where all things need to be TLS, is the warning about not using TLS in that patch correct? | 21:26 |
| timburke | thanks for continuing to push on it mattoliver | 21:26 |
| timburke | jrosser, i think i'd still recommend using an external TLS-terminator (haproxy, nginx, hitch, etc.) over eventlet's (which is what you'd get by configuring cert_file/key_file in swift) | 21:27 |
| jrosser | the thing is we can run into, for example, regulatory requirements that mandate TLS everywhere | 21:28 |
| mattoliver | jrosser: the proxy already does a bunch cpu (we find) so we use a terminator that speaks the proxy protocol | 21:28 |
| mattoliver | the terminator can be on the same server. so the last hop is loop back. | 21:28 |
| mattoliver | I gues | 21:29 |
| mattoliver | I guess that'll still be TLS everywhere | 21:29 |
| timburke | and you can couple it with something like the proxy protocol from https://github.com/openstack/swift/commit/661838d968c163cec716cd5c8bcce9f5c69d18d4 if you're worried about preserving client ip info | 21:29 |
| jrosser | yes, we would normally do that with uwsgi for other services in OSA | 21:29 |
| jrosser | i am just checking if there is a preferred / tested setup for these scenarios | 21:30 |
| mattoliver | Well we have a very large (proably the biggest) prod and thats what we do. We use hitch on each proxy node and use the proxy protocol that tim mentions. If that helps. | 21:32 |
| jrosser | ok thats useful info - thanks | 21:32 |
| timburke | "external terminator using proxy protocol v1" is the best solution i know of. fwiw, i run hitch at work (as mattoliver mentioned) and haproxy at home. i seem to recall OVH doing soemthing similar (though if they told me which terminator they use, i don't recall) | 21:33 |
| timburke | the bigger thing to watch out for with "TLS everywhere" is backend communications. might be able to call it good with a wireguard network, though? | 21:34 |
| jrosser | tbh we need to look at this in more detail | 21:35 |
| mattoliver | good point. and yeah, if you need/want a software solution. | 21:35 |
| jrosser | the focus so far has been getting everything between the frontend loadbalancer (haproxy) and the backends covered | 21:36 |
| timburke | i seem to recall some interested parties years ago looking at using https connections for proxy <-> backend communication, but don't think it gained much traction (unfortunately) | 21:37 |
| jrosser | anyway - this has been insightful and gives a direction to look in, thankyou | 21:37 |
| mattoliver | nps, hit us up if you need any more info or anything else | 21:38 |
| jrosser | sure, will do | 21:38 |
| timburke | happy to help! keep us posted | 21:38 |
| timburke | all right, i think i'll wrap it up a little early | 21:40 |
| mattoliver | timburke: I think that's all I got, or I can think of atm.. been a bit distracted (as I've said). | 21:40 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:40 |
| timburke | #endmeeting | 21:40 |
| opendevmeet | Meeting ended Wed May 3 21:40:19 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:40 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2023/swift.2023-05-03-21.00.html | 21:40 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2023/swift.2023-05-03-21.00.txt | 21:40 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2023/swift.2023-05-03-21.00.log.html | 21:40 |
| *** zaitcev_ is now known as zaitcev | 22:50 | |
| opendevreview | Tim Burke proposed openstack/swift master: ring: Introduce a v2 ring format https://review.opendev.org/c/openstack/swift/+/834261 | 22:55 |
| opendevreview | Tim Burke proposed openstack/swift master: ring-v2: Make RingReader a subclass of ZlibReader https://review.opendev.org/c/openstack/swift/+/882184 | 22:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!