| timburke | kota, i know clayg likes https://www.irccloud.com/ -- i think he's found a few converts | 00:32 |
|---|---|---|
| timburke | mcape, those *are* a little concerning -- it's probably worth dumping shard ranges for that DB and trying to sort out why the audit failed. if you post the output of `swift-manage-shard-ranges /path/to/db show`, i can probably help dig into it | 00:32 |
| timburke | the "refusing to replicate" messages should be benign as long as the sharder *does* get around to successfully processing the DBs fairly quickly -- if they're hanging around or your sharder's logs are full of errors, that's definitely worth investigating | 00:32 |
| timburke | DHE, yeah, storage servers first. ideally, you upgrade all object-servers, then all container-servers, then all account-servers, and finally all proxies -- but if you can at least get all backend servers, then all proxies, you're doing pretty good | 00:33 |
| timburke | i've done what i can to make the python upgrade smooth -- you should be able to run mixed py2/py3 just fine. i recommend upgrading swift first, having all nodes on a single, new-as-possible swift, then switching over to py3 on the same version of swift, then potentially upgrading swift further. that may be a little overly-cautious, though | 00:34 |
| opendevreview | Merged openstack/swift stable/train: Fix stable/train gate https://review.opendev.org/c/openstack/swift/+/871232 | 00:38 |
| DHE | oh no it's not py2. I've standardized on py 3.6.10 exactly, to the point I'm distributing pre-packaged python+swift pre-installed tarballs to all the hosts. | 01:51 |
| DHE | you probably don't remember. I was here some time ago about my proxy servers jamming randomly on EC-based GET requests. never did find a solution (yet). | 01:52 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Proxy: restructure cached updating shard ranges https://review.opendev.org/c/openstack/swift/+/870886 | 06:55 |
| mcape | timburke, thank you very much for stepping in! here's the log message and output of swift-manage-shard-ranges tool: https://pastebin.com/dcNDLLM0 | 14:05 |
| opendevreview | Alistair Coles proposed openstack/swift master: maybe squash? use a NamespaceBoundList class https://review.opendev.org/c/openstack/swift/+/871742 | 16:44 |
| opendevreview | Merged openstack/swift stable/ussuri: s3api: Prevent XXE injections https://review.opendev.org/c/openstack/swift/+/871243 | 20:08 |
| timburke | DHE, oh, right! sorry that we haven't run that down yet :-/ | 20:59 |
| seongsoocho | Hi~! | 21:00 |
| timburke | fwiw, i'm not *near* as worried about python upgrades between 3.x releases | 21:00 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Jan 25 21:00:22 2023 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift team meeting? | 21:00 |
| seongsoocho | o/ | 21:00 |
| indianwhocodes | o/ | 21:01 |
| zaitcev | Well, I am worried. The data model is kept, yes. So not as much forceful disruption. But they were throwing a bug down every goddamn release! Always something is screwed up in their eventlet or HTTP client. | 21:01 |
| acoles | o/ | 21:01 |
| timburke | zaitcev, fair point -- good chance DHE will need to upgrade eventlet too | 21:02 |
| timburke | as usual, the agenda's at | 21:02 |
| timburke | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | first up | 21:02 |
| timburke | #topic stable gate testing | 21:03 |
| opendevreview | Merged openstack/swift stable/train: s3api: Prevent XXE injections https://review.opendev.org/c/openstack/swift/+/871244 | 21:03 |
| timburke | just an fyi -- last week i talked about possibly removing integrated testing from older stable branches | 21:03 |
| timburke | i went ahead and did that for train and ussuri. since they're in extended-maintenance mode already, i don't think there's much other notification we need to give, but wanted to make sure y'all are aware | 21:04 |
| timburke | that was done to work around some broken jobs that were blocking... | 21:05 |
| timburke | #topic CVE fixes | 21:05 |
| timburke | as a refresher (i think seongsoocho may not have been here for the other meetings talking about it), a vulnerability was found in s3api's XML handling | 21:07 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/1998625 | 21:07 |
| timburke | it allows authenticated clients to read arbitrary files off swift proxy servers | 21:07 |
| seongsoocho | Yes. Now I patch it in our production swift . I tested it before, The CVE only occurs if the s3_acl option is enabled. the default is false. | 21:08 |
| timburke | patches have now merged to master and most open stable branches -- zed through train | 21:08 |
| timburke | seongsoocho, i'm pretty sure it would be exploitable via the delete-objects API -- i don't think that would be impacted by s3_acl | 21:09 |
| timburke | but i suppose as long as your swift is updated, it doesn't matter too much now :-) | 21:10 |
| timburke | patches have also been proposed to rocky and stein; i'll keep on them to get them merged | 21:10 |
| timburke | any comments or questions about the CVE? | 21:11 |
| seongsoocho | oh.. ok.. I've only reproduced it with xml files in the body of launchpad. It can also be exploited with the delete-objects API.... | 21:11 |
| timburke | the unit test that was merged uses that api, fwiw -- in case you want a starting point to try it out with s3_acl disabled | 21:12 |
| seongsoocho | ok I will check it. thanks | 21:13 |
| timburke | also on my list is to get a release together, so we have a tag we can point to that isn't affected. i'll likely also propose stable releases back through xena | 21:15 |
| timburke | the CVE's been my main focus for most of the last week -- i'm afraid i still haven't started on PTG prep, but i left it on the agenda to remind myself about it | 21:18 |
| timburke | so i think that's all i've got | 21:18 |
| timburke | #topic open discussion | 21:18 |
| timburke | anything else we should bring up this week? | 21:18 |
| acoles | timburke: thanks for all your work on the CVE and tests - seems like you got your priorities right :) | 21:18 |
| seongsoocho | 👍 thanks timburke | 21:19 |
| indianwhocodes | +1 timburke | 21:20 |
| timburke | all right, i think i'll call it then | 21:22 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:22 |
| timburke | #endmeeting | 21:22 |
| opendevmeet | Meeting ended Wed Jan 25 21:22:37 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:22 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2023/swift.2023-01-25-21.00.html | 21:22 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2023/swift.2023-01-25-21.00.txt | 21:22 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2023/swift.2023-01-25-21.00.log.html | 21:22 |
| opendevreview | Merged openstack/python-swiftclient master: Allow tempurl to be used to sign /info requests https://review.opendev.org/c/openstack/python-swiftclient/+/850786 | 23:13 |
| opendevreview | Merged openstack/python-swiftclient master: Switch to 2023.1 Python3 unit tests and generic template name https://review.opendev.org/c/openstack/python-swiftclient/+/856704 | 23:16 |
| opendevreview | Merged openstack/python-swiftclient master: Back-fill a bunch of ChangeLog releases https://review.opendev.org/c/openstack/python-swiftclient/+/869506 | 23:16 |
| opendevreview | Merged openstack/swift stable/stein: s3api: Prevent XXE injections https://review.opendev.org/c/openstack/swift/+/871501 | 23:44 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!