Wednesday, 2024-11-06

nathang15Hi everyone, I wonder if it is normal if only the TestReplicatorFunctions within probe tests fails? I did not modify any code and I tried running resetswift, reboot my pc but it still fails. Every other tests passed, some were skipped though. Here is the output of my probe tests: https://paste.opendev.org/show/bAkeq8da8gwLhL9rZClY/02:33
nathang15Thank you02:34
opendevreviewMerged openstack/swift feature/mpu: mpu: add x-backend-etag-is-at header to GET/HEAD requests  https://review.opendev.org/c/openstack/swift/+/93393202:45
timburkenathang15, is this off master, or some change you're working on? i'd expect the tests to all pass...03:12
timburkehave you tried running that test in isolation? `pytest test_replication_servers_working.py::TestReplicatorFunctions::test_main` should target just the one. then it's easier to check logs after the failure -- though off hand, i'm not sure what could have caused the failure03:13
nathang15Yep this is off master, maybe I'll try stable 2024.203:20
nathang15I did not work on any changes yet03:20
opendevreviewMatthew Oliver proposed openstack/swift master: Configurable expiring_objects_task_container_per_day  https://review.opendev.org/c/openstack/swift/+/93337304:13
mattolivernathang15: it seems to indicate that the directory is still there. If you just run that probe test (as Tim showed), it should leave it in the state it failed. and we can look at the directory and try and figure out why it wasn't removed. 04:34
opendevreviewMatthew Oliver proposed openstack/swift master: db_auditor: add vacuum support  https://review.opendev.org/c/openstack/swift/+/91686106:31
mattoliver^ just moving the guage definition to somewhere else in the file to stop a downstream conflict / collision. fun times06:32
opendevreviewJianjian Huo proposed openstack/swift master: Object-server: add periodic greenthread yielding during file write  https://review.opendev.org/c/openstack/swift/+/93420006:49
opendevreviewMatthew Oliver proposed openstack/swift master: db_auditor: add vacuum support  https://review.opendev.org/c/openstack/swift/+/91686108:31
opendevreviewMatthew Oliver proposed openstack/swift master: add statds guage and use it for auditor vacuum sizes  https://review.opendev.org/c/openstack/swift/+/93420608:31
opendevreviewMerged openstack/swift feature/mpu: mpu: fix param passing to sub requests  https://review.opendev.org/c/openstack/swift/+/93393312:31
opendevreviewAlistair Coles proposed openstack/swift master: reconciler: Record queue-clean-up later than old-policy-clean-up  https://review.opendev.org/c/openstack/swift/+/93368314:33
opendevreviewYan Xiao proposed openstack/swift master: proxy-logging: Add real-time transfer bytes counters  https://review.opendev.org/c/openstack/swift/+/93091814:59
opendevreviewMerged openstack/swift feature/mpu: mpu: change some response codes to keep s3api happy  https://review.opendev.org/c/openstack/swift/+/93393416:29
opendevreviewAlistair Coles proposed openstack/swift feature/mpu: mpu-auditor: audit sessions  https://review.opendev.org/c/openstack/swift/+/93151716:40
opendevreviewMerged openstack/swift master: Prevent installation with pyeclib 1.6.2 and 1.6.3  https://review.opendev.org/c/openstack/swift/+/93320318:45
opendevreviewASHWIN A NAIR proposed openstack/swift master: s3api: improve error handling for part-num req's  https://review.opendev.org/c/openstack/swift/+/93415418:48
opendevreviewASHWIN A NAIR proposed openstack/swift master: s3api: improve error handling for part-num req's  https://review.opendev.org/c/openstack/swift/+/93415419:06
opendevreviewTim Burke proposed openstack/swift master: utils: Add crc32c function  https://review.opendev.org/c/openstack/swift/+/90980020:53
opendevreviewTim Burke proposed openstack/swift master: s3api: Add support for additional checksums  https://review.opendev.org/c/openstack/swift/+/90980120:53
mattoliverMorning21:02
timburkeoh, right!21:04
timburke#startmeeting swift21:04
opendevmeetMeeting started Wed Nov  6 21:04:13 2024 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.21:04
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:04
opendevmeetThe meeting name has been set to 'swift'21:04
timburkewho's here for the swift team meeting?21:04
mattolivero/21:04
nathang15hi everyone, this is my first time joining the meeting21:05
timburkehi nathang15!21:05
boosungkim-dsonHello, I am Boosung from Dickinson College.21:05
jamesngHi everyone, I'm James, from Dickinson College. This is my first time either.21:05
timburkesorry i wasn't around longer to help debug your probe test issue yesterday -- i had to get my kids down for bed :-)21:05
nathang15No worries, I somehow fixed it  :)21:06
timburkehi boosungkim-dson, jamesng!21:06
nathang15all the tests passed now21:06
timburkenathang15, oh good! i wonder what the issue was21:06
timburkeeh, as long as it works now, i suppose :-)21:06
fulecorafaHey everbody21:07
nathang15Yeah, so I head over to install vsaio, and the tests for saio passed after I finished setting up vsaio.21:07
nathang15So I think this might be some issue with my local environment21:07
mattoliverNice21:07
timburkefulecorafa, o/21:07
mattoliverSome race condition.. keep and eye on it, and if it happens again might be nice to know what it was :)21:08
nathang15Yep, I'll keep an eye on it21:08
timburkei suppose it could have been the random chance to re-hash... not sure how best to write the test to avoid that, though21:08
timburkehttps://github.com/openstack/swift/blob/2.34.0/swift/obj/replicator.py#L666-L66821:09
timburkei'm afraid i haven't updated the agenda21:10
timburkebut i *did* finally get around to digging into https://bugs.launchpad.net/swift/+bug/202817521:10
patch-botBug #2028175 - intermittent probe test failure: test_reconciler_move_object_twice (In Progress)21:10
timburkeand i think i have a fix21:11
timburke#link https://review.opendev.org/c/openstack/swift/+/93368321:11
patch-botpatch 933683 - swift - reconciler: Record queue-clean-up later than old-p... - 2 patch sets21:11
mattoliverNice21:11
timburkei wanted to be sure to bring it up since it changed how the reconciler bumps timestamp offsets, which i don't think has ever changed since we introduced storage policies21:12
timburkeer, changes* -- not actually merged yet, though acoles likes the look of it21:13
mattoliverthanks for going to the effort of groking the whole reconciler offset stuff!21:13
mattoliverI'll take a look today and see if we can get it over the line21:13
timburkethanks mattoliver!21:14
timburkein related gate-health news, i think i managed to nerd-snipe someone into looking at https://bugs.launchpad.net/openstacksdk/+bug/2085654 -- so hopefully those openstacksdk-functional-devstack failures we've been seeing will be fewer and further between sometime soon 🤞21:15
patch-botBug #2085654 - Two intermittent functional test failures (Confirmed)21:15
mattolivernerd-snipe for the win.. and it wasn't me! so extra good news :P 21:16
timburkethe one other top-of-my-head topic i've got is on https://review.opendev.org/c/openstack/swift/+/93369721:17
patch-botpatch 933697 - swift - Pull out a bunch of liberasurecode requirements - 3 patch sets21:17
mattoliveroh cool, and pete has already +2ed it21:18
timburkenow that pyeclib is publishing binary wheels, we have an opportunity to remove some of the liberasurecode stuff from the swift repo, since we won't need to build pyeclib from sdists21:18
timburkenote that it doesn't remove the need for a build environment entirely (though that would be a nice end-goal) since various other entries in test-requirements.txt pull in deps (i remember seeing netifaces at least) that still need to be built from source21:21
mattoliverkk, might be a nice side project or task for someone. 21:22
timburkeit also doesn't change anything for https://docs.openstack.org/swift/latest/development_saio.html -- though maybe we could?21:22
mattoliverideally we shouhld probably build liberasurecode manually in SAIO seeing as it is one of our projects we maintain.. and so having the build env there is probably a good thing21:23
timburkefigured i'd see what people thought about how integral it is to easily hack on pyeclib in an SAIO21:23
mattoliverbut maybe I only say that because I've been looking at it lately :P 21:24
timburkeyeah, the "it is one of our projects we maintain" is an excellent point -- i was thinking maybe we could treat pyeclib/liberasurecode something like how we treat eventlet, but maybe that's the big dividing line21:25
mattoliverI guess we could have a developement_pyeclib and link to that.. but if a SAIO is a main dev env, might be nice to put it in there too. 21:26
timburkenot changing things is pretty cheap & easy :-) we'll keep it as it is21:27
mattoliverto build it, it's only installing the compiler tools and make (build-essential for ubuntu or "Development tools" in fedora/centos) then a make, make install.21:28
mattoliversounds like another side task.. we should write these down somewhere.. on the ideas page maybe in a new section. like project low hanging fruit :) 21:28
timburkehuh, interesting... looks like pyeclib is only mentioned on the SAIO page to install it from system packages on Fedora, while liberasurecode-dev(el) is mentioned for all of them (presumably because you used to need it to install pyeclib from pypi)21:30
mattoliverno wonder we never show pyeclib no love :( 21:31
timburkewriting up bugs on launchpad and tagging it "low-hanging-fruit" seems like a reasonable plan21:32
mattoliverkk21:32
timburkeone or two other things on my mind (since we've got the Dickinson College folks)21:33
timburkefirst, (carrying on from thinking about low-hanging-fruit) what projects might be nice for them to work on?21:34
boosungkim-dsonYeah, we just finished our presentation and small demo and are looking to start contributing soon!21:35
timburkeand second, do y'all have any questions about swift already? would it be nice to have an overview of the architecture or anything?21:35
timburkenice! what was the presentation about?21:35
boosungkim-dsonThe presentation was about introducing swift to our classmates, swift architecture, and a small swift client application we made (upload/download/auth).21:36
zaitcevnice21:36
zaitcevhopefully better than this https://www.youtube.com/watch?v=lSUSZ8xGF9o21:37
mattoliverhey zaitcev o/ 21:37
timburkeyou did great, zaitcev!21:37
zaitcevthanks21:37
boosungkim-dsonWe watched that video to prepare for our presentation!21:38
zaitcevBut boosungkim-dson had a demo!21:38
zaitcevOh, really?21:38
boosungkim-dsonWe watched a few reference videos to prepare21:38
mattoliverdid we want to organise a jitsi (meetpad) video chat sometime soon if that's an easier medium to ask questions in? worked well for the qatar students. I found them useful for the qatar students as they did their projects.21:39
boosungkim-dsonYes, that would be very helpful!21:39
mattolivermaybe we can find some smaller low langing fruit to wet their appitites and then find something more meaty for them to really dig into. So they can start getting used to the gerrit and openstack dev workflows. 21:40
opendevreviewASHWIN A NAIR proposed openstack/swift master: s3api: improve s3 sigv4 error responses  https://review.opendev.org/c/openstack/swift/+/93408721:40
timburkewhen would be a good time for you guys? i might recommend some afternoon M-Th; then we could try to have both mattoliver and i there21:41
mattoliverkk. how about I set up an etherpad (that we can link to any future meetpad sync meetings) and I'll seed it with some ideas. 21:41
mattoliver+121:41
timburkeif you wanted any more reference videos, https://www.youtube.com/@OpenInfraFoundation/search?query=swift has a bunch of good talks from years past21:42
boosungkim-dsonThank you!21:43
opendevreviewASHWIN A NAIR proposed openstack/swift master: s3api: improve error handling for part-num requests  https://review.opendev.org/c/openstack/swift/+/93415421:43
mattoliverI can also do a tour with the tracing code so you can watch some paths.. although there probably already is a talk about that already (this is a bit old, but can do a live one with newer code) https://www.youtube.com/watch?v=iP_qy8KKDno 21:44
mattoliverThe code has mostly compeltely changed from there, because it's now all open-telemtry and using a decorator implementation rather then mixins, but you get so see some traces :) 21:45
timburkeanother fun thing to do is look at extending swift with your own custom middleware -- teaches you a good bit about how the WSGI pipeline works21:45
nathang15awesome thank you timburke, we also read through the swift book so we have a general idea about the architecture. Anyway, Monday 7PM, Tuesday 5PM, and Wednesday 5PM EST works for us21:45
mattoliveryeah!21:46
timburkemattoliver, have any preference? for reference, 5PM EST is ~12mins away21:48
mattoliveroh that's 9am this time of year. thats not bad at all!21:49
timburkei think any of those would work well for me21:49
mattoliverso for me, that's 11am Tues, 9am weds or 9 am thurs. I can make them work. I do have a downstream meeting at Tues 9am EST.. so maybe one of the other 2? 21:50
mattoliverbut I run it, so can always move it ;) 21:50
timburkeand it rarely goes two hours ;-P21:51
mattoliveroh no i dont. that's monday 5pm EST (man timezones are hard)21:51
mattoliveryup they;re all ok for me too!21:51
mattoliverWhatever is easiest for the students. would you rather a 5pm or 7pm? :P 21:52
nathang15Monday 7pm works best for us, thank you21:53
timburkesounds good! mattoliver will work on getting an etherpad/meetpad seeded, and we'll meet monday 👍21:54
boosungkim-dsonSounds great!21:54
mattolivergreat! let's lock that in!21:54
mattoliverI'll paste a link to the ehterpad/meetpad here when we create it. 21:54
timburkethanks mattoliver21:55
timburkeall right, we're running a little low on time -- anything else we ought to discuss this week?21:55
boosungkim-dsonI have a quick question!21:55
timburkego ahead!21:56
boosungkim-dsonI am also trying out the vagrant install, but I am noticing that not all the functional tests pass for me, and the probe tests are really really slow (over an hour).21:56
boosungkim-dsonNot sure what I exactly did wrong though...21:56
timburkedid you create your own localrc based on https://github.com/NVIDIA/vagrant-swift-all-in-one/blob/master/localrc-template? i feel like the default VAGRANT_CPUS=1 is probably too low these days...21:57
timburkefunc test failures are a little surprising to me -- do you remember which ones failed?21:58
boosungkim-dsonEh I just cloned the repo and did "vagrant up"21:58
timburkemakes sense -- we should probably also fix the default ;-)21:58
boosungkim-dsonI'm trying to get the terminal output history for the functional tests, since the probe functions are still running21:58
mattoliveryou can copy rclocal.template to rclocal. and modify it. Then just source rclocal then vagrant up.21:59
indianwhocodesmissed it :-/21:59
timburkeno rush -- we can also help debug it more21:59
timburkeindianwhocodes, can always read scrollback ;-)21:59
mattoliveryeah probe test take a while too because it calls resetswift inbetween and the vsaio's resetswift tis more involved22:00
mattoliverI did rework the vacuum patch, and ended up splitting it into 2. one with the vacuum stuff and recon dumps to dump the bucket sizes and then a follow to add a statsd guage and start using it. I split it because of scope and because the guage stuff really conflicts with the current labeled metric work.. so thought it can be more of a follow up.22:00
boosungkim-dsonI will try that and see if that changes things, thank you mattoliver22:00
mattoliverp  https://review.opendev.org/c/openstack/swift/+/91686122:00
patch-botpatch 916861 - swift - db_auditor: add vacuum support - 13 patch sets22:00
mattoliverand 22:01
mattoliver#link https://review.opendev.org/c/openstack/swift/+/93420622:01
patch-botpatch 934206 - swift - add statds guage and use it for auditor vacuum sizes - 1 patch set22:01
mattoliverbut that's all I got and we're at time22:01
timburke(oh yeah, if anyone doesn't have an irc bouncer to help catch messages when you're not around, there's also https://meetings.opendev.org/irclogs/%23openstack-swift/ which has years of channel logs)22:02
timburkeyou're right mattoliver -- thank you all for coming, and thank you for working on swift!22:02
mattoliveroh good point. otherwise just run a bouncer, or a matrix client and connect with the irc bridge :) 22:02
timburke#endmeeting22:02
opendevmeetMeeting ended Wed Nov  6 22:02:45 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)22:02
opendevmeetMinutes:        https://meetings.opendev.org/meetings/swift/2024/swift.2024-11-06-21.04.html22:02
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/swift/2024/swift.2024-11-06-21.04.txt22:02
opendevmeetLog:            https://meetings.opendev.org/meetings/swift/2024/swift.2024-11-06-21.04.log.html22:02
boosungkim-dsonThank you, and see you on Monday!22:03
timburkei figured it'd be good to mention since we have some new faces22:03
timburkeboosungkim-dson, nathang15, jamesng thanks for coming! looking forward to Monday!22:03
jamesngSee you on Monday.22:04
nathang15likewise, thank you!22:04
boosungkim-dsonBye!22:04
opendevreviewShreeya Deshpande proposed openstack/swift master: Remove PrefixLoggerAdapter and SwiftLoggerAdapter  https://review.opendev.org/c/openstack/swift/+/93106022:22
opendevreviewTim Burke proposed openstack/swift master: s3api: Add support for additional checksums  https://review.opendev.org/c/openstack/swift/+/90980122:36
opendevreviewASHWIN A NAIR proposed openstack/swift master: s3api: improve error handling for part-num requests  https://review.opendev.org/c/openstack/swift/+/93415423:15
opendevreviewASHWIN A NAIR proposed openstack/swift master: s3api: error handling of complete mpu and part-num  https://review.opendev.org/c/openstack/swift/+/93415423:19

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!