| *** thurloat7 has joined #openstack-swift | 00:53 | |
| *** thurloat7 is now known as thurloat_ | 00:53 | |
| *** thurloat8 has joined #openstack-swift | 00:53 | |
| *** thurloat8 is now known as thurloat_ | 00:53 | |
| *** thurloat_ has quit IRC | 00:54 | |
| *** thurloat6 has joined #openstack-swift | 00:54 | |
| *** thurloat6 is now known as thurloat__ | 00:54 | |
| *** thurloat has quit IRC | 00:56 | |
| mattoliverau | TrevizeGP_: well adding more nodes and disks could balance the iops and distribute better. Extending a FS would give you more space on that node, and give more space for the partitions on each disk, so will help in that sense. As those grow bigger and bigger and if you have alot of small files, you'd want to watch your inode cache, as that may become an issue (depending on the size of RAM). In general more disks and | 02:33 |
|---|---|---|
| mattoliverau | nodes is better and makes things faster and more durable. | 02:33 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Increase some middleware test coverage https://review.openstack.org/652185 | 02:53 |
| *** psachin has joined #openstack-swift | 03:29 | |
| *** rcernin has quit IRC | 05:08 | |
| *** rcernin has joined #openstack-swift | 05:10 | |
| *** tkajinam has quit IRC | 05:28 | |
| *** tkajinam has joined #openstack-swift | 05:54 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: Add a basic partition diffing tool https://review.openstack.org/539466 | 06:06 |
| mattoliverau | ^ just a rebase | 06:06 |
| *** pcaruana has joined #openstack-swift | 06:19 | |
| kota_ | timburke, alecuyer, rledisez: small note for this week, I'm short off substitute vacation during Tue-Thu in this week to work on PTG week (it's whole week holidays in JP) so I'll be absent from weekly meeting. | 06:41 |
| kota_ | will be back on Friday and work on the next week. sorry for the inconvinience. | 06:41 |
| *** tkajinam_ has joined #openstack-swift | 06:48 | |
| *** e0ne has joined #openstack-swift | 06:48 | |
| *** e0ne has quit IRC | 06:49 | |
| *** tkajinam has quit IRC | 06:51 | |
| *** e0ne has joined #openstack-swift | 06:52 | |
| *** rdejoux has joined #openstack-swift | 07:01 | |
| TrevizeGP_ | mattoliverau: thanks! I'm definitely adding more nodes in the future (IOPS is fine for now) I was just curious if swift cares about when underlying FS changes size suddenly, can it cause any problems? I'm glad it doesn't:) | 07:04 |
| *** rcernin has quit IRC | 07:05 | |
| *** e0ne has quit IRC | 07:12 | |
| openstackgerrit | zhufl proposed openstack/swift master: Add missing ws separator between words https://review.openstack.org/652583 | 07:13 |
| *** spotz has joined #openstack-swift | 07:22 | |
| *** e0ne has joined #openstack-swift | 07:31 | |
| *** e0ne has quit IRC | 07:52 | |
| *** e0ne has joined #openstack-swift | 07:57 | |
| *** gkadam has joined #openstack-swift | 08:04 | |
| *** gkadam is now known as gkadam-afk | 08:06 | |
| *** tkajinam_ has quit IRC | 08:20 | |
| *** gkadam-afk is now known as gkadam | 08:36 | |
| openstackgerrit | zhufl proposed openstack/swift master: Add missing ws separator between words https://review.openstack.org/652583 | 09:08 |
| *** e0ne has quit IRC | 09:09 | |
| *** e0ne has joined #openstack-swift | 09:18 | |
| *** TrevizeGP_ has quit IRC | 09:25 | |
| *** e0ne has quit IRC | 09:55 | |
| *** e0ne has joined #openstack-swift | 10:06 | |
| *** baojg has joined #openstack-swift | 10:16 | |
| *** e0ne has quit IRC | 10:26 | |
| *** e0ne has joined #openstack-swift | 10:35 | |
| *** joeljwright has joined #openstack-swift | 10:40 | |
| *** ChanServ sets mode: +v joeljwright | 10:40 | |
| *** e0ne has quit IRC | 11:10 | |
| *** thurloat__ is now known as thurloat | 11:20 | |
| *** kazsh has quit IRC | 11:46 | |
| *** mvkr has joined #openstack-swift | 11:48 | |
| *** kazsh has joined #openstack-swift | 11:49 | |
| *** e0ne has joined #openstack-swift | 12:16 | |
| *** e0ne has quit IRC | 12:40 | |
| *** e0ne has joined #openstack-swift | 12:41 | |
| *** e0ne has quit IRC | 12:58 | |
| *** e0ne has joined #openstack-swift | 13:17 | |
| *** e0ne has quit IRC | 13:28 | |
| *** e0ne has joined #openstack-swift | 13:34 | |
| *** e0ne has quit IRC | 14:06 | |
| *** e0ne has joined #openstack-swift | 14:21 | |
| *** zaitcev has joined #openstack-swift | 15:09 | |
| *** ChanServ sets mode: +v zaitcev | 15:09 | |
| *** e0ne has quit IRC | 15:14 | |
| *** e0ne has joined #openstack-swift | 15:23 | |
| *** e0ne has quit IRC | 15:34 | |
| *** gyee has joined #openstack-swift | 15:38 | |
| *** zaitcev_ has joined #openstack-swift | 15:41 | |
| *** ChanServ sets mode: +v zaitcev_ | 15:41 | |
| *** zaitcev has quit IRC | 15:45 | |
| *** zaitcev_ has quit IRC | 16:08 | |
| *** zaitcev_ has joined #openstack-swift | 16:21 | |
| *** ChanServ sets mode: +v zaitcev_ | 16:21 | |
| tdasilva | just saw the email about dropping py35 testing, should we make py36 voting? | 16:32 |
| *** psachin has quit IRC | 16:42 | |
| *** e0ne has joined #openstack-swift | 16:51 | |
| timburke | tdasilva, what was clayg saying about me having a patch for everything? p 649485 | 17:04 |
| patchbot | https://review.openstack.org/#/c/649485/ - swift - Move from py35 to py37 for gating py3 jobs - 1 patch set | 17:04 |
| timburke | kota_, no worries! enjoy your early vacation :-) | 17:05 |
| tdasilva | timburke: lol, you certainly do! | 17:07 |
| *** gkadam has quit IRC | 17:15 | |
| *** zaitcev_ has quit IRC | 17:40 | |
| timburke | :-( so i spotted a traceback in a live cluster... http://paste.openstack.org/show/749317/ | 17:40 |
| timburke | all of the servers agree, that DB just doesn't exist. not on *any* device | 17:40 |
| timburke | so it's probably an account that got deleted... but there were some container dbs that were sitting on some offline disk so they never got reaped properly... | 17:41 |
| timburke | and when they finally *did* come back, they just waste time in the updater | 17:42 |
| timburke | ...which, since it doesn't have debug logging turned on, *doesn't even mention* the error, so i don't know where the containers are that are failing to report :-/ | 17:43 |
| *** zaitcev_ has joined #openstack-swift | 17:53 | |
| *** ChanServ sets mode: +v zaitcev_ | 17:53 | |
| *** e0ne has quit IRC | 18:00 | |
| *** rdejoux has quit IRC | 18:10 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: Be able to parse non-RFC-compliant request lines https://review.openstack.org/645284 | 18:42 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: Stop using stdlib's putrequest(); it only does ASCII https://review.openstack.org/642893 | 18:42 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Get functional/tests.py running under py3 https://review.openstack.org/642520 | 18:42 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: Cover account/container func tests https://review.openstack.org/645388 | 18:42 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add py37 func test job https://review.openstack.org/645856 | 18:42 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: Bring functional/test_object.py under test; add func-ec-py37 job https://review.openstack.org/645895 | 18:42 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: port staticweb and domain_remap func tests https://review.openstack.org/647923 | 18:42 |
| *** zaitcev_ has quit IRC | 18:42 | |
| clayg | timburke: that traceback sucks! | 18:53 |
| clayg | timburke: you're saying that when debugging logging is turned on the updater would say something useful tho? | 18:53 |
| clayg | timburke: seems totally reasonable to handle the db does not exist case on the account server side with a 404 or something | 18:54 |
| clayg | do we have a bug filed or anything? | 18:54 |
| timburke | clayg, on debug logging, yeah: https://github.com/openstack/swift/blob/2.21.0/swift/container/updater.py#L288-L290 | 18:55 |
| timburke | there's also https://github.com/openstack/swift/blob/2.21.0/swift/container/updater.py#L341-L343 but i don't think it would've tripped in this case -- the server responds just fine, but with a 500 | 18:56 |
| *** zaitcev_ has joined #openstack-swift | 18:56 | |
| *** ChanServ sets mode: +v zaitcev_ | 18:56 | |
| timburke | on having a bug to point to, i haven't even gotten to searching for whether it's a known issue yet | 18:56 |
| timburke | i'm still debating about how to address the problem in *this particular case*, though. not sure how best to search for whether the account really was marked deleted and reaped, and i'm debating about reviving it just to let updates land and see how many objects are supposedly in there... | 18:58 |
| clayg | could possibly be some .data files around too... re-creating the account db doesn't sound like such a bad idea... | 19:01 |
| *** e0ne has joined #openstack-swift | 19:02 | |
| timburke | hey, sounds kinda like https://bugs.launchpad.net/swift/+bug/1647840 ... | 19:05 |
| openstack | Launchpad bug 1647840 in OpenStack Object Storage (swift) "Object Updater should log when all primary missing" [Medium,Confirmed] | 19:05 |
| *** e0ne has quit IRC | 19:06 | |
| timburke | maybe also related? https://bugs.launchpad.net/swift/+bug/1293945 | 19:09 |
| openstack | Launchpad bug 1293945 in OpenStack Object Storage (swift) "Error code on putting container when update process to account fail" [Undecided,In progress] - Assigned to Takashi Kajinami (kajinamit) | 19:09 |
| timburke | then there's also https://bugs.launchpad.net/swift/+bug/1514528 and https://bugs.launchpad.net/swift/+bug/1300850 for another possible way this cluster may have gotten into this situation... | 19:11 |
| openstack | Launchpad bug 1514528 in OpenStack Object Storage (swift) "Some choices of delay_reaping & reclaim_age values can lead to container databases which are never deleted." [Medium,Confirmed] | 19:11 |
| openstack | Launchpad bug 1300850 in OpenStack Object Storage (swift) "Different reclaim ages for accounts and containers can result in un-reclaimable containers" [Undecided,In progress] - Assigned to Donagh McCabe (donagh-mccabe) | 19:11 |
| clayg | sweet! long and storied history. | 19:20 |
| timburke | ha! i can't read my logs :P of *course* we log the container name -- it's in the account-server's put path! looks like the container that's trying to send the update *is* deleted, has been for like a month, and the container-server reclaim age is like 6 weeks | 19:47 |
| timburke | wat. and the account-server reclaim age is *an hour*!? o.O | 19:48 |
| *** baojg has quit IRC | 20:14 | |
| *** pcaruana has quit IRC | 20:38 | |
| clayg | https://bugs.launchpad.net/swift/+bug/1300850 | 21:09 |
| openstack | Launchpad bug 1300850 in OpenStack Object Storage (swift) "Different reclaim ages for accounts and containers can result in un-reclaimable containers" [Undecided,In progress] - Assigned to Donagh McCabe (donagh-mccabe) | 21:09 |
| *** mvkr has quit IRC | 21:38 | |
| clayg | timburke: did you already figure out in your mind how you wanted to handle symloop_max? | 21:42 |
| clayg | timburke: unfortunately the best I seem to be coming up with is like "if this symlink was created by versioned writes (there's some sysmeta on it) decrement _loop_counter" | 21:44 |
| clayg | the only reason that feels wonky is symlink re-growing knowledge of versioned writes implementation | 21:45 |
| *** mvkr has joined #openstack-swift | 21:51 | |
| timburke | clayg, VW could *always* include the ?symlink=get query param when hitting a versioned container... and drop it when following the link | 22:16 |
| timburke | maybe there's also something to be done involving like a X-Backend-Symloop-Max header? in case the link we find wasn't put there by versioned_writes... | 22:17 |
| timburke | it kinda bugs me how coupled these things are... makes me think of the copy_hook crap we finally managed to get rid of | 22:19 |
| timburke | i might also be content to say, "this is just how symlinks work; if you're using symlinks for versions, consider increasing symloop_max by one" | 22:21 |
| timburke | test_multiphase_put_drains_extra_commit_junk_disconnect is really getting on my nerves... | 22:22 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Wait longer for log lines in unit test https://review.openstack.org/652791 | 22:44 |
| *** tkajinam has joined #openstack-swift | 22:53 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: func tests: Check for xattr support before env setUp https://review.openstack.org/652794 | 22:54 |
| mattoliverau | morning | 23:11 |
| *** rcernin has joined #openstack-swift | 23:24 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!