| *** Suyi_ has quit IRC | 00:06 | |
| *** tqtran has quit IRC | 00:22 | |
| *** asettle has joined #openstack-swift | 00:23 | |
| *** asettle has quit IRC | 00:28 | |
| *** thumpba has quit IRC | 00:29 | |
| timburke | oh yeah, i nearly forgot about https://review.openstack.org/#/c/331369/ | 00:41 |
|---|---|---|
| patchbot | patch 331369 - swift - Always set swift processes to use UTC | 00:41 |
| timburke | what a headache that was to hunt down! | 00:41 |
| clayg | do you have a bug report? | 00:42 |
| timburke | hadn't bothered. apparently no one else uses email.utils.mktime_tz on pre-2.7.4 python? (what suckers!) | 00:43 |
| clayg | like it's a great story reading the commit message - but I have no idea how to duplicate the bad and validate this patch fixes it? | 00:44 |
| clayg | was the problem inthe response headers? the logging? | 00:44 |
| clayg | is it purly a belts and braces correctness thing (those are good too!) | 00:44 |
| clayg | awesome - and the tests just validate that the code calls the method that it was changed to call ;) | 00:45 |
| clayg | *you* probably don't even remember :P | 00:45 |
| clayg | maybe you do - you've got a better memeory than the rest of us :P | 00:45 |
| timburke | clayg: the problem was in parsing a client-provided date header. you'll want a precise saio and some custom middleware... | 00:45 |
| clayg | rofl - no i definately do not want that :D | 00:46 |
| clayg | but it would be nice to understand the pre-req to review the patch - maybe comments? nope... how do you remember why this patch exists?! did you write down - am i missing it? Do you really have that good of a memory!? | 00:47 |
| timburke | even if the test actually described the problem, you'd need some old python to make it fail before the patch. probably better to view it as a belt & braces-style patch | 00:47 |
| timburke | clayg: there's the other patch! https://review.openstack.org/#/c/330888/ | 00:51 |
| patchbot | patch 330888 - swift3 - Stop using email.utils.mktime_tz | 00:51 |
| clayg | timburke: so it *is* just a belts and braces thing? or it would be if the swift3 thing got merged? | 00:52 |
| clayg | timburke: oh i guess anyone calling email.utils.mktime_tz from middleware would get the weird/mixed results on precise python? | 00:53 |
| timburke | the second one. also, it'll save anyone who writes some custom middleware that... yeah | 00:53 |
| clayg | timburke: so it's probably not outside of the relam of possibility to write a unittest that would fail before your change on oldish-precise-python then yeah? | 00:54 |
| *** m_kazuhiro has joined #openstack-swift | 00:55 | |
| clayg | even if only one other person verifies it fails one time - it'd still probably be helpful to read the asserts and understand what the failure mode looks like then | 00:55 |
| *** thumpba has joined #openstack-swift | 00:55 | |
| clayg | "w/o this change this stdlib function call breaks on old python because of stuff you do in process init" seems like more than belts and braces - it's a sound fix for a terrible weird edge case that I'm sure was annoying to track down! | 00:56 |
| *** gyee has quit IRC | 00:56 | |
| timburke | plus i'm sure there are some lurking pitfalls (in god-knows where) to do with that crazy tz-schizophrenia we've got going on | 00:56 |
| clayg | :'( | 00:57 |
| *** asettle has joined #openstack-swift | 00:57 | |
| timburke | i'll see what i can do to make the tests a little more useful | 00:57 |
| pdardeau | clayg: timburke: when you guys say "old" python, how far back is that? 2.6.x? | 00:58 |
| timburke | pdardeau: 2.7.3, like precise ships | 00:59 |
| clayg | is the '+0000' format that we stick into TZ from the gmtime call just like stright up wrong? the UTC+0 format seems to make the time.tzname attr more better? | 01:00 |
| clayg | timburke: like i'm not sure I buy the os.environ change at all? sigh | 01:01 |
| *** asettle has quit IRC | 01:01 | |
| clayg | portante: ^ *you* should review this TZ nonsense when timburke gets done figuring it out :P | 01:01 |
| timburke | clayg: that's how you change TZ, though -- https://docs.python.org/2/library/time.html#time.tzset | 01:02 |
| timburke | "The environment variable TZ specifies how this is done." | 01:02 |
| clayg | timburke: you should link to those docs in the updated change/bug report | 01:04 |
| timburke | i did! it was footnote 3! | 01:04 |
| *** neha__ has quit IRC | 01:05 | |
| clayg | Further, while we would change os.environ['TZ'], we would *not* call | 01:05 |
| clayg | time.tzset [3], which seems like a Bad Thing | 01:05 |
| clayg | instead say "like it says in the docs [3]" | 01:05 |
| clayg | IMHO - sorry I missed it | 01:05 |
| clayg | also the *3rd* footnote reminds me of http://blog.equalrightsinstitute.com/wp-content/uploads/2015/06/IM.png | 01:07 |
| kota_ | good morning | 01:08 |
| timburke | clayg: ok, you wanna see the bad? `python -c 'import os, time; print "Before:\n %s\n %s\n %s\n" % (time.gmtime(0).tm_year, time.localtime(0).tm_year, time.timezone); os.environ["TZ"] = "+0000"; print "After:\n %s\n %s\n %s\n" % (time.gmtime(0).tm_year, time.localtime(0).tm_year, time.timezone)'` | 01:14 |
| timburke | i think i can make a unittest out of that... | 01:14 |
| clayg | timburke: yeah I mean I was really just trying to understand the problem - sometimes the unittests make it obvious what was broke - but in this case the unitests were not the answer - talkig with you about it was | 01:18 |
| clayg | ... sort of - i mean - I guess the weird internal state of the time module getting out of whack with the env it somewhat easy to understand as wrong | 01:18 |
| clayg | in that light I think doing what the docs say is a pretty clear win? | 01:19 |
| clayg | I think that as long as your system was already configured for that timezone the change should have basically no effect? | 01:19 |
| clayg | presumably the time module calls tzset once when it's started - and it's only when the TZ environ is modified during runtime that this matters? | 01:20 |
| clayg | timburke: maybe we should just do it earlier and import time later? | 01:20 |
| clayg | ... well i'm fine with tzset too if that's the only problem - how does this only effect old python tho? | 01:20 |
| timburke | nope, this (may) affect current python code, too. swift3's use of mktime_tz was only bad on old python, though | 01:22 |
| clayg | interesting.. | 01:22 |
| timburke | so the unittest i eventually write (demonstrating the time module getting out of whack) will pop on modern platforms | 01:22 |
| clayg | timburke: cool! | 01:23 |
| clayg | take that swift process initilization code! no more screwing up stdlib module internal state for you!~ | 01:23 |
| *** vinsh has joined #openstack-swift | 01:24 | |
| clayg | timburke: I wonder if when portante added that the time module was being initialized at a diffent time in the process lifetype and somehow maybe this was just a regression or if this never really worked right? | 01:24 |
| *** _JZ__ has joined #openstack-swift | 01:27 | |
| timburke | well, it certainly did what we wanted it to do (stop stat'ing /etc/blahblahblah all the time), and since we generally recommend running in UTC anyway (makes merging log files *so* much easier) i'm guessing no one really noticed | 01:29 |
| *** _JZ___ has joined #openstack-swift | 01:30 | |
| *** _JZ_ has quit IRC | 01:30 | |
| *** _JZ__ has quit IRC | 01:33 | |
| *** _JZ___ has quit IRC | 01:36 | |
| *** tsg has joined #openstack-swift | 01:45 | |
| *** _JZ_ has joined #openstack-swift | 01:49 | |
| *** mingyu has joined #openstack-swift | 01:54 | |
| *** ANTI-torture has joined #openstack-swift | 01:56 | |
| *** ANTI-torture has quit IRC | 02:02 | |
| *** manous has quit IRC | 02:05 | |
| *** _JZ_ has quit IRC | 02:07 | |
| *** _JZ_ has joined #openstack-swift | 02:07 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Always set swift processes to use UTC https://review.openstack.org/331369 | 02:08 |
| *** thumpba has quit IRC | 02:08 | |
| *** mingyu_ has joined #openstack-swift | 02:09 | |
| *** mingyu has quit IRC | 02:09 | |
| *** tsg has quit IRC | 02:10 | |
| *** Jeffrey4l has quit IRC | 02:16 | |
| *** manous has joined #openstack-swift | 02:17 | |
| *** manous has quit IRC | 02:22 | |
| *** manous has joined #openstack-swift | 02:25 | |
| *** vinsh has quit IRC | 02:27 | |
| *** catintheroof has joined #openstack-swift | 02:27 | |
| *** bkeller` has quit IRC | 02:31 | |
| *** vinsh has joined #openstack-swift | 02:33 | |
| *** bkeller` has joined #openstack-swift | 02:34 | |
| *** thumpba has joined #openstack-swift | 02:35 | |
| *** sgundur_ has joined #openstack-swift | 02:35 | |
| *** vinsh has quit IRC | 02:36 | |
| *** vinsh has joined #openstack-swift | 02:36 | |
| * portante wonders about that portante | 02:40 | |
| *** vinsh has quit IRC | 02:40 | |
| *** sgundur_ has quit IRC | 02:44 | |
| *** sgundur_ has joined #openstack-swift | 02:49 | |
| *** klamath has quit IRC | 02:53 | |
| *** asettle has joined #openstack-swift | 02:59 | |
| *** sgundur_ has quit IRC | 03:00 | |
| *** mingyu_ has quit IRC | 03:02 | |
| *** asettle has quit IRC | 03:03 | |
| *** nadeem has joined #openstack-swift | 03:13 | |
| *** mingyu has joined #openstack-swift | 03:23 | |
| *** tsg has joined #openstack-swift | 03:30 | |
| *** manous has quit IRC | 03:32 | |
| *** vinsh has joined #openstack-swift | 03:36 | |
| *** vinsh has quit IRC | 03:37 | |
| *** vinsh has joined #openstack-swift | 03:37 | |
| *** _JZ_ has quit IRC | 03:38 | |
| *** thumpba has quit IRC | 03:40 | |
| *** thumpba has joined #openstack-swift | 03:40 | |
| *** vinsh has quit IRC | 03:41 | |
| *** mingyu_ has joined #openstack-swift | 03:42 | |
| *** mingyu has quit IRC | 03:43 | |
| *** manous has joined #openstack-swift | 03:44 | |
| *** mingyu_ has quit IRC | 03:45 | |
| *** manous has quit IRC | 03:49 | |
| *** manous has joined #openstack-swift | 03:50 | |
| *** tsg_ has joined #openstack-swift | 03:53 | |
| *** tsg has quit IRC | 03:53 | |
| *** mingyu has joined #openstack-swift | 03:54 | |
| *** manous has quit IRC | 03:55 | |
| *** thumpba has quit IRC | 03:58 | |
| *** thumpba has joined #openstack-swift | 03:58 | |
| *** thumpba has quit IRC | 03:58 | |
| *** thumpba has joined #openstack-swift | 03:59 | |
| *** thumpba has quit IRC | 03:59 | |
| *** manous has joined #openstack-swift | 03:59 | |
| *** thumpba has joined #openstack-swift | 03:59 | |
| *** thumpba has quit IRC | 04:00 | |
| *** mingyu has quit IRC | 04:06 | |
| *** mingyu has joined #openstack-swift | 04:07 | |
| *** ukaynar has joined #openstack-swift | 04:23 | |
| *** ppai has joined #openstack-swift | 04:27 | |
| *** dmorita has quit IRC | 04:30 | |
| *** psachin has joined #openstack-swift | 04:36 | |
| *** rcernin has quit IRC | 04:37 | |
| *** caiobrentano_ has joined #openstack-swift | 04:39 | |
| *** caiobrentano__ has quit IRC | 04:41 | |
| *** klamath has joined #openstack-swift | 04:41 | |
| mahatic | clayg: oh going by your last comment, I thought we're gonna call it a day by invalidating_hash for this lp bug #1301728. I thought of writing another patch on splitting out stuff | 04:42 |
| openstack | Launchpad bug 1301728 in OpenStack Object Storage (swift) "tombstone (.ts) object will never be deleted if the hash_suffix exists in the hashes.pkl " [High,In progress] https://launchpad.net/bugs/1301728 - Assigned to Mahati Chamarthy (mahati-chamarthy) | 04:43 |
| *** nadeem has quit IRC | 04:45 | |
| mahatic | clayg: I updated the patch 346865 to only invalidate hash. We're not messing up by making listdir calls with that I believe. Do you still think splitting out get_ondisk_files and cleanup_disk_files in this patch is better? | 04:45 |
| patchbot | https://review.openstack.org/#/c/346865/ - swift - Delete old tombstones | 04:45 |
| *** klamath has quit IRC | 04:46 | |
| mahatic | notmyname: awesome t-shirts! do you have a designer? ;) | 04:48 |
| *** tsg_ has quit IRC | 04:57 | |
| *** chsc has joined #openstack-swift | 04:59 | |
| *** asettle has joined #openstack-swift | 05:01 | |
| *** _JZ_ has joined #openstack-swift | 05:05 | |
| *** asettle has quit IRC | 05:05 | |
| *** _JZ_ has quit IRC | 05:07 | |
| *** manous has quit IRC | 05:07 | |
| *** chsc has quit IRC | 05:11 | |
| *** tqtran has joined #openstack-swift | 05:13 | |
| *** baojg_ has quit IRC | 05:13 | |
| *** baojg has joined #openstack-swift | 05:14 | |
| *** manous has joined #openstack-swift | 05:19 | |
| *** dmorita has joined #openstack-swift | 05:30 | |
| *** ChubYann has quit IRC | 05:31 | |
| *** dmorita has quit IRC | 05:35 | |
| *** rcernin has joined #openstack-swift | 05:43 | |
| *** silor has joined #openstack-swift | 05:44 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: *WIP* Try to fix doubtful code https://review.openstack.org/371274 | 05:48 |
| *** silor1 has joined #openstack-swift | 05:57 | |
| *** silor has quit IRC | 05:57 | |
| *** silor1 is now known as silor | 05:57 | |
| *** thumpba has joined #openstack-swift | 06:01 | |
| *** ukaynar has quit IRC | 06:02 | |
| *** thumpba has quit IRC | 06:06 | |
| *** klrmn has quit IRC | 06:11 | |
| *** rcernin has quit IRC | 06:14 | |
| *** rcernin has joined #openstack-swift | 06:19 | |
| *** pcaruana has joined #openstack-swift | 06:23 | |
| *** ukaynar has joined #openstack-swift | 06:28 | |
| *** klamath has joined #openstack-swift | 06:30 | |
| *** manous has quit IRC | 06:33 | |
| *** klamath has quit IRC | 06:35 | |
| *** silor has quit IRC | 06:37 | |
| *** vern has quit IRC | 06:40 | |
| *** vern has joined #openstack-swift | 06:43 | |
| *** hseipp has joined #openstack-swift | 06:44 | |
| *** manous has joined #openstack-swift | 06:45 | |
| *** manous has quit IRC | 06:50 | |
| *** manous has joined #openstack-swift | 06:50 | |
| *** manous has quit IRC | 06:55 | |
| *** tgtanya has joined #openstack-swift | 06:57 | |
| *** manous has joined #openstack-swift | 06:58 | |
| *** david-lyle_ has joined #openstack-swift | 06:59 | |
| *** david-lyle has quit IRC | 07:00 | |
| *** CrackerJackMack has quit IRC | 07:00 | |
| *** sams-gleb has joined #openstack-swift | 07:10 | |
| *** tmoreira has quit IRC | 07:18 | |
| *** manous has quit IRC | 07:18 | |
| *** cschwede has quit IRC | 07:19 | |
| *** tmoreira has joined #openstack-swift | 07:24 | |
| *** cschwede has joined #openstack-swift | 07:25 | |
| *** geaaru has joined #openstack-swift | 07:26 | |
| *** rledisez has joined #openstack-swift | 07:26 | |
| *** david-lyle has joined #openstack-swift | 07:28 | |
| *** oshritf__ has joined #openstack-swift | 07:28 | |
| *** oshritf_ has joined #openstack-swift | 07:28 | |
| *** ukaynar has quit IRC | 07:28 | |
| *** david-lyle_ has quit IRC | 07:29 | |
| *** manous has joined #openstack-swift | 07:31 | |
| *** CrackerJackMack has joined #openstack-swift | 07:31 | |
| *** tqtran has quit IRC | 07:33 | |
| *** Jeffrey4l has joined #openstack-swift | 07:37 | |
| *** kong has quit IRC | 07:41 | |
| *** joeljwright has joined #openstack-swift | 07:42 | |
| *** ChanServ sets mode: +v joeljwright | 07:42 | |
| *** oshritf_ has quit IRC | 07:56 | |
| *** oshritf__ has quit IRC | 07:56 | |
| *** oshritf__ has joined #openstack-swift | 08:00 | |
| *** oshritf_ has joined #openstack-swift | 08:00 | |
| *** thumpba has joined #openstack-swift | 08:01 | |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** openstackgerrit has joined #openstack-swift | 08:04 | |
| *** jordanP has joined #openstack-swift | 08:11 | |
| *** klamath has joined #openstack-swift | 08:19 | |
| *** asettle has joined #openstack-swift | 08:20 | |
| *** klamath has quit IRC | 08:24 | |
| *** oshritf__ has quit IRC | 08:29 | |
| *** oshritf_ has quit IRC | 08:29 | |
| *** thumpba has quit IRC | 08:30 | |
| *** tgtanya has quit IRC | 08:42 | |
| *** tgtanya has joined #openstack-swift | 08:42 | |
| *** manous has quit IRC | 09:17 | |
| *** daemontool has joined #openstack-swift | 09:49 | |
| *** donagh has joined #openstack-swift | 09:51 | |
| *** tgtanya has quit IRC | 09:57 | |
| *** daemontool has quit IRC | 09:59 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: *WIP* Try to fix doubtful code https://review.openstack.org/371274 | 10:01 |
| *** klamath has joined #openstack-swift | 10:08 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Enable object server to return non-durable data https://review.openstack.org/215276 | 10:12 |
| *** klamath has quit IRC | 10:12 | |
| kota_ | acoles: nice, do you find the reason why the unit test fails? | 10:12 |
| kota_ | i'm just looking for with looking at the code. | 10:13 |
| *** acoles_ is now known as acoles | 10:13 | |
| acoles | kota_: not yet, i just added an exception handler to get soem more detail when the test fails | 10:14 |
| kota_ | ok, good call. maybe similar thing with mine at patch 371274 | 10:14 |
| patchbot | https://review.openstack.org/#/c/371274/ - swift - *WIP* Try to fix doubtful code | 10:14 |
| kota_ | I also am digging... | 10:14 |
| *** tsg has joined #openstack-swift | 10:16 | |
| *** tsg_ has joined #openstack-swift | 10:20 | |
| *** tsg has quit IRC | 10:20 | |
| acoles | kota_: I don't like the 0 in this line https://github.com/openstack/swift/blob/79a8388fbf1e0a43abe196ad2f35e94414963c19/test/unit/proxy/controllers/test_obj.py#L2230-L2230, I think that could result in no fragments being generated, still looking into that. No idea why that would just start burning us though. | 10:22 |
| *** ppai has quit IRC | 10:22 | |
| kota_ | looking | 10:23 |
| kota_ | hmm, sounds reasonable, it would make an empty string. | 10:24 |
| kota_ | try it with my local | 10:24 |
| kota_ | acoles:^^ | 10:24 |
| acoles | yep. I will change it to 1 once the current patchset runs in jenkins, so we can see the new logs before and after the change | 10:25 |
| *** daemontool has joined #openstack-swift | 10:26 | |
| *** thumpba has joined #openstack-swift | 10:27 | |
| acoles | kota_: here's latest test failure log - the object doe not seem to have any fragments ?!? http://logs.openstack.org/76/215276/42/check/gate-swift-python27-ubuntu-xenial/eda0258/console.html#_2016-09-16_10_19_16_481902 | 10:27 |
| acoles | kota_: I'll change that 0 and see if it helps | 10:28 |
| kota_ | acoles: your thought is correct, I can see same error if the string is empty. wait a while, I'm preparing the snipets | 10:29 |
| acoles | kota_: okay. I need a coffee so bbiab | 10:29 |
| *** dmorita has joined #openstack-swift | 10:31 | |
| *** thumpba has quit IRC | 10:32 | |
| kota_ | acoles: it was reproduced, https://gist.github.com/bloodeagle40234/b9c3677fddd7deecb7efcc96a42b5025 | 10:32 |
| kota_ | "[:1]" does work, so that "random.randint(1, 1000)" for that seems to work | 10:34 |
| kota_ | or "random.randin(1, segment_size)" is better? | 10:35 |
| kota_ | randint | 10:35 |
| *** dmorita has quit IRC | 10:36 | |
| acoles | kota_: with your local diff, do other tests fail as well? | 10:36 |
| kota_ | hmm... interesting if it is the reason why gerrit fails, we've not been doing actually *random* content body | 10:36 |
| acoles | kota_: I mean this diff https://gist.github.com/bloodeagle40234/b9c3677fddd7deecb7efcc96a42b5025 | 10:37 |
| kota_ | wait a sec, i ran just a test. | 10:37 |
| acoles | kota_: yeah, are we just getting unlucky that that one test repeatedly gets a 0 from the randint? | 10:37 |
| kota_ | nosetests reports errors=18 yes, other tests also fail if we use the stub. | 10:38 |
| *** kong has joined #openstack-swift | 10:38 | |
| kota_ | maybe, something exits from fixed random seed? | 10:38 |
| acoles | yes, I just did it here too | 10:38 |
| kota_ | interesting | 10:38 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Enable object server to return non-durable data https://review.openstack.org/215276 | 10:40 |
| acoles | kota_: let's see what jenkins thinks ^^ | 10:40 |
| kota_ | woo, i made a wrong patch when running other tests, but anyway, other tests got failures or errors (5 errors and 12 failures actually) | 10:41 |
| kota_ | if i tested with my diff above | 10:41 |
| acoles | kota_: same for me, 5 + 12 | 10:45 |
| kota_ | acoles: ok, I'm starting to look at again with the diff from patch set 40. | 10:46 |
| kota_ | it looks small enaough +146, -52 :D | 10:46 |
| *** silor has joined #openstack-swift | 10:47 | |
| kota_ | acoles: ah, you addressed my comment for etag!? thanks! when reading subscription mail from gerrit including your response, I thought it could be in follow up | 10:50 |
| acoles | kota_: re. your comments about the scenarios passed to _test_get_ondisk_files - I agree that the scenarions ('.durable') are a little confusing, but I'd rather not make the test more complex. as a follow up I would rather clean up the scenario declarations. | 10:51 |
| acoles | kota_: re. etags - I added a check (well, I spent a day thinking about how to manage buckets for each (timestamp, etag) tuple, wrote code, deleted code, gave up and simply added a check :) | 10:52 |
| acoles | kota_: and a test. so now, even in worst case, pyeclib will not be passed frags from different etag objects | 10:52 |
| kota_ | acoles: yey, simple solution. | 10:53 |
| acoles | kota_: maybe random seed gets set to a fixed value during test run - https://github.com/openstack/swift/blob/488f88e30abe2b9d67ce43697e3abc63c75699b7/test/unit/cli/test_ring_builder_analyzer.py#L131-L131 | 10:53 |
| acoles | kota_: which calls https://github.com/openstack/swift/blob/89388bf232b54ec0adf5cb815efff23cd479e2c1/swift/common/ring/builder.py#L430-L431 | 10:53 |
| kota_ | acoles: yeah, and I had looked other place which setting the random seed for concrete testing result (can not point out for now though) | 10:54 |
| *** kei_yama has quit IRC | 10:57 | |
| kota_ | acolse: btw, it is a bit far from EC but in replication case with same timestamp but different etag | 10:58 |
| acoles | thats the only place I can find random.seed() in swift | 10:59 |
| acoles | kota_: yep? | 10:59 |
| kota_ | acoles: replication case seems safe to get the object becase one object-server should have complete object inside (some other object-server has different object though. | 10:59 |
| kota_ | acoles: and then | 10:59 |
| kota_ | acoles: if proxy got a failure (e.g. timeout) and doing fast_forward (redirecting to another object-server) | 11:00 |
| *** silor1 has joined #openstack-swift | 11:00 | |
| kota_ | acoles: I'm wondering we could get mixed content... maybe? | 11:00 |
| *** silor has quit IRC | 11:00 | |
| *** silor1 is now known as silor | 11:00 | |
| kota_ | am i getting crazy thought? | 11:00 |
| rsFF | ahale_: | 11:01 |
| kota_ | so e.g. object-server A: ts1 etag:a object-sever B:ts1, etag:b. proxy got a failure when draining etag a object and redirecting object-server B with the range for the rest. | 11:02 |
| acoles | kota_: looking | 11:02 |
| kota_ | it looks jut forwarding the range... | 11:03 |
| kota_ | exactly as you said, that situation (same timestamp but different etag) happens so rarely though. | 11:05 |
| acoles | kota_: looks like there is a check for etag equality when doing a fast_forward to a new node https://github.com/openstack/swift/blob/44a861787a60346aded8be2a54ef4e16accccbb6/swift/proxy/controllers/base.py#L1212-L1220 | 11:06 |
| acoles | phew! | 11:06 |
| kota_ | ah yeah! | 11:07 |
| kota_ | sorry, i missed it | 11:07 |
| acoles | kota_: in general, different etags at same timestamp breaks the fundamental presumption of swifts consistency model | 11:07 |
| acoles | kota_: with replica policy the object will survive and be readable but will never become consistent so no claim can be made over its durability (there may only ever be one replica of one content) | 11:08 |
| kota_ | yes | 11:09 |
| acoles | in other words, 'don't go there' :D | 11:09 |
| acoles | but I feel better that we now at least have a guard in the EC getter bucket | 11:10 |
| kota_ | completely agree, great. | 11:11 |
| kota_ | and thanks acoles! | 11:11 |
| acoles | kota_: success! http://logs.openstack.org/76/215276/43/check/gate-swift-python27-ubuntu-xenial/af85962/console.html#_2016-09-16_10_44_58_394299 | 11:11 |
| kota_ | acoles: yey!!!!! | 11:11 |
| acoles | kota_: I suspect that the one new test I added in patchset 41 caused the randint to fall on 0 for test_GET_with_only_handoffs, because the new test comes first alphabetically. | 11:14 |
| * acoles hates test coupling | 11:15 | |
| acoles | kota_: but still curious that it only worked out that way on jenkins | 11:16 |
| kota_ | exactly. | 11:17 |
| kota_ | ah, I learned, self.assertEqual({obj1['etag'], None}, collected_etags) <- this is equal!!! that is an expression of "set" | 11:17 |
| *** suresh__ has joined #openstack-swift | 11:25 | |
| kota_ | done. that looks the greatest one from ever pacht sets. | 11:26 |
| suresh__ | Hii all, What is the default chunksize of object when size is less than 5 GB | 11:26 |
| suresh__ | pls someone help | 11:26 |
| kota_ | suresh__: what do you mean the "chunk size"? the objects in your hardware devices? or network transfer chunk size? | 11:29 |
| suresh__ | kota: if we upload > 5GB file we will use "swift upload test_container -S 1073741824 large_file" | 11:31 |
| suresh__ | here 1073741824 is chunksize right? | 11:31 |
| kota_ | suresh__: got it, it's a size for each segments of static large object | 11:31 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: EC Fragment Duplication - Foundational Global EC Cluster Support https://review.openstack.org/219165 | 11:32 |
| suresh__ | like this if i want to upload 3GB file what is the default chunk size? | 11:32 |
| kota_ | suresh__: let me check | 11:33 |
| suresh__ | kota: pls let me know if you got the answer | 11:35 |
| kota_ | suresh__: it looks like no default value, if you don't specify the option, swift client doesn't break the original object into chunks, so you will see the raw object in the container. | 11:35 |
| *** sanchitmalhotra has quit IRC | 11:36 | |
| kota_ | just a 3GB file in the swift, if i read the code correctly. | 11:36 |
| kota_ | jowelright might know more detail, I'm not so good guy about swift client. | 11:37 |
| * kota_ is thinking it seems the NW for developers in my office is going to down, so it's time to go back home! | 11:39 | |
| *** catintheroof has quit IRC | 11:39 | |
| * kota_ is using mobile NW tentatively | 11:40 | |
| *** suresh__ has quit IRC | 11:40 | |
| *** klamath has joined #openstack-swift | 11:56 | |
| *** m_kazuhiro has quit IRC | 11:58 | |
| *** joeljwright1 has joined #openstack-swift | 12:00 | |
| *** asettle has quit IRC | 12:01 | |
| *** klamath has quit IRC | 12:01 | |
| *** asettle has joined #openstack-swift | 12:01 | |
| *** asettle has quit IRC | 12:06 | |
| *** pdardeau has quit IRC | 12:07 | |
| *** baojg has quit IRC | 12:10 | |
| *** baojg has joined #openstack-swift | 12:12 | |
| *** asettle has joined #openstack-swift | 12:12 | |
| *** joeljwright1 has quit IRC | 12:21 | |
| *** catintheroof has joined #openstack-swift | 12:27 | |
| *** porunov has joined #openstack-swift | 12:44 | |
| *** thumpba has joined #openstack-swift | 12:44 | |
| porunov | Hello everyone! Does somebody use JOSS (Java Client for Swift)? Could you please help me to figure out how to upload a manifest file? I want to use SLO but it missed in documentation how to use SLO. | 12:49 |
| *** StraubTW has joined #openstack-swift | 13:09 | |
| *** thumpba has quit IRC | 13:17 | |
| *** janonymous has quit IRC | 13:28 | |
| *** klamath has joined #openstack-swift | 13:45 | |
| *** daemontool has quit IRC | 13:46 | |
| *** klamath has quit IRC | 13:50 | |
| *** vinsh has joined #openstack-swift | 13:52 | |
| *** asettle has quit IRC | 14:06 | |
| *** tongli has joined #openstack-swift | 14:06 | |
| *** asettle has joined #openstack-swift | 14:07 | |
| *** asettle has quit IRC | 14:07 | |
| *** asettle has joined #openstack-swift | 14:07 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Unset random seed after rebalancing ring https://review.openstack.org/371564 | 14:08 |
| *** asettle has quit IRC | 14:12 | |
| *** _JZ_ has joined #openstack-swift | 14:12 | |
| *** asettle has joined #openstack-swift | 14:13 | |
| jrichli | porunov: SLO docs are here http://docs.openstack.org/developer/swift/overview_large_objects.html#module-swift.common.middleware.slo | 14:16 |
| jrichli | porunov: but I don't know about JOSS | 14:17 |
| *** asettle has quit IRC | 14:18 | |
| *** asettle has joined #openstack-swift | 14:18 | |
| *** porunov has quit IRC | 14:19 | |
| *** asettle has quit IRC | 14:23 | |
| *** asettle has joined #openstack-swift | 14:23 | |
| *** porunov has joined #openstack-swift | 14:29 | |
| *** sgundur_ has joined #openstack-swift | 14:31 | |
| *** sams-gleb has quit IRC | 14:31 | |
| porunov | jrichli: I mean using SLO with JOSS. Do you know how to use SLO with JOSS? | 14:32 |
| jrichli | porunov: no, sorry. I don't know about JOSS. | 14:33 |
| *** vinsh has quit IRC | 14:35 | |
| *** sgundur_ has quit IRC | 14:36 | |
| *** sgundur_ has joined #openstack-swift | 14:45 | |
| *** vinsh has joined #openstack-swift | 14:47 | |
| *** vinsh has quit IRC | 14:47 | |
| *** vinsh has joined #openstack-swift | 14:48 | |
| tdasilva | porunov: looking quickly through the joss api, it seems like their upload object api doesn't take metadata | 14:48 |
| *** janonymous has joined #openstack-swift | 14:48 | |
| tdasilva | porunov: http://joss.javaswift.org/future-features.html lists support for large objects as being a future thing, but seems to focus more on DLO | 14:48 |
| tdasilva | or actually for slo, what you need is not a header, but a query param | 14:50 |
| tdasilva | so yeah, sounds like you would need a special api for that | 14:50 |
| *** kong has quit IRC | 14:51 | |
| *** tsg_ has quit IRC | 14:55 | |
| *** klamath has joined #openstack-swift | 14:57 | |
| *** nikivi has joined #openstack-swift | 15:04 | |
| *** philipw_ has quit IRC | 15:05 | |
| *** philipw has joined #openstack-swift | 15:05 | |
| *** rcernin has quit IRC | 15:15 | |
| acoles | notmyname: clayg: if you want to discuss patch 215276 I can be available later (after noon your time), ping me on signal/text. looks like clayg might have some concerns about the overall strategy. | 15:17 |
| patchbot | https://review.openstack.org/#/c/215276/ - swift - Enable object server to return non-durable data | 15:17 |
| *** _JZ_ has quit IRC | 15:18 | |
| *** acoles is now known as acoles_ | 15:18 | |
| *** chsc has joined #openstack-swift | 15:20 | |
| *** klrmn has joined #openstack-swift | 15:23 | |
| *** klrmn has quit IRC | 15:29 | |
| *** tsg_ has joined #openstack-swift | 15:30 | |
| *** tsg__ has joined #openstack-swift | 15:32 | |
| porunov | tdasilva: Here is their supported features. http://joss.javaswift.org/feature-matrix.html . There is "Large Object Creation". I think that they support SLO | 15:32 |
| *** dmorita has joined #openstack-swift | 15:33 | |
| tdasilva | porunov: good point, still it could refer to DLO, it's somewhat vague, but you could be right | 15:35 |
| *** tsg_ has quit IRC | 15:35 | |
| tdasilva | porunov: javadocs does list a ObjectManifest and QueryParameter class | 15:36 |
| *** silor has quit IRC | 15:36 | |
| tdasilva | porunov: but it's not very clear to me how to use them....have you tried opening an issue on their github page and asking the question there? | 15:36 |
| tdasilva | i'm not sure if joss devs hangout here | 15:37 |
| *** dmorita has quit IRC | 15:37 | |
| mahatic | clayg: I'll check back your comments if any, on patch 346865 over the weekend, fyi. | 15:39 |
| patchbot | https://review.openstack.org/#/c/346865/ - swift - Delete old tombstones | 15:39 |
| *** ukaynar has joined #openstack-swift | 15:40 | |
| *** ukaynar has quit IRC | 15:41 | |
| notmyname | good morning | 15:41 |
| notmyname | near-final summit room layout has been published http://lists.openstack.org/pipermail/openstack-dev/2016-September/103851.html | 15:43 |
| *** psachin has quit IRC | 15:44 | |
| notmyname | mahatic: my sister is the tshirt designer | 15:44 |
| *** nadeem has joined #openstack-swift | 15:47 | |
| porunov | tdasilva: I've opened 2 issues (it was another questions). It seems that they abandoned this project. But nevertheless it is the only Java API which I found to be workable with tempauth. I also tried jclouds and openstack4j but they didn't work. | 15:49 |
| *** oshritf_ has joined #openstack-swift | 15:50 | |
| *** oshritf__ has joined #openstack-swift | 15:50 | |
| tdasilva | porunov: mm..too bad if they did abandon the project, although if you look at the history in github it seems they have had recent commits and even a release only 15 days ago | 15:51 |
| *** pdardeau has joined #openstack-swift | 15:57 | |
| *** pdardeau has quit IRC | 15:57 | |
| *** pdardeau has joined #openstack-swift | 15:57 | |
| porunov | tdasilva: Yes, but they hadn't any commits from Nov 30, 2014 till Jan 12, 2016. Also they don't answer questions on GitHub and don't have a documentation about their features. Of course I may be wrong but who knows. Nevertheless it is still nice API | 15:59 |
| *** oshritf_ has quit IRC | 16:00 | |
| *** oshritf__ has quit IRC | 16:00 | |
| *** oshritf_ has joined #openstack-swift | 16:01 | |
| *** oshritf__ has joined #openstack-swift | 16:01 | |
| *** tsg_ has joined #openstack-swift | 16:03 | |
| *** rledisez has quit IRC | 16:03 | |
| *** tsg__ has quit IRC | 16:06 | |
| clayg | acoles_: i'm around; reading backlog | 16:10 |
| *** oshritf_ has quit IRC | 16:12 | |
| *** oshritf__ has quit IRC | 16:12 | |
| *** ukaynar has joined #openstack-swift | 16:15 | |
| zaitcev | porunov: It's quite sad that there's no good library. Did you try to develop a fix and submit it (e.g. where the query parameter is needed)? Some maintainers are more receptive if they see a patch. | 16:19 |
| openstackgerrit | Travis McPeak proposed openstack/swift: Updating Bandit config file https://review.openstack.org/365065 | 16:20 |
| *** hseipp has quit IRC | 16:21 | |
| *** asettle has quit IRC | 16:22 | |
| *** asettle has joined #openstack-swift | 16:23 | |
| clayg | apparently I don't know how timzones work - when I was in Central time UK was only 6 hours away :'( | 16:23 |
| zaitcev | Sounds about right. | 16:23 |
| zaitcev | Mountain is 7, Pacific is 8 | 16:24 |
| clayg | right - need to remember *8* hours different now | 16:24 |
| zaitcev | Shift by 1 in winter. | 16:24 |
| zaitcev | Except in Arizona | 16:24 |
| notmyname | and Indiana, I think | 16:24 |
| zaitcev | When I fly, I immediately set my watch to GMT zone, because otherwise it's easy to set oneself up for a massive miscalculation. Also weather broadcast by ground stations has timestamps in GMT, so you know immediately how current an observation is. | 16:26 |
| *** nadeem has quit IRC | 16:26 | |
| *** nadeem has joined #openstack-swift | 16:27 | |
| *** asettle has quit IRC | 16:27 | |
| *** sgundur_ has quit IRC | 16:32 | |
| *** nikivi has quit IRC | 16:32 | |
| *** nikivi has joined #openstack-swift | 16:34 | |
| *** nowz has joined #openstack-swift | 16:37 | |
| nowz | HI everyone | 16:38 |
| nowz | I'm new User | 16:38 |
| *** Suyi_ has joined #openstack-swift | 16:41 | |
| MooingLemur | hellp | 16:50 |
| MooingLemur | hello, even | 16:51 |
| timburke | kota_: suresh_: yeah, with swiftclient if you don't specify -S <chunk_size>, no chunking occurs and you'll have a single 3GB object | 16:51 |
| *** sgundur_ has joined #openstack-swift | 16:53 | |
| *** tqtran has joined #openstack-swift | 16:53 | |
| *** klrmn has joined #openstack-swift | 16:53 | |
| MooingLemur | does swift-object-replicator use ephemeral IPv4 loopback sockets to talk to itself? | 16:54 |
| zaitcev | Do you mean to talk to swift-object-server on the same node or something? I don't think swift-object-replicator listens at all. | 16:55 |
| MooingLemur | I'm running into a situation where the object replicator seems to hang very easily (CentOS 7) and running strace shows most of the forked processes in futex(), one of them is in select(0,... and the remaining one is poll([fd=5... | 16:56 |
| patchbot | Error: Missing "]". You may want to quote your arguments with double quotes in order to prevent extra brackets from being evaluated as nested commands. | 16:56 |
| MooingLemur | that process doing the poll has these in the output of lsof | 16:57 |
| MooingLemur | swift-obj 16427 swift 5u IPv4 82879052 0t0 TCP 127.0.0.1:37528->127.0.0.1:56907 (ESTABLISHED) | 16:57 |
| MooingLemur | swift-obj 16427 swift 6u IPv4 82879053 0t0 TCP 127.0.0.1:56907->127.0.0.1:37528 (ESTABLISHED) | 16:57 |
| notmyname | hello nowz | 16:57 |
| *** janonymous has quit IRC | 16:58 | |
| MooingLemur | I haven't yet seen this problem on my Gentoo hosts, but it's happening all the time on the new CentOS hosts. I'm guessing this local socket is a side effect of some python library that swift is using, and the fact that it's using localhost sockets may not even be pointing to the actual problem. :P | 16:59 |
| MooingLemur | When it gets into this state, swift-object-replicator ignores SIGTERM | 16:59 |
| *** _JZ_ has joined #openstack-swift | 17:00 | |
| MooingLemur | and I don't see anything in syslog from swift-object-replicator to indicate any particular event | 17:00 |
| MooingLemur | it simply stops logging | 17:00 |
| zaitcev | A library is a reasonable guess, but I don't know what library triggers that. | 17:01 |
| zaitcev | And I see that it's the same PID. | 17:01 |
| *** nowz has quit IRC | 17:02 | |
| *** geaaru has quit IRC | 17:03 | |
| MooingLemur | https://bpaste.net/show/2c08a731a980 <-- strace -fp of the main pid of swift-object-replicator left for a few minutes | 17:05 |
| *** tsg_ has quit IRC | 17:06 | |
| *** _JZ_ has quit IRC | 17:07 | |
| *** dmorita has joined #openstack-swift | 17:08 | |
| MooingLemur | it does have a number of TCP connections open to various object server instances in the cluster (port 6000) and netstat shows 0 in the sendq and recvq for all established connections | 17:09 |
| MooingLemur | is there any introspection interface to the object replicator that I can use that might tell me more about the state that it's in? Maybe a SIGUSR1 or something? :) | 17:10 |
| MooingLemur | heh, I guess for now I'll have to write a cron to restart the object replicator if it hasn't seen any syslog lines from it in the last 10 minutes | 17:14 |
| zaitcev | Or add a watchdog into its main loop. It's an open source, you know. | 17:15 |
| MooingLemur | :) | 17:15 |
| zaitcev | I tend to see these things a challenges, so I guess I would resurrect my old patch that tracebacks every thread and at least figure out where the Python goes. | 17:17 |
| zaitcev | It was called "guru meditation" | 17:17 |
| *** tsg_ has joined #openstack-swift | 17:17 | |
| zaitcev | I can throw it up into a gist for you to look at, but if you reach for cron first, this may not be your style | 17:18 |
| zaitcev | http://www.zaitcev.us/things/swift/swift-2.7-nogurumed.diff | 17:20 |
| zaitcev | It's basically diff -R of what you need and you have to apply it to replicator in particular | 17:20 |
| *** jordanP has quit IRC | 17:20 | |
| MooingLemur | it's more about deadlines at the moment. I'd love to take time debug, but I've already got a framework for checking syslog (an enhanced version of swift-drive-audit that knows our syslog format, is aware of dm-cache, and handles lighting the fault light on the chassis) | 17:22 |
| MooingLemur | s/debug/to debug/ | 17:22 |
| MooingLemur | but I will grab that patch. Thanks for the info :) | 17:23 |
| *** rvasilets___ has joined #openstack-swift | 17:25 | |
| *** nikivi has quit IRC | 17:29 | |
| *** ChubYann has joined #openstack-swift | 17:37 | |
| clayg | whoa | 17:48 |
| *** nadeem has quit IRC | 17:49 | |
| clayg | the loop back socket stuff is eventlet threadpools | 17:49 |
| clayg | I think earlier versions used different stuff - but loopback sockets were more crossplatform? | 17:50 |
| clayg | I seem to recall not seeing them for a long time - then they showed up - i freaked out - torgomatic_ said it was cool tho and that was the resonablable explination he gave | 17:50 |
| zaitcev | So basically maybe steal an eventlet from Fedora 23 and stuck it into CentoOS 7, see if that magically cures eveything. | 17:51 |
| clayg | at somepoint I think I did verify that if your eventlet uses it's tpool you see those connections | 17:51 |
| clayg | MooingLemur: i've seen the object replicator lock up for ... different reasons | 17:52 |
| clayg | MooingLemur: I'm not sure the loopback/eventlet-threadpool has anything to do with it | 17:52 |
| *** joeljwright has quit IRC | 17:53 | |
| clayg | MooingLemur: there's a log line that you sometimes see emitted in this state -> https://bugs.launchpad.net/swift/+bug/1575277 | 17:53 |
| openstack | Launchpad bug 1575277 in OpenStack Object Storage (swift) "object-replicator goes into bad state when lockup timeout < rsync timeout" [High,Confirmed] | 17:53 |
| clayg | MooingLemur: can you grep for "Lockup detected.. killing live coros" | 17:53 |
| clayg | MooingLemur: by all means work around what ever issues you find in production in whatever way is convient for you to do - but please please please value us as a community enough to make sure any issues you work around have good bugs open for them! | 17:55 |
| clayg | MooingLemur: thank you! | 17:56 |
| *** porunov has quit IRC | 18:03 | |
| clayg | timburke: you've been reviewing patch 215276 - and I assume have some familiarity with the issues it's addressing? | 18:07 |
| patchbot | https://review.openstack.org/#/c/215276/ - swift - Enable object server to return non-durable data | 18:07 |
| clayg | lp bug #1469094 - the oldest and most commmon | 18:07 |
| openstack | Launchpad bug 1469094 in OpenStack Object Storage (swift) "Timeout writing .durable can cause error on GET (under failure)" [High,Confirmed] https://launchpad.net/bugs/1469094 - Assigned to paul luse (paul-e-luse) | 18:07 |
| clayg | lp bug #1624176 - newerish bug also quite common (broken in proxy already addressed in the reconstructor) | 18:08 |
| openstack | Launchpad bug 1624176 in OpenStack Object Storage (swift) "500 ServerError trying to read EC object with duplicate frags" [High,Confirmed] https://launchpad.net/bugs/1624176 | 18:08 |
| clayg | lp bug #1624088 - a *durability* issue | 18:08 |
| openstack | Launchpad bug 1624088 in OpenStack Object Storage (swift) "EC missing durable can prevent reconstruction" [Critical,Confirmed] https://launchpad.net/bugs/1624088 | 18:08 |
| clayg | lp bug #1484598 - a complicated web of not-well-understood spookyness that introduces a bunch of complexity | 18:08 |
| openstack | Launchpad bug 1484598 in OpenStack Object Storage (swift) "Proxy server ignores additional fragments on primary nodes" [High,In progress] https://launchpad.net/bugs/1484598 - Assigned to paul luse (paul-e-luse) | 18:08 |
| clayg | anyway - Al's patch got a +2 from Kota - I was thinking about running in through another round adding some followups and sticking a +A on it | 18:09 |
| clayg | ... any obvious objections or thoughts? | 18:09 |
| clayg | ... anyone else? notmyname? | 18:09 |
| notmyname | oh hai | 18:09 |
| clayg | notmyname: !!!! | 18:09 |
| clayg | notmyname: I'm in a bad mood today! :D | 18:10 |
| notmyname | then I'm glad you're WFH and can't throw things at me ;-) | 18:11 |
| timburke | clayg: seems like a plan. i'd kinda written off that review as a fait accompli between your previous +2 and kota_'s current +2 | 18:11 |
| clayg | you know me - i mostly just yell and curse - but I do that when i'm in a good mood - so what can you do | 18:11 |
| clayg | timburke: rofl! | 18:12 |
| clayg | If you stick enough Closes-Bug on a patch it is *definately* a fait accompli | 18:12 |
| clayg | I | 18:12 |
| clayg | I've never heard that term before - thank you | 18:12 |
| timburke | least i could do for introducing me to lagniappe | 18:14 |
| clayg | haha | 18:14 |
| clayg | sigh... we're going to have to start writing developer docs for our internal API's :'( | 18:15 |
| clayg | even ssync is FAR to complicated to not have descriptive prose - MIME expect 100 continuing object servers - whatever the heck that json RPC thing the containers servers' do | 18:16 |
| timburke | yep. over and above what we might want to write for middleware developers | 18:17 |
| timburke | http://docs.openstack.org/developer/swift/development_middleware.html#creating-your-own-middleware is ok, but it doesn't even *touch* on trying to write your own auth middleware, say, or how/when to use footers, or... | 18:17 |
| clayg | footers :'( | 18:18 |
| timburke | what? they're great! love 'em! but without already knowing about them or digging through a lot of code, there's no way i'd discover them | 18:19 |
| notmyname | timburke: can you hop in #openstack-operators please? | 18:23 |
| clayg | timburke: the idea of adding metadata not known when you start a PUT to an object as an atomic item is fine (vs say a non-atomic PUT/POST on the same connection pipeline with the same timestamp) - but I don't like the implementation of MIME streaming - as far as inventing a protocol to support footers I think we could have done better | 18:25 |
| *** chlong_ has quit IRC | 18:27 | |
| timburke | clayg: ah, but the beauty is, from a middleware developer's perspective, it doesn't matter! | 18:39 |
| clayg | haha | 18:41 |
| *** nikivi has joined #openstack-swift | 18:42 | |
| timburke | man, i love some of the things we've done with swiftclient lately! now we can say things like `swift download $(swift stat "$CONTAINER" "$DLO" | grep Manifest | sed -e 's/^[^:]*: //' -e 's!/! --prefix=!')` | 19:05 |
| clayg | timburke: that sed syntax is sorta spooky to me - but from context it looks roughly like you're downloading all of the segments of a DLO by pulling it's Manifest value out of the HEAD/stat request? | 19:08 |
| timburke | yup! | 19:08 |
| clayg | fwiw - i also like --prefix on download - is that new(ish)? | 19:08 |
| timburke | oxinabox might be interested as a temporary workaround for https://bugs.launchpad.net/python-swiftclient/+bug/1621562 | 19:09 |
| openstack | Launchpad bug 1621562 in python-swiftclient "Download large objects concurrently" [Wishlist,Confirmed] | 19:09 |
| timburke | clayg: yep; added when i did...umm... where was it.... | 19:09 |
| *** ukaynar has quit IRC | 19:10 | |
| *** ukaynar has joined #openstack-swift | 19:10 | |
| timburke | nope! i was thinking of delete growing --prefix in patch 190887 | 19:12 |
| patchbot | https://review.openstack.org/#/c/190887/ - python-swiftclient - Use bulk-delete middleware when available (MERGED) | 19:12 |
| timburke | download --prefix has been around a while; swifterdarrell added it for 1.5.0! | 19:13 |
| clayg | oh that swifterdarrell - he's so great | 19:13 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Add probetest for not uncommon EC failure https://review.openstack.org/371771 | 19:13 |
| *** McMurlock1 has joined #openstack-swift | 19:21 | |
| *** sgundur_ has quit IRC | 19:25 | |
| MooingLemur | clayg: Thanks for the bug pointer, my rsync timeout is set to 3600 at the moment due to inter-region replication and long-running rsyncs were getting killed prematurely when it was set to 900. I will increase the lockup_timeout and see what happens | 19:28 |
| *** nikivi has quit IRC | 19:28 | |
| *** david-lyle has quit IRC | 19:30 | |
| *** david-lyle has joined #openstack-swift | 19:30 | |
| MooingLemur | zaitcev, clayg: sorry if my tone was a little short. Sometimes I feel like I shouldn't bother opening bugs unless I have something more useful to say than "it locks up". I wish I had more time to dedicate to working on these issues :) | 19:31 |
| MooingLemur | clayg: I couldn't find that string in the logs, but I do remember seeing a message similar to that. However, it could have just been in the replicator source. | 19:32 |
| *** StraubTW has quit IRC | 19:33 | |
| *** sgundur_ has joined #openstack-swift | 19:37 | |
| clayg | MooingLemur: sometimes a bug report is just the start of a conversation - someone later might say "I saw it too!" - https://xkcd.com/979/ | 19:38 |
| *** StraubTW has joined #openstack-swift | 19:38 | |
| clayg | MooingLemur: and you weren't short/curt/whatever - don't worry about it - it's irc | 19:38 |
| *** StraubTW has quit IRC | 19:39 | |
| *** StraubTW has joined #openstack-swift | 19:39 | |
| *** StraubTW_ has joined #openstack-swift | 19:40 | |
| *** david-lyle has quit IRC | 19:43 | |
| *** StraubTW has quit IRC | 19:44 | |
| *** ukaynar has quit IRC | 19:46 | |
| tdasilva | notmyname: is there a policy or process for closing bugs that we think are no longer valid? | 19:47 |
| tdasilva | or that might have been fixed by an unknown patch? | 19:47 |
| notmyname | tdasilva: beyond simply marking them as closed? nope. no other process | 19:48 |
| notmyname | tdasilva: so either mark it as fix released or invalid and leave a comment on the bug with your thinking | 19:48 |
| clayg | tdasilva: if there's not a good way to a/b test "yes, this is fixed on master" - you could try to add your thinking/tests and set it to incomplete | 19:49 |
| clayg | but if it's fixed you can just say how you know it's fixed and then close it | 19:49 |
| clayg | ... that's what'd I'd do | 19:49 |
| clayg | notmyname: lol - i just said the same thing you did - sorry | 19:49 |
| clayg | tdasilva: listen to notmyname | 19:49 |
| tdasilva | clayg, notmyname: ok, my other question was 'what's launchpad for close', i guess notmyname said: 'fix released' | 19:50 |
| notmyname | yeah. fix released | 19:50 |
| tdasilva | notmyname, clayg: thanks! | 19:51 |
| openstackgerrit | Thiago da Silva proposed openstack/swift: Return 404 on a GET if tombstone is newer https://review.openstack.org/371150 | 19:53 |
| clayg | tdasilva: PROBETEST !! that's awesome! | 19:57 |
| *** openstackstatus has quit IRC | 19:58 | |
| tdasilva | clayg: thanks for the tip, i hope it's going in the right direction | 20:00 |
| *** openstackstatus has joined #openstack-swift | 20:01 | |
| *** ChanServ sets mode: +v openstackstatus | 20:01 | |
| clayg | hrmm... apparently my tox.ini isn't flagging H233 like it should e | 20:09 |
| *** chsc has quit IRC | 20:09 | |
| clayg | or at least not automatically on buffer save :\ | 20:10 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Add probetest for not uncommon EC failure https://review.openstack.org/371771 | 20:10 |
| clarkb | clayg: you may need to rebuild the virtualenv if its a new hacking rule (tox -r will do this) | 20:10 |
| notmyname | looks like the tox.ini comments say we ignore H233, but it's not in the ignore line :-( | 20:11 |
| notmyname | ah. it's timburke's fault | 20:12 |
| timburke | yup, forgot to remove it | 20:12 |
| openstackgerrit | John Dickinson proposed openstack/swift: remove comment saying we ignore H233. we actually check it https://review.openstack.org/371790 | 20:13 |
| notmyname | CRITICAL BUG FIX | 20:13 |
| notmyname | timburke: ;-) | 20:13 |
| notmyname | timburke: I'm definitely not casting any stones here | 20:13 |
| *** sgundur_ has quit IRC | 20:15 | |
| tdasilva | timburke: you should have put a -1 and added a comment about missing #Closes-Bug flag :P | 20:15 |
| *** AndyWojo has quit IRC | 20:16 | |
| *** AndyWojo has joined #openstack-swift | 20:17 | |
| clayg | idk, it's not like my flake8 plugin to vim is readying the comment? | 20:21 |
| clayg | it's unltimatley flake8 that reads the tox.ini - and it blew up from the command line - i'm not sure why my vim is missing it | 20:21 |
| timburke | clayg: does your flake8 plugin to vim know about hacking? | 20:22 |
| clayg | timburke: yes, it seems that somehow what I thought used to work for starting my editor to respect the local flake8 in the virtualenv is no longer working - i was only getting the checks of my system flake8 | 20:30 |
| *** Jeffrey4l has quit IRC | 20:35 | |
| *** Jeffrey4l has joined #openstack-swift | 20:35 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Turn on F812 check https://review.openstack.org/371799 | 20:36 |
| notmyname | timburke: uh oh. I think you might have stepped in it | 20:37 |
| timburke | that list is gonna keep getting shorter and shorter! eventually we won't even be worth *mentioning* on http://flake8.readthedocs.io/en/latest/user/configuration.html ! | 20:37 |
| notmyname | IIRC that was the biggest point of contention back when we went from whitelist checks to blacklist checks | 20:37 |
| timburke | *why*? there were only *three* violations, most of them in tests | 20:38 |
| *** baojg_ has joined #openstack-swift | 20:40 | |
| *** baojg has quit IRC | 20:43 | |
| MooingLemur | clayg: I might have found something relevant: warning Unexpected file /srv/node/f1s23/objects/785930/04a/2ff82a8a0ae1c31397e2fce2edc4104a/.1420009299.86614.data.fLw0Bm: Invalid Timestamp value in filename u'.1420009299.86614.data.fLw0Bm' | 20:44 |
| MooingLemur | last log line on one of the boxes before it froze | 20:44 |
| patchbot | froze | 20:44 |
| * MooingLemur eyes patchbot :> | 20:45 | |
| MooingLemur | err, not froze, but last line before object-replicator decided not to do any more work | 20:45 |
| notmyname | MooingLemur: patchbot is rather neurotic these days | 20:47 |
| notmyname | I think my next step is to write an IRC bot from scratch to be as simple as possible | 20:47 |
| MooingLemur | hehe | 20:48 |
| MooingLemur | clayg: however, that might be a red herring, since that's not like that on other boxes that have stalled replicators | 20:50 |
| *** myoonnyc has quit IRC | 20:50 | |
| *** baojg_ has quit IRC | 20:51 | |
| clayg | timburke: what the passive agressive who now? why are we on flake8's docs?! | 20:52 |
| clayg | WHY ARE ANY OPNSTACK CUSTOM H RULES ON FLAKE8'S DOCS!? | 20:52 |
| *** baojg has joined #openstack-swift | 20:52 | |
| notmyname | wow | 20:53 |
| zaitcev | OpenStack is a major project, you know. | 20:53 |
| zaitcev | Look at the Summit attendance. | 20:54 |
| notmyname | "let's take a look at an open source project that's using our tool. look at how terrible they are. instead of proposing a fix, we'll just keep it as-is as a bad example and warning to others" | 20:54 |
| MooingLemur | Was there ever code released in swift to clean up stale rsync dotfiles in object directories? | 20:57 |
| notmyname | https://review.openstack.org/#/c/293177/ | 20:57 |
| MooingLemur | I remember being around for the discussion of the issue | 20:57 |
| patchbot | patch 293177 - swift - Auditor will clean up stale rsync tempfiles (MERGED) | 20:57 |
| notmyname | that was in 2.7.0 | 20:58 |
| MooingLemur | roger :) | 20:58 |
| MooingLemur | thanks | 20:58 |
| *** sgundur_ has joined #openstack-swift | 21:00 | |
| *** baojg has quit IRC | 21:00 | |
| *** dmorita has quit IRC | 21:00 | |
| *** dmorita has joined #openstack-swift | 21:07 | |
| *** sgundur_ has quit IRC | 21:08 | |
| *** StraubTW_ has quit IRC | 21:09 | |
| notmyname | https://gitlab.com/pycqa/flake8/issues/228 | 21:12 |
| *** _JZ_ has joined #openstack-swift | 21:17 | |
| *** wasmum has quit IRC | 21:18 | |
| *** nadeem has joined #openstack-swift | 21:20 | |
| *** wasmum has joined #openstack-swift | 21:20 | |
| *** _JZ_ has quit IRC | 21:22 | |
| pdardeau | torgomatic_: notmyname: on comments on patch 336323, i'll be looking into some md5 related stuff soon | 21:26 |
| patchbot | https://review.openstack.org/#/c/336323/ - swift - Add checksum to object extended attributes | 21:26 |
| *** dmorita has quit IRC | 21:28 | |
| *** tsg__ has joined #openstack-swift | 21:30 | |
| *** caiobrentano_ has quit IRC | 21:32 | |
| pdardeau | notmyname: torgomatic_: i created lp 1624556 (wishlist) for utility to add metadata checksum on existing objects | 21:32 |
| openstack | Launchpad bug 1624556 in OpenStack Object Storage (swift) "Create utility to add metadata checksums" [Undecided,New] https://launchpad.net/bugs/1624556 | 21:32 |
| patchbot | https://review.openstack.org/#/c/1624556/ | 21:32 |
| *** tsg_ has quit IRC | 21:33 | |
| *** tsg__ has quit IRC | 21:36 | |
| *** klamath has quit IRC | 21:37 | |
| *** klamath has joined #openstack-swift | 21:37 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Store SLO Etag and size in sysmeta https://review.openstack.org/347538 | 21:38 |
| openstackgerrit | Tim Burke proposed openstack/swift: Turn on F812 check https://review.openstack.org/371799 | 21:45 |
| timburke | ok, so there were *four* places... | 21:45 |
| *** dmorita has joined #openstack-swift | 21:47 | |
| *** tsg__ has joined #openstack-swift | 21:58 | |
| openstackgerrit | Merged openstack/swift: Enable object server to return non-durable data https://review.openstack.org/215276 | 22:11 |
| *** catintheroof has quit IRC | 22:31 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Include metadata in PUT/POST responses https://review.openstack.org/357559 | 22:37 |
| *** tsg__ has quit IRC | 22:44 | |
| *** david-lyle has joined #openstack-swift | 22:48 | |
| *** Jeffrey4l has quit IRC | 22:48 | |
| *** Jeffrey4l has joined #openstack-swift | 22:49 | |
| *** McMurlock1 has quit IRC | 22:59 | |
| *** Jeffrey4l has quit IRC | 23:21 | |
| *** Jeffrey4l has joined #openstack-swift | 23:22 | |
| *** nadeem has quit IRC | 23:24 | |
| *** rvasilets___ has quit IRC | 23:27 | |
| *** Suyi_ has quit IRC | 23:44 | |
| *** dmorita has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
