| *** petertr7 has quit IRC | 00:11 | |
| *** annegentle has joined #openstack-swift | 00:16 | |
| *** annegentle has quit IRC | 00:22 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift: EC: support multiple ranges for GET requests https://review.openstack.org/166576 | 00:25 |
|---|---|---|
| *** chlong has joined #openstack-swift | 00:25 | |
| clayg | vishy: thanks for the confirmation! | 00:28 |
| clayg | i have this script that's basically a for loop around for sha in ec_review; git checkout sha; test; done | 00:35 |
| clayg | it's *great* for finding little pep8 shit that slips in between fixup commits | 00:35 |
| clayg | but it takes forever - and worse tends to pause when my laptop enters power save mode | 00:36 |
| clayg | is there a setting for "don't take a nap when your master told you do something you stupid mackbook" ??? | 00:37 |
| *** vinsh has joined #openstack-swift | 00:45 | |
| *** LCissoto has joined #openstack-swift | 00:52 | |
| *** km has quit IRC | 00:53 | |
| *** km has joined #openstack-swift | 00:53 | |
| *** kei_yama has quit IRC | 00:53 | |
| *** kei_yama_ has joined #openstack-swift | 00:53 | |
| *** annegentle has joined #openstack-swift | 00:57 | |
| *** thumpba has quit IRC | 01:16 | |
| *** thumpba has joined #openstack-swift | 01:17 | |
| *** annegentle has quit IRC | 01:22 | |
| *** Gue______ has joined #openstack-swift | 01:26 | |
| peluse | clayg, 10-4 on the rabbit hole | 01:32 |
| peluse | clayg, but you dug the first one :) | 01:32 |
| clayg | peluse: no but that was my point - I sorta had to - I was going to pass int(policy) to get_object_ring just so I could have policy.object_ring | 01:33 |
| clayg | er.. wasn't going to :P | 01:33 |
| peluse | heh | 01:37 |
| *** Gue______ has quit IRC | 01:40 | |
| *** Gue______ has joined #openstack-swift | 01:42 | |
| *** annegentle has joined #openstack-swift | 01:52 | |
| clayg | i am a little disconcerted that POLICIES[0] == 0 returns True :\ | 01:53 |
| *** winggundamth has joined #openstack-swift | 01:54 | |
| clayg | i'm not so sure I want to do anything about it - but it makes test fakes that were built to expect an int(policy) all to happy when they get a policy - and in this case none-too-upset when they just keep on getting an int :\ | 01:57 |
| winggundamth | hi I got inconsistent between each storage container as shown here http://pastebin.com/j3YqDMJ9. How can I fix this? | 01:58 |
| clayg | winggundamth: doesn't look so bad, what *one* container replica is missing *one* row? I've seen way worse. Has it been like this for awhile? Are the container replicators running? what version of swift, what's the environment? | 02:00 |
| winggundamth | hi clayg: I got inconsistent that cause from object expirer. I'm using Swift that came with Ubuntu 14.10 that's 2.2.0. I have 1 proxy node with 3 storage nodes | 02:03 |
| winggundamth | I'm going to recheck replicators process now | 02:03 |
| winggundamth | swift 80 1 0 Apr08 ? 00:00:02 /usr/bin/python /usr/bin/swift-container-replicator /etc/swift/container-server.conf | 02:04 |
| winggundamth | this is what process looks like right? | 02:04 |
| winggundamth | I see it's running on every storage nodes | 02:05 |
| clayg | it makes sense to me that an object write (put or delete) - from the object expirer or anywhere else - could (under failure) cause the container db's to get out of sync - it's an expected failure mode - but either the object updater or the container repliator should fix it | 02:05 |
| clayg | winggundamth: idk, maybe hasn't been running very long? maybe it's in some sort of restart loop? | 02:05 |
| clayg | oh hrmm... maybe not - i guess it's the 9th already in utc | 02:07 |
| winggundamth | I try to find some errors or process restart in log. but nothing too | 02:08 |
| *** kota_ has joined #openstack-swift | 02:08 | |
| winggundamth | this inconsistent already been for a night already. after I test object expirer last night | 02:09 |
| clayg | ... that's strange, I'd typically jump right into some pretty low level container replication debugging - although there's probably a simple explination | 02:13 |
| winggundamth | how can I turn on that? | 02:13 |
| clayg | maybe to start just turn on DEBUG logging for the container servers and grep around for the container db's hash | 02:13 |
| winggundamth | ok let me try it | 02:14 |
| clayg | did you find those container replicas with swift-get-nodes? It'll should also have the on-disk path for the sqlite db | 02:14 |
| clayg | swift-container-info /path/to/hash.db <- has useful info something might jump out at you | 02:14 |
| clayg | if you say you noticed it when the expirer killed off something it sounds like you believe that the container with 617 rows is correct and the two with 618 are out-of-date? | 02:15 |
| winggundamth | yes | 02:16 |
| winggundamth | did you find those container replicas with swift-get-nodes? <---- yes | 02:17 |
| winggundamth | where should hash.db should be? | 02:18 |
| clayg | /srv/node/<device>/<part>/<suffix>/<hash(account/container)>.db | 02:19 |
| clayg | swift-get-nodes will spit it out | 02:20 |
| winggundamth | ok let me see | 02:23 |
| winggundamth | it's weird that I never found this command in any document | 02:23 |
| *** annegentle has quit IRC | 02:24 | |
| clayg | it says there's a man page for it - dunno how you're supposed to find the man page tho | 02:24 |
| *** Gue______ has quit IRC | 02:24 | |
| winggundamth | it should be in troubleshoot document somewhere in openstack document :) | 02:29 |
| *** kei_yama_ has quit IRC | 02:29 | |
| clayg | ohhh... troubleshooting - i like it! | 02:29 |
| clayg | oh fooey - someone must have already had that idea -> http://docs.openstack.org/developer/swift/admin_guide.html?highlight=troubleshoot#debugging-tips-and-tools | 02:30 |
| clayg | that page is pretty dense tho, could probably use some organizational love by someone that knows something about tech writing | 02:31 |
| clayg | winggundamth: you wouldn't happen to know anything about tech writing do you!? | 02:31 |
| *** km_ has joined #openstack-swift | 02:31 | |
| winggundamth | sorry. not at all | 02:31 |
| *** kei_yama has joined #openstack-swift | 02:32 | |
| winggundamth | http://pastebin.com/juK0HB5b this is output from swift-container-info. I don't see anything clue to inconsistent problem | 02:32 |
| winggundamth | except X-Container-Sync-Point2: -1. Is this normal? | 02:33 |
| *** km has quit IRC | 02:33 | |
| clayg | yeah that's for the container-sync feature - if you're not using it that value is -1 | 02:38 |
| winggundamth | i see | 02:39 |
| winggundamth | so do you find anything from the output? | 02:39 |
| winggundamth | if not I think I really need dig into debug log :) | 02:39 |
| clayg | not really - chexor value correctly shows that they are out-of-sync - I would be expecting 18b496c0-1ab8-40b4-97e4-b4113d5873ce to be trying to ship the delete row to 55c35773-dd23-4da4-b79b-468e335b57a5 and 33a60613-0b1c-4b05-8cbd-88f2c90a7305 | 02:41 |
| clayg | so yeah on the 18b496c0-1ab8-40b4-97e4-b4113d5873ce node if you could just grep for 8ac649c9807a5c6b49307d5a6f4bed9b in the container-replicator logs - should show something about some kind of error something - or with debug turned on it would show at least that it's trying to sync that container and maybe we could tease out some idea why it's not | 02:42 |
| clayg | sqlite --line /path/to/hash.db "select * from incoming_sync" might be interesting - or at least confirm these nodes aren't talking to each other about this db for some reason | 02:43 |
| winggundamth | what is chexor? | 02:45 |
| clayg | it's this rolling checksum the container keeps track of (using sqlite triggers) of all the rows in the database to accelerate replication by just comparing the checksum value rather than shipping over a bunch of rows that might already be in sync with the remote node | 02:46 |
| winggundamth | ic | 02:46 |
| clayg | redbo thought it was a neat idea at somepoint - seems to do the trick | 02:47 |
| winggundamth | I'm trying to do sqlite command on 8ac649c9807a5c6b49307d5a6f4bed9b.db but it said Unable to open database "8ac649c9807a5c6b49307d5a6f4bed9b.db": file is encrypted or is not a database | 02:52 |
| clayg | er... sqlite3 :\ | 02:53 |
| clayg | sorry - the name of the command on my machine is *acctually* sql<tab> ;) | 02:53 |
| winggundamth | lol | 02:56 |
| winggundamth | sqlite3 8ac649c9807a5c6b49307d5a6f4bed9b.db "select * from incoming_sync" no any input from this command | 03:02 |
| clayg | well that's impressive | 03:05 |
| clayg | makes me wonder how you made it this long? | 03:05 |
| clayg | wtf is the container-replicator doing? | 03:05 |
| winggundamth | really -_- | 03:05 |
| clayg | can you stop them and run `swift-init container-replicator once -nv` to see in the console wtf it thinks it's doing? | 03:05 |
| winggundamth | I already ran it for almost half year | 03:05 |
| winggundamth | until I found object expirer feature and try it | 03:06 |
| clayg | wtg object-updaters? | 03:06 |
| winggundamth | ok let me try container-replicator first | 03:07 |
| winggundamth | http://pastebin.com/mzXZHLRb | 03:08 |
| winggundamth | here you go | 03:08 |
| clayg | lol? | 03:09 |
| winggundamth | actually I always see this output from log | 03:09 |
| clayg | ok - i'm reaching here - but maybe the devices option is wrong in the container-replicator section? | 03:11 |
| winggundamth | [container-replicator] this is the only line I have | 03:12 |
| winggundamth | without any configuraton | 03:12 |
| winggundamth | http://pastebin.com/dPtfKXGy my configuration right now | 03:13 |
| clayg | that's weird - i'm not even sure why an invalid devices path is not an OSError | 03:13 |
| clayg | winggundamth: seems reasonable | 03:14 |
| clayg | next most likely culprit is something with whataremyips or more generally the "what devices are local" check | 03:14 |
| clayg | in your container.builder is the replication_ip and replication_port the same as the ip/port for all devices? | 03:15 |
| winggundamth | oh I forget that I ran on these in Docker | 03:15 |
| winggundamth | so the ip will be 172.17.x.x and NAT with host IP | 03:15 |
| clayg | DOCKER DOCKER DOCKER! | 03:16 |
| clayg | python -c 'from swift.common.utils import whataremyips; print whataremyips()' | 03:17 |
| clayg | ^ something in that list should be in the ring | 03:17 |
| clayg | perferably one that would also allow the nodes to talk to each other | 03:18 |
| clayg | or "containers" or whatever | 03:18 |
| winggundamth | output from that python command ['127.0.0.1', '::1', '172.17.0.122', 'fe80::42:acff:fe11:7a'] | 03:19 |
| *** sorrison has joined #openstack-swift | 03:20 | |
| winggundamth | so this is the cause of problem? | 03:20 |
| sorrison | notmyname: I'm here | 03:20 |
| notmyname | sorrison: hello :-) | 03:20 |
| clayg | ok, so that 172 looks like a decent choice - can the nodes talk to each other on those ips? | 03:20 |
| notmyname | sorrison: so https://bugs.launchpad.net/swift/+bug/1441926 | 03:20 |
| openstack | Launchpad bug 1441926 in OpenStack Object Storage (swift) "rebalance fails once upgraded to juno" [Undecided,New] | 03:20 |
| clayg | yuk :'( | 03:20 |
| winggundamth | clayg: no way | 03:20 |
| notmyname | sorrison: I'm pulling up the code | 03:20 |
| sorrison | notmyname: I get that in our prod and test envs so i'm hoping it's not something specific to us | 03:21 |
| clayg | winggundamth: well *I* don't know man!? is there *any* ip the nodes can flipping talk to each other on? you'll have to figure out what stupid thing we're doing that's keeping that ip out of that list | 03:21 |
| winggundamth | on *.builder file I'm using the public ip that can talk each other. just wonder why it works for very long time | 03:23 |
| notmyname | sorrison: definitely looks like that code has moved on since 2.2.0, so let met see (1) what the issue is and (2) if it's still in the current version of swift | 03:24 |
| clayg | well the proxy -> node communication isn't the same as node -> node communication | 03:24 |
| winggundamth | I mean everything still works fine if I'm not using object expirer | 03:24 |
| winggundamth | ic | 03:24 |
| clayg | winggundamth: the fack that incoming_sync is empty means it hasn't been doing *squat* for a really long time | 03:24 |
| clayg | winggundamth: but so the nodes *can* talk to each other using the public ip - they just don't *know* what they're public ip is because it's all DOCKER DOCKER DOCKER | 03:25 |
| clayg | s/they're/their/ | 03:25 |
| clayg | sorrison: so maybe there's a zero weight device that's like the only device in some region or zone tier? | 03:27 |
| *** gyee has quit IRC | 03:27 | |
| clayg | it seems like any device with weight will have a key in tiers2sort_key for their tier | 03:27 |
| winggundamth | clayg: NAT doesn't help on this? Swift still sending their ip to other node with that python command? | 03:27 |
| sorrison | clayg: Yeah we actually have a region where all the devices have a weight of 0 | 03:28 |
| sorrison | clayg: but in our test env it still fails http://paste.openstack.org/show/200929/ | 03:29 |
| clayg | sorrison: so I checked out 2.2.0 and a lot of that code is changed in 2.2.2 - i'm not sure we still have this bug (not that it helps you, and I'm not trying to dodge, i'm just saying we may have fixed this already on purpose because it didn't work for a time) | 03:29 |
| notmyname | sorrison: have you added or removed any devices? | 03:29 |
| sorrison | no not yet, I just did a rebalance first to see if that was needed | 03:30 |
| sorrison | so you're saying try remove a device then rebalance and see if that fixes it? | 03:30 |
| notmyname | ah ha!! | 03:30 |
| clayg | is that how folks normally do zones - reuse the same id from another region in the next one? | 03:30 |
| *** sorrison has quit IRC | 03:30 | |
| *** sorrison has joined #openstack-swift | 03:31 | |
| notmyname | he dropped. | 03:31 |
| notmyname | sorrison: there he is | 03:31 |
| notmyname | sorrison: check out the commit message on https://github.com/openstack/swift/commit/d9bfa06be026e8acdb75fed0ae130b5c209251ce | 03:31 |
| *** erlon has quit IRC | 03:31 | |
| sorrison | back, client crashed | 03:31 |
| notmyname | sorrison: when torgomatic wrote that fix, he was convinced that it was just swiftstack who would see that bug that was fixed ;-) | 03:31 |
| notmyname | sorrison: so try changing the weight of a device and then rebalancing | 03:32 |
| clayg | notmyname: idk, people love to rebalancei | 03:32 |
| sorrison | haha, great! I'll give it a go | 03:32 |
| clayg | notmyname: it's what cpu's are for | 03:32 |
| notmyname | clayg: I thought CPUs were for EC calculations! | 03:32 |
| clayg | maybe next month | 03:32 |
| notmyname | sorrison: please let us know if that works | 03:33 |
| clayg | winggundamth: so anyway - the replicator is pretty sure that to find databases that it needs to replicate it needs to be able to self identify devices in the ring that are assigned to him - that means some sort of matching with the ip in the ring to the ip's it's able to identify as being "me" | 03:34 |
| *** ozialien has joined #openstack-swift | 03:35 | |
| clayg | winggundamth: whataremyips just uses netifaces.interfaces - which is fairly reasonble - but if the ip's the nodes need to talk to each other on is NAT'd - then you'll probably need something else - not sure what - probably a whole flipping spec file on "making swift replication work with NAT" | 03:36 |
| winggundamth | clayg: haha. sound big | 03:36 |
| clayg | winggundamth: fwiw I'm sure your object replicators are having the same issue - and you can't rebalance - i guess async pendings don't work either | 03:37 |
| clayg | well.. no... i ... so you have the NAT'd ips in the ring? | 03:37 |
| sorrison | notmyname, clayg all good, once I added a device it allowed me to rebalance | 03:37 |
| clayg | so it's just the is_local check that fails... | 03:37 |
| clayg | sorrison: that torgomatic is a smart muther - no lie | 03:38 |
| clayg | sorrison: maybe you could hack whataremyips to take a "default" that you can configure and make it say "no really trust me, this ip is me" then stick it in the config so it can self-match to the NAT'd ring entries | 03:39 |
| clayg | ^ winggundamth | 03:39 |
| clayg | sorrison: sry, i shouldn't be talking in here at all - i'm trying to fix test failures | 03:39 |
| *** LCissoto has quit IRC | 03:39 | |
| sorrison | ha | 03:39 |
| winggundamth | clayg: [async_pending] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3 this is output from swift-recon -a | 03:40 |
| winggundamth | well.. no... i ... so you have the NAT'd ips in the ring? <--- yes | 03:40 |
| notmyname | sorrison: awesome | 03:40 |
| notmyname | sorrison: mind if I close the bug? or would you like to leave a comment on it? | 03:40 |
| winggundamth | clayg: here all output from swift-recon http://pastebin.com/0P7QUZ2J | 03:43 |
| sorrison | yeah close it, cheers | 03:43 |
| notmyname | sorrison: ok. glad it works! | 03:43 |
| sorrison | notmyname: also one other question. We're adding a new region and I read that swift now does this a bit more gracefully? | 03:44 |
| sorrison | notmyname: do I need to do anything special? | 03:44 |
| notmyname | sorrison: well, you might need to use something later than 2.2.0. let me check the logs | 03:45 |
| sorrison | https://github.com/openstack/swift/blob/2.2.0.rc1/CHANGELOG#L1-L173 | 03:45 |
| notmyname | I hate that I can't close that bug with anything other than "invalid". There isn't a "this totally was reasonable, but it's already been fixed" | 03:45 |
| winggundamth | clayg: and I'm trying to turn on debugging on container-replicator by this http://pastebin.com/dPtfKXGy but it didn't show anything increasing as I expect like how's connect to another node | 03:45 |
| *** jrichli has quit IRC | 03:46 | |
| notmyname | sorrison: ah, yes. that's exactly what I was looking for. just that it was in 2.2.0. so looks like you're good | 03:46 |
| notmyname | sorrison: now, when adding a region, swift won't put half your capacity assigned to the new region the moment you add one drive there | 03:46 |
| notmyname | sorrison: instead, you'll get only what the weight of that drive allows for | 03:47 |
| sorrison | ah so when adding the new devices I should start with small weights and then slowly increase? | 03:47 |
| clayg | winggundamth: it's not trying to connect to anyone - it doesn't think it has any devices in the ring worth replicating | 03:48 |
| notmyname | sorrison: depends on what the rest of the cluster is. eg if you have 1000 drives in one region and you add one drive with it's full weight, that's not a lot relative to the rest of the cluster | 03:48 |
| notmyname | sorrison: point is, yes you shoudl gradually add capacity, but relative to the total of the existing weights in existing tiers. eg if you have 3 racks and add another, add the capacity for the new rack gradually. if you have 30 racks and add one, you don't have to add as gradually. (up to the limit of the network for ingest on the new capacity) | 03:50 |
| sorrison | notmyname: ok gotcha, we currently have 2 regions with approx same size and we are adding a third of approx same size but with 4T drives as opposed to 2T everywhere else | 03:50 |
| notmyname | sorrison: does that make sense? | 03:50 |
| sorrison | notmyname: yeah understand | 03:51 |
| notmyname | half the drives or the same number? | 03:51 |
| winggundamth | clayg: so how's it think that there's device in the ring that need to replicate? | 03:51 |
| notmyname | in the new zone? | 03:51 |
| sorrison | notmyname: half the drives | 03:51 |
| notmyname | err..region | 03:51 |
| notmyname | sorrison: ah, ok. good | 03:51 |
| notmyname | sorrison: I'm glad to hear that you are keeping the regions the same size. that's good | 03:52 |
| clayg | sorrison: is doing it right | 03:54 |
| *** foexle_ has joined #openstack-swift | 03:55 | |
| notmyname | sorrison: are you going to be in vnacouver? | 03:57 |
| notmyname | clayg: you might not know this, but sorrison is a pretty cool guy. he's running a swift cluster across all of australia, and he's the guy that scaled http://adrift.org.au by making it talk directly to swift | 03:57 |
| *** foexle has quit IRC | 03:58 | |
| notmyname | clayg: if he's in vancouver, you should definitely get a beer with him | 04:00 |
| sorrison | notmyname: yeah I'll be there, swiftstack having a party? | 04:01 |
| notmyname | sorrison: you know, that's a good question. I actually don't know | 04:03 |
| sorrison | notmyname: I'm sure we'll have a chance to have a beer either way | 04:03 |
| notmyname | yes, I hope so :-) | 04:04 |
| *** ppai has joined #openstack-swift | 04:08 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:10 | |
| mattoliverau | Maybe I should buy sorrison a beer, that is if he's ever in the Melbourne area ;) | 04:18 |
| clayg | sorrison: i'll get my people on it | 04:21 |
| sorrison | mattoliverau: does every day count? | 04:22 |
| mattoliverau | sorrison: then were in the same city.. so yes.. yes it does ;) | 04:23 |
| sorrison | mattoliverau: great, good to know someone else here knows about swift, you a swift-core? | 04:24 |
| notmyname | sorrison: yes he is :-) | 04:25 |
| mattoliverau | sorrison: yup | 04:25 |
| clayg | sorrison: I'd consider it a personal failing if mattoliverau was not | 04:26 |
| * clayg can think of at least one other failing he would like to rectify during liberty | 04:27 | |
| mattoliverau | clayg: aww shucks. | 04:27 |
| *** raginbajin has quit IRC | 04:32 | |
| *** raginbajin has joined #openstack-swift | 04:35 | |
| *** ppai has quit IRC | 04:41 | |
| *** _1_fahimabrar12 has joined #openstack-swift | 04:42 | |
| *** _1_fahimabrar12 has left #openstack-swift | 04:42 | |
| *** proteusguy has joined #openstack-swift | 04:43 | |
| *** ppai has joined #openstack-swift | 04:57 | |
| *** ho has quit IRC | 05:05 | |
| *** ozialien has quit IRC | 05:06 | |
| *** chlong has quit IRC | 05:09 | |
| *** ppai has quit IRC | 05:32 | |
| *** bkopilov has quit IRC | 05:41 | |
| *** bkopilov has joined #openstack-swift | 05:43 | |
| *** ppai has joined #openstack-swift | 05:44 | |
| *** yuan has joined #openstack-swift | 05:48 | |
| *** dmorita has joined #openstack-swift | 05:51 | |
| *** thumpba has quit IRC | 05:52 | |
| *** wasmum has quit IRC | 05:52 | |
| dmorita | notmyname: are you around? | 06:08 |
| *** ppai has quit IRC | 06:40 | |
| *** joeljwright1 has quit IRC | 06:42 | |
| clayg | acoles_away: can you let me know when you come online - I think i'm crossing the barrier of number of review rabbit holes which will push me outside of a reasonable ability to make significant progress on the reconstructor refactor before exahustion kicks in :\ | 06:43 |
| *** krykowski has joined #openstack-swift | 06:51 | |
| *** ppai has joined #openstack-swift | 06:52 | |
| openstackgerrit | Kamil Rykowski proposed openstack/swift: More user-friendly output for object metadata https://review.openstack.org/164019 | 06:54 |
| *** wasmum has joined #openstack-swift | 06:58 | |
| *** jordanP has joined #openstack-swift | 07:07 | |
| *** winggundamth has quit IRC | 07:15 | |
| *** jistr has joined #openstack-swift | 07:15 | |
| *** bkopilov has quit IRC | 07:18 | |
| *** fanyaohong has quit IRC | 07:19 | |
| *** geaaru has joined #openstack-swift | 07:31 | |
| openstackgerrit | Kamil Rykowski proposed openstack/swift: More user-friendly output for object metadata https://review.openstack.org/164019 | 07:53 |
| *** acoles_away is now known as acoles | 08:00 | |
| acoles | clayg: ping | 08:00 |
| clayg | you guys were busy while i was out | 08:01 |
| clayg | it's been slow going :P | 08:01 |
| *** jamielennox is now known as jamielennox|away | 08:01 | |
| clayg | I think i'm on the down hill strech | 08:01 |
| acoles | clayg: so what can i do to help | 08:03 |
| clayg | well until the other reviewers come online this morning and you guys can discuss the review comments I have not been able to address (i've been adding bullet lists) - maybe you could help me get a jump start on this plan I had to fixup the reconstructor revert cleanup? | 08:04 |
| acoles | clayg: sure. | 08:05 |
| clayg | acoles: oh wow, i didn't get nearly as far as I thought -> https://github.com/clayg/swift/commit/6d574d64a3e1f4f98f25138831c7cf201b08b4a3 | 08:05 |
| acoles | clayg: ok i can pick up the TODOs in there ^^ like move the delete out of ssync into recontructor, fix the purge and HCL cleanup of durables etc | 08:08 |
| clayg | yeah that's the idea - I'm still working through review comments - and need to do a bit of rebase work (and tests) before I can push - so you're not going to have a good clean place to branch from that won't be moving from underneath you just yet | 08:09 |
| clayg | not sure if you've made it through emails and got coffee yet - that sort of thing :) | 08:09 |
| acoles | clayg: coffee consumed, emails deleted, ready to roll ;) | 08:10 |
| *** proteusguy has quit IRC | 08:10 | |
| acoles | clayg: i'll work from latest gerrit review patchset and fix up once you push your changes | 08:11 |
| clayg | sounds great good luck! | 08:12 |
| acoles | clayg: get some sleep when you can | 08:12 |
| *** mmcardle has joined #openstack-swift | 08:13 | |
| *** winggundamth has joined #openstack-swift | 08:13 | |
| *** jasondotstar has quit IRC | 08:20 | |
| *** ppai has quit IRC | 08:22 | |
| *** clyps___1 has joined #openstack-swift | 08:28 | |
| *** proteusguy has joined #openstack-swift | 08:29 | |
| *** ppai has joined #openstack-swift | 08:36 | |
| clayg | wth M$ - nano-server sounds like a good idea almost - "To achieve these benefits, we removed the GUI stack, 32 bit support (WOW64), MSI and a number of default Server Core components" | 08:46 |
| clayg | you know how windows server sucks for being like - a server - yeah ftfy | 08:46 |
| *** Akshat has joined #openstack-swift | 09:10 | |
| *** kota_ has quit IRC | 09:11 | |
| *** joeljwright has joined #openstack-swift | 09:13 | |
| *** pcaruana has joined #openstack-swift | 09:17 | |
| *** bkopilov has joined #openstack-swift | 09:19 | |
| clayg | acoles: my tests are wrapping up - about to publish the updates to the chain | 09:19 |
| clayg | is there anything helpful I could do with the reconstructor branch? maybe fix some of those failing unittests on that wip branch - or refactor _get_job_info to address some of the outstanding comments? | 09:20 |
| acoles | clayg: ok | 09:20 |
| acoles | clayg: looking... | 09:23 |
| acoles | clayg: i'm not going to touch _get_job_info (i think). i am doing stuff related to deleting reverted objs, so will be making small change to process_jobs and ssync_sender and probably adding some tests. | 09:26 |
| clayg | alright sweet - sounds like a winner then | 09:26 |
| acoles | so once i am done, reconstructor will get set of sync jobs back from ssync_sender and then purge them. | 09:27 |
| acoles | and 'purge' will not touch .durable. | 09:27 |
| clayg | yeah - and :) | 09:27 |
| acoles | and .durable will be cleaned up in HCL | 09:27 |
| acoles | s/sync jobs/sync objs/ | 09:28 |
| clayg | acoles: the other part of the equation - and maybe it's a seperate issue that the good work you're doing - is the final rmtree to the part on the subsequent run | 09:30 |
| acoles | clayg: yeah, i can leave that ^^ to you if u like | 09:30 |
| clayg | I think it's going to work out that in reconstruct, under the for part_info loop, when you call build_reconstruction_jobs - if that list comes up empty - you can kick the tree out | 09:31 |
| clayg | i'm feeling a little humbled by the ineffectiveness today - i thought i was only off one day - but I guess it was probably closer to two - or four depending on how you define "day" | 09:32 |
| clayg | so - by all means don't "save" anything for me except whatever you can't get to - i'll be back in the office in the am to pick up from however far you get | 09:33 |
| clayg | no worries either way | 09:33 |
| clayg | - one last ask tho | 09:33 |
| acoles | clayg: nah you're doing a great job, dealing with comments is really time consuming | 09:33 |
| clayg | heh, leads to the next ask :P | 09:34 |
| acoles | nooooo!! | 09:34 |
| acoles | ;) | 09:34 |
| clayg | can you look over my bullet lists on my responses to the last set of revieww | 09:34 |
| clayg | some of them were like "well, yeah I can see that, but I'm also not sure" | 09:34 |
| clayg | any we can reach concensus on I'd love to just have as much "make the code like this and everyone will be least-unhappy" as possible | 09:35 |
| clayg | where we need to keep arguing - or I need to kick something out for folks to mill over some more - we can do that too | 09:35 |
| acoles | sure. btw i suspect we may need to riage some stuff - like, 'must fix for release' or 'must have a trello todo card so we don't forget' | 09:35 |
| acoles | triage | 09:36 |
| clayg | so if you could familiarize yourself with the questions I'm seeking answers to - and make whatever effort you could to build concensus I'd consdier it true kindness my friend | 09:36 |
| clayg | acoles: yeah that's the ticket! | 09:36 |
| acoles | clayg: ok will do my best, have a good rest | 09:36 |
| clayg | k, thanks bro - i'll let you get to it - i hope to catch you in my am, but don't stick around for me - i can take the handoff async via channel or email | 09:37 |
| clayg | awesome! | 09:37 |
| acoles | k | 09:37 |
| *** haigang has joined #openstack-swift | 09:39 | |
| *** krykowski_ has joined #openstack-swift | 10:06 | |
| *** krykowski has quit IRC | 10:09 | |
| *** ppai has quit IRC | 10:19 | |
| *** ppai has joined #openstack-swift | 10:32 | |
| *** dmorita has quit IRC | 10:38 | |
| *** winggundamth has quit IRC | 10:40 | |
| *** jamielennox|away is now known as jamielennox | 10:42 | |
| *** joeljwright1 has joined #openstack-swift | 10:45 | |
| *** joeljwright has quit IRC | 10:48 | |
| *** aix has joined #openstack-swift | 10:51 | |
| *** joeljwright has joined #openstack-swift | 10:56 | |
| *** joeljwright1 has quit IRC | 10:56 | |
| *** joeljwright has quit IRC | 11:08 | |
| *** ppai has quit IRC | 11:14 | |
| *** haigang has quit IRC | 11:24 | |
| *** winggundamth has joined #openstack-swift | 11:25 | |
| *** ppai has joined #openstack-swift | 11:28 | |
| *** joeljwright has joined #openstack-swift | 11:28 | |
| *** silor has joined #openstack-swift | 11:34 | |
| *** Akshat has quit IRC | 11:35 | |
| *** SkyRocknRoll has quit IRC | 11:38 | |
| *** SkyRocknRoll has joined #openstack-swift | 11:38 | |
| *** joeljwright has quit IRC | 11:44 | |
| *** joeljwright has joined #openstack-swift | 11:45 | |
| *** joeljwright1 has joined #openstack-swift | 11:49 | |
| *** joeljwright has quit IRC | 11:50 | |
| *** jamielennox is now known as jamielennox|away | 11:53 | |
| *** kei_yama has quit IRC | 12:03 | |
| *** nshaikh has joined #openstack-swift | 12:12 | |
| *** km_ has quit IRC | 12:31 | |
| *** ppai has quit IRC | 12:34 | |
| *** ozialien has joined #openstack-swift | 12:43 | |
| *** ppai has joined #openstack-swift | 12:46 | |
| *** aix has quit IRC | 12:53 | |
| *** dencaval has joined #openstack-swift | 12:54 | |
| tdasilva | good morning! just caught up with the scrollback, you guys were busy last night | 12:56 |
| *** ppai has quit IRC | 13:02 | |
| *** aix has joined #openstack-swift | 13:03 | |
| *** cdelatte has joined #openstack-swift | 13:03 | |
| *** delatte has joined #openstack-swift | 13:03 | |
| *** delattec has quit IRC | 13:05 | |
| *** winggundamth has quit IRC | 13:08 | |
| *** jkugel has joined #openstack-swift | 13:10 | |
| cschwede | i’m wondering if we should fix typos (and maybe other smaller issues) in the final docs patch (or another follow up) to avoid extra work with rebasing all patches over and over? | 13:10 |
| tdasilva | cschwede: you mean typos to non-docs files? | 13:11 |
| cschwede | tdasilva: yes | 13:12 |
| *** nshaikh has quit IRC | 13:13 | |
| acoles | tdasilva: cschwede : morning/afternoon! | 13:15 |
| cschwede | Hello Alistair! | 13:16 |
| *** mahatic has joined #openstack-swift | 13:16 | |
| *** ppai has joined #openstack-swift | 13:16 | |
| *** ppai has joined #openstack-swift | 13:16 | |
| acoles | cschwede: idk, i think clayg is intending to do a rebase every night (whenever his night is !), so what might help most is to supply diffs with typos fixed that he can apply | 13:17 |
| acoles | cschwede: but i could be wrong! | 13:17 |
| cschwede | acoles: ah, ok, makes sense too. i’m just reviewing my previous +2, checking the diffs and re-apply my +2 if it looks good to me | 13:18 |
| acoles | cschwede: tdasilva : one think clayg does want help with is triage on his bullet lists on patch 169988 and patch 169989 in particular | 13:18 |
| patchbot | acoles: https://review.openstack.org/#/c/169988/ | 13:18 |
| cschwede | yes, i saw that - it’s on my list | 13:19 |
| tdasilva | acoles: and by triage we mean "must fix" or "add to trello for liberty" right? | 13:20 |
| acoles | tdasilva: cschwede : i'm wondering how best to discuss/agree those points - maybe an etherpad with a section for each review/point? | 13:20 |
| acoles | tdasilva: exactly - we need to sift out what *has* to be done before release vs 'must have but can wait' | 13:20 |
| *** mahatic has quit IRC | 13:21 | |
| tdasilva | acoles: I think that could be done as comments on the review itself...makes it easier I think to lookup later on why we made a decision | 13:21 |
| acoles | tdasilva: yep, i think you are right. maybe someone could add number to clay's points so we can refer to them | 13:22 |
| tdasilva | I think between trello and the review itself we can probably capture that info...i'm always hesitant to spread info in too many places | 13:22 |
| acoles | tdasilva: me too. post it note in front of my nose is the only one that persists :P | 13:23 |
| acoles | tdasilva: cschwede : so i *think* if we can give clayg some sense of 'must do'/'can wait' on those patches for tomorrow that will be a help | 13:24 |
| *** mahatic has joined #openstack-swift | 13:25 | |
| cschwede | acoles: tdasilva: yes, i was like „ok, any typos and small nits that don’t break functionality or tests“ fall in the nice-to-have category | 13:25 |
| acoles | cschwede: yes and feel free to add a trello card for something that needs to be done/discussed after kilo | 13:26 |
| acoles | 'after kilo'/'during kilo' idk, i mean after april 30th !) | 13:27 |
| tdasilva | cschwede: agree | 13:32 |
| tdasilva | cschwede: I wouldn't want to discourage people from keeping reviewing and finding the nits, but it's probably something that we can just record now and fix after the merge to master | 13:33 |
| *** Guest___ has joined #openstack-swift | 13:40 | |
| *** Guest___ has quit IRC | 13:51 | |
| *** openstackgerrit has quit IRC | 13:53 | |
| *** openstackgerrit has joined #openstack-swift | 13:53 | |
| peluse | acoles, so what are you working on right now then? | 13:58 |
| acoles | peluse: the fixes to delete reverted frags that clayg didn't get to last night | 13:59 |
| *** krykowski has joined #openstack-swift | 14:01 | |
| peluse | acoles, OK cool. I have another half day of mtgs this morning but will be back after noon. I didn't quite the follow the conversation/plan on categorizing/identifying required work to be done immeidately vs later. Can you summarize who is doing what? | 14:01 |
| *** Guest___ has joined #openstack-swift | 14:02 | |
| peluse | gotta run, back in a few hours... | 14:03 |
| acoles | peluse: sure, clayg has left some bullet points on reviews where he wants consensus on what needs to be done, now or later. tdasilva and cschwede are looking over those, i will too once i am done with this diff | 14:03 |
| acoles | then clayg can pick up again his morning | 14:03 |
| peluse | OK, will go through them starting at the top then when I get back. Thanks!! | 14:03 |
| *** krykowski_ has quit IRC | 14:04 | |
| *** ppai has quit IRC | 14:07 | |
| *** erlon has joined #openstack-swift | 14:10 | |
| *** jrichli has joined #openstack-swift | 14:13 | |
| *** lcurtis has joined #openstack-swift | 14:17 | |
| *** pcaruana has quit IRC | 14:17 | |
| *** joeljwright has joined #openstack-swift | 14:18 | |
| *** joeljwright1 has quit IRC | 14:20 | |
| *** Guest___ has quit IRC | 14:23 | |
| *** krykowski has quit IRC | 14:36 | |
| *** krykowski has joined #openstack-swift | 14:37 | |
| *** lpabon has joined #openstack-swift | 14:37 | |
| tdasilva | acoles: soo....changing my mind here...I think you are correct :) An etherpad might be easier to iterate things over, then we can go back and add comments (final decision) to either trello or the patch itself | 14:48 |
| tdasilva | acoles, cschwede: so I created this: https://etherpad.openstack.org/p/swift_ec_triage | 14:49 |
| tdasilva | peluse: ^^^ | 14:49 |
| cschwede | tdasilva: woa, awesome, thx! | 14:49 |
| *** eikke has quit IRC | 14:55 | |
| *** krykowski has quit IRC | 15:10 | |
| *** jistr is now known as jistr|mtg | 15:15 | |
| peluse | cool | 15:17 |
| *** fanyaohong has joined #openstack-swift | 15:21 | |
| *** joeljwright1 has joined #openstack-swift | 15:27 | |
| *** jistr|mtg is now known as jistr | 15:27 | |
| *** joeljwright has quit IRC | 15:28 | |
| acoles | tdasilva: nice! thx | 15:29 |
| *** mahatic has quit IRC | 15:33 | |
| *** takoTuesday has joined #openstack-swift | 15:39 | |
| *** jistr has quit IRC | 15:40 | |
| takoTuesday | cschwede: thanks for posting the demo to djangp-storage-swift | 15:40 |
| takoTuesday | cschwede: Im still not 100% sure that it is the best option for serving files from my swift container. I have a django application that Im trying to modify that pulls down source code from a git repo, generates documentation from that code, and then serves the documentation. In development mode it copys the documentation to a local dir and serves from there. So far, I have modified it to copy the documentation to a swift container, but now I | 15:43 |
| takoTuesday | need to figure out how to serve from that container | 15:43 |
| cschwede | takoTuesday: you’re welcome. no need for the module, just make the container public and access it directly then | 15:44 |
| cschwede | takoTuesday: make the container public with „swift post -r ".r:*" containername“, and get the URL with „swift stat -v containername | grep URL“ | 15:45 |
| *** eikke has joined #openstack-swift | 15:46 | |
| notmyname | good morning | 15:47 |
| eikke | good late-afternoon | 15:47 |
| notmyname | :-) | 15:48 |
| notmyname | looks like, again, everyone was hard at work while I was sleeping | 15:48 |
| *** SkyRocknRoll has quit IRC | 15:50 | |
| notmyname | dmorita brought a patch to my attention last nght | 15:51 |
| notmyname | https://review.openstack.org/#/c/138697/ would be good to get landed, if possible, on master before the release | 15:51 |
| *** mahatic has joined #openstack-swift | 15:53 | |
| notmyname | hello mahatic | 15:53 |
| mahatic | notmyname, hello! good morning. What's up? | 15:56 |
| mahatic | notmyname, I'm all set to travel for the summit! :) | 15:56 |
| notmyname | I'm just doing a little stuff before I catch a bus to the office | 15:56 |
| notmyname | mahatic: nice!! | 15:56 |
| *** gyee has joined #openstack-swift | 15:59 | |
| btorch | notmyname: you guys aware of a reaper + async bug where asyncs may stay on forever due to the reaper removing the an account db | 16:00 |
| notmyname | btorch: doesn't ring a bell. has there been a bug filed for it? | 16:01 |
| btorch | where do you guys keep issues ? I don't see it on gh | 16:03 |
| mahatic | notmyname, yeah, see ya'll there. Meanwhile this week I have been brushing up my linux kernel and C skills. I have been recently asked by someone if I'm interested to work in those areas. I'm revisiting to assess :D | 16:03 |
| notmyname | btorch: launchpad | 16:03 |
| mahatic | I should be back picking back on Swift next week :) | 16:03 |
| notmyname | btorch: https://bugs.launchpad.net/swift | 16:04 |
| notmyname | mahatic: cool | 16:04 |
| peluse | FYI I've oone through the first 4 on the etherpad and added my corresponding +2 on each accordingly https://etherpad.openstack.org/p/swift_ec_triage | 16:07 |
| *** ChanServ changes topic to "Soft freeze of master in effect | EC Merge plan: https://etherpad.openstack.org/p/ec_merge_plan | Review Dashboard: http://goo.gl/vysJqI | Summit scheduling: https://etherpad.openstack.org/p/liberty-swift-summit-topics | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/" | 16:16 | |
| notmyname | summit scheduling etherpad, as discussed in the meeting yesterday https://etherpad.openstack.org/p/liberty-swift-summit-topics | 16:16 |
| mahatic | notmyname, working sessions, will that be listed out in detail? Is that for all levels of contributors? | 16:22 |
| *** Nadeem has joined #openstack-swift | 16:24 | |
| notmyname | mahatic: we need to propose both the fishbowl and working session ideas. and yes, everything at the summit should be accessible to as many people as possible. | 16:25 |
| *** ozialien has quit IRC | 16:26 | |
| mahatic | ah okay | 16:26 |
| notmyname | that being said, we can't spend all our time on "intro to swift..." ;-) | 16:31 |
| notmyname | mahatic: but I have zero concerns about you with that | 16:31 |
| *** silor has quit IRC | 16:33 | |
| mahatic | notmyname, :) yeah, of course. I didn't have a great idea about the sessions, was just checking | 16:33 |
| *** foexle_ has quit IRC | 16:34 | |
| notmyname | ok, time to go to the office, I think | 16:36 |
| notmyname | be back online later | 16:36 |
| *** mriedem has joined #openstack-swift | 16:46 | |
| mriedem | so...a packaging question related to erasure codes and pyeclib - from a guy that doesn't know jack about swift or configuring it for this stuff, | 16:48 |
| mriedem | but with swift 2.2.2, are (1) erasure codes a thing officially since i still see reviews up for the bp and the docs talk about future support | 16:48 |
| *** tdasilva has quit IRC | 16:49 | |
| peluse | mriedem, we will be at beta | 16:49 |
| mriedem | ok, i'm on this thread b/c of https://github.com/openstack/swift/commit/3d3db0ab789f37a69fc178eeb74c8cc61e4c6c1b | 16:49 |
| peluse | so it will al work but there will likely be tweaking needed in various areas - performance for example | 16:49 |
| mriedem | ok, but still this is bleeding edge it sounds like | 16:49 |
| peluse | well, beta, yes | 16:50 |
| mriedem | the issue we've run into is when packaging pyeclib, it requires liberasurecode, gf-complete and jerasure: https://bitbucket.org/kmgreen2/pyeclib/src/bb07ee0dda5b33668db67f43467aaa514285b3cf/setup.py?at=v1.0.6#cl-211 | 16:51 |
| peluse | and the docs for EC are still in review as well here: https://review.openstack.org/#/c/169990/ | 16:51 |
| mriedem | for legal reasons, we can't ship gf-complete and jerasure | 16:51 |
| peluse | mriedem, there's another EC library option as well, ISA-L | 16:51 |
| mriedem | reading the docs on liberasure code, it sounds like those are pluggable | 16:52 |
| mriedem | http://www.bytebucket.org/tsg-/liberasurecode | 16:52 |
| mriedem | yeah | 16:52 |
| peluse | mriedem, and I think we do need a bit more in the docs on how to install ISA-L as a back end | 16:52 |
| mriedem | so pyeclib doesn't necessarily need gf-complete/jerasure | 16:52 |
| peluse | correct | 16:52 |
| mriedem | ok | 16:52 |
| mriedem | they must just be the default/reference libs? | 16:52 |
| peluse | pyeclib is a python front-end for liberacecode which is the pluggable framework that takes different EC lirbaries | 16:52 |
| peluse | correct, jerasure is the default | 16:53 |
| mriedem | ok, and it's setup.py is overzealous | 16:53 |
| mriedem | anyway, we can patch around that | 16:53 |
| peluse | let me grab some ISA-L directiosn for you real quick... one sec | 16:53 |
| peluse | download https://01.org/sites/default/files/downloads/intelr-storage-acceleration-library-open-source-version/isa-lopensrc2.13.tar.gz | 16:54 |
| peluse | make clean && make && make install | 16:54 |
| peluse | run "sudo ldconfig" | 16:54 |
| peluse | and then change your swift.conf to use isa-l instead of jerasure... pasting example here in a sec | 16:54 |
| mriedem | ah, ok, that backend config change was my question | 16:55 |
| mriedem | how everything knew i was not using jerasure | 16:55 |
| peluse | [storage-policy:2] | 16:55 |
| peluse | name = ec42 | 16:55 |
| peluse | default = yes | 16:55 |
| peluse | policy_type = erasure_coding | 16:55 |
| peluse | ec_type = isa_l_rs_vand | 16:55 |
| peluse | ec_num_data_fragments = 4 | 16:55 |
| peluse | ec_num_parity_fragments = 2 | 16:55 |
| peluse | do all that and you won't be using jerasure :) | 16:55 |
| mriedem | if gf-complete a dep of jerasure then? | 16:55 |
| mriedem | looks like it | 16:56 |
| mriedem | given the jerasure docs | 16:56 |
| mriedem | ok | 16:56 |
| peluse | I think only if you want SIMD support and maybe some other schemes, I'm not an expert there though | 16:56 |
| mriedem | ok, that's all greek to me, i'm just packaging (or helping) :) | 16:56 |
| mriedem | but this has been super helpful | 16:57 |
| peluse | ISA-L is the Intel EC library that is optmized for Intel architecture | 16:57 |
| peluse | :) | 16:57 |
| mriedem | so if i were crazy and wanted to use this on ppc64 or s390x, no dice | 16:57 |
| mriedem | i'm not even sure if we support that | 16:57 |
| peluse | that would be nuts for sure :) | 16:57 |
| mriedem | ha, ok | 16:57 |
| mriedem | alright, well peluse thanks a ton! | 16:57 |
| peluse | just kidding, I'm not sure if that would work or not to tell you the truth. If it doesn't it would be massively slower | 16:57 |
| peluse | welcome! | 16:58 |
| *** delattec has joined #openstack-swift | 16:58 | |
| *** cdelatte has quit IRC | 17:01 | |
| *** delatte has quit IRC | 17:01 | |
| *** mmcardle has quit IRC | 17:03 | |
| *** takoTuesday has quit IRC | 17:06 | |
| acoles | clayg: in case you don't know, this exists thanks to tdasilva https://etherpad.openstack.org/p/swift_ec_triage | 17:06 |
| * notmyname back | 17:13 | |
| * peluse out for a bit, commute :) | 17:13 | |
| *** openstack has quit IRC | 17:13 | |
| *** openstack has joined #openstack-swift | 17:14 | |
| *** delattec has quit IRC | 17:15 | |
| *** tdasilva has joined #openstack-swift | 17:22 | |
| *** mriedem has left #openstack-swift | 17:23 | |
| *** jordanP has quit IRC | 17:24 | |
| *** ozialien has joined #openstack-swift | 17:25 | |
| *** shri1 has joined #openstack-swift | 17:28 | |
| *** fanyaohong has quit IRC | 17:29 | |
| *** zhill_ has joined #openstack-swift | 17:33 | |
| notmyname | tdasilva: that's a great etherpad! thanks! | 17:33 |
| *** geaaru has quit IRC | 17:35 | |
| *** zhill has joined #openstack-swift | 17:35 | |
| notmyname | acoles: do you know the hacking rule import issue clayg is referring to? https://review.openstack.org/#/c/169988/ | 17:36 |
| notmyname | acoles: you just added those bullets on the etherpad | 17:36 |
| *** zhill_ has quit IRC | 17:36 | |
| *** aix has quit IRC | 17:37 | |
| acoles | notmyname: thinks its whether any rule would enforce using (multi line imports) vs. \ separated | 17:37 |
| acoles | notmyname: and if not then do we enforce it manually in review as swift style thing | 17:37 |
| notmyname | I think there is a hacking rule about that, but we haven't added it to what we gate on | 17:38 |
| notmyname | IMO I don't really care. I prefer multiline with \ instead of () but it doesn't really matter to me | 17:38 |
| acoles | notmyname: k. well i vote for discussing that 'later' | 17:38 |
| notmyname | acoles: oh I 100% agree that it's a distraction for now :-) | 17:38 |
| acoles | heh. i prefer (). there you go! one for over beers... | 17:38 |
| notmyname | which is why I was curious to see it on a list of questions to address | 17:39 |
| acoles | notmyname: those are clayg's bullets from the patch review | 17:39 |
| *** gyee has quit IRC | 17:40 | |
| *** vinsh has quit IRC | 17:40 | |
| notmyname | torgomatic: you had said something in here (IRC) about why you didn't want to use swift.common.utils json (and instead imported directly) | 17:40 |
| notmyname | torgomatic: I totally get the "let's stop using simlplejson asap" but isn't it easier to switch out if we all depend on one import? | 17:41 |
| notmyname | ie swift.common.util.json? | 17:41 |
| tdasilva | notmyname: it was acoles idea, so thanks to him :-) | 17:41 |
| notmyname | thanks acoles | 17:41 |
| * tdasilva read something the other day about shifting blame :P | 17:42 | |
| acoles | tdasilva: lol | 17:42 |
| notmyname | tdasilva: "How to management"? | 17:43 |
| tdasilva | 101 | 17:43 |
| torgomatic | notmyname: harder, not easier; the problem is that you have to carefully test each consumer of JSON-serialized data to make sure the introduction of unicode-all-the-time behavior doesn't goof it up | 17:43 |
| notmyname | torgomatic: ie not using a common place makes it easier to do "rolling upgrade" patches? | 17:43 |
| torgomatic | simply running "perl -pi -e 's/simplejson/json/g'" across the Swift source tree would be far too big for anyone to test the whole thing | 17:44 |
| torgomatic | notmyname: exactly; that way, I can start with, say, SLO, and make it use stdlib json, and then functionally test it with various object and container names in there | 17:45 |
| torgomatic | and then tempauth | 17:45 |
| torgomatic | and then X, and then Y, and so on until it's done | 17:45 |
| notmyname | torgomatic: ok, so what about the fact that right now you're only importing json and not the try/except with simplejson? | 17:46 |
| torgomatic | notmyname: that's one fewer place I have to go change later | 17:46 |
| torgomatic | it's a small payload, so the speedup from simplejson is negligible | 17:47 |
| notmyname | torgomatic: what issues does it cause today with still nominally supporting py26? | 17:47 |
| *** rmcall has joined #openstack-swift | 17:47 | |
| notmyname | ah ok | 17:47 |
| torgomatic | and the speedup only applies to python 2.6 anyway; stdlib json and simplejson perform very similarly in 2.7. | 17:47 |
| *** dencaval has quit IRC | 17:48 | |
| *** vinsh has joined #openstack-swift | 18:01 | |
| *** joeljwright has joined #openstack-swift | 18:02 | |
| *** joeljwright has quit IRC | 18:03 | |
| clayg | morning! | 18:04 |
| *** ultgeek has joined #openstack-swift | 18:04 | |
| *** joeljwright1 has quit IRC | 18:05 | |
| jrichli | clayg: morning | 18:07 |
| *** proteusguy_ has joined #openstack-swift | 18:07 | |
| ultgeek | Do you anticipate Swift 2.3.0 being released prior to 4/30 Kilo release? | 18:08 |
| clayg | I like () better - hacking has no rules for those kinds of imports because they say always import he module - my point was just that we can't make a thing about it if we don't enforce it | 18:09 |
| clayg | so even if we had an opionin (which we don't) - step 0 would be to write a script that can enforce it | 18:09 |
| *** proteusguy has quit IRC | 18:11 | |
| *** proteusguy_ has quit IRC | 18:12 | |
| *** takoTuesday has joined #openstack-swift | 18:13 | |
| clayg | cschwede: I think you asked about fixing nits in the review chain - please paste/diff's only unless it's docs | 18:15 |
| *** ultgeek has left #openstack-swift | 18:16 | |
| clayg | that py26 failure is surprising | 18:16 |
| clayg | was someone saying they were having trouble with deadsnakes? vagrant-swift-all-in-one should square it for you - i forget what all had to be done - it was something like add a key for the ppa, add the ppa, apt-get install the package | 18:17 |
| clayg | god i am *so* sick of seeing installs of python libraries fail because of pbr | 18:19 |
| clayg | fucking python has been around for 20 years - we are *not* supposed to be one of those languages that doesn't have packing in a reliable working state - regradless of cruft - it should fucking *work* | 18:20 |
| *** proteusguy_ has joined #openstack-swift | 18:21 | |
| acoles | clayg: hi. well i didn't get as far as i'd hoped but there's diff linked off patch 170339 | 18:34 |
| patchbot | acoles: https://review.openstack.org/#/c/170339/ | 18:34 |
| clayg | acoles: you're a champ! | 18:34 |
| * clayg has the ball | 18:34 | |
| clayg | torgomatic: can you chime in on the patch 169989 changeset 4 proxy.controllers.obj L1272 - there's a conversation ongoing and you should make the call | 18:35 |
| patchbot | clayg: https://review.openstack.org/#/c/169989/ | 18:35 |
| acoles | clayg: when i applied your diff (from your gist) i saw KeyErrors - see my comment on patch - due to ring.get_more_nodes not adding the node[index] key :? | 18:35 |
| clayg | fuck that noise - I don't want no indexes in my handoff nodes - they're no good for anything but confusing the issue | 18:36 |
| clayg | acoles: which patch is the good one (the one you wrote) - diff_wrt_first_patch or first_patch | 18:39 |
| clayg | I think first_patch (i.e. the second one) | 18:39 |
| acoles | clayg: first_patch is good, tests all pass | 18:39 |
| clayg | perfecto! | 18:40 |
| clayg | k, i'm applying now | 18:40 |
| acoles | clayg: diff_wrt_first_patch has your diffs manually applied to first_patch, tests fail | 18:40 |
| *** BAKfr has quit IRC | 18:40 | |
| clayg | damnit - i don't know how to apply email formatted patches :P | 18:40 |
| torgomatic | clayg: blathering now | 18:41 |
| clayg | I normally just pipe the diff to git apply | 18:41 |
| acoles | clayg: git apply | 18:41 |
| acoles | just works no? | 18:41 |
| clayg | acoles: like curl gist | git apply - says it doesn't like it - i'm assuming it's because it's email formatted? | 18:41 |
| acoles | let me try it omm | 18:42 |
| *** ozialien has quit IRC | 18:42 | |
| acoles | clayg: works from local file | 18:43 |
| *** BAKfr has joined #openstack-swift | 18:44 | |
| clayg | yeah i had to -3 - fixing up conflicts now - maybe my working tree is whack? | 18:44 |
| *** ozialien has joined #openstack-swift | 18:44 | |
| clayg | acoles: doh! yeah I had my wip-fix-ec-recon changes let me get back in line with reality | 18:45 |
| clayg | sorry for the noise | 18:45 |
| acoles | clayg: oh crap, so diff_wrt_first_patch is only a diff from first_patch, not from the review | 18:47 |
| acoles | clayg: you'll have to stack them | 18:47 |
| clayg | i'll get it squared - thanks a ton | 18:48 |
| acoles | clayg: ok you have the ball. imho we could leave the durable delete in purge() for timebeing, in favor of other must-have stuff. it bugs me i didn't get it done but hey! | 18:49 |
| clayg | acoles: np, well see how far I get today | 18:49 |
| acoles | clayg: k, i'm calling it a day | 18:50 |
| *** mahatic has quit IRC | 18:53 | |
| *** aix has joined #openstack-swift | 18:55 | |
| clayg | acoles night' sorry to keep you up | 18:55 |
| *** Nadeem has quit IRC | 19:01 | |
| acoles | clayg: just spotted, ObjectReplicator.ssync() in my patch needs to check in_sync_objs is not empty before the zip | 19:07 |
| acoles | clayg: oops | 19:07 |
| clayg | acoles: I'll find it - now worries - thanks for the heads up | 19:08 |
| *** acoles is now known as acoles_away | 19:08 | |
| clayg | oh i see - my pbr problems was because I was running an aggregiously old distrobution like *precise* | 19:14 |
| clayg | ya'll remember when we used to pretend we could support lucid against the onslaught of infra's "newest all the things"! | 19:15 |
| * clayg was so naive | 19:15 | |
| clayg | torgomatic: so... json | 19:17 |
| clayg | whoa aws efs - nice | 19:20 |
| egon | like Manila? | 19:25 |
| *** Nadeem_ has joined #openstack-swift | 19:27 | |
| *** tacotuesday has joined #openstack-swift | 19:37 | |
| *** proteusguy has joined #openstack-swift | 19:38 | |
| *** takoTuesday has quit IRC | 19:40 | |
| *** proteusguy has quit IRC | 19:44 | |
| *** thumpba has joined #openstack-swift | 19:58 | |
| *** thumpba_ has joined #openstack-swift | 19:59 | |
| *** proteusguy has joined #openstack-swift | 20:00 | |
| *** Nadeem_ has quit IRC | 20:01 | |
| *** thumpba has quit IRC | 20:02 | |
| torgomatic | clayg: so? | 20:03 |
| *** ozialien has quit IRC | 20:03 | |
| *** shri2 has joined #openstack-swift | 20:04 | |
| *** shri1 has quit IRC | 20:04 | |
| clayg | torgomatic: what? aws has cool tech - i find it interesting | 20:05 |
| torgomatic | clayg: oh, not that. I was talking about "so... json" | 20:06 |
| clayg | oh - sorry - yeah ummm... so swift.common.utils is like a bad idea? | 20:06 |
| clayg | well, rather from swift.common.utils import json is a bad idea | 20:08 |
| tacotuesday | okay guys I think I almost have this, still trying to serve docs from my swift container (set to globally readable). Im in the process of learning python/django as well. I am modifying an application that uses django.views.static serve to serve the files | 20:08 |
| clayg | ? | 20:08 |
| *** shri2 has quit IRC | 20:08 | |
| tacotuesday | but putting the raw url for basepath doesnt work as an argument for serve | 20:09 |
| tacotuesday | is there a simpler way to redirect to my swift container objects than serve? | 20:09 |
| tacotuesday | I think I need whatever object serve returns though | 20:09 |
| tdasilva | peluse, clayg: quick question: I'm trying to review reconstructor code and keep going back between doc and the code itself. One thing is not clear to me, we say a lot that a node will contact only it's adjacent nodes to "sync", but I'm assuming that if it needs to reconstruct a whole fragment archive, then it would need to get at least ec_ndata fragments, so it would need to connect to a lot more nodes, what am i miss | 20:13 |
| tdasilva | ing? | 20:13 |
| peluse | the "contact" we're talking about in the docs there is to compare hashes, not to actually perform reconstruction | 20:17 |
| peluse | because for sure you have to go to enough survinving nodes to read in their frag archives :) | 20:17 |
| tdasilva | peluse: thanks | 20:22 |
| peluse | sure | 20:24 |
| peluse | clayg, on that py26 deadsnakes thing that was me - its a proxy problem when I'm at work. Just got home and got it fine, ran full tox and that patch and it runs fine (updated ether as well) | 20:25 |
| *** ozialien has joined #openstack-swift | 20:26 | |
| *** tsg has joined #openstack-swift | 20:33 | |
| *** delatte has quit IRC | 20:33 | |
| *** ozialien has quit IRC | 20:34 | |
| tacotuesday | clayg: can you comment on how to access objects in my swift container via url instead of file path? I have a serve function that serves from a filepath but I would like to routh that to my swift container url | 20:40 |
| *** lpabon has quit IRC | 20:46 | |
| *** Nadeem_ has joined #openstack-swift | 20:58 | |
| *** zhill has quit IRC | 21:00 | |
| *** Nadeem_ has quit IRC | 21:13 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: Compare each chunk of large objects when downloading https://review.openstack.org/172212 | 21:17 |
| *** tacotuesday has quit IRC | 21:21 | |
| *** vinsh has quit IRC | 21:39 | |
| *** gyee has joined #openstack-swift | 21:41 | |
| *** jkugel has quit IRC | 21:45 | |
| *** jkugel has joined #openstack-swift | 21:45 | |
| *** jkugel has quit IRC | 21:50 | |
| clayg | tacotuesday - you left! | 22:00 |
| clayg | peluse: what's the difference between flat_xor_hd_4 and flat_xor_hd_3? | 22:02 |
| klrmn | clayg: maybe they are changing their nick to tacothursday? | 22:02 |
| *** vinsh has joined #openstack-swift | 22:02 | |
| clayg | peluse: tsg: yuan: I'm getting pyeclib.Error: Invalid arguments passed to liberasurecode_instance_create instead of the expected ECDriverError on pyeclib 1.0.6 when I try to specifiy flat_xor_hd_4 or flat_xor_hd_3 instead of jerasure_rs_vand | 22:06 |
| tsg | clayg: looking .. 1 moment | 22:08 |
| *** vinsh has quit IRC | 22:10 | |
| *** vinsh_ has joined #openstack-swift | 22:10 | |
| *** zhill has joined #openstack-swift | 22:11 | |
| tsg | clayg: there is restriction on what k and m values can be used with flat_xor_hd | 22:13 |
| tsg | clayg: the minimum we test with is (6, 6) .. but I am trying to find out if we can use another | 22:13 |
| clayg | something that adds up to 6 would be ideal :\ | 22:13 |
| tsg | clayg: ok :) /me checking | 22:14 |
| *** jrichli has quit IRC | 22:17 | |
| tsg | clayg: the minimum we have implemented in the XOR liberasurecode backend is (5, 5) | 22:20 |
| tsg | I am checking with Kevin if we can quickly add a (3, 3) | 22:21 |
| tsg | (it should just be adding a new spec to a header) | 22:21 |
| clayg | tsg: may be moot if it's not already in 1.0.6 | 22:21 |
| clayg | w | 22:21 |
| tsg | clayg: we are about to release 1.0.7 for some fixes Pete suggested - and Kevin is getting ready to add (3, 3) to that codebase along with some unit tests | 22:22 |
| tsg | but for now, (5, 5) should work | 22:22 |
| notmyname | yo. clayg just caught me up to what tsg and peluse are talking about ? | 22:25 |
| clayg | i think vagrant-swift-all-in-one can do it easy! | 22:26 |
| clayg | ... probetests won't pass :\ | 22:26 |
| notmyname | tsg: if xor codes require 10 devs (for now), let's stick with jerasure in the sample config. and document jerasure + isa-l + xor. | 22:26 |
| *** vinsh has joined #openstack-swift | 22:27 | |
| *** vinsh_ has quit IRC | 22:27 | |
| notmyname | and if we can simplify dependencies later (and keep the saio requirements reasonable), then let's update it later | 22:27 |
| tsg | notmyname, clayg: sounds fine to me. the only reason this is an issue is because for some reason, folks are using the pypi version of pyeclib to create their rpms/debs (which includes jerasure) | 22:28 |
| notmyname | tsg: "some reason" ;-) | 22:28 |
| clayg | tsg: convience? | 22:28 |
| notmyname | tsg: like, maybe because that's where everyone gets pything dependencies? ;-) | 22:28 |
| tsg | notmyname, clayg: zaitcev categorically told me "no pypi" for Red hat packaging :) | 22:29 |
| clayg | zaitcev is always saying stuff like that | 22:29 |
| peluse | never a dull moment! | 22:32 |
| notmyname | well yeah, don't make your packages pull from pypi. nobody wants that | 22:32 |
| notmyname | but pypi is where people look for dependencies that are newer versions that what's available from a distro | 22:33 |
| mattoliverau | Morning | 22:34 |
| notmyname | hi mattoliverau | 22:35 |
| mattoliverau | Looks like everyone was busy again last night | 22:37 |
| notmyname | mattoliverau: https://etherpad.openstack.org/p/swift_ec_triage is a pretty good summary (courtesy of tdasilva and acoles_away) | 22:38 |
| mattoliverau | Thanks, saw that in scroll back, nice work tdasilva! | 22:40 |
| notmyname | mattoliverau: also, it's friday for you, so you're time for ec is almost up! ;-) | 22:42 |
| clayg | oh - wait - I think we need something about the list re: how much data to expose in /info about ec policies? | 22:43 |
| mattoliverau | notmyname: lol, it is Friday, but the wife is in Spain so I'm free to work on Saturday ;) | 22:44 |
| notmyname | whoa. just a little weekend jaunt to 12 time zones away ;-) | 22:45 |
| clayg | aww yeah 12 devices 10 replicas flat_xor_hd_4 (5,5) was pretty easy on vagrant swift all in one | 22:46 |
| *** Trixboxer has quit IRC | 22:49 | |
| mattoliverau | notmyname: lol, yeah, you know how it is, a quick trip to barcelona every now and again is good for the frequent flying points :P | 22:49 |
| *** gyee has quit IRC | 22:51 | |
| *** thumpba has joined #openstack-swift | 23:06 | |
| *** thumpba_ has quit IRC | 23:08 | |
| MooingLemur | it would be nice if swift-object-auditor -o -z would exit with a different exit status than if it exited by finishing the pass. | 23:09 |
| MooingLemur | I can send it a TERM and it exits with status 0, so I don't know if it actually completed, or got interrupted | 23:10 |
| MooingLemur | is there a better way (besides tailing syslog? | 23:10 |
| MooingLemur | ehh, such a kludge: swift-object-auditor /etc/swift/object-server.conf -z 100 -d R17 -v -o 2>&1 | fgrep -q '"once" mode completed'; echo $? | 23:15 |
| MooingLemur | :) | 23:15 |
| *** gyee has joined #openstack-swift | 23:16 | |
| *** lcurtis has quit IRC | 23:18 | |
| mattoliverau | But a return of 0 is success and >0 is usually an error.. So 0 means it successfully finished | 23:19 |
| MooingLemur | What I'm trying to achieve is making sure a zbf audit completes after a crash. Before mounting, I am testing the mount dev with file -Ls and grep to see if it needs ext4 journal recovery. If it does, drop a marker file, then mount it, and then fire off the zbf auditor. | 23:20 |
| MooingLemur | but only remove the marker file if the zbf pass actually finishes and not if it gets interrupted by an interim reboot | 23:20 |
| MooingLemur | I think my kludge will work | 23:20 |
| clayg | i think swift-ring-builder has that weird thing where there's a "success but with important detail" "errorcode" | 23:21 |
| clayg | I think drive-audit also has some fancy return code | 23:21 |
| *** mjfork has joined #openstack-swift | 23:28 | |
| *** mjfork has quit IRC | 23:29 | |
| clayg | I need to take off, and won't have internet - I'll be working on reconstructor changes I didn't get to day wrt job building | 23:33 |
| clayg | ... then in the am I have someone coming by to work on my furnace | 23:34 |
| *** zhill has quit IRC | 23:34 | |
| clayg | acoles_away: I'm going to focus adding the job/part/suffix handling - building your revert cleanup | 23:34 |
| notmyname | clayg: will you be pushing new patch sets tonight? | 23:34 |
| notmyname | clayg: do you have the internets yet? | 23:34 |
| clayg | notmyname: don't get a head of the story - new patch set coming up in a few | 23:34 |
| *** thumpba_ has joined #openstack-swift | 23:34 | |
| notmyname | lol, ok :-) | 23:34 |
| notmyname | clayg: now you know how I feel in the meetings when you start doing that ;-) | 23:35 |
| clayg | notmyname: basically all the nits we checked off the etherpad, plus sam's fixes, and acoles fixes | 23:35 |
| *** jrichli has joined #openstack-swift | 23:35 | |
| clayg | notmyname: lol - yeah i totally do that | 23:35 |
| clayg | umm... anyway - i guess that's it mostly | 23:36 |
| *** kei_yama has joined #openstack-swift | 23:36 | |
| clayg | acoles_away: so if you want to keep chipping away at the purge stuff that'd be sweet - but mostly you'll probably have to review all my changes to your diff | 23:36 |
| clayg | I apparently think a set of 2-tuples isn't as good as a sict | 23:36 |
| clayg | s/sict/dict | 23:36 |
| clayg | OMG pep8 is slow | 23:36 |
| clayg | i'm sure it's fine right? | 23:36 |
| *** thumpba has quit IRC | 23:37 | |
| notmyname | sict? static dict. sounds interesting. it's like a dictionary, but faster! | 23:37 |
| notmyname | or in our case, it's probably a "swift dictionary" (and not fast, but it is really really durable) | 23:38 |
| clayg | mattoliverau: ok - new changes up - it's all you now | 23:38 |
| clayg | notmyname: did we talk about how we're not going to be done tomorrow? | 23:38 |
| notmyname | clayg: ya. targeting tuesday | 23:39 |
| clayg | notmyname: it's not that I don't think we have a ton of patches that are already getting the +2's the deserve (but like we're also still targeting a few things in etherpad) | 23:39 |
| clayg | oh, yeah I made _reroute raise a 503 | 23:40 |
| *** zhill has joined #openstack-swift | 23:40 | |
| clayg | torgomatic: how did we not get an answer on common.utils.json? | 23:40 |
| mattoliverau | clayg: thanks will take a look, have a great evening | 23:50 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!