| *** mmcardle has joined #openstack-swift | 00:16 | |
| *** mmcardle1 has joined #openstack-swift | 00:18 | |
| *** mmcardle has quit IRC | 00:18 | |
| *** matsuhashi has joined #openstack-swift | 00:22 | |
| *** mmcardle1 has quit IRC | 00:23 | |
| *** mmcardle has joined #openstack-swift | 00:26 | |
| *** mmcardle has quit IRC | 00:31 | |
| *** matsuhashi has quit IRC | 00:32 | |
| *** matsuhas_ has joined #openstack-swift | 00:34 | |
| openstackgerrit | Alex Pecoraro proposed a change to openstack/swift: Allow hostname for nodes in Ring https://review.openstack.org/74542 | 00:47 |
|---|---|---|
| *** bsdkurt has quit IRC | 01:05 | |
| *** russellb has quit IRC | 01:05 | |
| *** bsdkurt has joined #openstack-swift | 01:06 | |
| *** jergerber has joined #openstack-swift | 01:21 | |
| *** zaitcev has quit IRC | 01:21 | |
| *** mmcardle has joined #openstack-swift | 01:28 | |
| *** mmcardle has quit IRC | 01:32 | |
| *** sfineberg has joined #openstack-swift | 01:34 | |
| *** nosnos has joined #openstack-swift | 01:36 | |
| *** shri1 has quit IRC | 01:42 | |
| *** mmcardle has joined #openstack-swift | 02:29 | |
| *** byeager has joined #openstack-swift | 02:29 | |
| *** mmcardle has quit IRC | 02:33 | |
| *** Cotes has quit IRC | 02:35 | |
| *** zackf has quit IRC | 02:37 | |
| *** jergerber has quit IRC | 02:42 | |
| creiht | mjseger: I still plan on doing more testing, that's just what I have done so far | 02:42 |
| *** peluse has quit IRC | 03:11 | |
| *** peluse has joined #openstack-swift | 03:12 | |
| *** yuan has quit IRC | 03:13 | |
| *** byeager has quit IRC | 03:20 | |
| openstackgerrit | Luis de Bethencourt proposed a change to openstack/python-swiftclient: make the help strings constant https://review.openstack.org/74578 | 03:24 |
| *** matsuhas_ has quit IRC | 03:29 | |
| *** mmcardle has joined #openstack-swift | 03:30 | |
| *** mmcardle has quit IRC | 03:34 | |
| *** byeager has joined #openstack-swift | 03:55 | |
| *** pandemicsyn has quit IRC | 04:00 | |
| *** gyee has quit IRC | 04:00 | |
| *** pandemicsyn has joined #openstack-swift | 04:07 | |
| *** ppai has joined #openstack-swift | 04:22 | |
| *** mmcardle has joined #openstack-swift | 04:32 | |
| *** matsuhashi has joined #openstack-swift | 04:35 | |
| *** mmcardle has quit IRC | 04:36 | |
| *** jokke_ has quit IRC | 04:43 | |
| *** jokke_ has joined #openstack-swift | 04:43 | |
| *** byeager has quit IRC | 04:49 | |
| *** yuan has joined #openstack-swift | 05:01 | |
| *** chandan_kumar has joined #openstack-swift | 05:26 | |
| *** nshaikh has joined #openstack-swift | 05:27 | |
| *** nshaikh has left #openstack-swift | 05:28 | |
| *** mmcardle has joined #openstack-swift | 05:34 | |
| *** mmcardle has quit IRC | 05:38 | |
| *** haomaiwang has quit IRC | 06:04 | |
| *** haomaiwang has joined #openstack-swift | 06:04 | |
| *** nshaikh has joined #openstack-swift | 06:22 | |
| *** mmcardle has joined #openstack-swift | 06:34 | |
| *** matsuhashi has quit IRC | 06:35 | |
| *** matsuhashi has joined #openstack-swift | 06:37 | |
| *** mmcardle has quit IRC | 06:39 | |
| *** Dharmit has joined #openstack-swift | 06:39 | |
| *** Dharmit has quit IRC | 06:45 | |
| *** Dharmit has joined #openstack-swift | 06:50 | |
| *** haomaiw__ has joined #openstack-swift | 07:12 | |
| *** haomaiwang has quit IRC | 07:13 | |
| *** basha has joined #openstack-swift | 07:16 | |
| *** chandan_kumar has quit IRC | 07:18 | |
| *** cschwede has left #openstack-swift | 07:20 | |
| *** basha has quit IRC | 07:20 | |
| *** nshaikh has quit IRC | 07:21 | |
| *** chandan_kumar has joined #openstack-swift | 07:25 | |
| *** Dharmit has quit IRC | 07:28 | |
| *** saju_m has joined #openstack-swift | 07:30 | |
| *** mmcardle has joined #openstack-swift | 07:36 | |
| *** mmcardle has quit IRC | 07:40 | |
| *** mmcardle has joined #openstack-swift | 07:50 | |
| *** basha has joined #openstack-swift | 08:08 | |
| *** cschwede has joined #openstack-swift | 08:13 | |
| *** saju_m has quit IRC | 08:21 | |
| *** Dharmit has joined #openstack-swift | 08:30 | |
| *** openstack has joined #openstack-swift | 08:42 | |
| *** zackmdavis has joined #openstack-swift | 08:43 | |
| *** alpha_ori has joined #openstack-swift | 08:43 | |
| *** chandan_kumar has joined #openstack-swift | 08:44 | |
| *** mlipchuk has joined #openstack-swift | 08:44 | |
| *** acorwin has joined #openstack-swift | 08:44 | |
| *** joeljwright has joined #openstack-swift | 08:44 | |
| *** amandap has joined #openstack-swift | 08:45 | |
| *** creiht has joined #openstack-swift | 08:45 | |
| *** ChanServ sets mode: +v creiht | 08:45 | |
| *** foexle has joined #openstack-swift | 08:46 | |
| *** otherjon has joined #openstack-swift | 08:46 | |
| *** swifterdarrell has joined #openstack-swift | 08:46 | |
| *** ChanServ sets mode: +v swifterdarrell | 08:46 | |
| *** anderstj has joined #openstack-swift | 08:47 | |
| *** mlanner has joined #openstack-swift | 08:48 | |
| *** hugokuo has joined #openstack-swift | 08:48 | |
| *** koolhead17 has joined #openstack-swift | 08:50 | |
| *** basha has joined #openstack-swift | 08:50 | |
| *** nosnos_ has joined #openstack-swift | 08:59 | |
| *** nosnos has quit IRC | 08:59 | |
| *** tane-away is now known as tane | 09:00 | |
| *** tane is now known as tanee-away | 09:00 | |
| *** tanee-away is now known as tane | 09:03 | |
| *** tane is now known as tanee | 09:03 | |
| *** chandan_kumar has quit IRC | 09:23 | |
| *** fbo_away is now known as fbo | 09:37 | |
| *** chandan_kumar has joined #openstack-swift | 09:41 | |
| *** chandankumar_ has joined #openstack-swift | 09:42 | |
| *** Midnightmyth has joined #openstack-swift | 09:45 | |
| *** chandan_kumar has quit IRC | 09:46 | |
| *** matsuhashi has quit IRC | 09:46 | |
| *** nosnos_ has quit IRC | 09:46 | |
| *** nosnos has joined #openstack-swift | 09:47 | |
| *** matsuhashi has joined #openstack-swift | 09:47 | |
| *** saju_m has joined #openstack-swift | 09:58 | |
| *** mlipchuk has quit IRC | 10:04 | |
| *** jsuchome has joined #openstack-swift | 10:04 | |
| tristanC | 'morning! | 10:13 |
| *** mlipchuk has joined #openstack-swift | 10:21 | |
| *** chandankumar_ has quit IRC | 10:32 | |
| *** mlipchuk1 has joined #openstack-swift | 10:43 | |
| *** mlipchuk has quit IRC | 10:47 | |
| *** chandankumar_ has joined #openstack-swift | 10:47 | |
| *** mkollaro has joined #openstack-swift | 10:49 | |
| *** Trixboxer has joined #openstack-swift | 10:56 | |
| *** basha has quit IRC | 11:11 | |
| *** chandankumar_ has quit IRC | 11:14 | |
| *** chandan_kumar has joined #openstack-swift | 11:15 | |
| *** matsuhashi has quit IRC | 11:28 | |
| *** matsuhashi has joined #openstack-swift | 11:31 | |
| *** ppai has quit IRC | 11:34 | |
| *** ctennis has quit IRC | 11:35 | |
| *** ctennis has joined #openstack-swift | 11:35 | |
| *** ppai has joined #openstack-swift | 11:51 | |
| *** Anju has quit IRC | 11:57 | |
| *** saurabh_ has quit IRC | 11:58 | |
| *** matsuhashi has quit IRC | 12:20 | |
| *** matsuhashi has joined #openstack-swift | 12:25 | |
| *** ppai has quit IRC | 12:40 | |
| mjseger | creiht: cschwede: good morning | 12:44 |
| cschwede | mjseger: good morning! | 12:44 |
| mjseger | I've been thinking a little more about why we're seeing differences and wonder if it might be configuration/version related? | 12:44 |
| cschwede | version related? you mean swift itself? | 12:48 |
| tristanC | mjseger: I gave getput.py a try and found similar results between 1.9 and 2.0. What is the worst case scenario in your environment ? | 12:52 |
| mjseger | swift and others things. for example we have separate physical proxy and object servers | 12:53 |
| mjseger | and we run container services on the object servers | 12:53 |
| tristanC | mjseger: I also experiments with requests.Session, could you try to apply this patch: https://review.openstack.org/#/c/74444/ ? | 12:55 |
| mjseger | unfortunately I don't really have any control over the software itself so I'm going to have to ask someone else to deploy that and get back to you. sorry... | 12:56 |
| tristanC | mjseger: it's for swiftclient | 12:56 |
| *** nosnos has quit IRC | 12:56 | |
| mjseger | hmm, when I try to access that page I'm getting an application error/page not found. I am logged in. | 12:58 |
| cschwede | tristanC: is that patch marked as draft? can't access it... | 12:58 |
| tristanC | oups sorry, I put it in review mode, it should be ok now | 12:59 |
| *** koolhead17 has quit IRC | 13:01 | |
| *** koolhead17 has joined #openstack-swift | 13:01 | |
| *** mlipchuk1 has quit IRC | 13:03 | |
| *** matsuhashi has quit IRC | 13:08 | |
| *** matsuhashi has joined #openstack-swift | 13:08 | |
| *** matsuhashi has quit IRC | 13:08 | |
| *** Anju has joined #openstack-swift | 13:10 | |
| *** saurabh_ has joined #openstack-swift | 13:10 | |
| cschwede | tristanC: http://paste.openstack.org/show/67263/ -> ~10% faster for puts | 13:15 |
| tristanC | That's good to hear... I'm having a hard time to get consistent results between two batch of tests on the same client version | 13:22 |
| cschwede | tristanC: yepp, me too, but then i increased the number of requests to 1000 and now the results are quite consistent | 13:25 |
| *** Anju has quit IRC | 13:26 | |
| *** saurabh_ has quit IRC | 13:27 | |
| *** pconstantine has quit IRC | 13:33 | |
| *** mmcardle has quit IRC | 13:36 | |
| *** mmcardle has joined #openstack-swift | 13:36 | |
| *** mmcardle has quit IRC | 13:41 | |
| mjseger | tristanC: I finally got around to that patch and it does seem to make a BIG difference. I'm going to run some longer tests to make sure... | 13:41 |
| tristanC | mjseger: cool! thanks you! | 13:43 |
| mjseger | but wait, there's no joy... I ran a 30 second test and it's slow again. seems like those few at a higher rate weren't representative | 13:44 |
| mjseger | the real problem with this sort of thing is performance does vary a lot from run to run. now I'm doing some quick 100 object puts and they are performign well so maybe it was the 30 second test that wasn't representative | 13:45 |
| tristanC | mjseger: well even on a local vm (devstack + getput.py), results vary a lot, sometimes better, sometimes worst | 13:47 |
| mjseger | exactly | 13:47 |
| mjseger | continuing to test and things are looking brighter ;) | 13:47 |
| *** Anju has joined #openstack-swift | 13:47 | |
| *** saurabh_ has joined #openstack-swift | 13:48 | |
| mjseger | tristanC: so in our configuration, with 1.9 I tend to see about 20 IOPS for 1k puts and almost 40 for 2K. using your patch for 2.0 I'm seeing numbers for both in the mid 30s so I think I'm happy | 13:49 |
| mjseger | right now I'm runniing my 30 second test 5 times (another switch lets me do that ;)) and I'll post the results when it completes | 13:50 |
| mjseger | btw I'm not bothering with unique container names, not because I don't want to use them but because I forgot to add the extra switch | 13:50 |
| tristanC | mjseger: my initial guess is that if you reuse the http_conn object it might improve performance (as session force connections pooling) | 13:51 |
| mjseger | not sure I understand. I get a connection and then do all my puts on that connection. is that what you mean or something else? | 13:52 |
| tristanC | yes | 13:52 |
| tristanC | (compared to a CLI usage where you do a 'put' per swift instance) | 13:53 |
| mjseger | I assume you're referring to things like curl, right or sticking the swift command in a loop or something like that? | 13:54 |
| mjseger | a real no-no | 13:54 |
| mjseger | check this out: http://paste.openstack.org/show/67268/ I'm a happy camper! this is with patch 74444 | 13:55 |
| mjseger | tristanC: have you found getput useful? | 13:57 |
| tristanC | mjseger: indeed it is useful to have a test to refer to for benchmarks | 14:01 |
| tristanC | mjseger: there is a meeting tonight for swift and I guess we are going to give swiftclient performance a topic | 14:02 |
| mjseger | if you get a chance, you shoudl have a look at the readme as it explains [i hope] how to set up suites of tests for running on parallel client | 14:02 |
| *** mmcardle has joined #openstack-swift | 14:04 | |
| mjseger | tristanC: what are the meeting coordinates? is it a conference bridge? | 14:04 |
| tristanC | mjseger: it's schedule at 19:00 UTC every wednesday, see https://wiki.openstack.org/wiki/Meetings/Swift | 14:05 |
| *** pconstantine has joined #openstack-swift | 14:05 | |
| mjseger | tristanC: ahh, so this is entirely on IRC then. I think I can make it | 14:07 |
| *** saju_m has quit IRC | 14:11 | |
| *** Midnightmyth has quit IRC | 14:23 | |
| *** dmsimard has joined #openstack-swift | 14:31 | |
| *** russellb has joined #openstack-swift | 14:47 | |
| *** mlipchuk has joined #openstack-swift | 14:56 | |
| *** lpabon has joined #openstack-swift | 15:00 | |
| *** zaitcev has joined #openstack-swift | 15:01 | |
| *** ChanServ sets mode: +v zaitcev | 15:01 | |
| *** Midnightmyth has joined #openstack-swift | 15:02 | |
| *** mkollaro has quit IRC | 15:02 | |
| *** mkollaro1 has joined #openstack-swift | 15:02 | |
| *** tongli has joined #openstack-swift | 15:05 | |
| *** openstackgerrit has joined #openstack-swift | 15:09 | |
| *** jergerber has joined #openstack-swift | 15:16 | |
| *** haomaiw__ has quit IRC | 15:17 | |
| *** zackf has joined #openstack-swift | 15:31 | |
| *** mkollaro has joined #openstack-swift | 15:32 | |
| *** haomaiwa_ has joined #openstack-swift | 15:32 | |
| *** otoolee has joined #openstack-swift | 15:33 | |
| *** mkollaro1 has quit IRC | 15:34 | |
| *** haomaiw__ has joined #openstack-swift | 15:37 | |
| *** haomaiwa_ has quit IRC | 15:39 | |
| openstackgerrit | Luis de Bethencourt proposed a change to openstack/python-swiftclient: make the help strings constant https://review.openstack.org/74578 | 15:41 |
| *** judd7 has joined #openstack-swift | 15:42 | |
| *** zackf has quit IRC | 15:44 | |
| *** fbo is now known as fbo_away | 15:50 | |
| luisbg | morning :) | 15:54 |
| notmyname | good morning everyone | 15:56 |
| *** joeljwright has quit IRC | 15:58 | |
| portante | notmyname: morning | 15:59 |
| luisbg | notmyname, morning :) | 15:59 |
| *** mlipchuk has quit IRC | 16:00 | |
| *** otoolee1 has joined #openstack-swift | 16:02 | |
| notmyname | what's new in the land of swift? | 16:03 |
| portante | notmyname: did some work on an optimization for the auditor to reduce disk-head seeks | 16:03 |
| *** otoolee1 has left #openstack-swift | 16:04 | |
| notmyname | ah, cool | 16:04 |
| portante | but have not had time to setup a good test environment to verify it actually works | 16:04 |
| notmyname | is it simple enough for creiht to like? ;-) (sorry creiht, couldn't resist) | 16:04 |
| portante | this is a play that comes from experience with a customer having a similar problem | 16:05 |
| portante | creiht: I am not going to propose it until I can show it works in my little test environment. :) | 16:05 |
| notmyname | that's always a good place to pull requirements from :-) | 16:05 |
| notmyname | portante: so I'm going to be in your area next week. how many toes am I going to lose to frostbite? | 16:06 |
| *** lpabon has quit IRC | 16:06 | |
| *** lpabon has joined #openstack-swift | 16:06 | |
| creiht | lol | 16:06 |
| portante | notmyname: it supposed to get to the 50s over the next copule of days, so bring more of that with you! | 16:06 |
| creiht | that's the type of auditor optimizing that I *want* to see :) | 16:06 |
| portante | creiht: did any more work on the bulk stat stuff proceed | 16:06 |
| portante | ? | 16:06 |
| creiht | bulk stat stuff? | 16:07 |
| notmyname | dfg's patch? | 16:07 |
| portante | using libxfs or something, bulk stat ioctl | 16:07 |
| notmyname | oh | 16:07 |
| creiht | oh | 16:07 |
| portante | oh | 16:07 |
| creiht | I have no idea | 16:07 |
| portante | redbo? | 16:07 |
| portante | maybe redbo worked on it at one point? | 16:07 |
| luisbg | notmyname, I am currently writing the media player simple web to play with the API, and sent a patch to fix some style/consistency issues | 16:07 |
| creiht | I know he has played around with it, but don't think he every got to anything official | 16:08 |
| luisbg | notmyname, since you asked "what is new?" | 16:08 |
| notmyname | luisbg: thanks. how's the media player project going? | 16:08 |
| luisbg | notmyname, learning how to do all I need with the API in efficient ways, by reading a lot of python-swiftclient code | 16:08 |
| notmyname | creiht: FYI I want to cover project IPA at the meeting today. if we as a group are pretty good with it, then I'll start talking about it with -infra. I'm imagining that there will be at least one more patch set to include comments/docstrings with stull like wkelly mentioned in his review (ie just like we have in swob and memcache) | 16:09 |
| luisbg | notmyname, but going well, got a bunch of albums in the storage, and I can retrieve them to play them with gstreamer | 16:09 |
| notmyname | luisbg: cool | 16:09 |
| redbo | the who with the what? I have a python interface to bulkstat that we use sometimes. | 16:09 |
| creiht | notmyname: k | 16:10 |
| luisbg | notmyname, and now thinking about what features could make the experience better | 16:10 |
| luisbg | notmyname, reading python-swiftclient code made me think though I should release this media player as a sample swift app, for well, having at least one sample app people can play with | 16:10 |
| luisbg | proof of concept, demo, etc | 16:11 |
| notmyname | I'd love to see the DiskFile stuff extended to include a full abstraction of the disk (ie replication and auditing too). that way we could have an xfs one that's more efficient and also ones for otherFS | 16:11 |
| notmyname | luisbg: that would be great | 16:11 |
| *** tanee is now known as tane-away | 16:13 | |
| luisbg | notmyname, :) doing it with pygtk now, since I had a simple python music player with that I co-wrote for a proof of concept a year ago | 16:14 |
| luisbg | notmyname, but ideally I should make it a html5 app, right? | 16:14 |
| portante | notmyname: let me know when you are in town | 16:17 |
| redbo | In theory you could "pip install xfs", then "xfs.bulkstat(mountpoint)". I need to figure out how to make the .c file get packaged in the .tar.gz so cython isn't required to build it. | 16:18 |
| notmyname | portante: going to be a whirlwind trip. fly into Hartford on Tuesday, NYC on Wed, and Philly on Friday | 16:18 |
| portante | redbo, we can manually define the data structures in python | 16:18 |
| portante | notmyname: you folks from large states, think that is considered "in your area"? :) | 16:19 |
| redbo | and hope they never change | 16:20 |
| portante | redbo, and then just make the ioctl calls directly maybe? | 16:20 |
| portante | well, there is that ... | 16:20 |
| notmyname | portante: heh. pretty much. it's all "somewhere up north, about 30 minutes for 6 different states" to me ;-) | 16:20 |
| redbo | I'm sure you could do it in pure python. But it'd be pretty ugly and you'd have to hope constants and data structures don't ever change. I'm fine with having my little binary module. | 16:22 |
| portante | :) | 16:24 |
| *** joeljwright has joined #openstack-swift | 16:28 | |
| *** chuck__ has joined #openstack-swift | 16:29 | |
| portante | redbo: have you made that bulk stat work available somewhere? | 16:29 |
| redbo | portante: https://github.com/redbo/python-xfs | 16:30 |
| portante | great, thanks | 16:30 |
| *** joeljwright has quit IRC | 16:39 | |
| *** gyee has joined #openstack-swift | 16:43 | |
| *** kris_h has joined #openstack-swift | 16:47 | |
| *** byeager has joined #openstack-swift | 16:48 | |
| *** byeager has quit IRC | 16:55 | |
| *** nacim has quit IRC | 17:05 | |
| *** jsuchome has quit IRC | 17:05 | |
| *** Dharmit has quit IRC | 17:07 | |
| *** zul has quit IRC | 17:08 | |
| *** chuck__ has quit IRC | 17:08 | |
| notmyname | swift team meeting is in about 2 hours in #openstack-meeting. I just updated the agenda https://wiki.openstack.org/wiki/Meetings/Swift | 17:09 |
| *** zul has joined #openstack-swift | 17:09 | |
| *** pberis_ has joined #openstack-swift | 17:12 | |
| *** pberis_ has quit IRC | 17:14 | |
| *** mjseger has quit IRC | 17:15 | |
| *** asselin_ has joined #openstack-swift | 17:15 | |
| *** asselin_ has left #openstack-swift | 17:17 | |
| *** chandankumar_ has joined #openstack-swift | 17:24 | |
| *** chandankumar_ has quit IRC | 17:27 | |
| *** ccarrizo has joined #openstack-swift | 17:31 | |
| *** gyee has quit IRC | 17:33 | |
| *** joeljwright has joined #openstack-swift | 17:35 | |
| *** shri has joined #openstack-swift | 17:36 | |
| *** joeljwright has quit IRC | 17:40 | |
| *** mvenesio has joined #openstack-swift | 17:41 | |
| *** basha has joined #openstack-swift | 17:48 | |
| notmyname | swift uncategorized errors in jenkins jobs: https://gist.github.com/notmyname/fe63420545f556cf6f7c | 18:01 |
| notmyname | clarkb: ^^ there are 3 there that look really weird to me, and most of the rest are "keystone didn't start" | 18:01 |
| *** tong_ has joined #openstack-swift | 18:05 | |
| *** tongli has quit IRC | 18:07 | |
| zaitcev | notmyname: What's the situation with swiftclient in the master nowadays? I remember I missed some kind of error when I reviewed the port to python-request, and I'm not talking about foo(..., files=content) that ate memory. We had some kind of worse problem that you apparently successfuly handled, but I completely blank on what it was. | 18:08 |
| notmyname | zaitcev: 2 things: the bin script didn't properly plumb through --insecure for v2, so it was broken against self-signed certs via the CLI (client.py worked). the other was that auth v1 just plain didn't work (client or CLI) with the --insecure setting. it wasn't plumbed through at all, so you couldn't do anything there | 18:10 |
| zaitcev | notmyname: Argh, sorry. One visit to "git log" and I have my answers... I was Tristan's fix, not yours. | 18:11 |
| zaitcev | notmyname: V.sorry for bothering you, thanks for clarification. | 18:11 |
| notmyname | yes, "argh" was close to how I reacted ;-) | 18:11 |
| notmyname | zaitcev: no bother. that's why I'm here :-) | 18:11 |
| zaitcev | yeah and it's like people who refuse to google before asking. kinda paniced here | 18:11 |
| notmyname | zaitcev: http://bit.ly/1hvIq2z | 18:12 |
| *** chandan_kumar has quit IRC | 18:12 | |
| *** basha has quit IRC | 18:13 | |
| notmyname | chmouel: when can we get tom on openstack reactions? https://lh3.googleusercontent.com/-1WsKe47_rVU/UwQYxK4iTyI/AAAAAAAAUzg/-3E1Oes0ANM/w1636-h1228-no/IMG_20140215_201702-MOTION.gif | 18:13 |
| pberis | Quick Q regarding object deletion: At what point is the object's entry in the containerdb deleted? | 18:13 |
| notmyname | pberis: when the container server gets the delete (pedantic, but let me explain) | 18:14 |
| notmyname | pberis: the object server will send a request to the container server after it deletes the object locally. but if that request fails, it will be queued for resending later. but also note that other replicas may have succeeded and replication may have propegated | 18:15 |
| zaitcev | notmyname: actually err, my question was different... *facepalm* I meant to ask how confident we are in current master that uses python-requests, now that 3 issues are fixed (Config-Length for Glance, auth v1 plumbing, files= to data=)? | 18:16 |
| notmyname | zaitcev: I'm confident in the current state of python-swiftclient 2.0.2 (wrt basic functionality support and the new feature of cert checking) | 18:18 |
| pberis | notmyname: So, the entry in the container is flagged as deleted until the first copy is physically deleted at which time the object server notifies the container ... regardless of presence of other copies? ... is this method applicable to containers removed from accounts as well? | 18:19 |
| *** mmcardle has quit IRC | 18:20 | |
| notmyname | pberis: no. the object is deleted. then the listing is updated (ie removed). if the listing update fails, it is retried again later. replication is also running in the background, so if one container replica gets the update, it may propagate before the retry happens | 18:22 |
| *** gvernik_ has joined #openstack-swift | 18:22 | |
| *** gvernik_ has quit IRC | 18:23 | |
| notmyname | pberis: I'm including that last detail simply to complete the picture. there are 3 ways a container listing gets updated: (1) synchronously with the request (2) asynchronously later (3) via replication, if another replica had methods 1 or 2 succeed | 18:23 |
| *** gvernik_ has joined #openstack-swift | 18:24 | |
| pberis | notmyname: OK ... got it. I gather that a similar process is used for removing the container listing from the accountdb ... bottom line, the lists contained in the containerdb and accountdb should not contain deleted instances, at least not for any extended period. Correct? | 18:25 |
| pberis | notmyname: IE entries are not simply flagged as deleted and left in the db ... | 18:29 |
| notmyname | correct (with handwavey about "any extended period") | 18:29 |
| notmyname | IIRC. (I should go look at that code) | 18:29 |
| pberis | notmyname: Guess it depends on what your definition of 'extended' is .. | 18:29 |
| pberis | notmyname: Thanks! | 18:32 |
| *** krtaylor has quit IRC | 18:34 | |
| *** krtaylor has joined #openstack-swift | 18:36 | |
| *** joeljwright has joined #openstack-swift | 18:36 | |
| *** donagh_ has joined #openstack-swift | 18:37 | |
| openstackgerrit | Alistair Coles proposed a change to openstack/swift: Fix invalid account acl generating 500 response. https://review.openstack.org/74417 | 18:37 |
| *** donagh has quit IRC | 18:39 | |
| *** donagh_ has quit IRC | 18:40 | |
| *** donagh has joined #openstack-swift | 18:40 | |
| *** joeljwright has quit IRC | 18:40 | |
| *** dtalton has joined #openstack-swift | 18:42 | |
| *** csd has joined #openstack-swift | 18:45 | |
| zaitcev | do we have a meeting in #openstack-meeting in a few minutes? | 18:51 |
| notmyname | zaitcev: yes | 18:52 |
| notmyname | pberis: looking through the code, I'm reminded of one other thing there. the container listing entry isn't immediately deleted. a "tombstone" row is inserted. I'm looking for when that is removed | 18:53 |
| *** joeljwright has joined #openstack-swift | 18:55 | |
| notmyname | pberis: the tombstone row is removed by replication after reclaim_age seconds have passed (defaults to 7 days) | 18:57 |
| notmyname | pberis: FWIW, this is exactly the same pattern as is used by objects themselves ("delete" by writing a tombstone, then remove the tombstone after a set time, default one week) | 18:58 |
| notmyname | ok, meeting time | 19:00 |
| notmyname | portante: around for the meeting? | 19:04 |
| openstackgerrit | Alex Pecoraro proposed a change to openstack/swift: Allow hostname for nodes in Ring https://review.openstack.org/74542 | 19:06 |
| cschwede | torgomatic: just thinking if it might be useful to include it in the container updater (to avoid walking all containers twice). well, i'll read your gist first and then start thinking :) | 19:14 |
| torgomatic | cschwede: imagine that you find 20 GB worth of misplaced objects; that'll put a nice big pause in your container update run :) | 19:14 |
| torgomatic | I'm not opposed to the idea though | 19:15 |
| creiht | what is the likely hood of this happening? | 19:15 |
| cschwede | torgomatic: ok, got it, makes a lot of sense to separete in that case :) | 19:15 |
| torgomatic | creiht: low | 19:15 |
| torgomatic | so maybe it should be all one daemon? | 19:15 |
| * torgomatic hasn't given this question enough thought | 19:16 | |
| creiht | just seems like a lot of work and stuff going on to catch something that isn't very likely to happen | 19:16 |
| creiht | or fix | 19:16 |
| creiht | dunno | 19:16 |
| creiht | need to think about it a bit | 19:16 |
| creiht | torgomatic: would it be better to have something that would report when this happen, then have a tool that can migrate objects from one policy to another? | 19:21 |
| torgomatic | creiht: I'm partial to automatically fixing it; if you've got a good ops team then moving stuff manually is fine, but not everyone has a good ops team | 19:22 |
| creiht | torgomatic: i would be for that if it were something that is likely to happen | 19:23 |
| omame | I take it as a compliment :) | 19:23 |
| torgomatic | creiht: I think it's more likely to happen in multi-region clusters | 19:25 |
| torgomatic | those slow flaky WAN links can give you little mini-splits all day long | 19:26 |
| *** judd7 has quit IRC | 19:26 | |
| gholt | Isn't .99999 one in 100,000 not a million, literally? On a serious note though, what is done about container servers arguing about what the storage policy is? | 19:27 |
| creiht | torgomatic: so what's bugging me a bit, is what is the use case or likely hood that a user (in a short period of time) is going to create the same container more than once on the separate network partitions with different storage policies? | 19:28 |
| torgomatic | gholt: bah, not enough nines ;) | 19:28 |
| gholt | :) | 19:28 |
| torgomatic | there's 1000 milliseconds per second and 1000 microseconds per millisecond, so it should work out to a million if I can do math :) | 19:29 |
| portante | creiht: I am not sure it is the case where somebody tries to make that happen, it seems to be more about the case where a mistake is made and then one can't recover | 19:29 |
| gholt | I can't really do math. I ask Google for help all the time. | 19:29 |
| *** judd7 has joined #openstack-swift | 19:29 | |
| creiht | portante: well that's what I'm trying to figure out is exactly what would have to happen to trigger this | 19:29 |
| creiht | I'm not saying that a user purposely is trying to cause an issue | 19:30 |
| portante | k | 19:30 |
| torgomatic | gholt: okay, I added more nines (if only it were always so easy...) | 19:30 |
| gholt | I'm trying to figure out how you have two different answers as to what the storage policy is for a container, and how that is resolved, and why that can't also start resolving misplaced objects. | 19:30 |
| torgomatic | gholt: hang on, lemme find the transcript... it involves a network partition | 19:30 |
| torgomatic | step 1: partition your cluster by hosing the network | 19:31 |
| torgomatic | step 2: on side A, create a container in policy 1 and upload an object | 19:31 |
| torgomatic | step 3: on side B, create the same container with policy 2 and upload an object | 19:31 |
| torgomatic | step 4: unhose the network | 19:31 |
| gholt | Yeah, so don't the container servers replicate and figure out the "right" answer for the policy for the container? How? | 19:32 |
| torgomatic | gholt: there's a proposed patch for that, but I think it's gone stale... it's basically like they do the metadata, but the older answer wins (not the newer) | 19:33 |
| torgomatic | the idea is that the older copy probably has more objects in it, so it's less crap to move doing it that way | 19:33 |
| gholt | I guess what I'm driving at is when that resolution occurs, you magically know you need to fix objects; no need to scan. | 19:34 |
| torgomatic | true | 19:35 |
| torgomatic | I'm sort of concerned about what happens if you crash midway through resolving stuff though; either you end up repeating work, or objects get lost | 19:36 |
| torgomatic | putting it in the container updater would at least save Swift from a new filesystem walker, and in the no-resolving-needed case it's just a read of container_stat (like it's already doing) | 19:37 |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add Storage Policy Support to Account HEAD https://review.openstack.org/73747 | 19:37 |
| peluse | torgomatic: ^ done :) | 19:37 |
| torgomatic | peluse: cool; will look soon | 19:37 |
| gholt | Yeah, kinda still reading the gist. | 19:41 |
| gholt | I'm not sure anything of this would be what the user wants. :) | 19:41 |
| torgomatic | gholt: the user just wants their objects back, so we can't lose them | 19:42 |
| torgomatic | that's the ultimate goal here: don't lose objects | 19:42 |
| gholt | Yeah, but one user wanted one thing for objects and the other wanted another thing. | 19:42 |
| torgomatic | thunderdome? | 19:43 |
| creiht | well if the storage policy was to /dev/null storage, then they are out of luck ;0 | 19:43 |
| gholt | If there was a way to just leave them separate and have the user resolve the issue, that'd be pretty nice. | 19:43 |
| gholt | I'm talking out of ignorance here, but one ends up the default and the other can be selected by header or something. | 19:43 |
| torgomatic | as soon as you ask the user if they want policy 1 or 2, they'll ask for policy 7, and they'll want it in a periwinkle blue ;) | 19:45 |
| torgomatic | I don't think we can rely on users to fix anything; mapping a container back to a human isn't always easy, and getting that human to do something is hard | 19:46 |
| * gholt wishes we reserved space to embed more things into the url path. | 19:48 | |
| gholt | But I'm evil that way, push things off to the data owner/creator. | 19:49 |
| redbo | /r/e/s/e/r/v/e/d/container/object | 19:49 |
| torgomatic | gholt: it's awesome when it works :) | 19:49 |
| torgomatic | req.path_info = base64.encode(json.dumps({'account': 'bob', 'container': ... | 19:49 |
| gholt | I didn't catch what you guys do with conflicting obj names | 19:50 |
| torgomatic | gholt: newest timestamp wins, same as before | 19:50 |
| gholt | Cool, nothing special there, just special for the policy, makes sense. I don't like the extras that entails, but I got nothing better. | 19:56 |
| torgomatic | I'm not especially enthused about all this either, but this policy stuff opens up a new consistency-failure mode | 19:58 |
| *** gvernik_ has quit IRC | 19:58 | |
| torgomatic | If you have a way that I could avoid writing all this code, I'm all ears :) | 19:59 |
| gholt | :) | 20:00 |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add Storage Policy Support to Account HEAD https://review.openstack.org/73747 | 20:03 |
| peluse | torgomatic: for real this time :) | 20:03 |
| torgomatic | peluse: :) | 20:03 |
| notmyname | FYI if you didn't see links for the reconciler gist or the swiftclient post-mortem, they're on the minutes http://eavesdrop.openstack.org/meetings/swift/2014/swift.2014-02-19-19.00.html | 20:06 |
| zaitcev | what if NTP goes mad on you and a node creates a bunch of junk with very fresh timestamps | 20:07 |
| zaitcev | am I overthinking | 20:07 |
| *** Trixboxer has quit IRC | 20:08 | |
| * notmyname steps away for a bit | 20:08 | |
| zaitcev | I just had something untarred and it cropped a bunch of "timestamp in the fuuuuuture!!" (time zone was incorrect somewhere) | 20:08 |
| zaitcev | I understand we do not support resistance against Byzantine failures, but I was just wondering if anyone at RAX ever had comething like this screwed up by mistake. | 20:10 |
| gholt | Only a little, since we accept x-timestamp headers users can override anything | 20:21 |
| gholt | So that can cause some interesting scenarios | 20:21 |
| *** foexle has quit IRC | 20:23 | |
| creiht | zaitcev: so I saw your comment in the code, and I'm tending to believe that ideally the guru stuff should be unobtrusive enough that it is always running | 20:45 |
| creiht | and not require a config var | 20:45 |
| zaitcev | right... otherwise Murphy says you won't have it when you really need it | 20:46 |
| creiht | haha | 20:46 |
| creiht | yes | 20:46 |
| zaitcev | I was on the fence about it but I'll drop it now. | 20:47 |
| creiht | cool | 20:47 |
| *** joeljwright has quit IRC | 21:00 | |
| *** dtalton has left #openstack-swift | 21:03 | |
| *** csd has quit IRC | 21:06 | |
| *** mvenesio has quit IRC | 21:07 | |
| *** joeljwright has joined #openstack-swift | 21:09 | |
| *** csd has joined #openstack-swift | 21:18 | |
| *** cschwede_ has joined #openstack-swift | 21:20 | |
| *** cschwede_ has quit IRC | 21:21 | |
| *** keving has joined #openstack-swift | 21:29 | |
| *** fbo_away is now known as fbo | 21:34 | |
| creiht | zaitcev: so I'm trying to enable the guru stuff on my saio | 21:36 |
| creiht | in one of my object server confs, I added | 21:36 |
| creiht | gurumed = yes | 21:37 |
| creiht | gurumed_dir = /var/lib/swift/dump | 21:37 |
| creiht | under DEFAULT | 21:37 |
| creiht | then I do | 21:37 |
| creiht | kill -s SIGUSR1 PID | 21:38 |
| openstackgerrit | Vyacheslav Rafalskiy proposed a change to openstack/swift: Add support for reading from archives https://review.openstack.org/73008 | 21:38 |
| creiht | but I don't get a dump | 21:38 |
| creiht | I see Removing dead child and Started child in the logs | 21:38 |
| creiht | am I missing something? | 21:38 |
| *** tong_ has quit IRC | 21:40 | |
| zaitcev | well... I only added hooks to small set of problematic daemons. Those that don't have the hook will die on SIRUSR1. | 21:42 |
| zaitcev | creiht: So, the only thing that works is swift-init object-replicator dump | 21:43 |
| creiht | ahh ok | 21:43 |
| zaitcev | I'm sorry, I'll add hookage across the board. | 21:43 |
| creiht | I think that will be useful | 21:44 |
| creiht | if all of this works | 21:44 |
| creiht | :) | 21:44 |
| zaitcev | yeah... I thought maybe find a common place instead of having 21 identical hook | 21:44 |
| creiht | yeah | 21:44 |
| openstackgerrit | Fabien Boucher proposed a change to openstack/python-swiftclient: Allow get_account and get_container to return an iterator https://review.openstack.org/74845 | 21:45 |
| luisbg | python-swiftclient in Jenkins has the python3.3 builds fail with: FileNotFoundError: [Errno 2] No such file or directory: './subunit_log.txt' | 21:51 |
| luisbg | :S | 21:51 |
| luisbg | why is that? | 21:51 |
| luisbg | they are marked as (non-vote) so builds still pass, but wondering why those tests are broken | 21:51 |
| torgomatic | luisbg: probably because swiftclient isn't py3 compatible | 21:52 |
| torgomatic | my guess is it doesn't even compile, so there's no chance for the testrunner to do anything, including write a log | 21:52 |
| *** foexle has joined #openstack-swift | 21:53 | |
| luisbg | torgomatic, ooooh, that makes sense, didn't knew | 21:53 |
| luisbg | torgomatic, and I guess Jenkins needs something, even if it setup to ignore it | 21:53 |
| torgomatic | folks are now working on making it py3 compatible, but it's not done yet AFAIK | 21:53 |
| *** cschwede_ has joined #openstack-swift | 21:57 | |
| notmyname | torgomatic: FYI on https://review.openstack.org/#/c/74705/ we don't currently use six at all, so this adds a test dependency (that's not referenced in test-requirements) | 22:02 |
| torgomatic | notmyname: ...then how do the tests pass? | 22:02 |
| torgomatic | what are they importing? | 22:03 |
| notmyname | you already have it installed? | 22:03 |
| notmyname | is six in the stdlib? | 22:03 |
| torgomatic | maybe...? what about the Jenkins stuff? | 22:03 |
| notmyname | ya, because jenkins uses the global requirements (instead of what the project says) | 22:03 |
| torgomatic | all the check jobs passed, so they must have done something reasonable when processing the "import six" directive | 22:03 |
| torgomatic | oh | 22:03 |
| notmyname | kinda like it always uses the tip of master for client libs, even if the requirements have a version cap (ie the problems they had with swiftclient in grizzly) | 22:04 |
| torgomatic | sure, what could POSSIBLY GO WRONG | 22:04 |
| gholt | There is no tip, this is git. ;) | 22:04 |
| notmyname | from Guido: " That's the problem with Twitter. You must lie to fit in 140 chars." | 22:13 |
| luisbg | notmyname, van Rosum? | 22:13 |
| notmyname | the one and only | 22:13 |
| luisbg | notmyname, :) | 22:13 |
| luisbg | notmyname, there is also this: http://gvr.sourceforge.net/ | 22:14 |
| luisbg | Guido van Robot, haha | 22:14 |
| *** foexle has quit IRC | 22:33 | |
| *** cschwede_ has quit IRC | 22:44 | |
| *** foexle has joined #openstack-swift | 22:57 | |
| *** byeager has joined #openstack-swift | 23:02 | |
| *** fbo is now known as fbo_away | 23:02 | |
| *** foexle has quit IRC | 23:05 | |
| *** joeljwright has quit IRC | 23:06 | |
| *** foexle has joined #openstack-swift | 23:06 | |
| *** foexle has quit IRC | 23:12 | |
| *** foexle has joined #openstack-swift | 23:12 | |
| *** foexle_ has joined #openstack-swift | 23:16 | |
| openstackgerrit | A change was merged to openstack/python-swiftclient: Remove useless statement https://review.openstack.org/74701 | 23:17 |
| *** Midnightmyth has quit IRC | 23:18 | |
| *** asselin_ has joined #openstack-swift | 23:20 | |
| *** asselin_ has left #openstack-swift | 23:21 | |
| *** dmsimard has quit IRC | 23:23 | |
| *** foexle_ has quit IRC | 23:24 | |
| *** foexle has quit IRC | 23:25 | |
| *** csd has quit IRC | 23:29 | |
| *** openstack has joined #openstack-swift | 23:34 | |
| *** kris_h has quit IRC | 23:41 | |
| *** jergerber has quit IRC | 23:46 | |
| *** mkollaro has quit IRC | 23:47 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
