| notmyname | ok, as of now, Launchpad answers have been closed. all questions have been migrated to http://ask.openstack.org | 00:03 |
|---|---|---|
| *** miurahr has quit IRC | 00:05 | |
| wer | I am trying to do some backend middleware magic. And I am not sure what I am going wrong with my swob.Request. I see my request hitting the proxy-server but am getting a 401. the container is public so I don't get it. | 00:19 |
| wer | requests that are identical with curl can access the files just fine. I've seen examples of middlewares using webob... one with swob. I chose swob cause it is there. But I need a simple example to "fix" an incoming request. and add the AUTH_ pieces and redo some of the url. The return that response. But I've been pulling my hair out a little :) | 00:23 |
| *** miurahr has joined #openstack-swift | 00:23 | |
| wer | use of the term identical is loose :) cause really I must be doing something stuupid. But it sure looks correct in the logs :) | 00:24 |
| notmyname | wer: normally I point people to the included catch_errors as a sample middleware | 00:24 |
| wer | gah. I have not looked there yet. Was just going through all of them :) | 00:24 |
| notmyname | catch_errors is fairly simple, so it's a good starting point | 00:25 |
| wer | I have written middlewares that can do things like return 302's and things. But I have never done one that does another call to swift... or somewhere else to get something. | 00:25 |
| notmyname | wer: ah. doing extra requests. like dlo, slo, bulk, staticweb | 00:26 |
| wer | yeah! | 00:26 |
| wer | I have been looking at bulk and basing my stuff on that and swob. And it returns a 401 which I think is from the new request. | 00:27 |
| wer | But I don't think I should be getting a 401 so something is wonky. | 00:27 |
| notmyname | are you using the pre-authed request mechanics? | 00:28 |
| wer | And in my proxy-server logs... I see the "malformed" original request.... and the newly formed nice looking request. And the new request should not be 401. But from my middleware it is. | 00:28 |
| wer | I don't think so. | 00:28 |
| notmyname | dfg: still around? in my mind you're the expert on pre authed requests in middleware | 00:29 |
| wer | The new request is hitting a public bucket... so I guess I don't get why I would need preauth at that point.... but it seems like you may be right. I've obviously missed a step :) | 00:30 |
| *** miurahr has quit IRC | 00:30 | |
| wer | err container I meant. | 00:30 |
| wer | I guess I should also take a harder look at swob too :( I am just not a very accomplished wsgi guy. So all this stuff feels highy abstracted to me. so many self.logger.debug lines :) | 00:32 |
| *** miurahr has joined #openstack-swift | 00:34 | |
| wer | notmyname: does pre auth refer to middleware in the pipeline before auth? I'm not finding things about it. | 00:35 |
| notmyname | wer: hmm..probably not. that doesn't make sense to me (I'm not looking at the code..I'm sitting in the middle of a conference right now) | 00:35 |
| wer | oh no worries :) I'll keep poking. | 00:36 |
| wer | and ty | 00:37 |
| *** miurahr has quit IRC | 00:41 | |
| *** openstackgerrit has quit IRC | 00:44 | |
| *** openstackgerrit has joined #openstack-swift | 00:44 | |
| *** miurahr has joined #openstack-swift | 00:45 | |
| openstackgerrit | A change was merged to openstack/swift: Use new style class for ringbuilder's Commands https://review.openstack.org/77463 | 00:55 |
| *** miurahr has quit IRC | 01:04 | |
| openstackgerrit | A change was merged to openstack/swift: Object server PUTs should respect client_timeout https://review.openstack.org/77226 | 01:05 |
| *** nosnos has joined #openstack-swift | 01:29 | |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Convert all old-style classes to new-style https://review.openstack.org/77745 | 01:34 |
| *** shri has quit IRC | 01:44 | |
| *** Dharmit has joined #openstack-swift | 01:51 | |
| *** Dharmit has quit IRC | 02:03 | |
| *** Dharmit has joined #openstack-swift | 02:06 | |
| clayg | torgomatic: why you gotta go kocking out the ducks? | 02:09 |
| creiht | pre-emptive strike | 02:18 |
| creiht | :) | 02:18 |
| *** lpabon has quit IRC | 02:35 | |
| *** yuanz has quit IRC | 02:49 | |
| *** yuan has joined #openstack-swift | 03:17 | |
| *** chandan_kumar has joined #openstack-swift | 03:37 | |
| *** annegentle has quit IRC | 03:42 | |
| *** mrsnivvel has joined #openstack-swift | 03:59 | |
| *** judd7 has quit IRC | 04:04 | |
| *** madhuri has quit IRC | 04:21 | |
| *** ppai has joined #openstack-swift | 04:30 | |
| *** lpabon has joined #openstack-swift | 04:34 | |
| *** madhuri has joined #openstack-swift | 04:38 | |
| *** chandan_kumar has quit IRC | 04:53 | |
| madhuri | Hi, can anyone please tell me what happens in case we submit a PUT request for a container and the account doesn't exist? | 04:54 |
| *** sungju_ has joined #openstack-swift | 05:35 | |
| *** sungju_ has quit IRC | 05:36 | |
| *** lpabon has quit IRC | 05:38 | |
| *** nshaikh has joined #openstack-swift | 05:40 | |
| *** bvandenh has joined #openstack-swift | 05:49 | |
| clayg | madhuri: 404 | 06:09 |
| clayg | creiht: no no no - we have to give them something to play with so they don't break our nice things | 06:10 |
| *** bvandenh has quit IRC | 06:11 | |
| madhuri | clayg: But I have tried using it but it doesn't show 404 | 06:14 |
| clayg | oh... well if you're authorized to create containers in that account and you have account_autocreate turned on - the account will just... get... auto... created. | 06:27 |
| madhuri | clayg: but no account database is created in that case :( | 06:29 |
| madhuri | clayg: http://pastebin.com/jydbLdwN | 06:32 |
| *** sungju has quit IRC | 06:33 | |
| *** gyee has quit IRC | 06:34 | |
| openstackgerrit | Christian Schwede proposed a change to openstack/swift: Add containeralias middleware https://review.openstack.org/62494 | 06:52 |
| *** odyssey4me has quit IRC | 06:57 | |
| clayg | madhuri: well neither of those is a container create request - that's just the difference between the 201 and 202? | 06:58 |
| clayg | madhuri: try running the same thing with -I, or make the same requests against a container. | 06:59 |
| *** SkyRocknRoll has joined #openstack-swift | 07:06 | |
| *** SkyRocknRoll has quit IRC | 07:06 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:06 | |
| ppai | Hi all, does anyone know what are the *.db.pending files for ? | 07:07 |
| openstackgerrit | Pete Zaitcev proposed a change to openstack/swift: Add Guru Meditation https://review.openstack.org/70513 | 07:12 |
| *** saju_m has joined #openstack-swift | 07:12 | |
| *** odyssey4me has joined #openstack-swift | 07:15 | |
| zaitcev | ppai: it's for updaters | 07:16 |
| madhuri | ppai: It contains all the pending commit request for account database or container database. | 07:17 |
| ppai | zaitcev, thanks, has it got anything to do with sqlite WAL feature ? | 07:18 |
| ppai | madhuri, at what point does it get committed to .db file ? | 07:18 |
| madhuri | ppai: After the limit of maximum pending entries is reached. | 07:20 |
| ppai | madhuri, thanks. Is it this http://docs.openstack.org/developer/swift/db.html#swift.common.db.PENDING_CAP ? | 07:22 |
| *** zaitcev has quit IRC | 07:22 | |
| madhuri | ppai: Yes | 07:22 |
| *** mlipchuk has joined #openstack-swift | 07:27 | |
| ppai | madhuri: thanks again, I set PENDING_CAP = 0 in db.py and now i can see object entries in container DBs :) | 07:28 |
| madhuri | ppai: anytime :) | 07:29 |
| *** bvandenh has joined #openstack-swift | 07:30 | |
| *** nosnos has quit IRC | 07:31 | |
| *** nosnos_ has joined #openstack-swift | 07:31 | |
| *** bvandenh has quit IRC | 07:36 | |
| *** ppai has quit IRC | 07:37 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: sanity check copy tests https://review.openstack.org/77811 | 07:38 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: refactor proxy and close connections https://review.openstack.org/77812 | 07:38 |
| *** ppai has joined #openstack-swift | 07:41 | |
| *** nosnos_ has quit IRC | 08:04 | |
| *** nosnos has joined #openstack-swift | 08:04 | |
| *** saju_m has quit IRC | 08:26 | |
| *** acoles has joined #openstack-swift | 08:30 | |
| *** haomaiwang has quit IRC | 08:57 | |
| *** haomaiwang has joined #openstack-swift | 09:00 | |
| *** foexle has joined #openstack-swift | 09:04 | |
| openstackgerrit | Zhang Hua proposed a change to openstack/swift: Add profiling middleware in Swift https://review.openstack.org/53270 | 09:08 |
| *** Midnightmyth has joined #openstack-swift | 09:20 | |
| *** miurahr has joined #openstack-swift | 09:20 | |
| *** miurahr has quit IRC | 09:25 | |
| *** nacim has joined #openstack-swift | 09:28 | |
| *** Trixboxer has joined #openstack-swift | 09:34 | |
| *** tanee-away is now known as tanee | 09:36 | |
| *** tanee is now known as tanee-away | 09:36 | |
| *** Midnightmyth has quit IRC | 09:37 | |
| *** sungju has joined #openstack-swift | 09:38 | |
| *** tanee-away is now known as tanee | 09:39 | |
| *** saju_m has joined #openstack-swift | 09:41 | |
| *** krtaylor has quit IRC | 09:44 | |
| *** fbo_away is now known as fbo | 09:45 | |
| *** sungju has quit IRC | 09:49 | |
| *** sungju has joined #openstack-swift | 09:55 | |
| *** Dharmit has quit IRC | 10:01 | |
| *** mkollaro has joined #openstack-swift | 10:26 | |
| *** jamieh has joined #openstack-swift | 10:49 | |
| *** sungju has quit IRC | 11:06 | |
| *** haomaiwang has quit IRC | 11:08 | |
| *** haomaiwang has joined #openstack-swift | 11:09 | |
| *** saschpe has quit IRC | 11:12 | |
| *** haomaiwang has quit IRC | 11:14 | |
| *** saschpe has joined #openstack-swift | 11:16 | |
| *** miurahr has joined #openstack-swift | 11:28 | |
| pconstantine | clayg: https://review.openstack.org/#/c/77812/ - closing all the connections to backends, finally! Bug #667956 caused me too much grief. | 11:39 |
| *** mrsnivvel has quit IRC | 11:39 | |
| *** ppai has quit IRC | 11:45 | |
| *** mkollaro has quit IRC | 11:47 | |
| *** miurahr has quit IRC | 11:48 | |
| *** miurahr has joined #openstack-swift | 11:53 | |
| *** mrsnivvel has joined #openstack-swift | 11:53 | |
| *** miurahr has quit IRC | 12:00 | |
| *** ppai has joined #openstack-swift | 12:02 | |
| *** ekarlso has quit IRC | 12:09 | |
| *** erlon has joined #openstack-swift | 12:12 | |
| *** nshaikh has left #openstack-swift | 12:23 | |
| *** miurahr has joined #openstack-swift | 12:33 | |
| *** gholt has quit IRC | 12:34 | |
| *** Midnightmyth has joined #openstack-swift | 12:36 | |
| *** saju_m has quit IRC | 12:40 | |
| *** ppai has quit IRC | 12:45 | |
| *** SkyRocknRoll has quit IRC | 12:48 | |
| *** saju_m has joined #openstack-swift | 12:51 | |
| *** ppai has joined #openstack-swift | 12:58 | |
| *** mkollaro has joined #openstack-swift | 13:01 | |
| *** saju_m has quit IRC | 13:04 | |
| *** bvandenh has joined #openstack-swift | 13:08 | |
| *** gholt has joined #openstack-swift | 13:13 | |
| *** ChanServ sets mode: +v gholt | 13:13 | |
| *** lpabon has joined #openstack-swift | 13:16 | |
| *** ppai has quit IRC | 13:18 | |
| *** ppai has joined #openstack-swift | 13:18 | |
| *** mkollaro has quit IRC | 13:18 | |
| *** mkollaro has joined #openstack-swift | 13:18 | |
| *** bvandenh has quit IRC | 13:18 | |
| *** bvandenh has joined #openstack-swift | 13:18 | |
| *** ekarlso has joined #openstack-swift | 13:23 | |
| *** nacim_ has joined #openstack-swift | 13:43 | |
| *** nacim has quit IRC | 13:44 | |
| *** krtaylor has joined #openstack-swift | 13:44 | |
| *** mkollaro1 has joined #openstack-swift | 13:46 | |
| *** mkollaro has quit IRC | 13:46 | |
| *** nosnos has quit IRC | 13:56 | |
| *** ppai has quit IRC | 13:58 | |
| *** annegentle has joined #openstack-swift | 14:09 | |
| *** mkollaro has joined #openstack-swift | 14:15 | |
| *** mkollaro1 has quit IRC | 14:16 | |
| *** mrsnivvel has quit IRC | 14:21 | |
| *** ccorrigan has joined #openstack-swift | 14:22 | |
| *** changbl has quit IRC | 14:35 | |
| *** mrsnivvel has joined #openstack-swift | 14:38 | |
| *** rturk has joined #openstack-swift | 14:49 | |
| *** haomaiwang has joined #openstack-swift | 14:49 | |
| *** rturk is now known as rturk-away | 14:50 | |
| *** judd7 has joined #openstack-swift | 14:56 | |
| mjseger | notmyname: are you around? | 14:56 |
| *** odyssey4me has quit IRC | 14:59 | |
| *** lpabon has quit IRC | 14:59 | |
| *** dmsimard has joined #openstack-swift | 15:00 | |
| *** hurricanerix has joined #openstack-swift | 15:06 | |
| creiht | clayg: hehe | 15:09 |
| *** haomaiw__ has joined #openstack-swift | 15:12 | |
| *** jergerber has joined #openstack-swift | 15:12 | |
| *** mkollaro1 has joined #openstack-swift | 15:13 | |
| *** mkollaro has quit IRC | 15:13 | |
| *** haomaiwang has quit IRC | 15:16 | |
| *** chandan_kumar has joined #openstack-swift | 15:24 | |
| *** tongli has joined #openstack-swift | 15:29 | |
| *** bvandenh has quit IRC | 15:38 | |
| *** bvandenh has joined #openstack-swift | 15:41 | |
| *** tdasilva has joined #openstack-swift | 15:42 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: refactor proxy and close connections https://review.openstack.org/77812 | 15:56 |
| *** otoolee has quit IRC | 16:12 | |
| *** jamieh has quit IRC | 16:15 | |
| *** haomaiw__ has quit IRC | 16:16 | |
| *** haomaiwang has joined #openstack-swift | 16:16 | |
| *** bvandenh has quit IRC | 16:17 | |
| clayg | ppai the .pending feature predates sqlite WAL, and redbo and dfg both tried to make WAL work IIRC it would never pause writes and force a flush (grow forever and ever) | 16:18 |
| clayg | so... we stick with .pendings - which is basically the same idea | 16:18 |
| clayg | i guess I turned of joins logging for this channel | 16:19 |
| *** haomaiwang has quit IRC | 16:21 | |
| *** otoolee has joined #openstack-swift | 16:24 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Update tox.ini so flake8 works even without hacking https://review.openstack.org/77072 | 16:29 |
| *** foexle has quit IRC | 16:31 | |
| *** changbl has joined #openstack-swift | 16:35 | |
| *** piyush1 has joined #openstack-swift | 16:39 | |
| *** gyee has joined #openstack-swift | 16:48 | |
| notmyname | good morning | 17:03 |
| dfg | clayg: are you going to see if the new vrs of sqlite works now? that would be awesome | 17:03 |
| luisbg | morning | 17:05 |
| clayg | dfg: maybe we can pair on it in Alt? | 17:21 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: refactor proxy and close connections https://review.openstack.org/77812 | 17:22 |
| clayg | pep8 whack-a-mole ^ | 17:22 |
| *** mlipchuk has quit IRC | 17:26 | |
| * dfg and clayg pair programming. there's nothing we couldn't solve. well as long as we didn't strangle each other first :p | 17:29 | |
| openstackgerrit | Fabien Boucher proposed a change to openstack/python-swiftclient: Allow get_account and get_container to return an iterator https://review.openstack.org/74845 | 17:31 |
| openstackgerrit | Fabien Boucher proposed a change to openstack/python-swiftclient: Allow get_account and get_container to return an iterator https://review.openstack.org/74845 | 17:33 |
| *** tanee is now known as tanee-away | 17:40 | |
| notmyname | please take a look at these patches today, if possible: https://review.openstack.org/#/q/status:open+project:openstack/swift+branch:feature/ec,n,z | 17:46 |
| *** wklely has joined #openstack-swift | 17:51 | |
| notmyname | torgomatic: peluse: probe tests fail on the ec branch in my saio. do they work for you? | 17:52 |
| *** chandan_kumar has quit IRC | 17:53 | |
| *** wkelly has quit IRC | 17:53 | |
| peluse | notmyname: I'll check | 17:57 |
| notmyname | peluse: thanks | 17:58 |
| mjseger | notmyname: I've got some really interesting stuff to show you if you have some time | 18:08 |
| *** wklely is now known as wkelly | 18:10 | |
| notmyname | mjseger: I am trying to stay focused on the storage policy feature set today. we've got several things to get done there for Icehouse | 18:13 |
| portante | notmyname: four of those are markek WIP | 18:13 |
| portante | marked | 18:14 |
| notmyname | portante: yes., but 2 of the WIP are for EC-specific stuff | 18:14 |
| notmyname | portante: and I want to know from torgomatic about his WIP, since it's in a dependency chain for the account HEAD one | 18:14 |
| portante | ah | 18:15 |
| mjseger | notmyname: ok, we can visit another time but to whet your appetite, I took the brute force approach to finding my timing problems and stuff some print statements into the obj.py code on the proxy and found a 2 second gap between 2 statements! | 18:15 |
| notmyname | the WIP by peluse looks like an enhancement for later, not required immediately | 18:15 |
| notmyname | mjseger: fun! | 18:15 |
| notmyname | portante: the root of the dep chain is https://review.openstack.org/#/c/71702/ | 18:15 |
| portante | thanks, that is helpful | 18:16 |
| mjseger | idea why this would take that long - conns = [conn for conn in pile if conn] | 18:16 |
| mjseger | anyhow we can dig some more another time and I'll try to get more data points | 18:16 |
| notmyname | mjseger: "pile" is a generator (iterator) that will only complete once all the greenthreads are done | 18:16 |
| *** csd has joined #openstack-swift | 18:17 | |
| mjseger | notmyname: so someone is slowing everybody else down then | 18:17 |
| portante | notmyname: I'll be available to do reviews again late this afternoon | 18:17 |
| notmyname | mjseger: ya, probably | 18:17 |
| notmyname | portante: thanks | 18:17 |
| mjseger | ;) | 18:17 |
| portante | mjseger: this is likely disk access or some other op that blocks the enire process | 18:18 |
| mjseger | I gotta run, but we can talk later | 18:18 |
| portante | couple that with strace to see | 18:18 |
| notmyname | mjseger: last thing | 18:18 |
| portante | mjseger: | 18:18 |
| notmyname | mjseger: see if the proxy config var https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L105 is affecting it | 18:18 |
| mjseger | portante: as I said I have to run but since I have detailed disk stats I can certainly look into it | 18:18 |
| portante | okay, thanks | 18:20 |
| *** piyush1 has quit IRC | 18:22 | |
| peluse | notmyname: stuck in meetings for a while but I am having issues also - will be a while before I can take a look but can't rule out my own SWAIO at the momeent (getting failures on master). I assume you see no issues on master? | 18:27 |
| notmyname | peluse: I haven't run on master today. I will when I get done with the current patch I'm looking at | 18:28 |
| peluse | notmyname: rock on | 18:28 |
| *** piyush1 has joined #openstack-swift | 18:36 | |
| *** nacim_ has quit IRC | 18:40 | |
| openstackgerrit | A change was merged to openstack/swift: Convert all old-style classes to new-style https://review.openstack.org/77745 | 18:43 |
| *** changbl has quit IRC | 18:50 | |
| *** changbl has joined #openstack-swift | 18:54 | |
| luisbg | any low hanging fruit bugs I could look into tonight? | 19:10 |
| *** jamieh has joined #openstack-swift | 19:26 | |
| *** bada has joined #openstack-swift | 19:29 | |
| notmyname | gholt: glange: creiht: dfg: redbo: swifterdarrell: clayg: torgomatic: portante: chmouel: please be looking at the feature/ec branch this week. once it's proposed to merge into master, having already looked at it will certainly help speed up the review process | 19:33 |
| notmyname | the goal is still to have it merged into master for the openstack icehouse release. the release candidates will start to be cut for Icehouse (in all the projects) on March 27 | 19:34 |
| *** bada has quit IRC | 19:39 | |
| mjseger | portante: back from my errand. I just looked at the second-by-second disk activity on the proxy and object servers. the service/wait times were ok but the util was close to 100% on all 3 of the object servers, just for those few seconds. | 19:41 |
| *** piyush1 has quit IRC | 19:41 | |
| mjseger | notmyname: in ref to the timeouts in proxy-server.conf, there weren't any explicitly specified so I guess it's using all the defaults | 19:42 |
| *** dmsimard1 has joined #openstack-swift | 19:43 | |
| *** dmsimard has quit IRC | 19:44 | |
| *** piyush1 has joined #openstack-swift | 19:49 | |
| *** gyee has quit IRC | 20:03 | |
| *** dmsimard1 has quit IRC | 20:11 | |
| *** dmsimard has joined #openstack-swift | 20:18 | |
| *** dmsimard1 has joined #openstack-swift | 20:21 | |
| *** dmsimard has quit IRC | 20:23 | |
| *** Midnightmyth has quit IRC | 20:26 | |
| openstackgerrit | A change was merged to openstack/swift: fix gatekeeper response headers behavior https://review.openstack.org/76709 | 20:28 |
| mjseger | portante: notmyname: one last piece of data and I'll leave you alone for the rest of the day. I just did 16500 PUTs in 48 threads. 19 took over 2 seconds and most of those took over a full second in that pile iterator. | 20:29 |
| mjseger | in looking at many of the shorter transactions, a LOT of them still took over 0.1 seconds which I also think is way too long, especially for a transaction that normally takes a few hundredths of a second. any suggestions on where to take it from here are all welcome. | 20:31 |
| mjseger | this is VERY easy to reproduce with getput if anyone else cares to try it out and if so I can give you some driving instructions | 20:32 |
| *** redbo has quit IRC | 20:47 | |
| *** redbo has joined #openstack-swift | 20:48 | |
| *** ChanServ sets mode: +v redbo | 20:48 | |
| *** jamieh has quit IRC | 21:09 | |
| *** Midnightmyth has joined #openstack-swift | 21:11 | |
| *** vu_ has joined #openstack-swift | 21:14 | |
| *** Trixboxer has quit IRC | 21:16 | |
| *** bada has joined #openstack-swift | 21:17 | |
| *** vu_ has quit IRC | 21:17 | |
| *** vu_ has joined #openstack-swift | 21:18 | |
| *** tsnider has joined #openstack-swift | 21:22 | |
| *** gyee has joined #openstack-swift | 21:58 | |
| *** vu_ has quit IRC | 21:58 | |
| *** zaitcev has joined #openstack-swift | 21:59 | |
| *** ChanServ sets mode: +v zaitcev | 21:59 | |
| *** vuvu has joined #openstack-swift | 22:01 | |
| *** vu_ has joined #openstack-swift | 22:01 | |
| *** Midnightmyth has quit IRC | 22:05 | |
| *** fbo is now known as fbo_away | 22:09 | |
| *** vuvu has quit IRC | 22:11 | |
| *** vu_ has quit IRC | 22:11 | |
| portante | mjseger, would love to see that compare to a run where the object server is configured with thread pools, and aanother where it is configured with sufficient workers with max_clients=1 | 22:13 |
| *** sungju has joined #openstack-swift | 22:13 | |
| portante | that would help ferrit out the behavior without getting into lots of strace analysis | 22:13 |
| *** shri has joined #openstack-swift | 22:15 | |
| peluse | notmyname: all tests (unit, func, probe) pass for me on both master and EC but only on a brand new install of SAIO. On my regular VM I get failures for some reason, haven't looked into why as I wanted to confirm whether a clean install worked or not | 22:27 |
| notmyname | peluse: I found that my issues on the probetests were because I had a SP configured but not a ring for it | 22:27 |
| notmyname | `cp object.ring.gz object-1.ring.gz` and it worked | 22:28 |
| peluse | notmyname: cool | 22:29 |
| peluse | I'm just going to migrate to my new SAIO as my old one is, well, old and who knows where its out of whack | 22:29 |
| notmyname | peluse: your account HEAD patch is on top of a dependency chain that has a couple of issues | 22:30 |
| *** vu_ has joined #openstack-swift | 22:30 | |
| *** vuvu has joined #openstack-swift | 22:30 | |
| notmyname | peluse: 1) a rebase causes unittest failures. I think I've got that straightened out, and I'm about to push a fix | 22:30 |
| notmyname | peluse: 2) torgomatic has a WIP patch in there because of reconciler work | 22:31 |
| *** shri1 has joined #openstack-swift | 22:31 | |
| notmyname | peluse: can your patch be rebased to the tip of the ec branch without any issues? | 22:31 |
| peluse | notmyname: yup, I know. I need to up date my acct rollup patch once he's ready | 22:31 |
| peluse | notmyname: sure | 22:31 |
| notmyname | peluse: or do you need some of the functionality? | 22:31 |
| openstackgerrit | John Dickinson proposed a change to openstack/swift: Send policy index to container on sync update https://review.openstack.org/71703 | 22:32 |
| openstackgerrit | John Dickinson proposed a change to openstack/swift: Send policy index on async update https://review.openstack.org/71702 | 22:32 |
| peluse | notmyname: wait... err, I already have a series of his patches as a dependency on that one and I'm not sure what his new patch does so I guess my answer is IDK | 22:32 |
| notmyname | :-) | 22:32 |
| notmyname | ok | 22:32 |
| notmyname | well I just broke your dep chain anyway ;-) | 22:33 |
| peluse | notmyname: :) | 22:33 |
| *** shri has quit IRC | 22:33 | |
| notmyname | rebasing one of the patches in there caused a unittest failure. cherry-picking it, though, worked | 22:34 |
| creiht | ugh gerrit formatting | 22:34 |
| peluse | torgomatic: you out there? | 22:35 |
| notmyname | he's busy doing storage policy coding. don't bother him ;-) | 22:35 |
| notmyname | he has his headphones on an everything | 22:36 |
| torgomatic | ¿que? no hablo ingles | 22:36 |
| creiht | torgomatic: estas alli? | 22:36 |
| creiht | :) | 22:36 |
| torgomatic | creiht: turns out my Spanish is worse than my jokes :) | 22:36 |
| creiht | hehe | 22:37 |
| *** piyush1 has quit IRC | 22:37 | |
| zaitcev | probably better than my Japanese | 22:37 |
| torgomatic | yeah, I'm working on the reconciliatortron-o-mat 5000 here | 22:37 |
| zaitcev | creiht: I fixed up Guru Meditation to work, it can be tried now. | 22:37 |
| creiht | oh cool | 22:37 |
| creiht | I've been poking at the profiling middleware today | 22:38 |
| creiht | some small cleanups and it should be good for a first pass | 22:38 |
| creiht | zaitcev: I also saw the snide remark on the mailing list | 22:38 |
| peluse | torgomatic: excelllent :) What impact is there to the acct and container DB schema? | 22:38 |
| zaitcev | It turned out that copying an import from eventlet was a wrong thing, which produced identically looking, useless tracebacks. I replaced it with a plain "import greenlet" and that fixed the issue that you pointed ou. | 22:39 |
| torgomatic | peluse: the container schema gets a new column in container_stat; nothing to account | 22:39 |
| creiht | zaitcev: cool | 22:39 |
| creiht | I'll play with it then | 22:39 |
| *** vuvu has quit IRC | 22:39 | |
| *** vu_ has quit IRC | 22:39 | |
| creiht | I also learned that RunSnakeRun is a pretty cool profiler visualization tool today | 22:40 |
| zaitcev | creiht: If you mean Dan insisting that we use Oslo, I'll let notmyname to decide on it. Current code obviously doesn't use openstack-wide classes. John told me do it. | 22:40 |
| *** vuvu has joined #openstack-swift | 22:40 | |
| *** vu_ has joined #openstack-swift | 22:40 | |
| creiht | zaitcev: yeah I'm fine with the direction that you took | 22:40 |
| *** vuvu has quit IRC | 22:40 | |
| *** vu_ has quit IRC | 22:40 | |
| creiht | it isn't much code | 22:40 |
| *** vuvu has joined #openstack-swift | 22:40 | |
| *** vu_ has joined #openstack-swift | 22:40 | |
| creiht | and it isn't like it is a library that we could just install and use | 22:40 |
| notmyname | creiht: runsnakerun is pretty cool. just by mentioning it, you won points with swifterdarrell ;-) | 22:40 |
| creiht | hah | 22:41 |
| notmyname | zaitcev: what did I do? | 22:41 |
| zaitcev | notmyname: told me that you'll veto if I import Oslo's implementation of Guru Meditation | 22:41 |
| notmyname | is it in oslo-incubator? | 22:41 |
| peluse | torgomatic: OK, thanks. Is policy index still a column? | 22:41 |
| torgomatic | peluse: in container_stat? yeah | 22:41 |
| torgomatic | in object? no | 22:42 |
| torgomatic | it's not necessary with the way things are going | 22:42 |
| notmyname | zaitcev: ah, yes. I see his comment and it is in oslo incubator. we don't want that in swift | 22:42 |
| zaitcev | notmyname: yes, it is in oslo-incubator in openstack/common/report/ | 22:42 |
| openstackgerrit | John Dickinson proposed a change to openstack/swift: Store policy index in container_stat table https://review.openstack.org/71704 | 22:43 |
| peluse | torgomatic: cool, shouldn't have impact on my acct rollup patch then,,, | 22:43 |
| zaitcev | anyhow, I'm mostly curious if gurumed actually solves problems, else I am in gholt's camp about excessive bloat | 22:43 |
| notmyname | torgomatic: peluse: ok, up to the WIP patch, the dep chain is up to date with the tip of ec (and tests pass) | 22:43 |
| creiht | zaitcev: I think something like that could be very handy | 22:44 |
| creiht | unless we spin out a swift-debug package or something like that | 22:44 |
| torgomatic | peluse: hopefully not; just the bits where I go moving X-S-P-I out of metadata should be all you need to worry about | 22:44 |
| notmyname | seems interesting that guru and profiling middleware seems ok because it's "handy" but container migration isn't ;-) | 22:44 |
| creiht | which I would also be alright with at some point | 22:44 |
| * creiht sighs | 22:44 | |
| *** shri1 has quit IRC | 22:45 | |
| creiht | I know it is a fine line | 22:46 |
| creiht | and like I said, I'm not against created a swift-debug repo that has the middleware and meditation stuff in it | 22:46 |
| creiht | creating | 22:46 |
| * notmyname wants profiling and migration containers in swift | 22:46 | |
| notmyname | and dynamic pipeline configs, too, cause I have an actual business case for that one (along with migration containers) | 22:47 |
| creiht | notmyname: well sounds like a discussion for the summit | 22:47 |
| creiht | notmyname: I also wanted a v1.1 api ;) | 22:47 |
| notmyname | so on the topic of 1.1 | 22:48 |
| creiht | oh noes | 22:48 |
| * torgomatic hides | 22:48 | |
| notmyname | heh | 22:48 |
| creiht | I'm still not a fan of the dynamic pipeline stuff | 22:48 |
| creiht | I want to figure out a better way to solve the problem without it having to be dynamic like that | 22:48 |
| creiht | just too many things to work on | 22:49 |
| creiht | :) | 22:49 |
| peluse | torgomatic: cool, but you had already done that in the patch I rebased from if you're talking about the entries in the container_stat metadata column | 22:49 |
| torgomatic | peluse: yeah, that one | 22:49 |
| torgomatic | that one's staying | 22:49 |
| notmyname | I was in the room with a bunch of openstack deployers yesterday. their perspective was very much in favor of static versions. it's an interesting data point, I thought | 22:49 |
| creiht | what do you mean by static versions? | 22:50 |
| *** otoolee has quit IRC | 22:50 | |
| *** dmsimard1 has quit IRC | 22:51 | |
| notmyname | peluse: http://d.not.mn/QA_cluster_back.jpg http://d.not.mn/QA_cluster_front.jpg | 22:51 |
| notmyname | creiht: mostly it was protest against so many projects changing API versions and dropping support for older versions. (I realize you've never said anything about dropping support for something) | 22:52 |
| creiht | hah | 22:52 |
| creiht | yeah | 22:52 |
| notmyname | creiht: reminds me of http://imgur.com/ALoSp | 22:53 |
| creiht | heh | 22:53 |
| creiht | the current nova v3 api discussion has been amusing | 22:54 |
| *** vuvu has quit IRC | 22:54 | |
| *** vu_ has quit IRC | 22:54 | |
| *** mkollaro1 has quit IRC | 22:54 | |
| notmyname | creiht: what was funny is the ops sitting in the room while that thread was developing were saying, basically, "surely they wouldn't ever drop support for v2. that would just be crazy" | 22:54 |
| creiht | heh | 22:54 |
| zaitcev | Keystone warns "v2 is going to be removed in K release" every time I run its own stupid keystone CLI beause it uses v2. So scripting anything floods logs with those lol | 22:55 |
| peluse | notmyname: very cool pics! Up and running yet? | 22:55 |
| creiht | heh | 22:55 |
| *** vuvu has joined #openstack-swift | 22:55 | |
| *** vu_ has joined #openstack-swift | 22:55 | |
| notmyname | peluse: racked and plugged in. next step is to get an OS on them. I need to go do that now, actually | 22:55 |
| *** gholt has quit IRC | 22:56 | |
| peluse | notmyname: coll, keep me posted... offline for a while. later | 22:56 |
| zaitcev | Better yet, they have a plugin that attempts to run openssl, fails, never prints command line that failed. So my cluster taht used keystone fell apart after I ran "yum update" | 22:56 |
| zaitcev | Good thing it was a testing cluster | 22:57 |
| zaitcev | Sorry, deviating from obsoleting APIs here | 22:57 |
| *** gholt has joined #openstack-swift | 22:58 | |
| *** ChanServ sets mode: +v gholt | 22:58 | |
| *** vuvu has quit IRC | 22:59 | |
| *** vu_ has quit IRC | 23:00 | |
| zaitcev | https://blueprints.launchpad.net/swift/+spec/ec-proxy-work -- very mysterious blueprint | 23:01 |
| zaitcev | holy mother of overflowing basket of gerrit patches | 23:02 |
| zaitcev | 8 different reviews to implement a blueprint | 23:02 |
| *** tsnider has quit IRC | 23:03 | |
| *** otoolee has joined #openstack-swift | 23:06 | |
| *** vuvu has joined #openstack-swift | 23:09 | |
| *** vu_ has joined #openstack-swift | 23:09 | |
| *** tongli has quit IRC | 23:10 | |
| *** shri has joined #openstack-swift | 23:13 | |
| *** jergerber has quit IRC | 23:25 | |
| creiht | zaitcev: http://paste.openstack.org/show/72359/ | 23:29 |
| creiht | there's a dump of an saio proxy server while running a swift-bench | 23:29 |
| zaitcev | creiht: thanks but brb | 23:29 |
| creiht | no rush | 23:29 |
| *** hurricanerix has quit IRC | 23:31 | |
| *** hurricanerix has joined #openstack-swift | 23:31 | |
| creiht | there's a mix, but a large number of the threads show up with a guru_meditation traceback | 23:31 |
| creiht | not sure if that is desired or just a weird artifact, or... | 23:35 |
| *** hurricanerix has quit IRC | 23:36 | |
| openstackgerrit | gro qez proposed a change to openstack/python-swiftclient: Decode HTTP responses, fixes bug #1282861 https://review.openstack.org/78039 | 23:39 |
| *** vuvu has quit IRC | 23:40 | |
| *** vu_ has quit IRC | 23:40 | |
| openstackgerrit | gro qez proposed a change to openstack/python-swiftclient: Decode HTTP responses, fixes bug #1282861 https://review.openstack.org/78040 | 23:41 |
| openstackgerrit | gro qez proposed a change to openstack/python-swiftclient: Decode HTTP responses, fixes bug #1282861 https://review.openstack.org/78043 | 23:46 |
| zaitcev | creiht: Honestly I do not know where the noise comes from. Solly had fewer threads in his dumps. | 23:49 |
| creiht | zaitcev: it may just be noise | 23:49 |
| creiht | but not sure | 23:49 |
| zaitcev | 48073184 seems to be doing something | 23:49 |
| creiht | how do you debug the debugger? :) | 23:49 |
| creiht | yeah there are threads that are doing things | 23:50 |
| creiht | and some of my other dumps are the same way | 23:50 |
| creiht | including replciation | 23:50 |
| zaitcev | Here's one thing. We get all the objects by groping through garbage collector and comparing if the instance type is an eventlet. So... Is it possible that old ones are stuck around? One problem I see is that they seem like not exiting, not looking like dead threads. | 23:55 |
| zaitcev | The real test is going to be if you hit a mysterious hang and if this thing helps to find a race, missing timeout, or some logic issue. | 23:55 |
| creiht | yeah | 23:56 |
| creiht | zaitcev: I wonder if there is a better way to crawl the current green threads | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
