Friday, 2017-06-09

*** jamielennox|away is now known as jamielennox00:02
*** lucasxu has joined #openstack-swift00:46
*** lucasxu has quit IRC00:46
*** zhurong has joined #openstack-swift00:49
*** zhengyin has joined #openstack-swift00:58
*** jamielennox is now known as jamielennox|away01:00
*** jamielennox|away is now known as jamielennox01:10
kota_good morning!01:14
kota_yey, https://review.openstack.org/439572 landed01:15
patchbotpatch 439572 - swift - Limit number of revert tombstone SSYNC requests (MERGED)01:15
kota_thanks clayg and mahatic01:15
charzclayg: thanks! checking01:16
*** zhurong has quit IRC01:29
*** mingyu has joined #openstack-swift02:03
*** mingyu has quit IRC02:04
*** zhurong has joined #openstack-swift02:04
charzLooks it passed.02:04
*** mingyu has joined #openstack-swift02:12
*** mingyu has quit IRC02:48
*** mingyu has joined #openstack-swift02:50
*** mingyu has quit IRC02:51
*** mingyu has joined #openstack-swift02:53
*** mingyu has quit IRC03:09
*** mingyu has joined #openstack-swift03:35
*** gkadam has joined #openstack-swift03:38
mahaticgood morning04:15
*** SkyRocknRoll_ has joined #openstack-swift04:15
mahaticyay patch 439572 landed04:15
patchbothttps://review.openstack.org/#/c/439572/ - swift - Limit number of revert tombstone SSYNC requests (MERGED)04:15
*** SkyRocknRoll has quit IRC04:17
*** mingyu has quit IRC04:19
*** links has joined #openstack-swift04:20
*** zhurong has quit IRC04:25
*** psachin has joined #openstack-swift04:31
*** zhurong has joined #openstack-swift04:34
*** zhengyin has quit IRC04:35
*** aselius has quit IRC04:36
*** zhengyin has joined #openstack-swift04:40
*** gyee has quit IRC04:49
*** gabor_antal has quit IRC05:07
*** skudlik has joined #openstack-swift05:09
*** links has quit IRC05:10
*** aselius_ has joined #openstack-swift05:12
*** links has joined #openstack-swift05:25
*** SkyRocknRoll_ has quit IRC05:43
*** cshastri has joined #openstack-swift05:48
*** klrmn has quit IRC06:02
*** ChubYann has quit IRC06:05
*** zhengyin has quit IRC06:08
*** zhengyin has joined #openstack-swift06:09
*** jaosorior has joined #openstack-swift06:18
*** zhengyin has quit IRC06:27
*** zhengyin has joined #openstack-swift06:28
*** zhengyin has quit IRC06:32
*** zhengyin has joined #openstack-swift06:34
*** cbartz has joined #openstack-swift06:37
*** zhengyin has quit IRC06:40
openstackgerritPallavi proposed openstack/swift master: Fix html_last_updated_fmt for Python3.  https://review.openstack.org/47253406:40
*** zhengyin has joined #openstack-swift06:40
*** zhengyin has quit IRC06:47
*** zhengyin has joined #openstack-swift06:47
*** hseipp has joined #openstack-swift06:48
*** geaaru has joined #openstack-swift06:49
*** hseipp has quit IRC06:51
*** hseipp has joined #openstack-swift06:51
*** zhengyin has quit IRC06:52
*** zhengyin has joined #openstack-swift06:52
*** tesseract has joined #openstack-swift06:53
openstackgerritTimur Alperovich proposed openstack/python-swiftclient master: Default to binary/octet-stream.  https://review.openstack.org/47254106:54
*** zhurong has quit IRC06:55
*** zhurong has joined #openstack-swift06:56
*** zhurong has quit IRC06:57
*** pcaruana has joined #openstack-swift07:05
*** early has quit IRC07:07
*** links has quit IRC07:09
*** rcernin has joined #openstack-swift07:09
*** early has joined #openstack-swift07:12
*** aselius_ has quit IRC07:21
*** links has joined #openstack-swift07:25
*** kei_yama has quit IRC07:37
*** kei_yama has joined #openstack-swift07:45
*** SkyRocknRoll has joined #openstack-swift07:58
*** oshritf has joined #openstack-swift08:13
*** qwerty has joined #openstack-swift08:21
*** qwerty has quit IRC08:38
*** qwerty has joined #openstack-swift08:41
*** oshritf has quit IRC08:48
*** qwerty has quit IRC08:55
*** Dw_Sn has joined #openstack-swift09:23
*** qwerty has joined #openstack-swift09:27
*** qwerty has quit IRC09:33
openstackgerritHieu LE proposed openstack/swift master: OSprofiler in OpenStack Swift  https://review.openstack.org/46831609:41
openstackgerritHieu LE proposed openstack/swift master: OSprofiler in OpenStack Swift  https://review.openstack.org/46831609:42
openstackgerritLingyong Xu proposed openstack/python-swiftclient master: Drop py34 target in tox.ini  https://review.openstack.org/47168509:43
*** links has quit IRC09:51
*** zhurong has joined #openstack-swift09:55
*** links has joined #openstack-swift10:03
*** zhurong has quit IRC10:06
openstackgerritLingyong Xu proposed openstack/swift master: Fix html_last_updated_fmt for Python3  https://review.openstack.org/47262110:09
*** mat128 has joined #openstack-swift11:08
*** psachin has quit IRC11:13
*** zhengyin has quit IRC11:33
*** SkyRocknRoll has quit IRC11:51
*** SkyRocknRoll has joined #openstack-swift11:55
rledisezclayg: agree that 10 devices per worker for 24 devices should give 8 devices per worker. seems reasonable. I don't see any reason someone would want 10+10+4, because the attribution may be "unpredictable" (well, it's not random, but you wouldn't be able do a real attribution)12:05
rledisezbut i agree that everywhere concurrency is configured in swift, it's about how many workers/thread/… to run. so it might be confusing for the operator that this particular item is not that way, but the opposite (we calculate the concurrency)12:07
*** kei_yama has quit IRC12:17
rledisezacoles: I had a look at https://bugs.launchpad.net/swift/+bug/1652323 , it's actually quite simple (too simple to be real?). i should write some new tests, but at least existing tests didn't break. i'll have a look on monday for tests. in the mean time i'm pushing the patch (see below)12:29
openstackLaunchpad bug 1652323 in OpenStack Object Storage (swift) "ssync syncs an expired object as a tombstone" [Medium,Confirmed]12:29
openstackgerritRomain LE DISEZ proposed openstack/swift master: Allow to rebuild a fragment of an expired object  https://review.openstack.org/47265912:29
*** jaosorior has quit IRC12:37
*** blair has quit IRC12:39
*** catintheroof has joined #openstack-swift12:40
*** catintheroof has quit IRC12:45
*** catintheroof has joined #openstack-swift12:46
*** links has quit IRC12:48
*** SkyRocknRoll has quit IRC12:55
*** joeljwright has joined #openstack-swift13:02
*** ChanServ sets mode: +v joeljwright13:02
*** lucasxu has joined #openstack-swift13:03
*** SkyRocknRoll has joined #openstack-swift13:03
*** blair has joined #openstack-swift13:10
*** lucasxu has quit IRC13:16
*** ujjain has quit IRC13:21
*** klamath has joined #openstack-swift13:30
*** klamath has quit IRC13:31
*** klamath has joined #openstack-swift13:32
*** klamath has quit IRC13:35
*** lcurtis has joined #openstack-swift13:45
*** chlong has joined #openstack-swift13:51
*** jaosorior has joined #openstack-swift13:56
*** lucasxu has joined #openstack-swift14:09
lcurtishello all...running into some sever performance problems on container servers14:10
lcurtiscould someone tell me the algorithm proxy server uses for choosing a container server?14:11
lcurtisor maybe some tweaks for faster container response?14:12
lcurtisusing ssds14:12
*** lucasxu has quit IRC14:14
*** lucasxu has joined #openstack-swift14:14
*** lucasxu has quit IRC14:16
*** tesseract has quit IRC14:16
*** aselius_ has joined #openstack-swift14:19
*** lucasxu has joined #openstack-swift14:20
*** tesseract has joined #openstack-swift14:23
*** lucasxu has quit IRC14:23
*** SkyRocknRoll has quit IRC14:25
*** gkadam_ has joined #openstack-swift14:26
*** gkadam has quit IRC14:26
*** SkyRocknRoll has joined #openstack-swift14:28
*** oshritf has joined #openstack-swift14:29
*** cshastri has quit IRC14:30
*** gkadam_ has quit IRC14:31
*** Dinesh_Bhor has quit IRC14:44
*** oshritf has quit IRC14:52
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted now to clear an issue arising from an unanticipated SSH API connection flood14:58
*** oshritf has joined #openstack-swift15:04
*** oshritf has quit IRC15:07
*** rcernin has quit IRC15:08
*** jaosorior has quit IRC15:09
Dw_Sntimburke: would you please leave me an answer regarding how can I use the swift3 middleware for pre-signed instead of tempurl  ? I have the module enabled, but can't find any docs on how to sign and generate the AWS S3 look alike URL ..15:09
*** oshritf has joined #openstack-swift15:15
*** oshritf has quit IRC15:17
*** oshritf has joined #openstack-swift15:17
*** oshritf has quit IRC15:25
*** aselius_ has quit IRC15:26
*** aselius has joined #openstack-swift15:26
*** oshritf has joined #openstack-swift15:28
*** oshritf has quit IRC15:30
clayglcurtis: what makes you think the issue is container server?15:36
openstackgerritGábor Antal proposed openstack/swift master: Use more specific asserts in test/unit/common  https://review.openstack.org/34278115:37
clayglcurtis: generally speaking on object PUT the container server selection is deterministic by container name.15:37
lcurtisclayg: getting timeouts and update failures, tons of async pendings15:39
lcurtisdefinitely containers15:39
lcurtisasync pending in the billions15:39
*** itlinux has joined #openstack-swift15:40
*** chsc has joined #openstack-swift15:40
timburkeDw_Sn: it works the same as is described at http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html -- the region name used in the credential needs to match the location configured in the proxy-server.conf, but aside from that, there isn't much special; <your-access-key-id> and <SecretAccessKey> come out of keystone (or whatever auth system)15:41
timburkefwiw, i've used boto to generate such urls before: https://github.com/tipabu/swift3-demo/blob/master/boto_demo.py#L1615:42
*** oshritf has joined #openstack-swift15:46
*** oshritf has quit IRC15:48
*** cbartz has quit IRC15:48
*** oshritf has joined #openstack-swift15:49
Dw_Sntimburke: cool ! I will give it a try next week !15:50
Dw_Snthanks a lot :)15:50
timburkesure! sorry it took so long to unblock you15:51
*** oshritf has quit IRC15:52
Dw_Sntimburke: no worries at all, I will look into it, not sure it will be the exact same as I expected but will see :)15:52
*** gyee has joined #openstack-swift15:53
Dw_Snalso I don't think I have location in my proxy-server.conf15:53
timburkeiirc it defaults to "US"15:53
*** SkyRocknRoll_ has joined #openstack-swift15:53
Dw_Sntimburke: cool ! thanks again, I will read about if changing a location now might mess up anything or not .. and read the link you sent :)15:55
*** SkyRocknRoll has quit IRC15:55
*** Dw_Sn has quit IRC15:58
*** oshritf has joined #openstack-swift16:02
notmynamegood morning16:04
*** oshritf has quit IRC16:06
*** chsc has quit IRC16:06
*** oshritf has joined #openstack-swift16:08
*** itlinux has quit IRC16:09
*** oshritf has quit IRC16:10
*** klrmn has joined #openstack-swift16:12
clayglcurtis: do you have a container server offline?  Are the update timeouts/error all pointing to the same node/ip/disk/part?16:13
*** oshritf has joined #openstack-swift16:14
*** klrmn has quit IRC16:14
lcurtisclayg: no....numerous containers16:14
lcurtisall container servers under heavy load16:14
claygon the container servers are the updates coming from the container updaters?16:15
*** oshritf has quit IRC16:15
*** joeljwright has quit IRC16:16
notmynamelcurtis: what version of swift?16:17
lcurtisclayg: seeing this on object servers16:17
lcurtiswe have proxy and object on same server16:17
lcurtiscontainers separate16:17
clayg"this" == ????16:17
lcurtissorry...the timeouts i mentioned16:18
lcurtisupdates and heads16:18
lcurtisversion is 2.2.0-0ubuntu1~cloud016:18
lcurtiswhich is prob quite old16:19
*** ChubYann has joined #openstack-swift16:20
*** oshritf has joined #openstack-swift16:21
claygok, so the proxy's are trying to HEAD containers for the existence check cache - the object servers are trying to update the containers with either new PUT/DELETE (or the existing async pending updates via the swift-object-updater which I called the "container updater" earlier because it updates the containers) - and both (all three) processes are on the same physical server and throwing timeouts talking to container16:21
claygservers.16:21
clayg... which you said were under heavy load16:21
claygwhat do things look like from the container-servers perspective?16:22
*** oshritf has quit IRC16:22
claygcan you get a sense if the majority of updates are coming from the proxy-server, object-server or object-updater?  The requests should say in the user-agent line...16:22
*** chsc has joined #openstack-swift16:24
lcurtiswe are trying to add more nodes into container layer16:25
lcurtisso replication also under heavy load on container servers16:25
lcurtiswe may just need to wait it out16:25
*** hseipp has quit IRC16:27
*** vint_bra has joined #openstack-swift16:32
*** tesseract has quit IRC16:32
*** pcaruana has quit IRC16:34
*** SkyRocknRoll_ has quit IRC16:37
*** oshritf has joined #openstack-swift16:38
claygmaybe, you might slow things down overall trying to do too many things at once.  Maybe stop the object-updater until the replicators finish?  Still not clear if the load is on disks or cpu.  Good luck!16:38
*** oshritf has quit IRC16:40
*** SkyRocknRoll has joined #openstack-swift16:40
*** rcernin has joined #openstack-swift16:45
*** honga has joined #openstack-swift16:46
lcurtisthanks clayg16:47
hongahello, is there a way I can view the contents of a dlo/slo manifest file instead of downloading the actual file?16:47
lcurtisload definitely on the ssds16:47
lcurtishigh iowait16:47
*** oshritf has joined #openstack-swift16:47
notmynamelcurtis: do you have a good sense of how many object write (PUT/DELETE) operations per second each container is getting? how many objects are in each container?16:48
notmynamehonga: add "?multipart-manifest=get" to the end of the GET request16:48
*** oshritf has quit IRC16:49
hongadoh, it was right in the docs :P thanks notmyname16:49
lcurtisnotmyname: what would be best way to check ops / sec on containers?16:52
*** oshritf has joined #openstack-swift16:53
notmynamethe best way? track the logs, feed it to some analysis, group by container, split by verb, watch the graphs16:53
notmynamelcurtis: the fastest way? grab some logs, count and divide by the seconds of logs you grabbed16:54
lcurtisi am told about 10k objects per container16:56
notmynamelcurtis: but where I'm going with this is checking to see if you're in the 10/sec, 100/sec, or 1000/sec range, and then compare that against how many objects are in the container16:56
lcurtisyes...checking16:56
lcurtisthanks much16:56
notmyname10k objects in a container isn't much16:59
notmynamehowever, if the usage pattern on the container is very ... idk ... "fluid"? ie instead of just adding stuff you're also deleting stuff17:00
notmynameeg you have 10k objects, but you're adding 1 million per day and deleting 1 million per day17:00
notmynamethat could end up with larger on-disk container DBs, which IIRC would result in longer lock times for sqlite, which results in more lock errors and timeouts and therefore async pendings17:01
notmyname(that's based on my assumption that a non-vacuumed container DB that is large on-disk is similarly slow to acquire a lock or do a DB index update as a DB that's big because it's got a lot of objects in it)17:02
*** klrmn has joined #openstack-swift17:10
openstackgerritTimur Alperovich proposed openstack/python-swiftclient master: Default to binary/octet-stream.  https://review.openstack.org/47254117:17
*** vint_bra has quit IRC17:17
*** tonanhngo has joined #openstack-swift17:21
*** vint_bra has joined #openstack-swift17:22
timburkenotmyname: there are a couple other complications with "fluid" containers -- we've gotta keep the deleted rows around for reclaim_age (which i'm fairly certain *must* have the same characteristics as a "normal" container with a lot of objects in it) and when we *do* call reclaim, we're doing a scan over *all* deleted objects -- there's no deleted, created_at index for containers (nor a deleted, delete_timestamp index for accounts)17:30
notmynameyep17:30
notmynameall of that. plus I was thinking about our customer who has large on-disk DBs even after the reclaim age, just because the DBs aren't vacuumed17:31
notmynameI think my average words-per-email that I send to the openstack ML must be measured in the 1000s :-(17:34
* notmyname is currently working on the email about patch 46810517:35
patchbothttps://review.openstack.org/#/c/468105/ - swift - Require that known-bad EC schemes be deprecated17:35
notmynameit's going to be long17:35
timburkeoh yeah, speaking of MLs -- notmyname, should we do a swiftclient release in the nearish future?17:35
notmynamelet me check my helper tool...17:38
*** mat128 has quit IRC17:39
notmynamehttps://gist.github.com/notmyname/e2b5292bad2891ed4b67f3b80780739a17:39
*** mat128 has joined #openstack-swift17:41
*** mat128 has quit IRC17:41
*** oshritf has quit IRC17:51
*** vint_bra has quit IRC18:11
*** vint_bra has joined #openstack-swift18:13
*** geaaru has quit IRC18:15
claygwhy did we put 'workers' in [DEFAULT]18:26
claygdo we have freedom!?18:26
notmynamehttp://www.alexmaximo.com/wp-content/uploads/2016/02/braveheart-freedom-400x250.jpg18:42
tdasilvaclayg: or free doom?18:42
notmynamehttps://upload.wikimedia.org/wikipedia/en/c/c8/Doom_gibs.png ?18:44
claygi think you are all mocking me18:44
claygbut srly, I can't name the reconstructors workers config option workers :\18:45
notmynameclayg: ah. that sthinks18:45
notmynamesorry, I just spend a lot of time concentrating, just finished, and now need to let off steam18:45
notmynameand I type with a lithp?18:45
claygok18:46
* notmyname should probably stand up and walk around instead of make comments in irc18:46
claygi could either call the option "processes" stealing from the expirer (where processes means soemthing else entirely) or I could require "use_workers = true" + "set workers = X" config syntax18:49
* clayg shrugs18:50
claygi'll use "reconstructor_workers" for now and sort it out during review18:50
notmynameI like "reconstructor_workers"18:51
*** itlinux_ has joined #openstack-swift19:27
tdasilvatimburke: about swift3 acls, my understanding is that swift3 has some acls support correct? what is missing is just the object-level ACL support?20:29
*** itlinux_ has quit IRC20:29
timburketdasilva: we've even got some support for that -- it's just much more wasteful. gotta go start reading the object, get some (sys?)meta to the proxy, then decide whether or not to allow access20:30
timburkei honestly don't remember the full state of acls in swift3. i know we don't pass all the ceph s3 tests about them, but as i recall we passed more than i was expecting20:31
tdasilvatimburke: right, that's what I was going to say next, I remember IBM??? was working on that piece, where you would have to essentially HEAD the object20:31
tdasilvatimburke: ah ok, cool20:31
tdasilvatimburke: are the ceph s3 tests results published anywhere?20:31
timburkemight be worth grepping for acl in https://github.com/openstack/swift3/blob/master/swift3/test/functional/conf/ceph-known-failures-tempauth.yaml ...20:32
timburkea lot of that just isn't going to happen -- we don't have a way to look up users, for example, so you can't really do user existence checks or grants by email20:33
timburkeother parts are tricky for other reasons -- all the "raw" stuff really means "unauthenticated" and we don't (currently) have a way to know what account to use20:34
tdasilvatimburke: so that's a list of known failures? is there a list of what's known to pass? :P or do i just have to collate the data?20:35
tdasilvatimburke: ahh,,that's the public container thing, right?20:35
timburkeone of those lists is much longer than the other :-)20:36
tdasilvaS3 having global namespace, you can just make container public and don't really have the concept of an account20:36
timburkei felt like it was better to carve out the exclusions20:36
timburkeyup20:36
*** honga has quit IRC20:36
timburkei've got a thought about making a .swift3-buckets account or somesuch but haven't found the time...20:36
timburkeand migrating existing buckets will be...interesting20:37
tdasilvahe20:37
*** itlinux_ has joined #openstack-swift20:39
*** SkyRocknRoll has quit IRC21:10
claygnotmyname: oh and maybe late Friday afternoon isn't the best time to send the ec warning - could wait till Monday morning...21:14
notmynameclayg: yep. that's exactly what I'm doing21:14
notmynamealso wanting to make sure rledisez thinks it's good from an ops perspective, and acoles and cschwede in EU21:14
claygrighto!21:23
*** chlong has quit IRC21:58
*** itlinux_ has quit IRC22:06
*** skudlik has quit IRC22:11
*** rcernin has quit IRC22:19
lcurtishmmm..an i correct in stating replication happens source to destination?22:27
lcurtisi think issue is that i weighted down 2 container servers at once to replace ssds22:27
lcurtiscontainer layer under heavy load22:27
lcurtisand now trying to weight these container servers back in22:28
lcurtisthey are under no load, but rest of container servers are22:28
lcurtiscan i do some kind of reverse rsync?22:28
timurtimburke: cschwede I looked into the python3 + requests issue with the Content-Type being set to application/x-www-form-urlencoded and found that the issue is fixed with requests versions > 2.4. I'd like to consider either bumping the requests version in requirements.txt or doing a version check and only sending "Content-Type: " when we detect an older requests library (and python3?)22:31
openstackgerritClay Gerrard proposed openstack/swift master: Untangle ServersPerPortStrategy.bind_ports  https://review.openstack.org/47285122:31
clayglcurtis: all swift replication is pushed22:32
lcurtisany utils to do reverse replication so can utilize cpu on nodes being weighted back in?22:33
*** vint_bra has quit IRC22:35
*** ^andrea^ has joined #openstack-swift22:41
*** ^_andrea_^ has joined #openstack-swift22:41
*** ^_andrea_^ has quit IRC22:41
*** catintheroof has quit IRC22:42
*** lcurtis has quit IRC22:49
openstackgerritTimur Alperovich proposed openstack/python-swiftclient master: Do not set Content-Type to ''.  https://review.openstack.org/47254122:50
openstackgerritTimur Alperovich proposed openstack/python-swiftclient master: Do not set Content-Type to '' with new requests.  https://review.openstack.org/47254122:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!