Friday, 2019-10-18

« Thursday, 2019-10-17 Index Saturday, 2019-10-19 »
*** mikecmpbll has quit IRC00:13
openstackgerritMerged openstack/swift stable/train: CHANGELOG for 2.23.1  https://review.opendev.org/68884800:18
*** NM has joined #openstack-swift00:38
*** nanzha has joined #openstack-swift01:06
*** NM has quit IRC01:32
*** manuvakery has quit IRC02:41
*** psachin has joined #openstack-swift03:37
*** diablo_rojo has quit IRC04:22
*** manuvakery has joined #openstack-swift05:43
*** manuvakery has quit IRC06:17
*** nanzha has quit IRC06:31
*** nanzha has joined #openstack-swift06:37
*** manuvakery has joined #openstack-swift06:42
openstackgerritOpenStack Proposal Bot proposed openstack/swift master: Imported Translations from Zanata  https://review.opendev.org/68888307:10
manuvakerydcourtoi using rabbimtq shovel on proxy-servers definitely help atleast the whole cluster wont fail when rabbitmq server is down. Still wondering even with nonblocking_notify=true why all proxy fails  completely07:14
*** tesseract has joined #openstack-swift07:22
*** pcaruana has joined #openstack-swift07:26
*** rdejoux has joined #openstack-swift07:30
*** rpittau|afk is now known as rpittau08:01
*** e0ne has joined #openstack-swift08:30
*** mikecmpbll has joined #openstack-swift08:39
*** mikecmpbll has quit IRC09:44
*** e0ne has quit IRC10:14
*** e0ne has joined #openstack-swift10:25
*** e0ne has quit IRC10:42
*** e0ne has joined #openstack-swift10:44
*** e0ne has quit IRC10:45
*** e0ne has joined #openstack-swift10:54
*** zigo has quit IRC10:54
*** e0ne has quit IRC10:59
*** zigo has joined #openstack-swift11:05
*** tkajinam has quit IRC11:16
*** e0ne has joined #openstack-swift11:18
*** e0ne_ has joined #openstack-swift11:20
*** e0ne has quit IRC11:23
*** e0ne_ has quit IRC11:25
*** manuvakery has quit IRC11:29
*** e0ne has joined #openstack-swift11:41
*** e0ne has quit IRC11:49
*** e0ne has joined #openstack-swift11:52
*** nanzha has quit IRC12:02
*** nanzha has joined #openstack-swift12:10
*** e0ne has quit IRC12:16
*** e0ne has joined #openstack-swift12:21
*** NM has joined #openstack-swift12:24
*** e0ne has quit IRC12:37
*** e0ne has joined #openstack-swift12:37
*** e0ne has quit IRC12:42
*** e0ne has joined #openstack-swift13:20
rledisezmanuvakery: after some testing and code analysis I came to the conclusion that nonblocking_notify=true is not working. I looked into it at some point to try to fix it, but it looked to me that it would imply major refactoring so I gave up. when we need to restart a rabbitmq (which is rare, we exclude the proxy-server from the load balancing) (as dcourtoi said we run rabbitmq locally to proxy-servers)13:27
*** e0ne has quit IRC13:33
*** e0ne has joined #openstack-swift13:34
*** e0ne has quit IRC13:34
*** e0ne has joined #openstack-swift13:35
*** rcernin has quit IRC13:40
*** e0ne has quit IRC13:42
*** e0ne has joined #openstack-swift13:42
*** manuvakery has joined #openstack-swift13:51
*** pcaruana has quit IRC13:53
*** psachin has quit IRC13:54
*** e0ne has quit IRC13:59
*** e0ne has joined #openstack-swift14:03
*** e0ne has quit IRC14:13
*** e0ne has joined #openstack-swift14:18
*** e0ne has quit IRC14:27
*** e0ne has joined #openstack-swift14:27
*** diablo_rojo has joined #openstack-swift14:51
claygi wonder how much I should be thinking about null-bytes in shard ranges?14:53
*** psachin has joined #openstack-swift14:57
*** FlorianFa has quit IRC15:01
*** gyee has joined #openstack-swift15:07
*** e0ne has quit IRC15:18
*** e0ne has joined #openstack-swift15:19
*** e0ne has quit IRC15:22
*** e0ne has joined #openstack-swift15:26
*** e0ne has quit IRC15:29
*** NM has quit IRC15:32
*** e0ne has joined #openstack-swift15:33
*** e0ne has quit IRC15:37
*** e0ne has joined #openstack-swift15:39
openstackgerritMerged openstack/swift master: Imported Translations from Zanata  https://review.opendev.org/68888315:52
*** e0ne_ has joined #openstack-swift15:56
*** e0ne has quit IRC15:58
*** rpittau is now known as rpittau|afk16:04
*** binwiederhier has joined #openstack-swift16:05
binwiederhierhey does swift support `Connection: keep-alive`/HTTP1.1?16:06
binwiederhierIt seems to keep the connection open after a PUT response, but then when I send another PUT it kills the connection on me16:06
*** nanzha has quit IRC16:11
*** e0ne_ has quit IRC16:16
*** e0ne has joined #openstack-swift16:17
*** rdejoux has quit IRC16:18
*** e0ne has quit IRC16:39
*** e0ne has joined #openstack-swift16:42
*** e0ne has quit IRC16:45
*** e0ne has joined #openstack-swift16:50
timburkebinwiederhier, yes, it should -- what do the proxy-server logs have to say?16:52
*** e0ne has quit IRC16:53
binwiederhierThey just show the single request even when I enabled DEBUG and eventlet_debug. I'll paste an example when I get to my desk16:57
*** NM has joined #openstack-swift16:58
binwiederhierhttps://pastebin.com/brs5Q6Ky17:04
binwiederhierhttps://pastebin.com/02rQJL4517:06
binwiederhierthe first one is a manual netcat with me pasting the PUT request into the terminal.17:06
binwiederhierthe connection is cut by the proxy after i send the second PUT17:07
*** e0ne has joined #openstack-swift17:11
timburkedo i see a newline between the "12345" and the "PUT /v1/..."? i think that might break the HTTP framing...17:15
*** e0ne has quit IRC17:16
timburkefwiw, i could do some pipelined requests with nc, but it's a little hard to read: http://paste.openstack.org/show/784733/17:16
*** tesseract has quit IRC17:18
timburkeadding a newline between my end of body and start of next request, i get a broken pipe, too: http://paste.openstack.org/show/784734/17:19
binwiederhieroh interesting.17:19
binwiederhierso you just continue with the next request after the content-length is reached.17:19
binwiederhierwill try17:19
binwiederhiertimburke: perfect. thank you!17:20
timburkechunk-encoded uploads might be somewhat more readable? i know there's a bunch more newlines with that17:21
timburkehappy to help! glad it's working better for you now :-)17:21
*** diablo_rojo has quit IRC17:25
binwiederhierNot sure if anyone is interested, but I made this plugin to protect from fatfinger deletes of accounts and objects: https://github.com/binwiederhier/swift-recycle17:48
binwiederhierCould be much more sophisticated, but this works :-)17:48
timburkecool! Thanks for putting it out there :-)17:50
timburkeout of curiosity, what are you using swift for? i love hearing user stories17:50
binwiederhierYou can ask Joe for more details :-) I work for Datto, a backup company. My company has a 700 node cluster and we're looking to use swift for another application.17:53
binwiederhier~ 150 PB17:53
tdasilvatimburke: with defaulter middleware and X-Object-Meta-Recycled could one apply that to every object PUT?17:54
timburkeoh, yeah! i've ben hearing a bit about you guys here at SwiftStack lately :-) definitely an impressive cluster!17:54
timburketdasilva ... yes? that'd be the idea, anyway17:55
binwiederhierWhile I have you guys, I two other questions: (1) Assuming no payload (empty/tiny objects), are there any metrics around what the performance of Swift can be? Our tests for with a 10 object-node cluster are not as exciting as I'd hope them to be. (2) The "small file prototype" seems to use a lot of Go and an arguably vastly different architecture than the rest of swift. How likely is that gonna get merged the way it is and when?17:59
timburkeon 2, i'm hoping that can be one of our main focuses for the next year or so. exact timelines will kinda depend on how much we get pulled in other directions -- things like s3 versioning and the like18:02
timburkei'm super-excited about it though -- feels like a model for how we could do more experimentation in the object servers18:03
binwiederhierSo I spent yesterday writing an inline packer that buffers and packs items as they are PUT and I was able get a 10x performance increase on my PUT tests.18:04
binwiederhierIt's just 200 lines of code or so and so far only handles PUTs and not GETs or DELETEs and repacking, but it is much simpler than what I saw the other code was. I'll share the code here when it's a little cleaned up.18:05
timburkenice! what's the behavior like in the event of a crash, though? i know rledisez is always trying to get rid of fsync calls, but then we get nervous about whether we really have durable data after returning a 201...18:07
timburkenow i'm curious. can't wait to take a look!18:08
binwiederhierIt doesnt return until the pack PUT is complete. It forwards the response to all original PUTs.18:09
timburkecool! love it!18:10
binwiederhierThat's why I'm asking for the PUTs/second, I am barely getting more than 1k/s without packing, and that's across 10 object nodes and proxies with 360 x 12T disks18:13
binwiederhierWith my inline packer I'm able to get 10k/s18:13
binwiederhier(form the application perspective)18:14
*** psachin has quit IRC18:15
timburkelooking at some data from when i was benchmarking encryption a few years ago, i think i was seeing ~100 PUTs/second, but i forget exactly what my setup was -- i wanna say just 5 nodes? not sure how many disks per node. also, i forget whether i was using a load balancer or just hitting one proxy...18:22
timburkelemme ask around, see if i can get you some better/more recent numbers18:22
rledisezbinwiederhier: basicilly, from a 12TB disk, you can expect about 40 PUT/s because of the way XFS and Swift works, it requires 3 sync per PUT (datafile, object dir, suffix dir). So, 40 PUT * 3 sync == 120 write IO, which is the IO capacity of a disk.18:27
rledisezto compare, on LOSF, we were reaching about 90 PUT/s (because it only requires 1 sync per PUT in the majority of cases)18:27
rledisezso, depending on the concurrency and numbers of disk, you can estimate your IO capacity18:27
timburkefwiw, i know the per-request overhead (HTTP parsing, spinning up back-end connections, fsyncs, etc.) dominates below about 1MB -- you'd get about the same req/s whether they're 0B, 1k, 100k. right around 1MB (in my testing, anyway), i'd expect to start to get network-bound18:29
timburkeoh hey, here's a nice graph! https://www.cisco.com/c/dam/en/us/td/docs/unified_computing/ucs/UCS_CVDs/ucs_s3260_m5_swiftstack.docx/_jcr_content/renditions/ucs_s3260_m5_swiftstack_89.png18:32
timburkelooks like that was 12 node, 28x 10TB drives/node18:33
timburkefound at https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/ucs_s3260_m5_swiftstack.html#_Toc53048760018:34
rledisez"+ 12 NVMe" => now I understand ;)18:35
rledisezhum, it says it's for A/C. I need to read this document because 13k PUT/s seems a lot on 112 drives (*3)18:37
timburkegotta love the 40Gb NICs :D18:43
binwiederhier(on my phone right now, so I can't get more numbers or details but we were not network bound at all. 10g NICs at all)18:50
timburkeyeah, i wouldn't expect you to be with small objects. just commenting/drooling :-)18:51
*** e0ne has joined #openstack-swift19:03
*** gyee has quit IRC19:39
*** gyee has joined #openstack-swift19:39
*** e0ne has quit IRC19:42
openstackgerritTim Burke proposed openstack/swift stable/stein: Sharding: Clean up old CleaveConext's during audit  https://review.opendev.org/68956719:45
*** e0ne has joined #openstack-swift19:47
openstackgerritTim Burke proposed openstack/swift stable/stein: versioned_writes: checks for SLO object before copy  https://review.opendev.org/68956819:54
openstackgerritTim Burke proposed openstack/swift stable/stein: dlo: Respond 200 on multi-range GETs  https://review.opendev.org/68957019:58
*** e0ne has quit IRC20:25
openstackgerritTim Burke proposed openstack/swift stable/stein: slo: Better handle non-manifest responses when refetching manifest  https://review.opendev.org/68958020:41
openstackgerritTim Burke proposed openstack/swift stable/stein: sharding: better handle get_shard_ranges failures  https://review.opendev.org/68958220:45
openstackgerritTim Burke proposed openstack/swift stable/stein: Fix invalid assert states  https://review.opendev.org/68958320:46
*** e0ne has joined #openstack-swift20:53
openstackgerritTim Burke proposed openstack/swift stable/stein: Fix incorrect setting of symlink_target_account  https://review.opendev.org/68958720:54
openstackgerritTim Burke proposed openstack/swift stable/stein: s3api: Fix ETag when copying a MU part from another MU  https://review.opendev.org/68958920:57
binwiederhierSo I can't see anything on that doc that says the did anything special here. I would have expected the nvme drives used as xfs logdev20:58
binwiederhierBut that doesn't seem to be the case.  It's PACO with nvmes for A/C20:59
openstackgerritTim Burke proposed openstack/swift stable/stein: Set Content-Type with s3api metadata updates.  https://review.opendev.org/68959121:00
openstackgerritTim Burke proposed openstack/swift stable/stein: S3Api: handle non-ASCII markers in v1 listings.  https://review.opendev.org/68959221:01
*** e0ne has quit IRC21:06
openstackgerritTim Burke proposed openstack/swift stable/stein: S3Api: handle non-ASCII markers in v1 listings.  https://review.opendev.org/68959221:20
*** NM has quit IRC21:49
*** joeljwright has quit IRC22:08
openstackgerritTim Burke proposed openstack/swift master: Add py38 unit test job  https://review.opendev.org/68959922:15
timburkelet's see if that works ;-)22:15
openstackgerritMerged openstack/swift stable/stein: Fix invalid assert states  https://review.opendev.org/68958323:00
openstackgerritMerged openstack/swift stable/stein: sharding: better handle get_shard_ranges failures  https://review.opendev.org/68958223:04
*** gyee has quit IRC23:56
« Thursday, 2019-10-17 Index Saturday, 2019-10-19 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!