Experimental radically improved throughput on S3/Elasticdrive/RAID0 (or not)
Experimental radically improved throughput on S3/Elasticdrive/RAID0 (or not)
Mon, 01/21/2008 - 03:20 — Derek AndersonAfter testing elasticdrive all week (painful and repetitive, but necessary), I stumbled upon an idea to radically improve throughput.
The original ED implementation was not a good performer for RAID5, due to it's multithreaded architecture, but the new version supports an ordered read/write model, and has a much lower cpu and memory footprint...
I built out a new elasticdrive where the same bucket was used for 4 different drives (labeled s3_0 through s3_3), and where the prefix was different...
Next I mounted each drive with losetup, and then used mdadm to create a raid0 array (yes, you can parallelize more than 2 disks).
Interesting... now I have redundant storage on S3, with ordered writes, and 4x the thoughput as a single drive (which means I should get around 32 MBytes/s, which compares favorably to a local USB drive). Then I found a dumb oversight (svn changelog #42, which was not a bug, but limited performance), then fixed it, then tested.
[4:15:27PM]
...
[7:17:39PM]
...
[10:11:00PM]
Damn. Not as fast as I had hoped :(


