Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have to copy 400G of files from an elastic block store volume to an s3 bucket... Those are about 300k files of ~1Mb

I've tried s3cmd and s3fuse, both of them are really, really slow.. s3cmd ran for a complete day, said it finished copying, and when I checked the bucket, nothing had happened (I suppose something went wrong, but at least s3cmd never complained of anything)

S3Fuse is working for an other complete day, and copied less than 10% of files...

Is there a better solution for this?

I'm running Linux (ubuntu 12.04) of course

share|improve this question
2  
Many benchmarks (e.g. this one) have demonstrated 3 determining factors of throughput to S3: 1) file size 2) number of parallel threads and 3) instance size. Between 64 and 128 parallel (simultaneous) uploads of 1MB objects should saturate the 1Gbps uplink that an m1.xlarge has and should even saturate the 10Gbps uplink of a cluster compute (cc1.4xlarge) instance. There should be many scripts with this in mind (e.g. this one or s3cmd-modification) –  cyberx86 May 8 '12 at 3:32
1  
s3-parallel-put did the trick! –  aseba May 8 '12 at 14:27

2 Answers 2

So, after a lot of testing s3-parallel-put did the trick awesomely. Clearly the solution if you need to upload a lot of files to S3. Thanks to cyberx86 for the comments.

share|improve this answer
3  
Out of curiosity, a) how long did it take to upload the 400GB b) how many threads did you use c) what instance size did you use? –  cyberx86 May 9 '12 at 2:18
    
@Cyberx86 I recently used s3-parallel-put on a Large Ec2 Instance. I used 5 threads and it copied 288.73 GB in 10.49 hours. –  Gortron Aug 28 '13 at 6:38

There are several key factors that determine throughput from EC2 to S3:

  • File size - smaller files require a larger number of requests and more overhead and transfer slower. The gain with filesize (when originating from EC2) is negligible for files larger than 256kB. (Whereas, transfering from a remote location, with higher latency, tends to continue showing appreciable improvements until between 1MiB and 2MiB).
  • Number of parallel threads - a single upload thread usually has a fairly low throughout - often below 5MiB/s. Throughput increases with the number of concurrent threads, and tends to peak between 64 and 128 threads. It should be noted that larger instances are able to handle a greater number of concurrent threads.
  • Instance size - As per the instance specifications, larger instances have more dedicated resources, including a larger (and less variable) allocation of network bandwidth (and I/O in general - including reading from ephemeral/EBS disks - which are network attached. Typical numbers values for each category are:
    • Very High: Theoretical: 10Gbps = 1250MB/s; Realistic: 8.8Gbps = 1100MB/s
    • High: Theoretical: 1Gbps = 125MB/s; Realistic: 750Mbps = 95MB/s
    • Moderate: Theoretical: 250Mbps; Realistic: 80Mbps = 10MB/s
    • Low: Theoretical: 100Mbps; Realistic: 10-15Mbps = 1-2MB/s

In cases of transferring large amounts of data, it may be economically practical to use a cluster compute instance, as the effective gain in throughput (>10x) is more than the difference in cost (2-3x).

While the above ideas are fairly logical (although, the per-thread cap may not be), it is quite easy to find benchmarks backing them up. One particularly detailed one can be found here.

Using between 64 and 128 parallel (simultaneous) uploads of 1MB objects should saturate the 1Gbps uplink that an m1.xlarge has and should even saturate the 10Gbps uplink of a cluster compute (cc1.4xlarge) instance.

While it is fairly easy to change instance size, the other two factors may be harder to manage.

  • File size is usually fixed - we cannot join files together on EC2 and have them split apart on S3 (so, there isn't much we can do about small files). Large files however, we can split apart on the EC2 side and reassemble on the S3 side (using S3's multi-part upload). Typically, this is advantageous for files that are larger than 100MB.
  • Parallel threads is a bit harder to cater to. The simplest approach comes down to writing a wrapper for some existing upload script that will run multiple copies of it at once. Better approaches use the API directly to accomplish something similar. Keeping in mind that the key is parallel requests, it is not difficult to locate several potential scripts, for example:
    • s3cmd-modification - a fork of an early version of s3cmd that added this functionality, but hasn't been updated in several years.
    • s3-parallel-put - reasonably recent python script that works well
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.