up vote 2 down vote favorite
share [g+] share [fb]

What's the best way to upload 200GB tar.gz files to S3 in Linux? On researching I found that S3 limit on objects has been increased to 5TB and came to know about the multi part upload mechanism to speed uploads. I found a python script boto which might help on this. Could someone shed more light on this topic?

link|improve this question
feedback

migrated from stackoverflow.com Dec 2 '11 at 21:35

This question came from our site for professional and enthusiast programmers.

3 Answers

up vote 10 down vote

Don't upload it. Post it. http://aws.amazon.com/importexport/

Amazon offer a service where you send them portable media, and they upload the data for you, from their fast backbone.

If you're really hellbent on doing it yourself, grab a copy of S3cmd and do s3cmd sync.

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." - Andrew S Tanenbaum

Edit: If you really want to be able to chunk the file upload, I suggest you do the following.

  1. Get hold of an AWS EC2 instance with enough ephemeral storage to hold the files you want to upload.
  2. Use GNU Split to divide the files into smaller chunks.
  3. Upload the chunks to your temporary EC2 instance.
  4. Reassemble the chunks with the split option to reaassemble.
  5. Upload the chunks to S3 from EC2 (blindingly fast!)
  6. Shutdown the EC2 instance, but keep it handy.
link|improve this answer
Thanks for your reply. Aws Import/Export option might not be an option as i might have many files further down the line. s3cmd options does not have the multipart option due to which any interruption would cause a huge problem as i need to restart my upload again. I found many tools for windows like s3browser,bucket explorer which support multi part but not anything for linux.So I am looking for somethinging linux where i can resume incase of interruptions or break down the big files into smaller chunks and upload in parallel and finally couple them into single file. On those lines i came to k – Prem Anand Jayaraman Dec 3 '11 at 0:15
+1 for Sneakernet. It's old, it's low-tech, and it just keeps going on and on and on... – Avery Payne Dec 3 '11 at 2:28
@prem I updated my answer with one alternative for uploading. You can also run s3cmd in a screen window, and hope that TCP checksums are enough to restart a stalled transfer. You might get better performance from rsync to an EC2 node, which then uses rsync's delta algorithms to restart from a decent position. S3cmd sync does similar things, but not as cleanly. – Tom O'Connor Dec 3 '11 at 23:27
feedback
up vote 2 down vote accepted

Thanks for your reply and options tom.I got was able to achieve a 20gb upload to s3 using mulipart upload.I needed python 2.5> + boto library + s3_mulitpart python script to do the upload. My references where

  1. http://code.google.com/p/boto/ - boto ( used 2.1.1)
  2. http://www.elastician.com/2010/12/s3-multipart-upload-in-boto.html : mulipart upload using boto.
  3. http://bcbio.wordpress.com/2011/04/10/parallel-upload-to-amazon-s3-with-python-boto-and-multiprocessing/ : parallel upload to Amazon S3 script

Hope these are useful.

Prem

link|improve this answer
feedback
up vote 0 down vote

Split your tar.gz files into smaller chunks and track their local and remote MD5 sums. Then you can just re-upload a smaller portion should something go wrong.

link|improve this answer
yup thats precisely the principle of parallel upload to Amazon S3 script . ( bcbio.wordpress.com/2011/04/10/…) . Divides into smaller chunks using split ,then does multiprocessing of the smaller chunks and finally combine them in s3. There is one another caveat the md5 of the multi-upload (i.e Etag in s3 for the files) is different from the original file.But if i download the s3 uploaded file and then compare with the md5 they match. – Prem Anand Jayaraman Dec 7 '11 at 20:49
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.