I have several terabytes of data I need to back up to S3/Glacier over a slow link. I want to upload from, say 9pm to 9am and resume from where it left off. Is there free software that would do that easily? I could use s3cmd/s3sync for the uploading, but is there a simple way to control it with cron? I'm OK with several days latency, since I have static data and am just adding another layer of redundancy.

Would it be best to just start a process and kill it, then restart sync? I'm mainly looking for a standard system, but I could probably just write some shell scripts. I'm using Ubuntu and want to use standard Linux tools.

  • Are there lots of small files, one or two really large files or a combination? – GeoSword Nov 12 '13 at 21:29
  • Lots of moderate size files, 10 megs up to 1 gig but nothing huge. I need to get about 600 gigs up in a few weeks, with the next several terabytes trickling up gradually. My connection speed is about 5 megs/sec upload. – sventechie Nov 12 '13 at 22:03
  • Does it need to be from an ubuntu box? There's a few Glacier clients that support bandwidth throttling, but I'm not sure how many are linux-compatible. Example: fastglacier.com – jlehtinen Nov 12 '13 at 22:12
  • More glacier clients here: labnol.org/internet/amazon-glacier-clients/25314 – jlehtinen Nov 12 '13 at 22:16

Of course any upload in the terrabyte range is going to take some time, how slow is your "slow" uplink?

A 1 terrabyte upload over a 10Mbit uplink will take 1024*1024 = 1048576 seconds = 290 hours = 12 full days.

I would take a look at Amazon's bulk upload by shipping a NAS appliance

Take a look at http://aws.amazon.com/importexport/

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.