I am running Linux on Amazon EC2 severs. I need to copy millions of files that total hundreds of gigabytes between two EC2 systems in the same availability zone. I don't need to sync directories, I just need to copy all the files in one directory over to an empty directory on the other machine.

What is the fastest way to do this? Has anyone seen or run performance tests?

rsync? scp? Should I zip them first? Should I detach the drive they are on and re-attach it to the machine I'm copying to, then copy them? Does transferring over the EC2's Private IP speed things up?

Any thoughts would be appreciated.

NOTE: Sorry this was unclear, but I'm copying data between two EC2 systems both in the same AWS availability zone.

  • Rackspace lets you FedEx them a hard drive.. Wonder if Amazon would go for this.. – Mike Christensen Feb 9 '12 at 0:29
  • 1
    Amazon offers a few services to fit this need. ( aws.amazon.com/importexport & aws.amazon.com/storagegateway ). Different situation though. My data is already in their network. – T. Brian Jones Feb 9 '12 at 0:32
  • 3
    If detaching the EBS volume and attaching it to another instance is an option for you, I can't really see anything else working faster than a simple cp between the two drives mounted on the same instance. (EBS volumes incur a network overhead as it is, adding more to that by copying over a network can't do anything except slow things down. That said, AWS internal network is reasonably fast - just don't use micro/small instances for the copy as they are i/o constrained.) – cyberx86 Feb 9 '12 at 1:13

If the files are already on an EBS volume (and if you care about them, why aren't they?):

  1. Create a snapshot of the EBS volume containing the files on the first instance.

  2. Create an EBS volume from that snapshot.

  3. Attach the EBS volume to the second instance.

The new EBS volume may be a bit slow for a bit while it's filling in blocks from the snapshot, but it will be usable right away.

ALTERNATIVE (If the files are not already on an EBS volume):

  1. Attach a new EBS volume to the first instance.

  2. Copy the files from other disks to the new EBS volume.

  3. Move the EBS volume to the second instance.

  • Great solution for a lot of cases, but doesn't exactly solve my problem. I'm combining files from many drives on systems that were manipulating them in parallel onto one ( or a few ) drives on a single system. – T. Brian Jones Feb 9 '12 at 4:47
  • @T.BrianJones You should still be able to use this. Mount the EBS volumes to the new instance with the few large drives and copy everything over. – ceejayoz Feb 9 '12 at 16:15

Use tar and netcat. If they're in the same subnet, and you're not too concerned about security.. This is a pretty neat solution. You can add stuff into the pipeline if you want security.. You could use gpg.. or compress first with gzip

On the receiving end do:

netcat -l -p 7000 | tar x

And on the sending end do:

tar cf - * | netcat otherhost 7000
  • 1
    +1 for coolness and simplicity . This will do the trick. – The Unix Janitor Feb 9 '12 at 16:20

You can use Amazon AWS Import/Export service. Ship the drive to them and let them do the copy for you. More expensive, but perfect in your situation when you need have many many GB of data transfer without waiting for weeks for the job to finish. Their link: http://aws.amazon.com/importexport/

  • Sorry I was unclear. I'm copying between EC2 systems in the same availability zone. – T. Brian Jones Feb 9 '12 at 1:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.