Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am running Linux on Amazon EC2 severs. I need to copy millions of files that total hundreds of gigabytes between two EC2 systems in the same availability zone. I don't need to sync directories, I just need to copy all the files in one directory over to an empty directory on the other machine.

What is the fastest way to do this? Has anyone seen or run performance tests?

rsync? scp? Should I zip them first? Should I detach the drive they are on and re-attach it to the machine I'm copying to, then copy them? Does transferring over the EC2's Private IP speed things up?

Any thoughts would be appreciated.

NOTE: Sorry this was unclear, but I'm copying data between two EC2 systems both in the same AWS availability zone.

share|improve this question

migrated from stackoverflow.com Feb 9 '12 at 0:49

This question came from our site for professional and enthusiast programmers.

    
Rackspace lets you FedEx them a hard drive.. Wonder if Amazon would go for this.. –  Mike Christensen Feb 9 '12 at 0:29
1  
Amazon offers a few services to fit this need. ( aws.amazon.com/importexport & aws.amazon.com/storagegateway ). Different situation though. My data is already in their network. –  T. Brian Jones Feb 9 '12 at 0:32
3  
If detaching the EBS volume and attaching it to another instance is an option for you, I can't really see anything else working faster than a simple cp between the two drives mounted on the same instance. (EBS volumes incur a network overhead as it is, adding more to that by copying over a network can't do anything except slow things down. That said, AWS internal network is reasonably fast - just don't use micro/small instances for the copy as they are i/o constrained.) –  cyberx86 Feb 9 '12 at 1:13
add comment

3 Answers

up vote 9 down vote accepted

If the files are already on an EBS volume (and if you care about them, why aren't they?):

  1. Create a snapshot of the EBS volume containing the files on the first instance.

  2. Create an EBS volume from that snapshot.

  3. Attach the EBS volume to the second instance.

The new EBS volume may be a bit slow for a bit while it's filling in blocks from the snapshot, but it will be usable right away.

ALTERNATIVE (If the files are not already on an EBS volume):

  1. Attach a new EBS volume to the first instance.

  2. Copy the files from other disks to the new EBS volume.

  3. Move the EBS volume to the second instance.

share|improve this answer
    
Great solution for a lot of cases, but doesn't exactly solve my problem. I'm combining files from many drives on systems that were manipulating them in parallel onto one ( or a few ) drives on a single system. –  T. Brian Jones Feb 9 '12 at 4:47
    
@T.BrianJones You should still be able to use this. Mount the EBS volumes to the new instance with the few large drives and copy everything over. –  ceejayoz Feb 9 '12 at 16:15
add comment

Use tar and netcat. If they're in the same subnet, and you're not too concerned about security.. This is a pretty neat solution. You can add stuff into the pipeline if you want security.. You could use gpg.. or compress first with gzip

On the receiving end do:

netcat -l -p 7000 | tar x

And on the sending end do:

tar cf - * | netcat otherhost 7000
share|improve this answer
1  
+1 for coolness and simplicity . This will do the trick. –  The Unix Janitor Feb 9 '12 at 16:20
add comment

You can use Amazon AWS Import/Export service. Ship the drive to them and let them do the copy for you. More expensive, but perfect in your situation when you need have many many GB of data transfer without waiting for weeks for the job to finish. Their link: http://aws.amazon.com/importexport/

share|improve this answer
    
Sorry I was unclear. I'm copying between EC2 systems in the same availability zone. –  T. Brian Jones Feb 9 '12 at 1:01
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.