I am running Linux on Amazon EC2 severs. I need to copy millions of files that total hundreds of gigabytes between two EC2 systems in the same availability zone. I don't need to sync directories, I just need to copy all the files in one directory over to an empty directory on the other machine.
What is the fastest way to do this? Has anyone seen or run performance tests?
rsync? scp? Should I zip them first? Should I detach the drive they are on and re-attach it to the machine I'm copying to, then copy them? Does transferring over the EC2's Private IP speed things up?
Any thoughts would be appreciated.
NOTE: Sorry this was unclear, but I'm copying data between two EC2 systems both in the same AWS availability zone.
cp
between the two drives mounted on the same instance. (EBS volumes incur a network overhead as it is, adding more to that by copying over a network can't do anything except slow things down. That said, AWS internal network is reasonably fast - just don't use micro/small instances for the copy as they are i/o constrained.)