I am running Linux on Amazon EC2 severs. I need to copy millions of files that total hundreds of gigabytes between two EC2 systems in the same availability zone. I don't need to sync directories, I just need to copy all the files in one directory over to an empty directory on the other machine.
What is the fastest way to do this? Has anyone seen or run performance tests?
rsync? scp? Should I zip them first? Should I detach the drive they are on and re-attach it to the machine I'm copying to, then copy them? Does transferring over the EC2's Private IP speed things up?
Any thoughts would be appreciated.
NOTE: Sorry this was unclear, but I'm copying data between two EC2 systems both in the same AWS availability zone.