1

I need to move multiple terabytes of information from one NFS system (The NFS system is in AWS) to amazon EFS, but I need it to move that that data the fastest way possible. I've tried rsync, but it takes too long, and I've also tried using parallel with rsync but it does not give me the results I need.

The data consists in multiple directories with many small files

Are there any tried and tested ways to move data very quickly while still making sure that it's not corrupted when it arrives (Like rsync)?

Improve this question
9
  • I can't imagine you are the first to need to do this. Have you tried contacting Amazon support to see what they recommend?
    – ivanivan
    Nov 15, 2017 at 19:43
  • We only have the basic support, I do not believe we are entitled to technical support at that level.
    – John Doe
    Nov 15, 2017 at 19:46
  • May be worth asking, certainly no harm in asking. I was in a similar situation with a different company, and casually mentioned that if I had to do all the work to move the data then I may as well move it to $other_provider .... even with no "enhanced" support contract, they had their engineers get with me right quick and they took care of everything - which got them another $750k of our business over a 5 year period.
    – ivanivan
    Nov 15, 2017 at 20:40
  • Interesting, i'll give it a try
    – John Doe
    Nov 15, 2017 at 20:46
  • Note that they may call your bluff... if it is a bluff.
    – ivanivan
    Nov 15, 2017 at 20:51

2 Answers 2

Reset to default
5

The fastest way to move such large volumes of data is still Sneakernet. Thus, Amazon Snowball. This device gets shipped to your location, where it can move up to 80 terabytes of data directly off your local network via a 10 Gigabit Ethernet connection. It then gets shipped back to Amazon where they upload the data into your Amazon account.

Improve this answer
4
  • I clarified my question, the NFS server is also in AWS. So it's moving the data from the NFS server in AWS to the EFS mount point. The options I mentioned above just aren't moving the data fast enough.
    – John Doe
    Nov 15, 2017 at 18:23
  • 1
    Never underestimate the bandwidth of a station wagon full of backup tapes!
    – ivanivan
    Nov 15, 2017 at 19:42
  • 1
    What does that mean: "the NFS server in AWS"?
    – Michael Hampton
    Nov 15, 2017 at 22:33
  • @MichaelHampton It means that the NFS server where the files are is an EC2 instance.
    – John Doe
    Nov 16, 2017 at 15:44
0

For very many small files the limit will be the IOPS for both the source and the destination storage. rsync won't speed things up as it has to read every file, its main benefit is not transferring already copied chunks of large files.

If you instead restore a backup archive, the reads are larger, sequential, and higher throughput:

cd /mnt/files/
tar -xzf /mnt/backup/files.tar.gz

However, you have to have the space for this backup archive.

Also, the point in time restored may be some time ago depending on how long it takes to make the backup.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .