Backing up large files using rsync to external drives (NTFS) taking into consideration performance & speed

Question

Currently files are backed up from one external hard drive to another as a means to provide the ability to restore from a secondary device should the primary external hard drive become unavailable e.g. hardware failure. The current method is predicated on the use of Microsoft Windows.

All backup is to be performed from Linux devices running Ubuntu 14.04 going forward.

All external hard drives are encrypted using TrueCrypt and have been formatted with NTFS since they are also used on other operating systems other than Linux such as Microsoft Windows.

Files are also created on Microsoft Windows and stored on these devices.

File sizes vary from as little as 100KB to as large as 10GB. There are approximately 1 million files and this continues to grow on a regular basis.

Requirements

• Performance on external hard drives. All current drives are USB 2
• Speed of transfer
• Compute thresholds or considerations. The devices that currently perform the backups are either i3, i5, i7 with a minimum of 8GB of RAM
• Preserving permissions and the like
• Deep paths. Some paths traverse as deep as 20 or 30 hierarchies
• Minimal impact on the lifespan of the external hard drives
• Avoidance of changing filesystems if possible
• Whole copies are fine and deltas or differentials are not required
• Checksums to ensure that files have been successfully committed
• Logs
• The ability to resume transfers in the event of a failure

Now there have been mention of using switches or options such as inplace however there is very little information available on additional considerations for encrypted drives that leverage NTFS filesystems.

EDIT

The current attempted command as an example is;

rsync -vrlptg /media/drive1 /media/drive2

What other switches/options must be included to support the requirements?

Answer 1

Considering you are using a slow USB 2.0 protocol, it is critical to transfer as little data as possible.

Rsync is your best friend here. However, NTFS is not making you favor. Using something as EXT4 enable you to leaverage hardlinks to only store the changed files. Take a look at rsnapshot for further information

If you can't change your filesystem, you can also have a look at attic-backup, so you can have deduplicated backup without resorting to hardlinks.

Answer 2

Most, if not all, of your requirements can be met by rsync:

Here's how you would use rsync with NTFS:

rsync -rvhc --size-only --progress /path/to/ntfs1/ /path/to/ntfs2/

rsync -rvhc --size-only --progress /path/to/ext4/ /path/to/ntfs/

Helpful hint: If you omit -c (checksum); and only use rsync flags -rvh; you'll find notable improvement in performance and speed.

flags v(verbose) and h(human readable) are optional. More info here