2

Currently files are backed up from one external hard drive to another as a means to provide the ability to restore from a secondary device should the primary external hard drive become unavailable e.g. hardware failure. The current method is predicated on the use of Microsoft Windows.

All backup is to be performed from Linux devices running Ubuntu 14.04 going forward.

All external hard drives are encrypted using TrueCrypt and have been formatted with NTFS since they are also used on other operating systems other than Linux such as Microsoft Windows.

Files are also created on Microsoft Windows and stored on these devices.

File sizes vary from as little as 100KB to as large as 10GB. There are approximately 1 million files and this continues to grow on a regular basis.

Requirements

  • Performance on external hard drives. All current drives are USB 2
  • Speed of transfer
  • Compute thresholds or considerations. The devices that currently perform the backups are either i3, i5, i7 with a minimum of 8GB of RAM
  • Preserving permissions and the like
  • Deep paths. Some paths traverse as deep as 20 or 30 hierarchies
  • Minimal impact on the lifespan of the external hard drives
  • Avoidance of changing filesystems if possible
  • Whole copies are fine and deltas or differentials are not required
  • Checksums to ensure that files have been successfully committed
  • Logs
  • The ability to resume transfers in the event of a failure

Now there have been mention of using switches or options such as inplace however there is very little information available on additional considerations for encrypted drives that leverage NTFS filesystems.

EDIT

The current attempted command as an example is;

rsync -vrlptg /media/drive1 /media/drive2

What other switches/options must be included to support the requirements?

6
  • 1
    You are looking for some software recommendation? – Jakuje Jan 7 '16 at 9:04
  • Are your files all new, or do old files change? rsync doesn't handle changes to large files quickly - for example, moving a file within a file system. There's a good discussion of rsync vs ZFS replication here: slashdot.org/story/15/12/22/026209/… – Andrew Henle Jan 7 '16 at 13:12
  • @Jakuje - No Jakuje. I am seeking to achieve this using rsync. If rsync isn't the ideal option, what other alternatives are there? – Motivated Jan 7 '16 at 18:42
  • @AndrewHenle - Both. There are new files as well as changes to old files. – Motivated Jan 7 '16 at 18:43
  • @Iain - As mentioned, there is very little information available on the additional considerations for encrypted drives that leverage NTFS filesystems, so commercial tools have been tested. This is however predicated on the use on the use of Microsoft Windows which is not something that is desired. – Motivated Jan 7 '16 at 18:45
1

Considering you are using a slow USB 2.0 protocol, it is critical to transfer as little data as possible.

Rsync is your best friend here. However, NTFS is not making you favor. Using something as EXT4 enable you to leaverage hardlinks to only store the changed files. Take a look at rsnapshot for further information

If you can't change your filesystem, you can also have a look at attic-backup, so you can have deduplicated backup without resorting to hardlinks.

6
  • Why is it critical to transfer as little data as possible? Are you suggesting that the rate of transfer be rate limited or throttled? If so what is the recommended rate of transfer? Using EXT4 unless supported by Windows isn't an option. – Motivated Jan 7 '16 at 23:20
  • 1
    Because USB 2.0 is a slow protocol, so reducing the amount of transferred data will speed up the backup process considerably – shodanshok Jan 7 '16 at 23:24
  • I appreciate it being a protocol that doesn't support high rates of transfer however, what is the recommended rate of transfer? – Motivated Jan 7 '16 at 23:25
  • The fastest possible available. To put things in perspective: transferring 400 GB of data via USB 2.0 (about 40 MB/s) needs about 3 hours. Using a USB 3.0 or SATA disk (over 100 MB/s, limited by the disk and not by the protocol) will need about 1 hour. – shodanshok Jan 8 '16 at 0:02
  • Thanks. What other switches or options should be used with rsync to meet the requirements? – Motivated Jan 8 '16 at 0:29
0

Most, if not all, of your requirements can be met by rsync:

Here's how you would use rsync with NTFS:

rsync -rvhc --size-only --progress /path/to/ntfs1/ /path/to/ntfs2/

rsync -rvhc --size-only --progress /path/to/ext4/ /path/to/ntfs/

Helpful hint: If you omit -c (checksum); and only use rsync flags -rvh; you'll find notable improvement in performance and speed.

flags v(verbose) and h(human readable) are optional. More info here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.