We're looking to backup about 100gb+ of data containing small files (10kb+) each. The backup needs to be done as fast as possible to another harddrive weekly. Which is the better (especially speed wise) way to backup in such scenario? Rsync, or tar?

  • Information about the files would be interesting. Are the existing ones static and only new ones are added or are all files prone to changes? – AndreasT Nov 23 '12 at 9:54

Definitely rsync.

The advantage of rsync is that it will copy only the files which have changed.

If you have 100GB+ of relatively small files, you don't want to copy them all each time.

Note: the first backup with rsync will be slow because all files are copied. Subsequently only the changed files are copied, and they can be compressed during the copy.

Be sure to familiarise yourself with all the options of rsync ... there are many.

Tar is an archive utility. You could conceivably create a tar file for the entire 100GB+, but you don't want to transfer it all, each time.

| improve this answer | |

I would like to add that, although in general I agree with pavium's reply and I would choose rsync, there are options in tar for incremental backups. From man:

-g, --listed-incremental F create/list/extract new GNU-format incremental backup

   -G, --incremental
      create/list/extract old GNU-format incremental backup

EDIT: Following a recent comment, I will further expand on how both backups work:

tar initially creates a large file, possibly compressed (-g gzip flag) with all backed up files. Then each incremental backup creates a new file only with the modified files, in which it also specifies which of them have been deleted.

rsync on the other hand initially keeps a second mirror directory with the exact tree and files of the source directory, uncompressed. Then with every incremental backup (-B flag), it continues to have a mirror copy of the source, keeping in another directory by date all changed files (both modified and deleted).

Therefore, one can understand that each method has its plus and minus. A tar backup is more difficult to be maintained in a medium with limited capacity, as it happens with the classic incremental method. rsync is not considered a classic backup solution. It requires more disk space for the mirror, since it is uncompressed. It requires more time to reconstruct a full backup of a previous date.

UPDATE: Since Mar 2016 a newer alternative came up: borg backup. I very strongly recommend it. It uses the 'deduplicating' method. More information on the link provided above.

| improve this answer | |
  • 1
    Why would you choose rsync over tar if both have incremental backups? – Nicolas Schmidt Jun 10 '19 at 9:34
  • 2
    I hope the recent edit covers your question. – Wtower Jun 10 '19 at 11:59

rsync can be somewhat painful if you have a very large number of files - especially if your rsync version is lower than 3. On the other hand: if you use tar, you would generate a very big resulting tar-file (unless the data may be compressed a lot). Personally, I would look at rdiff-backup, but make sure that you test your restore situation: rdiff-backup can be very memory demanding when restoring.

| improve this answer | |

if your files do not change much - i would vote for rsync.

| improve this answer | |

Do you need history (multiple backups) or just a plain copy of your data to some other disk? Backing up 100GB of 10KB files would take ages if you don't use a block level backup. Think about making block level snapshots or some other block level based solution, if you really need a fast solution.

| improve this answer | |
  • Do not need a history, just a plain copy of data to a secondary harddisk mounted on the server. Any suggestioon on a faster solution? – Paatrick Jan 5 '10 at 13:51
  • ``dd=/dev/sdX of=/dev/sdY'' in a cron job should be the fastest solution since it's block level copy if sdX being copied to sdY. Benchmark that against a tar'ed or rsync'd copy. – pfo Jan 5 '10 at 14:03
  • 1
    for a general use (home directories) rsync works fine even for several terabytes of data (including a mix of small and big files). – wazoox Jan 5 '10 at 14:09
  • Can you take a block level backup of just certain folders? – Svish Jan 5 '10 at 14:12
  • 1
    Block-level backups by design only work for entire filesystems, not single directories. This is especially true for simple solutions like "dd if=/dev/foo of=/dev/bar" but AFAIK also for the more advanced Snapshot-based products from NetApp, EMC and the like. – daff Jan 11 '10 at 5:49

Take a look at rsnapshot, it's just a script that you can use as front for rsync. It will only backup stuff that has changed and rotate your backups.

| improve this answer | |

Consider just using the RAID1 capabilities of your controller, a linux softraid that you mount in that directory (can even be done on imagefiles with loopback devices) or use btrfs on files (has the added beauty of COW Snapshots, but is still to be considered experimental). This link should give you some ideas. This way you create your backups as you go.

EDIT: Ok, before I get more (somewhat angry) downvotes I need to clarify: The idea is to swap out one of the discs in the RAID 1 with a blank one.

The one you yanked out is your backup, (done in constant amortized time), the other one should be happily hot-syncing. In linux I can do that completely automated in software and it works. RAID hot-sync might just be the slowest discussed backup method, but it happens concurrently, and easily within one week. This answer fits the requirements of the OP perfectly, so I do not see a reason for the downvotes.

I have to admit though, yanking a disk out of a hot (meaning heavily trafficked) array feels somewhat risky, it should work fine though, since all components are just doing what they were built to be doing. If RAID wouldn't resync, you could just as well not use it at all.

One could argue that this is abuse of a one-time fail-safe mechanism for regular use. Tell that to professional parachuters. We live in the digital age, if this wouldn't work, it would be broken.

| improve this answer | |
  • 2
    For the millionth time: RAID IS NOT BACKUP – Tonny Nov 23 '12 at 10:25
  • It is, if you know how to use it. You didn't even bother to read through my answer to the btrfs cow bits. A full sync takes time and causes system load, but you can do it live, as long as hot-sync is well supported on whichever technology you use. – AndreasT Nov 23 '12 at 10:53
  • I did read and comprehend your answer. I am fully familiar with the in's and out's of BTRFS. (Hell, I've written parts of BTRFS.) You can use BTRFS with a combination of snapshots and RAID to achieve something that would act as a backup, but it is finicky, uses experimental features and IN GENERAL it is not a solution for a generic backup problem. I stand by my statement: RAID by itself is not backup. RAID+snapshot can be backup, but it is very hard to do it right. Just snapshot+blockbases image-copy of the snapshot is a much simpler approach to the posters question. – Tonny Nov 23 '12 at 22:10
  • If computer strikes by thunder or fire, will your raid 1 backup surrive? – Yuan May 5 '13 at 22:52
  • @Yuan: Of course it would survive, you just swap out one Hard disk with a blank one, done. It is definately the quickest method. Albeit I wouldn't have the guts to yank out a disk from an actively used RAID, I have done this to backup our backup RAID and it worked like a charm. – AndreasT May 24 '13 at 7:08