We're looking to backup about 100gb+ of data containing small files (10kb+) each. The backup needs to be done as fast as possible to another harddrive weekly. Which is the better (especially speed wise) way to backup in such scenario? Rsync, or tar?
The advantage of rsync is that it will copy only the files which have changed.
If you have 100GB+ of relatively small files, you don't want to copy them all each time.
Note: the first backup with
rsync will be slow because all files are copied. Subsequently only the changed files are copied, and they can be compressed during the copy.
Be sure to familiarise yourself with all the options of
rsync ... there are many.
Tar is an archive utility. You could conceivably create a tar file for the entire 100GB+, but you don't want to transfer it all, each time.
I would like to add that, although in general I agree with pavium's reply and I would choose
rsync, there are options in
tar for incremental backups. From man:
-g, --listed-incremental F create/list/extract new GNU-format incremental backup
-G, --incremental create/list/extract old GNU-format incremental backup
EDIT: Following a recent comment, I will further expand on how both backups work:
tar initially creates a large file, possibly compressed (
-g gzip flag) with all backed up files. Then each incremental backup creates a new file only with the modified files, in which it also specifies which of them have been deleted.
rsync on the other hand initially keeps a second mirror directory with the exact tree and files of the source directory, uncompressed. Then with every incremental backup (
-B flag), it continues to have a mirror copy of the source, keeping in another directory by date all changed files (both modified and deleted).
Therefore, one can understand that each method has its plus and minus. A
tar backup is more difficult to be maintained in a medium with limited capacity, as it happens with the classic incremental method.
rsync is not considered a classic backup solution. It requires more disk space for the mirror, since it is uncompressed. It requires more time to reconstruct a full backup of a previous date.
UPDATE: Since Mar 2016 a newer alternative came up: borg backup. I very strongly recommend it. It uses the 'deduplicating' method. More information on the link provided above.
rsync can be somewhat painful if you have a very large number of files - especially if your rsync version is lower than 3. On the other hand: if you use tar, you would generate a very big resulting tar-file (unless the data may be compressed a lot). Personally, I would look at rdiff-backup, but make sure that you test your restore situation: rdiff-backup can be very memory demanding when restoring.
Take a look at rsnapshot, it's just a script that you can use as front for rsync. It will only backup stuff that has changed and rotate your backups.
Consider just using the RAID1 capabilities of your controller, a linux softraid that you mount in that directory (can even be done on imagefiles with loopback devices) or use btrfs on files (has the added beauty of COW Snapshots, but is still to be considered experimental). This link should give you some ideas. This way you create your backups as you go.
EDIT: Ok, before I get more (somewhat angry) downvotes I need to clarify: The idea is to swap out one of the discs in the RAID 1 with a blank one.
The one you yanked out is your backup, (done in constant amortized time), the other one should be happily hot-syncing. In linux I can do that completely automated in software and it works. RAID hot-sync might just be the slowest discussed backup method, but it happens concurrently, and easily within one week. This answer fits the requirements of the OP perfectly, so I do not see a reason for the downvotes.
I have to admit though, yanking a disk out of a hot (meaning heavily trafficked) array feels somewhat risky, it should work fine though, since all components are just doing what they were built to be doing. If RAID wouldn't resync, you could just as well not use it at all.
One could argue that this is abuse of a one-time fail-safe mechanism for regular use. Tell that to professional parachuters. We live in the digital age, if this wouldn't work, it would be broken.