We're looking to backup about 100gb+ of data containing small files (10kb+) each. The backup needs to be done as fast as possible to another harddrive weekly. Which is the better (especially speed wise) way to backup in such scenario? Rsync, or tar?
The advantage of rsync is that it will copy only the files which have changed.
If you have 100GB+ of relatively small files, you don't want to copy them all each time.
Note: the first backup with
rsync will be slow because all files are copied. Subsequently only the changed files are copied, and they can be compressed during the copy.
Be sure to familiarise yourself with all the options of
rsync ... there are many.
Tar is an archive utility. You could conceivably create a tar file for the entire 100GB+, but you don't want to transfer it all, each time.
1why would you "definitely" use rsync if you could use tar.gz and combine it with find to create incremental compressed backups?– asaMay 31, 2021 at 18:37
@asa more work i assume Jul 22, 2021 at 20:45
I would like to add that, although in general I agree with pavium's reply and I would choose
rsync, there are options in
tar for incremental backups. From man:
-g, --listed-incremental F create/list/extract new GNU-format incremental backup
-G, --incremental create/list/extract old GNU-format incremental backup
EDIT: Following a recent comment, I will further expand on how both backups work:
tar initially creates a large file, possibly compressed (
-g gzip flag) with all backed up files. Then each incremental backup creates a new file only with the modified files, in which it also specifies which of them have been deleted.
rsync on the other hand initially keeps a second mirror directory with the exact tree and files of the source directory, uncompressed. Then with every incremental backup (
-B flag), it continues to have a mirror copy of the source, keeping in another directory by date all changed files (both modified and deleted).
Therefore, one can understand that each method has its plus and minus. A
tar backup is more difficult to be maintained in a medium with limited capacity, as it happens with the classic incremental method.
rsync is not considered a classic backup solution. It requires more disk space for the mirror, since it is uncompressed. It requires more time to reconstruct a full backup of a previous date.
UPDATE: Since Mar 2016 a newer alternative came up: borg backup. I very strongly recommend it. It uses the 'deduplicating' method. More information on the link provided above.
1Why would you choose
tarif both have incremental backups? Jun 10, 2019 at 9:34
2I hope the recent edit covers your question.– WtowerJun 10, 2019 at 11:59
rsync can be somewhat painful if you have a very large number of files - especially if your rsync version is lower than 3. On the other hand: if you use tar, you would generate a very big resulting tar-file (unless the data may be compressed a lot). Personally, I would look at rdiff-backup, but make sure that you test your restore situation: rdiff-backup can be very memory demanding when restoring.
if your files do not change much - i would vote for rsync.
Do you need history (multiple backups) or just a plain copy of your data to some other disk? Backing up 100GB of 10KB files would take ages if you don't use a block level backup. Think about making block level snapshots or some other block level based solution, if you really need a fast solution.
Do not need a history, just a plain copy of data to a secondary harddisk mounted on the server. Any suggestioon on a faster solution?– PaatrickJan 5, 2010 at 13:51
``dd=/dev/sdX of=/dev/sdY'' in a cron job should be the fastest solution since it's block level copy if sdX being copied to sdY. Benchmark that against a tar'ed or rsync'd copy.– pfoJan 5, 2010 at 14:03
1for a general use (home directories) rsync works fine even for several terabytes of data (including a mix of small and big files).– wazooxJan 5, 2010 at 14:09
Can you take a block level backup of just certain folders?– SvishJan 5, 2010 at 14:12
1Block-level backups by design only work for entire filesystems, not single directories. This is especially true for simple solutions like "dd if=/dev/foo of=/dev/bar" but AFAIK also for the more advanced Snapshot-based products from NetApp, EMC and the like.– daffJan 11, 2010 at 5:49
Take a look at rsnapshot, it's just a script that you can use as front for rsync. It will only backup stuff that has changed and rotate your backups.
Consider just using the RAID1 capabilities of your controller, a linux softraid that you mount in that directory (can even be done on imagefiles with loopback devices) or use btrfs on files (has the added beauty of COW Snapshots, but is still to be considered experimental). This link should give you some ideas. This way you create your backups as you go.
EDIT: Ok, before I get more (somewhat angry) downvotes I need to clarify: The idea is to swap out one of the discs in the RAID 1 with a blank one.
The one you yanked out is your backup, (done in constant amortized time), the other one should be happily hot-syncing. In linux I can do that completely automated in software and it works. RAID hot-sync might just be the slowest discussed backup method, but it happens concurrently, and easily within one week. This answer fits the requirements of the OP perfectly, so I do not see a reason for the downvotes.
I have to admit though, yanking a disk out of a hot (meaning heavily trafficked) array feels somewhat risky, it should work fine though, since all components are just doing what they were built to be doing. If RAID wouldn't resync, you could just as well not use it at all.
One could argue that this is abuse of a one-time fail-safe mechanism for regular use. Tell that to professional parachuters. We live in the digital age, if this wouldn't work, it would be broken.
2For the millionth time: RAID IS NOT BACKUP– TonnyNov 23, 2012 at 10:25
It is, if you know how to use it. You didn't even bother to read through my answer to the btrfs cow bits. A full sync takes time and causes system load, but you can do it live, as long as hot-sync is well supported on whichever technology you use.– AndreasTNov 23, 2012 at 10:53
I did read and comprehend your answer. I am fully familiar with the in's and out's of BTRFS. (Hell, I've written parts of BTRFS.) You can use BTRFS with a combination of snapshots and RAID to achieve something that would act as a backup, but it is finicky, uses experimental features and IN GENERAL it is not a solution for a generic backup problem. I stand by my statement: RAID by itself is not backup. RAID+snapshot can be backup, but it is very hard to do it right. Just snapshot+blockbases image-copy of the snapshot is a much simpler approach to the posters question.– TonnyNov 23, 2012 at 22:10
If computer strikes by thunder or fire, will your raid 1 backup surrive?– YuanMay 5, 2013 at 22:52
@Yuan: Of course it would survive, you just swap out one Hard disk with a blank one, done. It is definately the quickest method. Albeit I wouldn't have the guts to yank out a disk from an actively used RAID, I have done this to backup our backup RAID and it worked like a charm.– AndreasTMay 24, 2013 at 7:08