Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a Linux server where I only store new files or rename directories and never edit files. It contains backups from other Linux servers.

Due to certain circumstances there are quite some duplicate files, often with different names.

Is there any free Linux tool which periodically scans the filesystem and has a database with filenames, sizes and maybe sha1sums and then identifies duplicates and replaces them with hardlinks?

share|improve this question
    
What do you use for to make your backups? –  Spack Apr 23 '13 at 19:33
    
I use rsync and I know that rsync can create hardlinks. But I still want to get the remaining duplicates. Also I learnt that it's a good idea to rotate logs not like syslog.123.gz, but instead let logrotate name them with the date, otherwise rsync copies them again and again... I use rsync because then I can easily do incemental, very efficient backup of all my Linus machines and I can recreate the system easily –  Christian Apr 23 '13 at 20:40

2 Answers 2

up vote 2 down vote accepted

some tools taken from http://unix.stackexchange.com/questions/3037/is-there-an-easy-way-to-replace-duplicate-files-with-hardlinks

  • trimtrees.pl
  • fduples -L
  • findup -m (from fslint)
  • rdfind -makehardlinks

You could run one of them in a cron job.

share|improve this answer

You can use a deduplicating filesystem. There are two main choices in Linux - btrfs and zfs.

With btrfs the drawback would be that it is still not marked as stable and has no fsck.

ZFS is not in the Linux kernel due to licensing issues but there is a kernel module with support for most Linux distributions. Also ZFS sports some kind of online-fsck with the scrub feature. You can have a look at the supported distros on zfsonlinux.org

Both have compression, deduplication and snapshotting features without the need of any additional userspace daemons - making them ideal for backup solutions.

share|improve this answer
    
I'm using ext4 at the moment. Currently only 2 TB. I'd rather use something in userspace if possible, but thanks anyway! –  Christian Apr 23 '13 at 20:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.