I host a lot of websites and our system makes it easy to duplicate items in these sites which is handy, but leads to lots of duplicated (and potentially quite large) files. I was wondering if these is any mechanism in linux (specifically Ubuntu) where the filesystem will only store the file once but link to it from all its locations.

I'd need this to be transparent, and also handle the case that if a user changes one of the files, it doesn't alter the contents of the main file but creates a new file for just this particular instance of the file.

The point of the exercise is to reduce wasted space used by duplicated files.

2 Answers 2


I'd need this to be transparent

ZFS-on-Linux × feature called "on-line deduplication".

UPD.: I've re-read your question once again now it looks like Aufs can be of help for you. It's very popular solution for hosting environments. And actually I can mention Btrfs by myself now as well — the pattern is you have some template sub-volume which you snapshot every time you need another instance. It's COW, so only changed file blocks would need more space. But keep in mind, Btrfs is, ergh… well, not too stable anyways. I'd use it in production only if data on it are absolutely okay to be gone.

  • Seems this is the better solution for online deduplication. I've decided to skip out on this in favour of bigger hard drives. Space is cheaper than the ram required to make this work. Jan 22, 2014 at 2:19
  • Well, yeah, ZFS is rather RAM-greedy thing. But can be controlled, so actually as always it is task-dependent. Ppl run on FreeBSD, some claim they're happy with it on Linux as well, and it think it's quite reasonable to run it on ≥ 16 GiB
    – poige
    Jan 22, 2014 at 5:51
  • Hehe, I have about 6GB and Rails want to chew a lot of that up! :) Thank you for revising your question. Hopefully it will help anyone else that comes across this requirement and doesn't know where to even start. Jan 22, 2014 at 22:05

There is a linux user space/fuse filesystem that will do this dedup.


Linux Journal has a good article on it in it's August 2011 issue. There are also various filesystem specific options with btrfs and zfs.

  • «There are also various filesystem specific options with btrfs» — ORLY?
    – poige
    Jan 20, 2014 at 8:57
  • btrfs.wiki.kernel.org/index.php/Deduplication and sdteffen.blogspot.com/2013/10/… would be good places to start reading about btrfs's dedup. Jan 21, 2014 at 13:29
  • And does any of them mention you can have on-line dedup with Btrfs? I don't think so, so, what's the better place to start then? )
    – poige
    Jan 21, 2014 at 14:17
  • 1
    Sure looks to me like the bedup stuff runs with filesystem mounted. Granted, you have to run a tool to go find the duplicates and tell btrfs about them, but the functionality is there. Not enough definition of "transparent" in the original request to know what is strictly required. Jan 22, 2014 at 14:36
  • Thanks @nneul. I guess I was hoping it would deduplicate on write but running something every so often wouldn't be a problem. I guess it's like how they do scrubbing (unless that's changed). Jan 22, 2014 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.