Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I host a lot of websites and our system makes it easy to duplicate items in these sites which is handy, but leads to lots of duplicated (and potentially quite large) files. I was wondering if these is any mechanism in linux (specifically Ubuntu) where the filesystem will only store the file once but link to it from all its locations.

I'd need this to be transparent, and also handle the case that if a user changes one of the files, it doesn't alter the contents of the main file but creates a new file for just this particular instance of the file.

The point of the exercise is to reduce wasted space used by duplicated files.

share|improve this question

2 Answers 2

up vote 3 down vote accepted

I'd need this to be transparent

ZFS-on-Linux × feature called "on-line deduplication".

UPD.: I've re-read your question once again now it looks like Aufs can be of help for you. It's very popular solution for hosting environments. And actually I can mention Btrfs by myself now as well — the pattern is you have some template sub-volume which you snapshot every time you need another instance. It's COW, so only changed file blocks would need more space. But keep in mind, Btrfs is, ergh… well, not too stable anyways. I'd use it in production only if data on it are absolutely okay to be gone.

share|improve this answer
    
Seems this is the better solution for online deduplication. I've decided to skip out on this in favour of bigger hard drives. Space is cheaper than the ram required to make this work. –  Brendon Muir Jan 22 at 2:19
    
Well, yeah, ZFS is rather RAM-greedy thing. But can be controlled, so actually as always it is task-dependent. Ppl run on FreeBSD, some claim they're happy with it on Linux as well, and it think it's quite reasonable to run it on ≥ 16 GiB –  poige Jan 22 at 5:51
    
Hehe, I have about 6GB and Rails want to chew a lot of that up! :) Thank you for revising your question. Hopefully it will help anyone else that comes across this requirement and doesn't know where to even start. –  Brendon Muir Jan 22 at 22:05

There is a linux user space/fuse filesystem that will do this dedup.

http://sourceforge.net/p/lessfs/wiki/Home/

Linux Journal has a good article on it in it's August 2011 issue. There are also various filesystem specific options with btrfs and zfs.

share|improve this answer
    
«There are also various filesystem specific options with btrfs» — ORLY? –  poige Jan 20 at 8:57
    
btrfs.wiki.kernel.org/index.php/Deduplication and sdteffen.blogspot.com/2013/10/… would be good places to start reading about btrfs's dedup. –  Nathan Neulinger Jan 21 at 13:29
    
And does any of them mention you can have on-line dedup with Btrfs? I don't think so, so, what's the better place to start then? ) –  poige Jan 21 at 14:17
1  
Sure looks to me like the bedup stuff runs with filesystem mounted. Granted, you have to run a tool to go find the duplicates and tell btrfs about them, but the functionality is there. Not enough definition of "transparent" in the original request to know what is strictly required. –  Nathan Neulinger Jan 22 at 14:36
    
Thanks @nneul. I guess I was hoping it would deduplicate on write but running something every so often wouldn't be a problem. I guess it's like how they do scrubbing (unless that's changed). –  Brendon Muir Jan 22 at 22:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.