NetApp provides block-level deduplication (ASIS). Do you know any filesystem (even FUSE-based) on Linux (or OpenSolaris, *BSD) that provides the same functionnality ?
(I'm not interested in false deduplication like hardlinks).
Tell me more
×
Server Fault is a question and answer site for
professional system and network administrators. It's 100% free, no registration required.
|
Check lessFS, data-deduplication filesystem, for Linux. It is still in beta but you can try it out: Regards, MV |
|||
|
|
Deduplication is coming to ZFS on OpenSolaris but that functionality is not currently available. It was prototyped by Jeff Bonwick and Bill Moore this past winter and they are working on integrating it this summer. So it should be available in the next release of OpenSolaris or sooner if you want to play around with the development branch. |
|||
|
|
|
For people who may be unfamiliar with data deduplication, it is a technique whereby data is analyzed at the file (or block, I suppose) level, and where identical files/blocks throughout the file system are replaced with a smaller token. This has the effect of greatly shrinking the effective size on disk. It could be considered a form of copy-on-write. Read the wiki page on it. There is no filesystem that I have heard of in Linux to do dedup, file or block level. Such a beast would be handy, although pretty processor intensive. |
|||
|
|
|
Deduplication is now available with ZFS on OpenSolaris (build 128a and newers). |
|||
|
|
|
A year later, but here is a solution for OpenBSD called Epitome: http://www.peereboom.us/epitome/. Provided it's liberal licensing, it could very well make it into the Linux kernel. |
|||
|
|
|
I just posted a project that I have been working on that does inline deduplication. You can take a look at it here if you are interrested. It is based on fuse and runs on linux. |
|||
|
|
|
I don't know of any free implementations of dedup for Linux. I have seen some storage vendors recommending using a HSM(hierarchical storage management) system with a VTL(Virtual storage Library) which does dedup. You could also consider an Occarina like system which is not transparent but can provide better results than dedup. |
|||
|
|
|
so ... no news about deduplication on Linux? opendedup might be a choice but giving the java platform it runs on, i don't wanna get headaches. I have tried it yes, but this java machine and the rest are not getting very well with my needs of storage response times and safety. |
|||
|
|
|
DRBD does just that and does it really well to ! Can do Master/Slave or Master/Master :-) |
|||||||||||
|
