Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Now I'm trying to configure ZFS with deduplication on my linux backup server and have a question. For example I have 1k files in backup1.tar and 1k same files and 1 new in backup2.tar. This files will be deduplicated or not? As I can see it doesn't work with tar archives. Maybe I do something wrong.

share|improve this question

2 Answers 2

up vote 6 down vote accepted

It's possible some blocks from the tars will be the same, but very unlikely. ZFS deduplicates at the block level (called the recordsize in ZFS parlance), so individual blocks need to be identical. The tars are essentially guaranteed to have runs of the exact same content, but whether that will compose a whole block and be block aligned is highly unlikely.

If you want to take advantage of ZFS dedupe for backups use snapshots and the send/receive functions. This has the normal drawback of using slack space (storage is allocated to files in recordsize blocks, if the file doesn't use the whole block, there's leftover space that goes to waste). Tar avoids this "problem", but you'd have to create incremental tar backups to recreate the deduplication effect.

share|improve this answer
    
I have 2Gb iso image, same iso image in tar and 2 copies of same iso image in tar. ZFS can't find same blocks? How could it be? –  cpt.Buggy Feb 12 '13 at 16:52
    
The tar header is likely a different size between the tar with one ISO and the tar with two ISOs. This couple byte difference would make ever data point from the ISOs offset throughout the archives, so no whole blocks would match. –  Chris S Feb 12 '13 at 17:00

ZFS dedupe is block-based, so it should account for the contents of your archive.

  • How large is the TAR archive?
  • Do you already have compression enabled?
  • What method are you using to determine the deduplication ratio? The standard methods are: zpool status -D poolname or zpool get dedupratio poolname
share|improve this answer
    
Files more then 1Gb. Cmpression disabled. zdb -S storage (dedup = 1.00) –  cpt.Buggy Feb 12 '13 at 15:00
    
I don't think this would work as the entire content of the whole block has to be the same for dedup to kick in. Any change e.g. in the file header inside the tar file would prevent it to work. –  SvW Feb 12 '13 at 15:01
    
@cpt.Buggy I wouldn't really bother with dedupe for this purpose. Compression is a far better bet, assuming you have compressible content. What are you backing up? –  ewwhite Feb 12 '13 at 15:02
    
@ewwhite large packs of photos. I think compressing them not a good idea. From time to time some photos are added to the package but I can't make incremental backup, can only download full tar archive. –  cpt.Buggy Feb 12 '13 at 15:05
    
Yeah, image files certainly don't compress well. But a large number of deduplicated images wouldn't work well either because of the RAM needs. If you stored these images flat, you'd run into other issues. –  ewwhite Feb 12 '13 at 15:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.