3

Now I'm trying to configure ZFS with deduplication on my linux backup server and have a question. For example I have 1k files in backup1.tar and 1k same files and 1 new in backup2.tar. This files will be deduplicated or not? As I can see it doesn't work with tar archives. Maybe I do something wrong.

Improve this question

2 Answers 2

Reset to default
6

It's possible some blocks from the tars will be the same, but very unlikely. ZFS deduplicates at the block level (called the recordsize in ZFS parlance), so individual blocks need to be identical. The tars are essentially guaranteed to have runs of the exact same content, but whether that will compose a whole block and be block aligned is highly unlikely.

If you want to take advantage of ZFS dedupe for backups use snapshots and the send/receive functions. This has the normal drawback of using slack space (storage is allocated to files in recordsize blocks, if the file doesn't use the whole block, there's leftover space that goes to waste). Tar avoids this "problem", but you'd have to create incremental tar backups to recreate the deduplication effect.

Improve this answer
2
  • I have 2Gb iso image, same iso image in tar and 2 copies of same iso image in tar. ZFS can't find same blocks? How could it be?
    – cpt.Buggy
    Feb 12, 2013 at 16:52
  • The tar header is likely a different size between the tar with one ISO and the tar with two ISOs. This couple byte difference would make ever data point from the ISOs offset throughout the archives, so no whole blocks would match.
    – Chris S
    Feb 12, 2013 at 17:00
3

ZFS dedupe is block-based, so it should account for the contents of your archive.

  • How large is the TAR archive?
  • Do you already have compression enabled?
  • What method are you using to determine the deduplication ratio? The standard methods are: zpool status -D poolname or zpool get dedupratio poolname
Improve this answer
5
  • Files more then 1Gb. Cmpression disabled. zdb -S storage (dedup = 1.00)
    – cpt.Buggy
    Feb 12, 2013 at 15:00
  • I don't think this would work as the entire content of the whole block has to be the same for dedup to kick in. Any change e.g. in the file header inside the tar file would prevent it to work.
    – Sven
    Feb 12, 2013 at 15:01
  • @cpt.Buggy I wouldn't really bother with dedupe for this purpose. Compression is a far better bet, assuming you have compressible content. What are you backing up?
    – ewwhite
    Feb 12, 2013 at 15:02
  • @ewwhite large packs of photos. I think compressing them not a good idea. From time to time some photos are added to the package but I can't make incremental backup, can only download full tar archive.
    – cpt.Buggy
    Feb 12, 2013 at 15:05
  • Yeah, image files certainly don't compress well. But a large number of deduplicated images wouldn't work well either because of the RAM needs. If you stored these images flat, you'd run into other issues.
    – ewwhite
    Feb 12, 2013 at 15:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .