Questions tagged [deduplication]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
33 views

How to automatically delete duplicate emails?

I am using the iredmail mail system. When I create a maillists and add a user, when I send an email to that maillist and add the user in that maillist, that user gets two identical emails. Is there a ...
user avatar
  • 1
0 votes
0 answers
9 views

How to containerize a Linux native application in order to have multiple versions of it running at the same time from mostly the same codebase?

I have a Linux server application that is no longer supported by the vendor. I need to run 2-3 versions of this application at the same time in parallel. I can set up different VMs for it, and it ...
user avatar
0 votes
1 answer
65 views

Windows hardlinks - why it consumes same space as original file

Im using finddupe tool to de-duplicate files and save some space on HDD. But checking before and after operation I cant see any free space released - see below. Can somebody advise what is wrong here ?...
user avatar
  • 123
-1 votes
2 answers
241 views

Find file duplicates and convert them into links [WINDOWS] [closed]

My users tend to save tons of duplicate files what consumes more and more space and generate HW and archiving cost. Im thinking to create some scheduled job, to: find duplicate files (check file MD5 ...
user avatar
  • 123
0 votes
1 answer
267 views

Workaround for Windows Search/Indexing on a deduplicated volume

According to Microsoft, Microsoft Windows Server 2019 still does not support Windows Search on Data Deduplication enabled volumes (source): Windows Search doesn't support Data Deduplication. Data ...
user avatar
1 vote
1 answer
297 views

Dedup.sys 0x7E after inplace upgrade Server 2016 => 19

I've recently performed an in-place upgrade from Server 2016 => 2019 (non-domain). 2 of the drives connected has been configured with data deduplication (mostly VHDs for Hyper-V). Ever since the ...
user avatar
3 votes
1 answer
2k views

Cheap/fast deduplication with hardlinks?

I've got shared hosting with a few thousand Wordpress installs and I've wanted for ages to have a nice way of removing all the duplicate files in a sensible and secure way. I'm looking for better disk ...
user avatar
  • 287
2 votes
1 answer
360 views

ZFS interpret output of zdb -S tank

I wanted to know if it would pay it out for me to activate the zfs deduplication so I ran the command zdb -S tank but know I need some help to interpret the output. Simulated DDT histogram: bucket ...
user avatar
  • 25
1 vote
0 answers
304 views

btrfs compress existing data with snapshots

I'm runninga a btrfs raid1 on 5 8 TB disks, where primarily urbackup stores backups to. Urbackup uses snapshots extensively. compsize produces the following output: Processed 106161732 files, 57335496 ...
user avatar
  • 111
3 votes
1 answer
76 views

Setup Raid 10 for backup store

We have HP DL380p Gen8 with Smart Array P420i: 10HDDx6TB, 2 RAID10 6 and 4 HDD, for backup store via Arcserve UDP, deduplication backup configured for 16 KB blocks, we are not satisfied with the speed ...
user avatar
  • 31
4 votes
0 answers
517 views

Windows NTFS Data Deduplication and Snapshot Backups

We have a file server (fs00) on Google Cloud Platform (GCP): Running Windows Server 2019 (with Desktop Experience installed). One OS / System disk (250GB SSD) An independent data disk (5TB standard ...
user avatar
0 votes
0 answers
1k views

ZFS send/receive fails consistently with "Input/output error"

I have a ZFS disk with deduplication turned on. I have copied data from many other old hard disks onto this ZFS disk, ultimately with the intention of getting rid of all the original disks. I need ...
user avatar
  • 615
2 votes
1 answer
374 views

ZFS deduplication

This is my first post on this website so please be kind with me :) On Centos 7, using ZFS deduplication in the past on a storage pool, I moved all file-systems that dedup was on to another storage ...
user avatar
2 votes
0 answers
98 views

Windows 2016 deduplication and application compatibility

I'm re-evaluating Windows 2016/2019 deduplication engine, which is way better (faster/more capable) than what shipped in previous Windows versions. I understand how it works (sparsifying files via ...
user avatar
  • 43.5k
0 votes
0 answers
33 views

postfix dovecot avoid duplicate sendings from two different IPs but same computer, same sender/recipients, same subject & body

Is there a way to avoid duplicate sendings from two different IPs but same computer, same sender, same recipients, same subject & body ? I'm using a postfix/dovecot server to which one of the ...
user avatar
5 votes
2 answers
1k views

Offline Deduplication Software for ZFS

I do have a 300TB Freenas server to backup several linux nodes. Backup works with daily snapshot and rsync tasks. The user often move big datasets (2-5TB) between the server - so often big files are ...
user avatar
  • 101
2 votes
1 answer
135 views

puppet code design: how to collect arguments into a string (avoiding possible duplicate declaration errros)

I'm having trouble with puppet (version 5.5) code design. I've written a component module to deal with ucarp. It make use of the module eyp-systemd to register an ucarp service within systemd. Now I ...
user avatar
1 vote
0 answers
113 views

Is Path-based Unoptimization Possible?

I have a deduplicated volume in Windows Server 2016 that some software is having problems dealing with. Specifically, Matlab 2015 and older couldn't read deduplicated files, and while updating to a ...
user avatar
1 vote
0 answers
270 views

Deduplication Savings differs a lot between same two volumes

I run this robocopy command to copy all data from the existing volume to my new volume which will replace the old one in the near future: robocopy I:\ E:\ /E /COPYALL /PURGE /MIR /B /R:1 /W:1 /XD "...
user avatar
6 votes
1 answer
2k views

Windows 2012 Robocopy with Deduplication active

Current Issue: File Cluster Disk of 14 TB Size needs an CHKDSK which can't be done because it lasts too long. So I'm going to replace this disk in the storage of the cluster. I run many times the ...
user avatar
1 vote
1 answer
32 views

VHDX drive space being consumed

I have a host running Server 2012 R2, and a file server VM also running 2012 R2. The VM has 3 drives, OS.vhdx, file-1.vhdx and file2.vhdx. Dedupe is running inside the VM on file-1.vhdx and file2....
user avatar
  • 1,120
13 votes
5 answers
18k views

Apparently Robocopy does not understand Windows Server 2016 deduplication. Is there a way to make it work without corrupting the volume?

Scenario: Doing a server migration from old Server 2008 R2 to new Server 2016, following this Server Fault guide: File server migration using Robocopy After Robocopy completes, enable deduplication ...
user avatar
1 vote
0 answers
47 views

Deduplication assistance

We have consolidated a couple of drives and NAS systems of a former colleague before the disks failed (some actually showed signs of degradation). Now, we know that the colleague has done "RAID by ...
user avatar
4 votes
2 answers
4k views

How to verify that a deduplication has taken place?

Microsoft Windows Server 2012 and onwards offers a de-duplication service that periodically scans files, find identical chunks and removes excessive copies to save space. To the user browsing the ...
user avatar
  • 139
1 vote
0 answers
459 views

How to calculate dedupe ratio per folder

Is there a way to determine the zfs dedup ratio on a per folder basis? I realize the potential difficulties in computing this as every block would need to be analyzed, but can such a method be ...
user avatar
0 votes
1 answer
2k views

Size on disk smaller than capacity but no free space

We have a Windows 2012 Veeam Server with a volume that we use to store our backups on. The volume has a capacity of 14.9 TB and has data deduplication enabled. On the volume there are folders for the ...
user avatar
0 votes
1 answer
1k views

Windows deduplication doesn't yield usable space

Dedup size picture Hi all, When I setup Windows depuplication, I get great savings. See the attached picture. 1.28 TB deduped down to 23 GB. However, the space savings are not real. Lets say that ...
user avatar
  • 65
0 votes
0 answers
37 views

recurring, queryable, cached checksumming of all files in a filesystem

I'm looking for a way to efficiently manage and leverage file-level checksums for all files in a filesystem over time. Goals: Configurable, fast refresh - only re-checksumming large files when ...
user avatar
3 votes
1 answer
3k views

Can an unoptimize dedup job be resumed?

I have a Start-DedupJob -type Unoptimize ... that's been running for 7 hours and is still at 0% progress. I'm reconfiguring the server and need to restart. Disk activity has been pegged to near ...
user avatar
2 votes
1 answer
491 views

How to read fully deduplicated files over SMB on macOS?

I have a volume in Windows Server 2016 that is a target of the data deduplication role. It works very well ~60% savings rate, with many files appearing as zero bytes on the disk. The volume is ...
user avatar
1 vote
1 answer
1k views

How to find duplicate files against a reference directory structure in Linux

There are a couple of duplicate file finders for Linux listed e.g. here. I have already tried fdupes and fslint. However, from what I have seen, these will find all duplicates of the selected ...
user avatar
  • 165
3 votes
1 answer
1k views

zfs wrong space usage

I have a backup server with ZFS (Ubuntu 16.04; 32GB RAM, 4x6TB HDD, raidz2). Recently I've found the problem with space available. # zpool list -v NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP ...
user avatar
0 votes
1 answer
713 views

Unable to mount borg backup repository residing on a ZFS snapshot

I've posted the question and my own solution to this problem, to make it available to others. This relates to borg backup and was posted to this borg github issue: Allow check on read-only ...
user avatar
  • 137
4 votes
1 answer
2k views

On deduped volumes, how can I determine space used in a folder with Measure-DedupFileMetadata

I'm trying to understand how Measure-DedupFileMetadata works so I can recursively go through some folders to report on how much space is actually used. I don't know how to interpret the output. If I ...
user avatar
3 votes
1 answer
4k views

How to use OverlayFS with Docker volumes?

For some use cases I'd like to have the possibility to create a volume with docker volume create, and fill it with data. Then I want to create a new volume that is only a copy of the first one, but no ...
user avatar
  • 131
1 vote
0 answers
388 views

Clarifying sizing of dedup table on ZFS when only some datasets are deduped

I am trying to be clear about how zfs dedup handles the case where some (but not all) datasets in a pool are deduped, from a dedup table/RAM impact perspective. I found this quote from the FreeBSD ...
user avatar
  • 664
6 votes
3 answers
5k views

Windows 2016 Storage Spaces Direct + Data Deduplication

Has anyone combined S2D (Storage Spaces Direct) with Data Deduplication? Is this even possible or recommended practice? Please elaborate on why this is a good idea or not. EDIT: Just stumbled ...
user avatar
1 vote
0 answers
72 views

Better to deduplicate inside VHDX on client or outside VHDX on host? [duplicate]

I have a 16TB array that I'm hosting in a virtual environment. It contains video and photos from our marketing department. The data resides on the Hyper-V host itself as a VHDX file on a 8-drive SATA ...
user avatar
0 votes
0 answers
53 views

Is there a way to measure or change how fast a Windows Server 2012 rehydrates data that it previously deduplicated?

Since the deduplication option was enabled on the Windows Server (currently at about 40 percent deduplication rate overall), the speed of the Backup Exec 16 job became much slower - less than 30 ...
user avatar
  • 101
7 votes
1 answer
6k views

NTFS Compression with Deduplication?

I have installed the Data Deduplication role on Windows Server 2016. Per Microsoft, it's using NTFS compression to help save space: Data deduplication stores more data in less physical space. It ...
user avatar
3 votes
2 answers
4k views

Deduplication and cost savings on IaaS object stores (S3/Azure Blobs)

Do any of the commercial IaaS object stores (S3, Azure Blobs etc.) avoid charging multiple times for storing duplicate data (identical files, or parts of files)? For instance, we have a 15 TB dataset ...
user avatar
  • 408
1 vote
1 answer
48 views

18.1 TB Drive Shows .75 TB Free but only 4.66 TB Deduped on Windows Server 2012 R2

I have a Dell PE R520 with 18.1 TB drive (built on RAID 6, disk internal to server). I setup Microsoft's Deduplication functionality on this drive and began moving large amounts of material onto it. ...
user avatar
1 vote
0 answers
2k views

Error 0x80070519 reading file: revision level is unknown

I have mounted a dedup volume in Windows Server 2012 R2. When I try to copy a file from the volume to another I get the error 0x80070519 revision level is unknown The volume seems to be healthy ...
user avatar
1 vote
0 answers
205 views

Stop Windows Backup from making Deduplication Exclusion for backup drive

With Windows Server Backup after every backup, the server puts an exclusion in the Deduplication list for the backup folder. Is there a way to stop this behavior? If not, is there a way to get ...
user avatar
  • 251
0 votes
1 answer
413 views

Does rsync -H convert file to hard link on destination after deduplication on source?

I've got two servers, the second one being synced via rsync every now and then. I've got some (few thousands ?) of big duplicate files on which I would like to use freedup to spare some space. If I ...
user avatar
0 votes
2 answers
157 views

With backups, is deduplication a function of the server independent of the backup software?

In order to save space with backups, I wanted to know if I can use any deduplication or if I have use a special implementation. In other words, say I use backup software x. If I ship those backups ...
user avatar
  • 2,248
1 vote
2 answers
4k views

Is there a way to get the cmdlet for data deduplication when running an Unoptimization job?

I deactivated data deduplication on one of my drives because it doesn't work well with the backup system. I then started an unoptimization job in Administrative PowerShell. After that I realized I ...
user avatar
  • 547
1 vote
0 answers
175 views

Is it safe to delete chunks.chk file while using SDFS?

I setup a temporary storage NAS using various hard drives and random components lying about my workshop. It is running CentOS 7, with a software RAID of 8 500 GB SATA drives connected as spare drives ...
user avatar
  • 141
2 votes
0 answers
263 views

2012 R2 DPM: massive increase in data transferred during syncronisation of a Deduplicated protected volume

I have been using 2012 R2 DPM to protect a 46 TB 2012 R2 deduplicated volume for the past 4 months without too much issue. The volume is a low access archive server. Current usage on the volume is 23 ...
user avatar
  • 133
2 votes
2 answers
626 views

Windows Server 2012R2 NTFS data deduplication and SVN clients

There seems to be an issue with using SVN on a NTFS deduped volume. To save space on my SSD for some huge development checkouts (> 5 GB each) i enabled data deduplication on the volume. This seems to ...
user avatar
  • 183