Questions tagged [deduplication]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
61 views

Dedup.sys 0x7E after inplace upgrade Server 2016 => 19

I've recently performed an in-place upgrade from Server 2016 => 2019 (non-domain). 2 of the drives connected has been configured with data deduplication (mostly VHDs for Hyper-V). Ever since the ...
0
votes
0answers
43 views

Unoptimize taking weeks

We have ReFS on a iSCSI NAS drive. We found that there are corrupted files. We found an article saying we could unoptimize it and that might stop the corruption. We started this process before ...
2
votes
1answer
1k views

Cheap/fast deduplication with hardlinks?

I've got shared hosting with a few thousand Wordpress installs and I've wanted for ages to have a nice way of removing all the duplicate files in a sensible and secure way. I'm looking for better disk ...
1
vote
1answer
96 views

ZFS interpret output of zdb -S tank

I wanted to know if it would pay it out for me to activate the zfs deduplication so I ran the command zdb -S tank but know I need some help to interpret the output. Simulated DDT histogram: bucket ...
1
vote
0answers
123 views

btrfs compress existing data with snapshots

I'm runninga a btrfs raid1 on 5 8 TB disks, where primarily urbackup stores backups to. Urbackup uses snapshots extensively. compsize produces the following output: Processed 106161732 files, 57335496 ...
0
votes
0answers
23 views

Bacula. Backup from deduplication client

i have bacula server 9.6.5 on Centos and i need to backup folders from Windows 2019 server client with deduplication on, Problem if i'm making Full backup, the archive is incomplete (files count match,...
3
votes
1answer
60 views

Setup Raid 10 for backup store

We have HP DL380p Gen8 with Smart Array P420i: 10HDDx6TB, 2 RAID10 6 and 4 HDD, for backup store via Arcserve UDP, deduplication backup configured for 16 KB blocks, we are not satisfied with the speed ...
4
votes
0answers
268 views

Windows NTFS Data Deduplication and Snapshot Backups

We have a file server (fs00) on Google Cloud Platform (GCP): Running Windows Server 2019 (with Desktop Experience installed). One OS / System disk (250GB SSD) An independent data disk (5TB standard ...
0
votes
0answers
559 views

ZFS send/receive fails consistently with “Input/output error”

I have a ZFS disk with deduplication turned on. I have copied data from many other old hard disks onto this ZFS disk, ultimately with the intention of getting rid of all the original disks. I need ...
1
vote
1answer
312 views

ZFS deduplication

This is my first post on this website so please be kind with me :) On Centos 7, using ZFS deduplication in the past on a storage pool, I moved all file-systems that dedup was on to another storage ...
2
votes
0answers
63 views

Windows 2016 deduplication and application compatibility

I'm re-evaluating Windows 2016/2019 deduplication engine, which is way better (faster/more capable) than what shipped in previous Windows versions. I understand how it works (sparsifying files via ...
0
votes
0answers
26 views

postfix dovecot avoid duplicate sendings from two different IPs but same computer, same sender/recipients, same subject & body

Is there a way to avoid duplicate sendings from two different IPs but same computer, same sender, same recipients, same subject & body ? I'm using a postfix/dovecot server to which one of the ...
4
votes
2answers
1k views

Offline Deduplication Software for ZFS

I do have a 300TB Freenas server to backup several linux nodes. Backup works with daily snapshot and rsync tasks. The user often move big datasets (2-5TB) between the server - so often big files are ...
2
votes
1answer
94 views

puppet code design: how to collect arguments into a string (avoiding possible duplicate declaration errros)

I'm having trouble with puppet (version 5.5) code design. I've written a component module to deal with ucarp. It make use of the module eyp-systemd to register an ucarp service within systemd. Now I ...
1
vote
0answers
82 views

Is Path-based Unoptimization Possible?

I have a deduplicated volume in Windows Server 2016 that some software is having problems dealing with. Specifically, Matlab 2015 and older couldn't read deduplicated files, and while updating to a ...
1
vote
0answers
234 views

Deduplication Savings differs a lot between same two volumes

I run this robocopy command to copy all data from the existing volume to my new volume which will replace the old one in the near future: robocopy I:\ E:\ /E /COPYALL /PURGE /MIR /B /R:1 /W:1 /XD "...
6
votes
1answer
1k views

Windows 2012 Robocopy with Deduplication active

Current Issue: File Cluster Disk of 14 TB Size needs an CHKDSK which can't be done because it lasts too long. So I'm going to replace this disk in the storage of the cluster. I run many times the ...
1
vote
1answer
30 views

VHDX drive space being consumed

I have a host running Server 2012 R2, and a file server VM also running 2012 R2. The VM has 3 drives, OS.vhdx, file-1.vhdx and file2.vhdx. Dedupe is running inside the VM on file-1.vhdx and file2....
13
votes
5answers
16k views

Apparently Robocopy does not understand Windows Server 2016 deduplication. Is there a way to make it work without corrupting the volume?

Scenario: Doing a server migration from old Server 2008 R2 to new Server 2016, following this Server Fault guide: File server migration using Robocopy After Robocopy completes, enable deduplication ...
1
vote
0answers
44 views

Deduplication assistance

We have consolidated a couple of drives and NAS systems of a former colleague before the disks failed (some actually showed signs of degradation). Now, we know that the colleague has done "RAID by ...
4
votes
2answers
3k views

How to verify that a deduplication has taken place?

Microsoft Windows Server 2012 and onwards offers a de-duplication service that periodically scans files, find identical chunks and removes excessive copies to save space. To the user browsing the ...
1
vote
0answers
348 views

How to calculate dedupe ratio per folder

Is there a way to determine the zfs dedup ratio on a per folder basis? I realize the potential difficulties in computing this as every block would need to be analyzed, but can such a method be ...
0
votes
1answer
2k views

Size on disk smaller than capacity but no free space

We have a Windows 2012 Veeam Server with a volume that we use to store our backups on. The volume has a capacity of 14.9 TB and has data deduplication enabled. On the volume there are folders for the ...
0
votes
1answer
1k views

Windows deduplication doesn't yield usable space

Dedup size picture Hi all, When I setup Windows depuplication, I get great savings. See the attached picture. 1.28 TB deduped down to 23 GB. However, the space savings are not real. Lets say that ...
0
votes
0answers
34 views

recurring, queryable, cached checksumming of all files in a filesystem

I'm looking for a way to efficiently manage and leverage file-level checksums for all files in a filesystem over time. Goals: Configurable, fast refresh - only re-checksumming large files when ...
3
votes
1answer
2k views

Can an unoptimize dedup job be resumed?

I have a Start-DedupJob -type Unoptimize ... that's been running for 7 hours and is still at 0% progress. I'm reconfiguring the server and need to restart. Disk activity has been pegged to near ...
2
votes
1answer
424 views

How to read fully deduplicated files over SMB on macOS?

I have a volume in Windows Server 2016 that is a target of the data deduplication role. It works very well ~60% savings rate, with many files appearing as zero bytes on the disk. The volume is ...
1
vote
1answer
1k views

How to find duplicate files against a reference directory structure in Linux

There are a couple of duplicate file finders for Linux listed e.g. here. I have already tried fdupes and fslint. However, from what I have seen, these will find all duplicates of the selected ...
3
votes
1answer
896 views

zfs wrong space usage

I have a backup server with ZFS (Ubuntu 16.04; 32GB RAM, 4x6TB HDD, raidz2). Recently I've found the problem with space available. # zpool list -v NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP ...
0
votes
1answer
607 views

Unable to mount borg backup repository residing on a ZFS snapshot

I've posted the question and my own solution to this problem, to make it available to others. This relates to borg backup and was posted to this borg github issue: Allow check on read-only ...
4
votes
1answer
2k views

On deduped volumes, how can I determine space used in a folder with Measure-DedupFileMetadata

I'm trying to understand how Measure-DedupFileMetadata works so I can recursively go through some folders to report on how much space is actually used. I don't know how to interpret the output. If I ...
3
votes
1answer
3k views

How to use OverlayFS with Docker volumes?

For some use cases I'd like to have the possibility to create a volume with docker volume create, and fill it with data. Then I want to create a new volume that is only a copy of the first one, but no ...
1
vote
0answers
357 views

Clarifying sizing of dedup table on ZFS when only some datasets are deduped

I am trying to be clear about how zfs dedup handles the case where some (but not all) datasets in a pool are deduped, from a dedup table/RAM impact perspective. I found this quote from the FreeBSD ...
6
votes
3answers
5k views

Windows 2016 Storage Spaces Direct + Data Deduplication

Has anyone combined S2D (Storage Spaces Direct) with Data Deduplication? Is this even possible or recommended practice? Please elaborate on why this is a good idea or not. EDIT: Just stumbled ...
1
vote
0answers
69 views

Better to deduplicate inside VHDX on client or outside VHDX on host? [duplicate]

I have a 16TB array that I'm hosting in a virtual environment. It contains video and photos from our marketing department. The data resides on the Hyper-V host itself as a VHDX file on a 8-drive SATA ...
0
votes
0answers
50 views

Is there a way to measure or change how fast a Windows Server 2012 rehydrates data that it previously deduplicated?

Since the deduplication option was enabled on the Windows Server (currently at about 40 percent deduplication rate overall), the speed of the Backup Exec 16 job became much slower - less than 30 ...
6
votes
1answer
5k views

NTFS Compression with Deduplication?

I have installed the Data Deduplication role on Windows Server 2016. Per Microsoft, it's using NTFS compression to help save space: Data deduplication stores more data in less physical space. It ...
3
votes
2answers
3k views

Deduplication and cost savings on IaaS object stores (S3/Azure Blobs)

Do any of the commercial IaaS object stores (S3, Azure Blobs etc.) avoid charging multiple times for storing duplicate data (identical files, or parts of files)? For instance, we have a 15 TB dataset ...
1
vote
1answer
45 views

18.1 TB Drive Shows .75 TB Free but only 4.66 TB Deduped on Windows Server 2012 R2

I have a Dell PE R520 with 18.1 TB drive (built on RAID 6, disk internal to server). I setup Microsoft's Deduplication functionality on this drive and began moving large amounts of material onto it. ...
1
vote
0answers
2k views

Error 0x80070519 reading file: revision level is unknown

I have mounted a dedup volume in Windows Server 2012 R2. When I try to copy a file from the volume to another I get the error 0x80070519 revision level is unknown The volume seems to be healthy ...
1
vote
0answers
188 views

Stop Windows Backup from making Deduplication Exclusion for backup drive

With Windows Server Backup after every backup, the server puts an exclusion in the Deduplication list for the backup folder. Is there a way to stop this behavior? If not, is there a way to get ...
0
votes
1answer
374 views

Does rsync -H convert file to hard link on destination after deduplication on source?

I've got two servers, the second one being synced via rsync every now and then. I've got some (few thousands ?) of big duplicate files on which I would like to use freedup to spare some space. If I ...
0
votes
2answers
147 views

With backups, is deduplication a function of the server independent of the backup software?

In order to save space with backups, I wanted to know if I can use any deduplication or if I have use a special implementation. In other words, say I use backup software x. If I ship those backups ...
1
vote
2answers
3k views

Is there a way to get the cmdlet for data deduplication when running an Unoptimization job?

I deactivated data deduplication on one of my drives because it doesn't work well with the backup system. I then started an unoptimization job in Administrative PowerShell. After that I realized I ...
1
vote
0answers
174 views

Is it safe to delete chunks.chk file while using SDFS?

I setup a temporary storage NAS using various hard drives and random components lying about my workshop. It is running CentOS 7, with a software RAID of 8 500 GB SATA drives connected as spare drives ...
2
votes
0answers
251 views

2012 R2 DPM: massive increase in data transferred during syncronisation of a Deduplicated protected volume

I have been using 2012 R2 DPM to protect a 46 TB 2012 R2 deduplicated volume for the past 4 months without too much issue. The volume is a low access archive server. Current usage on the volume is 23 ...
2
votes
2answers
605 views

Windows Server 2012R2 NTFS data deduplication and SVN clients

There seems to be an issue with using SVN on a NTFS deduped volume. To save space on my SSD for some huge development checkouts (> 5 GB each) i enabled data deduplication on the volume. This seems to ...
13
votes
3answers
3k views

Windows Server 2012 R2 Deduped 356GB to 1.32GB

I'm experimenting with deduplication on a Server 2012 R2 storage space. I let it run the first dedupe optimisation last night, and I was pleased to see that it claimed a 340GB reduction. However, I ...
6
votes
2answers
2k views

How to interpret the results of DDPEval.exe?

I have read the documentation for DDPEval.exe and various blogs and so on and cannot find any information on how to interpret the results. I ran the tool on a drive on one of our servers and found ...
0
votes
1answer
143 views

HUGE ammount of duplicate files in this directory

I maintain our companies SBS2011 server, and I have duplicate file reports emailed to me on a monthly basis. I've noticed an increasing build up of duplicated files in the following directories: NT ...