Questions tagged [deduplication]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
21 views

Effective way of replicating data deduplication reparse point

Recently we enabled Windows Data Deduplication feature in all of our back-end database servers and it has be working extremely well in compressing the data storage. However, since one of our daily ...
0
votes
0answers
108 views

Method for creating accessible, compressed and deduplicated long-term archives

I am looking for a good method of archiving larger amounts of data (multiple 10 GB of mixed data, partially containing duplicates) over a longer time frame. The data should be stored as efficiently as ...
1
vote
1answer
141 views

ZFS deduplication

This is my first post on this website so please be kind with me :) On Centos 7, using ZFS deduplication in the past on a storage pool, I moved all file-systems that dedup was on to another storage ...
2
votes
0answers
36 views

Windows 2016 deduplication and application compatibility

I'm re-evaluating Windows 2016/2019 deduplication engine, which is way better (faster/more capable) than what shipped in previous Windows versions. I understand how it works (sparsifying files via ...
0
votes
0answers
26 views

postfix dovecot avoid duplicate sendings from two different IPs but same computer, same sender/recipients, same subject & body

Is there a way to avoid duplicate sendings from two different IPs but same computer, same sender, same recipients, same subject & body ? I'm using a postfix/dovecot server to which one of the ...
4
votes
1answer
682 views

Offline Deduplication Software for ZFS

I do have a 300TB Freenas server to backup several linux nodes. Backup works with daily snapshot and rsync tasks. The user often move big datasets (2-5TB) between the server - so often big files are ...
2
votes
1answer
65 views

puppet code design: how to collect arguments into a string (avoiding possible duplicate declaration errros)

I'm having trouble with puppet (version 5.5) code design. I've written a component module to deal with ucarp. It make use of the module eyp-systemd to register an ucarp service within systemd. Now I ...
1
vote
0answers
55 views

Is Path-based Unoptimization Possible?

I have a deduplicated volume in Windows Server 2016 that some software is having problems dealing with. Specifically, Matlab 2015 and older couldn't read deduplicated files, and while updating to a ...
1
vote
0answers
185 views

Deduplication Savings differs a lot between same two volumes

I run this robocopy command to copy all data from the existing volume to my new volume which will replace the old one in the near future: robocopy I:\ E:\ /E /COPYALL /PURGE /MIR /B /R:1 /W:1 /XD "...
6
votes
1answer
439 views

Windows 2012 Robocopy with Deduplication active

Current Issue: File Cluster Disk of 14 TB Size needs an CHKDSK which can't be done because it lasts too long. So I'm going to replace this disk in the storage of the cluster. I run many times the ...
1
vote
1answer
28 views

VHDX drive space being consumed

I have a host running Server 2012 R2, and a file server VM also running 2012 R2. The VM has 3 drives, OS.vhdx, file-1.vhdx and file2.vhdx. Dedupe is running inside the VM on file-1.vhdx and file2....
13
votes
4answers
12k views

Apparently Robocopy does not understand Windows Server 2016 deduplication. Is there a way to make it work without corrupting the volume?

Scenario: Doing a server migration from old Server 2008 R2 to new Server 2016, following this Server Fault guide: File server migration using Robocopy After Robocopy completes, enable deduplication ...
1
vote
0answers
43 views

Deduplication assistance

We have consolidated a couple of drives and NAS systems of a former colleague before the disks failed (some actually showed signs of degradation). Now, we know that the colleague has done "RAID by ...
4
votes
2answers
2k views

How to verify that a deduplication has taken place?

Microsoft Windows Server 2012 and onwards offers a de-duplication service that periodically scans files, find identical chunks and removes excessive copies to save space. To the user browsing the ...
0
votes
0answers
271 views

How to calculate dedupe ratio per folder

Is there a way to determine the zfs dedup ratio on a per folder basis? I realize the potential difficulties in computing this as every block would need to be analyzed, but can such a method be ...
0
votes
1answer
2k views

Size on disk smaller than capacity but no free space

We have a Windows 2012 Veeam Server with a volume that we use to store our backups on. The volume has a capacity of 14.9 TB and has data deduplication enabled. On the volume there are folders for the ...
0
votes
1answer
822 views

Windows deduplication doesn't yield usable space

Dedup size picture Hi all, When I setup Windows depuplication, I get great savings. See the attached picture. 1.28 TB deduped down to 23 GB. However, the space savings are not real. Lets say that ...
0
votes
0answers
28 views

recurring, queryable, cached checksumming of all files in a filesystem

I'm looking for a way to efficiently manage and leverage file-level checksums for all files in a filesystem over time. Goals: Configurable, fast refresh - only re-checksumming large files when ...
3
votes
1answer
2k views

Can an unoptimize dedup job be resumed?

I have a Start-DedupJob -type Unoptimize ... that's been running for 7 hours and is still at 0% progress. I'm reconfiguring the server and need to restart. Disk activity has been pegged to near ...
2
votes
1answer
370 views

How to read fully deduplicated files over SMB on macOS?

I have a volume in Windows Server 2016 that is a target of the data deduplication role. It works very well ~60% savings rate, with many files appearing as zero bytes on the disk. The volume is ...
1
vote
1answer
890 views

How to find duplicate files against a reference directory structure in Linux

There are a couple of duplicate file finders for Linux listed e.g. here. I have already tried fdupes and fslint. However, from what I have seen, these will find all duplicates of the selected ...
3
votes
1answer
700 views

zfs wrong space usage

I have a backup server with ZFS (Ubuntu 16.04; 32GB RAM, 4x6TB HDD, raidz2). Recently I've found the problem with space available. # zpool list -v NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP ...
0
votes
1answer
491 views

Unable to mount borg backup repository residing on a ZFS snapshot

I've posted the question and my own solution to this problem, to make it available to others. This relates to borg backup and was posted to this borg github issue: Allow check on read-only ...
4
votes
1answer
2k views

On deduped volumes, how can I determine space used in a folder with Measure-DedupFileMetadata

I'm trying to understand how Measure-DedupFileMetadata works so I can recursively go through some folders to report on how much space is actually used. I don't know how to interpret the output. If I ...
3
votes
1answer
3k views

How to use OverlayFS with Docker volumes?

For some use cases I'd like to have the possibility to create a volume with docker volume create, and fill it with data. Then I want to create a new volume that is only a copy of the first one, but no ...
1
vote
0answers
337 views

Clarifying sizing of dedup table on ZFS when only some datasets are deduped

I am trying to be clear about how zfs dedup handles the case where some (but not all) datasets in a pool are deduped, from a dedup table/RAM impact perspective. I found this quote from the FreeBSD ...
6
votes
3answers
5k views

Windows 2016 Storage Spaces Direct + Data Deduplication

Has anyone combined S2D (Storage Spaces Direct) with Data Deduplication? Is this even possible or recommended practice? Please elaborate on why this is a good idea or not. EDIT: Just stumbled ...
1
vote
0answers
68 views

Better to deduplicate inside VHDX on client or outside VHDX on host? [duplicate]

I have a 16TB array that I'm hosting in a virtual environment. It contains video and photos from our marketing department. The data resides on the Hyper-V host itself as a VHDX file on a 8-drive SATA ...
0
votes
0answers
50 views

Is there a way to measure or change how fast a Windows Server 2012 rehydrates data that it previously deduplicated?

Since the deduplication option was enabled on the Windows Server (currently at about 40 percent deduplication rate overall), the speed of the Backup Exec 16 job became much slower - less than 30 ...
6
votes
1answer
4k views

NTFS Compression with Deduplication?

I have installed the Data Deduplication role on Windows Server 2016. Per Microsoft, it's using NTFS compression to help save space: Data deduplication stores more data in less physical space. It ...
3
votes
2answers
3k views

Deduplication and cost savings on IaaS object stores (S3/Azure Blobs)

Do any of the commercial IaaS object stores (S3, Azure Blobs etc.) avoid charging multiple times for storing duplicate data (identical files, or parts of files)? For instance, we have a 15 TB dataset ...
1
vote
1answer
40 views

18.1 TB Drive Shows .75 TB Free but only 4.66 TB Deduped on Windows Server 2012 R2

I have a Dell PE R520 with 18.1 TB drive (built on RAID 6, disk internal to server). I setup Microsoft's Deduplication functionality on this drive and began moving large amounts of material onto it. ...
1
vote
0answers
1k views

Error 0x80070519 reading file: revision level is unknown

I have mounted a dedup volume in Windows Server 2012 R2. When I try to copy a file from the volume to another I get the error 0x80070519 revision level is unknown The volume seems to be healthy ...
1
vote
0answers
178 views

Stop Windows Backup from making Deduplication Exclusion for backup drive

With Windows Server Backup after every backup, the server puts an exclusion in the Deduplication list for the backup folder. Is there a way to stop this behavior? If not, is there a way to get ...
0
votes
1answer
342 views

Does rsync -H convert file to hard link on destination after deduplication on source?

I've got two servers, the second one being synced via rsync every now and then. I've got some (few thousands ?) of big duplicate files on which I would like to use freedup to spare some space. If I ...
0
votes
2answers
144 views

With backups, is deduplication a function of the server independent of the backup software?

In order to save space with backups, I wanted to know if I can use any deduplication or if I have use a special implementation. In other words, say I use backup software x. If I ship those backups ...
1
vote
2answers
3k views

Is there a way to get the cmdlet for data deduplication when running an Unoptimization job?

I deactivated data deduplication on one of my drives because it doesn't work well with the backup system. I then started an unoptimization job in Administrative PowerShell. After that I realized I ...
1
vote
0answers
171 views

Is it safe to delete chunks.chk file while using SDFS?

I setup a temporary storage NAS using various hard drives and random components lying about my workshop. It is running CentOS 7, with a software RAID of 8 500 GB SATA drives connected as spare drives ...
2
votes
0answers
244 views

2012 R2 DPM: massive increase in data transferred during syncronisation of a Deduplicated protected volume

I have been using 2012 R2 DPM to protect a 46 TB 2012 R2 deduplicated volume for the past 4 months without too much issue. The volume is a low access archive server. Current usage on the volume is 23 ...
2
votes
2answers
572 views

Windows Server 2012R2 NTFS data deduplication and SVN clients

There seems to be an issue with using SVN on a NTFS deduped volume. To save space on my SSD for some huge development checkouts (> 5 GB each) i enabled data deduplication on the volume. This seems to ...
13
votes
3answers
3k views

Windows Server 2012 R2 Deduped 356GB to 1.32GB

I'm experimenting with deduplication on a Server 2012 R2 storage space. I let it run the first dedupe optimisation last night, and I was pleased to see that it claimed a 340GB reduction. However, I ...
5
votes
2answers
2k views

How to interpret the results of DDPEval.exe?

I have read the documentation for DDPEval.exe and various blogs and so on and cannot find any information on how to interpret the results. I ran the tool on a drive on one of our servers and found ...
0
votes
1answer
143 views

HUGE ammount of duplicate files in this directory

I maintain our companies SBS2011 server, and I have duplicate file reports emailed to me on a monthly basis. I've noticed an increasing build up of duplicated files in the following directories: NT ...
1
vote
1answer
443 views

Mavericks permission issues with Windows Server deduplicated shares

We have a number of 10.9-10.9.3 - Mavericks - machines installed throughout our facility. Much of the user content is pulled from shares stored on our Windows Server 2012 fileservers with ...
1
vote
4answers
6k views

How to cut of Windows 2012 Server to save the disk space?

I have VPS Hosting windows 2012 Server + MSSQL Server 40GB storage. I am running out of space and I don't want to upgrade the hosting to more expensive variant. I have cleaned logs, temporary files, ...
6
votes
4answers
6k views

ZFS: Memory issues with dedup even though zdb -DD looks fine

I've been experimenting with ZFS on a machine with Ubuntu 12.10, 32GB RAM (non-ECC, production system will have ECC) and a 2x2TB Linux-managed RAID1 (will be moved to RAIDZ1 for production). I just ...
6
votes
2answers
3k views

Server 2012 deduplication: run on Hyper-V host or guest VM?

I have a Windows Server 2012 Hyper-V host with VMs on a dedicated RAID-5 volume. One VM is also running Server 2012 and is being used to serve files. The other VMs are smaller and mostly used for ...
3
votes
1answer
827 views

Production-ready disk deduplication for KVM (Linux)? [closed]

Multiple Windows VMs, all of them with identical software installed (they're clones in fact) that I have to run for administrative+licensing reasons, give me a headache storage-wise. KVM on Linux runs ...
0
votes
0answers
219 views

Data deduplication works vice versa

I have enabled data deduplication on one of my SSD's volumes and then run Start-DedupJob cmdlet with optimizastion parameter on this volume (Windows Server 2012). After dedup completed i check Get-...
5
votes
3answers
1k views

Store multiple versions of large binary file with minimal data duplication (preferably Linux)

I need to store multiple versions of a ~ 150 GB binary file (qcow2) on Linux servers with local storage, and was hoping there is some solution that involves just keeping diffs that can be merged as ...