Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [deduplication]

The tag has no usage guidance.

0
votes
0answers
9 views

Block level deduplication estimate on ext4

I'm thinking of changing an ext4 filesystem to a btrfs one, the only gain I hope to have is deduplication. I know there's no dedupe on ext4, but is there some tool that can give me an estimate of what ...
2
votes
1answer
39 views

puppet code design: how to collect arguments into a string (avoiding possible duplicate declaration errros)

I'm having trouble with puppet (version 5.5) code design. I've written a component module to deal with ucarp. It make use of the module eyp-systemd to register an ucarp service within systemd. Now I ...
0
votes
0answers
27 views

Windows Server: File verification check post un-dedupe vs. before (deduped)

We have a disk that was previously deduped (windows server 2012 dedupe), but we have since re-hydrated back out. There is still 5TB left in system\chunkstore which no amount of scrubbing/GC has ...
0
votes
0answers
27 views

Is Path-based Unoptimization Possible?

I have a deduplicated volume in Windows Server 2016 that some software is having problems dealing with. Specifically, Matlab 2015 and older couldn't read deduplicated files, and while updating to a ...
1
vote
0answers
64 views

Deduplication Savings differs a lot between same two volumes

I run this robocopy command to copy all data from the existing volume to my new volume which will replace the old one in the near future: robocopy I:\ E:\ /E /COPYALL /PURGE /MIR /B /R:1 /W:1 /XD "...
6
votes
1answer
113 views

Windows 2012 Robocopy with Deduplication active

Current Issue: File Cluster Disk of 14 TB Size needs an CHKDSK which can't be done because it lasts too long. So I'm going to replace this disk in the storage of the cluster. I run many times the ...
1
vote
1answer
21 views

VHDX drive space being consumed

I have a host running Server 2012 R2, and a file server VM also running 2012 R2. The VM has 3 drives, OS.vhdx, file-1.vhdx and file2.vhdx. Dedupe is running inside the VM on file-1.vhdx and file2....
13
votes
4answers
4k views

Apparently Robocopy does not understand Windows Server 2016 deduplication. Is there a way to make it work without corrupting the volume?

Scenario: Doing a server migration from old Server 2008 R2 to new Server 2016, following this Server Fault guide: File server migration using Robocopy After Robocopy completes, enable deduplication ...
1
vote
0answers
40 views

Deduplication assistance

We have consolidated a couple of drives and NAS systems of a former colleague before the disks failed (some actually showed signs of degradation). Now, we know that the colleague has done "RAID by ...
4
votes
2answers
567 views

How to verify that a deduplication has taken place?

Microsoft Windows Server 2012 and onwards offers a de-duplication service that periodically scans files, find identical chunks and removes excessive copies to save space. To the user browsing the ...
0
votes
0answers
310 views

Dovecot and deduplication based on the email message content, not the dovecot's Id

Problem I have an IMAP-only server and I want it to automatically de-duplicate messages. I am unable to make dovecot remove copies of "identical messages" based on their content. PD: I only need to ...
0
votes
0answers
168 views

How to calculate dedupe ratio per folder

Is there a way to determine the zfs dedup ratio on a per folder basis? I realize the potential difficulties in computing this as every block would need to be analyzed, but can such a method be ...
0
votes
1answer
570 views

Size on disk smaller than capacity but no free space

We have a Windows 2012 Veeam Server with a volume that we use to store our backups on. The volume has a capacity of 14.9 TB and has data deduplication enabled. On the volume there are folders for the ...
0
votes
1answer
401 views

Windows deduplication doesn't yield usable space

Dedup size picture Hi all, When I setup Windows depuplication, I get great savings. See the attached picture. 1.28 TB deduped down to 23 GB. However, the space savings are not real. Lets say that ...
0
votes
0answers
23 views

recurring, queryable, cached checksumming of all files in a filesystem

I'm looking for a way to efficiently manage and leverage file-level checksums for all files in a filesystem over time. Goals: Configurable, fast refresh - only re-checksumming large files when ...
3
votes
1answer
605 views

Can an unoptimize dedup job be resumed?

I have a Start-DedupJob -type Unoptimize ... that's been running for 7 hours and is still at 0% progress. I'm reconfiguring the server and need to restart. Disk activity has been pegged to near ...
1
vote
1answer
243 views

How to read fully deduplicated files over SMB on macOS?

I have a volume in Windows Server 2016 that is a target of the data deduplication role. It works very well ~60% savings rate, with many files appearing as zero bytes on the disk. The volume is ...
0
votes
1answer
478 views

How to find duplicate files against a reference directory structure in Linux

There are a couple of duplicate file finders for Linux listed e.g. here. I have already tried fdupes and fslint. However, from what I have seen, these will find all duplicates of the selected ...
3
votes
1answer
387 views

zfs wrong space usage

I have a backup server with ZFS (Ubuntu 16.04; 32GB RAM, 4x6TB HDD, raidz2). Recently I've found the problem with space available. # zpool list -v NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP ...
0
votes
1answer
304 views

Unable to mount borg backup repository residing on a ZFS snapshot

I've posted the question and my own solution to this problem, to make it available to others. This relates to borg backup and was posted to this borg github issue: Allow check on read-only ...
4
votes
1answer
837 views

On deduped volumes, how can I determine space used in a folder with Measure-DedupFileMetadata

I'm trying to understand how Measure-DedupFileMetadata works so I can recursively go through some folders to report on how much space is actually used. I don't know how to interpret the output. If I ...
0
votes
1answer
1k views

How to use OverlayFS with Docker volumes?

For some use cases I'd like to have the possibility to create a volume with docker volume create, and fill it with data. Then I want to create a new volume that is only a copy of the first one, but no ...
1
vote
0answers
277 views

Clarifying sizing of dedup table on ZFS when only some datasets are deduped

I am trying to be clear about how zfs dedup handles the case where some (but not all) datasets in a pool are deduped, from a dedup table/RAM impact perspective. I found this quote from the FreeBSD ...
6
votes
3answers
4k views

Windows 2016 Storage Spaces Direct + Data Deduplication

Has anyone combined S2D (Storage Spaces Direct) with Data Deduplication? Is this even possible or recommended practice? Please elaborate on why this is a good idea or not. EDIT: Just stumbled ...
1
vote
0answers
51 views

Better to deduplicate inside VHDX on client or outside VHDX on host? [duplicate]

I have a 16TB array that I'm hosting in a virtual environment. It contains video and photos from our marketing department. The data resides on the Hyper-V host itself as a VHDX file on a 8-drive SATA ...
0
votes
0answers
45 views

Is there a way to measure or change how fast a Windows Server 2012 rehydrates data that it previously deduplicated?

Since the deduplication option was enabled on the Windows Server (currently at about 40 percent deduplication rate overall), the speed of the Backup Exec 16 job became much slower - less than 30 ...
5
votes
1answer
3k views

NTFS Compression with Deduplication?

I have installed the Data Deduplication role on Windows Server 2016. Per Microsoft, it's using NTFS compression to help save space: Data deduplication stores more data in less physical space. It ...
3
votes
2answers
2k views

Deduplication and cost savings on IaaS object stores (S3/Azure Blobs)

Do any of the commercial IaaS object stores (S3, Azure Blobs etc.) avoid charging multiple times for storing duplicate data (identical files, or parts of files)? For instance, we have a 15 TB dataset ...
1
vote
1answer
39 views

18.1 TB Drive Shows .75 TB Free but only 4.66 TB Deduped on Windows Server 2012 R2

I have a Dell PE R520 with 18.1 TB drive (built on RAID 6, disk internal to server). I setup Microsoft's Deduplication functionality on this drive and began moving large amounts of material onto it. ...
1
vote
0answers
861 views

Error 0x80070519 reading file: revision level is unknown

I have mounted a dedup volume in Windows Server 2012 R2. When I try to copy a file from the volume to another I get the error 0x80070519 revision level is unknown The volume seems to be healthy ...
1
vote
0answers
158 views

Stop Windows Backup from making Deduplication Exclusion for backup drive

With Windows Server Backup after every backup, the server puts an exclusion in the Deduplication list for the backup folder. Is there a way to stop this behavior? If not, is there a way to get ...
0
votes
1answer
281 views

Does rsync -H convert file to hard link on destination after deduplication on source?

I've got two servers, the second one being synced via rsync every now and then. I've got some (few thousands ?) of big duplicate files on which I would like to use freedup to spare some space. If I ...
0
votes
2answers
137 views

With backups, is deduplication a function of the server independent of the backup software?

In order to save space with backups, I wanted to know if I can use any deduplication or if I have use a special implementation. In other words, say I use backup software x. If I ship those backups ...
1
vote
2answers
2k views

Is there a way to get the cmdlet for data deduplication when running an Unoptimization job?

I deactivated data deduplication on one of my drives because it doesn't work well with the backup system. I then started an unoptimization job in Administrative PowerShell. After that I realized I ...
1
vote
0answers
146 views

Is it safe to delete chunks.chk file while using SDFS?

I setup a temporary storage NAS using various hard drives and random components lying about my workshop. It is running CentOS 7, with a software RAID of 8 500 GB SATA drives connected as spare drives ...
2
votes
0answers
218 views

2012 R2 DPM: massive increase in data transferred during syncronisation of a Deduplicated protected volume

I have been using 2012 R2 DPM to protect a 46 TB 2012 R2 deduplicated volume for the past 4 months without too much issue. The volume is a low access archive server. Current usage on the volume is 23 ...
2
votes
2answers
520 views

Windows Server 2012R2 NTFS data deduplication and SVN clients

There seems to be an issue with using SVN on a NTFS deduped volume. To save space on my SSD for some huge development checkouts (> 5 GB each) i enabled data deduplication on the volume. This seems to ...
13
votes
3answers
3k views

Windows Server 2012 R2 Deduped 356GB to 1.32GB

I'm experimenting with deduplication on a Server 2012 R2 storage space. I let it run the first dedupe optimisation last night, and I was pleased to see that it claimed a 340GB reduction. However, I ...
2
votes
1answer
2k views

How to interpret the results of DDPEval.exe?

I have read the documentation for DDPEval.exe and various blogs and so on and cannot find any information on how to interpret the results. I ran the tool on a drive on one of our servers and found ...
0
votes
1answer
142 views

HUGE ammount of duplicate files in this directory

I maintain our companies SBS2011 server, and I have duplicate file reports emailed to me on a monthly basis. I've noticed an increasing build up of duplicated files in the following directories: NT ...
1
vote
1answer
433 views

Mavericks permission issues with Windows Server deduplicated shares

We have a number of 10.9-10.9.3 - Mavericks - machines installed throughout our facility. Much of the user content is pulled from shares stored on our Windows Server 2012 fileservers with ...
1
vote
3answers
5k views

How to cut of Windows 2012 Server to save the disk space?

I have VPS Hosting windows 2012 Server + MSSQL Server 40GB storage. I am running out of space and I don't want to upgrade the hosting to more expensive variant. I have cleaned logs, temporary files, ...
6
votes
4answers
5k views

ZFS: Memory issues with dedup even though zdb -DD looks fine

I've been experimenting with ZFS on a machine with Ubuntu 12.10, 32GB RAM (non-ECC, production system will have ECC) and a 2x2TB Linux-managed RAID1 (will be moved to RAIDZ1 for production). I just ...
6
votes
2answers
3k views

Server 2012 deduplication: run on Hyper-V host or guest VM?

I have a Windows Server 2012 Hyper-V host with VMs on a dedicated RAID-5 volume. One VM is also running Server 2012 and is being used to serve files. The other VMs are smaller and mostly used for ...
3
votes
1answer
724 views

Production-ready disk deduplication for KVM (Linux)? [closed]

Multiple Windows VMs, all of them with identical software installed (they're clones in fact) that I have to run for administrative+licensing reasons, give me a headache storage-wise. KVM on Linux runs ...
0
votes
0answers
213 views

Data deduplication works vice versa

I have enabled data deduplication on one of my SSD's volumes and then run Start-DedupJob cmdlet with optimizastion parameter on this volume (Windows Server 2012). After dedup completed i check Get-...
5
votes
3answers
1k views

Store multiple versions of large binary file with minimal data duplication (preferably Linux)

I need to store multiple versions of a ~ 150 GB binary file (qcow2) on Linux servers with local storage, and was hoping there is some solution that involves just keeping diffs that can be merged as ...
20
votes
2answers
13k views

How large is my ZFS dedupe table at the moment?

I have read a lot of information about planning RAM requirements forZFS deduplication. I've just upgraded my file server's RAM to support some very limited dedupe on ZFS zvols which I cannot use ...
7
votes
5answers
10k views

What's the best way to copy deduplicated files onto a new Server 2012 drive?

We have a deduplicated volume on a Windows Server 2012 machine that is approaching it's limits. It is a 1.3TB drive with ~10TB of duplicated data. We want to copy all of this data onto a larger 4TB ...
1
vote
1answer
543 views

Backing up multiple git repos of the same project (deduplication)

I use a central git server (a gitlab instance). Developers clone projects to a samba share on another server. At the moment, I make backups of the gitlab server and all of the developers directorys on ...