Questions tagged [deduplication]
The deduplication tag has no usage guidance.
99
questions
25
votes
2
answers
23k
views
How large is my ZFS dedupe table at the moment?
I have read a lot of information about planning RAM requirements forZFS deduplication. I've just upgraded my file server's RAM to support some very limited dedupe on ZFS zvols which I cannot use ...
- 9,218
13
votes
5
answers
22k
views
Apparently Robocopy does not understand Windows Server 2016 deduplication. Is there a way to make it work without corrupting the volume?
Scenario:
Doing a server migration from old Server 2008 R2 to new Server 2016, following this Server Fault guide: File server migration using Robocopy
After Robocopy completes, enable deduplication ...
- 755
13
votes
3
answers
3k
views
Windows Server 2012 R2 Deduped 356GB to 1.32GB
I'm experimenting with deduplication on a Server 2012 R2 storage space. I let it run the first dedupe optimisation last night, and I was pleased to see that it claimed a 340GB reduction.
However, I ...
- 69.1k
12
votes
1
answer
614
views
Technical details for Server 2012 de-duplication feature
Now that Windows Server 2012 comes with de-duplication features for NTFS volumes I am having a hard time finding technical details about it. I can deduce from the TechNet documentation that the de-...
- 41k
11
votes
10
answers
12k
views
Block-level deduplication on Linux
NetApp provides block-level deduplication (ASIS). Do you know any filesystem (even FUSE-based) on Linux (or OpenSolaris, *BSD) that provides the same functionnality ?
(I'm not interested in false ...
- 1,351
11
votes
2
answers
4k
views
ZFS - destroying deduplicated zvol or data set stalls the server. How to recover?
I'm using Nexentastor on a secondary storage server running on an HP ProLiant DL180 G6 with 12 Midline (7200 RPM) SAS drives. The system has an E5620 CPU and 8GB RAM. There is no ZIL or L2ARC device.
...
- 198k
9
votes
3
answers
5k
views
Backup storage server with ZFS
I am IT everything man at a small company. I want to design a new infrastructure including a new server and a separate backup server with company wide backup policy.
The most important thing in the ...
- 190
8
votes
2
answers
380
views
What is "deduplication"?
I mean, I can look up the dictionary definition, but why is everyone suddenly talking about it in reference to virtual tape libraries? What's "new" here so that it's so much in the news lately?
- 238
8
votes
5
answers
16k
views
What's the best way to copy deduplicated files onto a new Server 2012 drive?
We have a deduplicated volume on a Windows Server 2012 machine that is approaching it's limits. It is a 1.3TB drive with ~10TB of duplicated data. We want to copy all of this data onto a larger 4TB ...
- 231
8
votes
1
answer
7k
views
NTFS Compression with Deduplication?
I have installed the Data Deduplication role on Windows Server 2016. Per Microsoft, it's using NTFS compression to help save space:
Data deduplication stores more data in less physical space. It ...
- 445
7
votes
2
answers
8k
views
How does ZFS Block Level Deduplication fit with Variable Block Size?
According to The First Google Result for "ZFS Deduplication"
...
What to dedup: Files, blocks, or bytes?
...
Block-level dedup has somewhat higher overhead than file-level dedup when whole ...
- 2,273
7
votes
3
answers
5k
views
Windows 2016 Storage Spaces Direct + Data Deduplication
Has anyone combined S2D (Storage Spaces Direct) with Data Deduplication?
Is this even possible or recommended practice?
Please elaborate on why this is a good idea or not.
EDIT: Just stumbled ...
- 722
6
votes
4
answers
6k
views
ZFS: Memory issues with dedup even though zdb -DD looks fine
I've been experimenting with ZFS on a machine with Ubuntu 12.10, 32GB RAM (non-ECC, production system will have ECC) and a 2x2TB Linux-managed RAID1 (will be moved to RAIDZ1 for production). I just ...
user206444
6
votes
2
answers
4k
views
Server 2012 deduplication: run on Hyper-V host or guest VM?
I have a Windows Server 2012 Hyper-V host with VMs on a dedicated RAID-5 volume. One VM is also running Server 2012 and is being used to serve files. The other VMs are smaller and mostly used for ...
- 217
6
votes
1
answer
2k
views
Windows 2012 Robocopy with Deduplication active
Current Issue: File Cluster Disk of 14 TB Size needs an CHKDSK which can't be done because it lasts too long. So I'm going to replace this disk in the storage of the cluster. I run many times the ...
- 73
6
votes
2
answers
3k
views
How to interpret the results of DDPEval.exe?
I have read the documentation for DDPEval.exe and various blogs and so on and cannot find any information on how to interpret the results.
I ran the tool on a drive on one of our servers and found ...
- 161
5
votes
18
answers
6k
views
How to Eliminate Tape Backup and Off-site Storage Service?
PLEASE READ UPDATE AT THE BOTTOM. THANKS! ;)
Environment Info (all Windows):
2 sites
30 servers site #1 (3TB of backup data)
5 servers site #2 (1TB of backup data)
MPLS backbone tunnel connecting ...
- 1,192
5
votes
6
answers
685
views
Capacity Optimization / Deduplication Options for Primary Storage
I'm exploring options for making more efficient use of our primary storage.
Our current NAS is an HP ProLiant DL380 G5 with an HP Storageworks MSA20, and one other disk shelf which I'm not sure what ...
- 231
5
votes
3
answers
2k
views
Proper use of disk to disk to tape backup using de-duplication and LTO5
I currently have ~12TB of data for a full disk to tape (LTO3) backup. Needless to say, it's now requiring over 16 tapes so I'm looking at other solutions. Here is what I've come up with. I'd like ...
- 506
5
votes
5
answers
382
views
Full-featured online backup providers for medium-sized enterprise? [closed]
I've been banging my head against the wall trying to find an online backup service that supports all of the following enterprisey features:
Full-system backup for Linux and Windows 2003/2008 servers, ...
- 3,792
5
votes
1
answer
226
views
Windows Data Deduplication: Determine which file/s use a given chunkfile?
Using Windows Server's data deduplication feature:
Given a specific chunk file in the chunk store, is it possible to determine which file/s include that chunk?
The use case here is that unreadable ...
- 181
5
votes
3
answers
2k
views
Store multiple versions of large binary file with minimal data duplication (preferably Linux)
I need to store multiple versions of a ~ 150 GB binary file (qcow2) on Linux servers with local storage, and was hoping there is some solution that involves just keeping diffs that can be merged as ...
user160910
5
votes
2
answers
2k
views
Offline Deduplication Software for ZFS
I do have a 300TB Freenas server to backup several linux nodes. Backup works with daily snapshot and rsync tasks.
The user often move big datasets (2-5TB) between the server - so often big files are ...
- 101
4
votes
2
answers
4k
views
Deduplication and cost savings on IaaS object stores (S3/Azure Blobs)
Do any of the commercial IaaS object stores (S3, Azure Blobs etc.) avoid charging multiple times for storing duplicate data (identical files, or parts of files)?
For instance, we have a 15 TB dataset ...
- 478
4
votes
2
answers
1k
views
File System with Real-Time Data Deduplication
Is there a file system that stores files under a hash so there are no duplicates? It can be under any operating system. I know Git does that, but I'm looking for something that can run in real-time.
- 199
4
votes
2
answers
5k
views
How to verify that a deduplication has taken place?
Microsoft Windows Server 2012 and onwards offers a de-duplication service that periodically scans files, find identical chunks and removes excessive copies to save space.
To the user browsing the ...
- 139
4
votes
1
answer
3k
views
On deduped volumes, how can I determine space used in a folder with Measure-DedupFileMetadata
I'm trying to understand how Measure-DedupFileMetadata works so I can recursively go through some folders to report on how much space is actually used. I don't know how to interpret the output.
If I ...
- 486
4
votes
0
answers
828
views
Windows NTFS Data Deduplication and Snapshot Backups
We have a file server (fs00) on Google Cloud Platform (GCP):
Running Windows Server 2019 (with Desktop Experience installed).
One OS / System disk (250GB SSD)
An independent data disk (5TB standard ...
- 310
3
votes
2
answers
3k
views
Deduplication of tar archives
Now I'm trying to configure ZFS with deduplication on my linux backup server and have a question.
For example I have 1k files in backup1.tar and 1k same files and 1 new in backup2.tar. This files ...
- 279
3
votes
2
answers
3k
views
Accessing windows server deduplicated files from other operating systems
If you enable data deduplication on the Windows Server 2012, it will change certain files (according to the policy) and introduce reparse points.
If the drive containing deduplicated volumes is ...
- 330
3
votes
1
answer
3k
views
Can an unoptimize dedup job be resumed?
I have a Start-DedupJob -type Unoptimize ... that's been running for 7 hours and is still at 0% progress.
I'm reconfiguring the server and need to restart.
Disk activity has been pegged to near ...
- 785
3
votes
1
answer
6k
views
How to use OverlayFS with Docker volumes?
For some use cases I'd like to have the possibility to create a volume with docker volume create, and fill it with data. Then I want to create a new volume that is only a copy of the first one, but no ...
- 131
3
votes
1
answer
950
views
Production-ready disk deduplication for KVM (Linux)? [closed]
Multiple Windows VMs, all of them with identical software installed (they're clones in fact) that I have to run for administrative+licensing reasons, give me a headache storage-wise. KVM on Linux runs ...
- 1,471
3
votes
1
answer
2k
views
How to find duplicate files against a reference directory structure in Linux
There are a couple of duplicate file finders for Linux listed e.g. here. I have already tried fdupes and fslint. However, from what I have seen, these will find all duplicates of the selected ...
- 185
3
votes
1
answer
3k
views
Extracting and saving attachments from maildir, deduplicated
I would like to know if there exists a solution for archiving email, stripping out the attachments and saving them separately, deduplicated?
The ideal would be a maildir-setup, with a script running ...
- 429
3
votes
1
answer
123
views
Setup Raid 10 for backup store
We have HP DL380p Gen8 with Smart Array P420i: 10HDDx6TB, 2 RAID10 6 and 4 HDD, for backup store via Arcserve UDP, deduplication backup configured for 16 KB blocks, we are not satisfied with the speed ...
- 31
3
votes
2
answers
5k
views
Mixing Volume Shadow Copy and Data Deduplication in Windows Server
I'm using Windows Server 2012 and I'm creating a scheduled backup for multiple Hyper-V guests (all of them are Windows Server 2012) to a dynamic VHDX on a removable USB drive (so that I can alternate ...
- 330
3
votes
1
answer
3k
views
Cheap/fast deduplication with hardlinks?
I've got shared hosting with a few thousand Wordpress installs and I've wanted for ages to have a nice way of removing all the duplicate files in a sensible and secure way.
I'm looking for better disk ...
- 287
3
votes
1
answer
1k
views
ZFS interpret output of zdb -S tank
I wanted to know if it would pay it out for me to activate the zfs deduplication so I ran the command
zdb -S tank but know I need some help to interpret the output.
Simulated DDT histogram:
bucket ...
- 135
3
votes
1
answer
2k
views
zfs wrong space usage
I have a backup server with ZFS (Ubuntu 16.04; 32GB RAM, 4x6TB HDD, raidz2). Recently I've found the problem with space available.
# zpool list -v
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP ...
2
votes
3
answers
3k
views
Fastest way to copy large file directory over slow internet
I need to copy a directory from one of our servers several hundred miles away. Both sites where I am and where the files are have business cable. The cable is 5 Mbps(Down) / 768 Kbps(Up). This poses a ...
- 1,985
2
votes
2
answers
677
views
Windows Server 2012R2 NTFS data deduplication and SVN clients
There seems to be an issue with using SVN on a NTFS deduped volume.
To save space on my SSD for some huge development checkouts (> 5 GB each) i enabled data deduplication on the volume. This seems to ...
- 183
2
votes
1
answer
373
views
ZFS for offline backups
I'm thinking of using ZFS to backups files to single drives that are going to be kept mostly offline. Does anyone know if the FreeBSD ZFS implementation still uses memory if a dedupped pool is ...
- 557
2
votes
1
answer
195
views
puppet code design: how to collect arguments into a string (avoiding possible duplicate declaration errros)
I'm having trouble with puppet (version 5.5) code design.
I've written a component module to deal with ucarp. It make use of the module eyp-systemd to register an ucarp service within systemd.
Now I ...
2
votes
2
answers
621
views
How to read fully deduplicated files over SMB on macOS?
I have a volume in Windows Server 2016 that is a target of the data deduplication role.
It works very well ~60% savings rate, with many files appearing as zero bytes on the disk.
The volume is ...
- 785
2
votes
1
answer
513
views
ZFS deduplication
This is my first post on this website so please be kind with me :)
On Centos 7, using ZFS deduplication in the past on a storage pool, I moved all file-systems that dedup was on to another storage ...
- 21
2
votes
0
answers
138
views
Windows 2016 deduplication and application compatibility
I'm re-evaluating Windows 2016/2019 deduplication engine, which is way better (faster/more capable) than what shipped in previous Windows versions. I understand how it works (sparsifying files via ...
- 50.7k
2
votes
0
answers
282
views
2012 R2 DPM: massive increase in data transferred during syncronisation of a Deduplicated protected volume
I have been using 2012 R2 DPM to protect a 46 TB 2012 R2 deduplicated volume for the past 4 months without too much issue. The volume is a low access archive server. Current usage on the volume is 23 ...
- 133
1
vote
1
answer
1k
views
How to activate deduplication in Windows Server 2012?
I just installed Windows Server 2012, and while I see and read about a new deduplication feature, the Server Manager keeps reporting empty fields in the columns for deduplication rate and savings. How ...
- 435
1
vote
1
answer
205
views
Windows hardlinks - why it consumes same space as original file
Im using finddupe tool to de-duplicate files and save some space on HDD.
But checking before and after operation I cant see any free space released - see below.
Can somebody advise what is wrong here ?...
- 133