Tag Info

Hot answers tagged

20

Buy an external hard drive. Copy data at site A. Mail it to site B. Copy to machine at site B. Don't underestimate the power of the postal service.


14

Deduplication is where you look at the content of a data set, note all the duplicate bits that are present, and store the data just once, replacing all those otherwise copies of data with a pointer back to the one copy. It is particularly helpful with backups because when you back up things like servers so much of the data is the same. Imagine, for instance, ...


13

This has been solved. They key is that deduplicated volumes need to have the dedup flag turned off before deletion. This should be done at the pool level as well as the zvol or filesystem level. Otherwise, the deletion is essentially being deduplicated. The process takes time because the ZFS deduplication table is being referenced. In this case, RAM helps. I ...


10

Certainly ZFS is plenty stable enough to do this kind of thing, there are many very large high-profile and reliable production platforms out there based entirely on ZFS and Nexenta. That said always like to have on-site disk-based backups such as the one you're suggesting AND removable-disk or tape based backups that go off-site daily to protect against ...


10

(assuming you're referring to using dedupe within ZFS versus your backup software) I would not recommend using ZFS native deduplication for your backup system unless you design your storage system specifically for it. Using dedupe in ZFS is extremely RAM intensive. Since the deduplication occurs in real-time as data is streamed/written to the storage ...


10

You can use the zpool status -D poolname command. The output would look similar to: root@san1:/volumes# zpool status -D vol1 pool: vol1 state: ONLINE scan: scrub repaired 0 in 4h38m with 0 errors on Sun Mar 24 13:16:12 2013 DDT entries 2459286, size 481 on disk, 392 in core bucket allocated referenced ...


9

As I suspected, it's based in the VSS subsystem (source) which also explains it's async nature. The de-dupe chunks are stored in \System Volume Information\Dedup\ChunkStore\*, with settings in \System Volume Information\Dedup\Settings\*. This has significant impacts to how your backup software interacts with such volumes, which is explained in the linked ...


7

Deduplication is coming to ZFS on OpenSolaris but that functionality is not currently available. It was prototyped by Jeff Bonwick and Bill Moore this past winter and they are working on integrating it this summer. So it should be available in the next release of OpenSolaris or sooner if you want to play around with the development branch.


6

Check lessFS, data-deduplication filesystem, for Linux. It is still in beta but you can try it out: http://www.lessfs.com/ Regards, MV


6

You need to install, enable, and configure the dedupe features. Since there is overhead associated with deduplication, and there are scrubbing and other jobs that need to be scheduled, it's not automagically enabled on a default install. You can do this from the GUI, or using PowerShell: Import-Module ServerManager Add-WindowsFeature -name ...


6

It's possible some blocks from the tars will be the same, but very unlikely. ZFS deduplicates at the block level (called the recordsize in ZFS parlance), so individual blocks need to be identical. The tars are essentially guaranteed to have runs of the exact same content, but whether that will compose a whole block and be block aligned is highly unlikely. ...


6

Deduplicating VHD and VHDX files is not supported. You should not dedupe on the Hyper-V host. Dedupe within the guest. Edit: Note that this has changed a bit since this answer was originally written. In certain configurations relating to VDI, dedupe of VHD/VHDXs is supported: ...


5

Windows Server 2003 R2 and later has support for DFSR, which I used extensively to sync and backup large amounts of data over a rather small pipe across three sites (80GB+ over a T1<-->T1<-->T1 topology). msdn.microsoft.com/en-us/library/bb540025(VS.85).aspx Replicating data to multiple servers increases data availability and gives users in remote ...


5

As a long time user of Sun/Oracle ZFS 7000-series appliances, I can tell you without question dedupe isn't polished. Never confuse sales with delivery! The salesguys will tell you "Oh, it's been fixed". In real life - my real life - I can tell you 24GB isn't enough to handle the "DDT tables". That is, the back end index which stores the dedupe table. That ...


5

Is there duplicate data? If not, then a dedup will help not at all; if there is, it will only be deduplicated over the wire if you transfer it with zfs send -D - since your destination is Windows, then that's probably a no. Don't forget good old fashioned compression.


5

ZFS does this, but it is not a file-level deduplication. It's two better: block level deduplication (the intermediary between block and file deduplication being byte deduplication). On Linux, there is SDFS; however ZFS has some better features like the ability to use a solid state drive as a hash table store so you're not eating up enormous amounts of RAM ...


4

Deduplication is now available with ZFS on OpenSolaris (build 128a and newers).


4

For people who may be unfamiliar with data deduplication, it is a technique whereby data is analyzed at the file (or block, I suppose) level, and where identical files/blocks throughout the file system are replaced with a smaller token. This has the effect of greatly shrinking the effective size on disk. It could be considered a form of copy-on-write. Read ...


4

I have found that most black-box solutions for de-duplication are not as effective or as efficient as the ones built directly into the storage. For example, a black-box de-dupe appliance will require all of your data pass through it in both directions before hitting whatever generic storage you are using, processing it all for de-dupe, whereas storage ...


4

I'm currently doing nightly backup of my datasystems, using mostly rsync and rsnapshot for some more 'user visible' volumes. The biggest volume has a capacity of 16TB, currently 9.5TB used. It first does a simple rsync to a separate disk array. This takes between typically 30-45minutes. Then, it does a second copy to an offsite server over a 100Mbit ...


4

ZFS deduplication works on blocks (recordlength) it does not know/care about files. Each block is checksummed using sha256 (by default changeable). If the checksum matches an other block it will just reference the same record and no new data will be written. One problem of deduplication with ZFS is that checksums are kept in memory so large pools will ...


4

If the user's file systems are built by cloning a template one, you'll have deduplication and encryption working fine together unless you change the file system keys. If the file systems have no relationship outside being in the same pool, the keys used will be different so the same data will create non deduplicable encrypted blocks. See this blog for ...


4

Look into ZFS on Linux. That will deduplicate all day long... You'll need a lot of RAM and may find that this may not be worth the effort (you could have sized appropriately for your needs), but it does work.


3

Windows isn't my area of expertise but rsync may help get the backups from one site to the other. Rsync works by breaking down files into smaller blocks and then only transfers the blocks that changed across the network. It can also compress the data as it sends it. There are some versions of it for Windows out there but I've never used them so I can't ...


3

It seems like you have done a lot of research on this and every service has been found to come up wanting :( Thinking of this in another way, is there a reason why you'd like to use an online service like this? I wonder if a more suitable option for you might be to have a remote cloud server running your favourite flavour of Linux with bacula installed? Your ...


3

Unfortunately, there is not. http://blogs.technet.com/b/filecab/archive/2012/05/21/introduction-to-data-deduplication-in-windows-server-2012.aspx 3) Portability: A volume that is under deduplication control is an atomic unit. You can back up the volume and restore it to another server. You can rip it out of one Windows 2012 server and move it to ...


3

you can actually install dedup in windows 8.0 or 8.1 http://forums.mydigitallife.info/threads/34417-Data-deduplication-for-Windows-8-x64


3

ZFS dedupe is block-based, so it should account for the contents of your archive. How large is the TAR archive? Do you already have compression enabled? What method are you using to determine the deduplication ratio? The standard methods are: zpool status -D poolname or zpool get dedupratio poolname


3

Dedupe on ZFS isn't always worth it. Okay, it's rarely worth it... I know it's appealing, sexy-sounding and seems to be a great selling point... but at what cost? Predictability. Stability. RAM usage. Planning and design. Performance. Also see: ZFS - destroying deduplicated zvol or data set stalls the server. How to recover? So let's examine your DDT ...


2

I know the MSA range well and I think you'll struggle to dedupe with what you have, for a start deduping is a reasonably slow and IO-intensive job that's best done by the actual SAN/NAS controllers. It's slightly different in a backup scenario as the backup media server can dedupe as it goes but with live data it's important to maintain data integrity and ...



Only top voted, non community-wiki answers of a minimum length are eligible