Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to copy a directory from one of our servers several hundred miles away. Both sites where I am and where the files are have business cable. The cable is 5 Mbps(Down) / 768 Kbps(Up). This poses a small problem considering I need to copy about 85GB of data.

I used a file transfer calculator and it said it would take 11 days and some change.

I was wondering if there is any way to leverage something like DeDup to speed up the transfer.

The server I would be copying from is Windows 2003 R2, my box is Windows 7 Pro x64.

I was going to transfer the data to my iSCSI target which is running Ubuntu Linux, with ZFS and SCST. Would the dedup being performed by ZFS aid in the transfer so does the data need to be DeDup'd on the sending end?

share|improve this question
Are you enable to employ a human at both ends? Not necessarily at the same time. – mdpc Sep 23 '11 at 22:26
I am looking into that, might be the best way. – ianc1215 Sep 23 '11 at 23:25 – Yousf Sep 24 '11 at 0:15
up vote 20 down vote accepted

Buy an external hard drive. Copy data at site A. Mail it to site B. Copy to machine at site B.

Don't underestimate the power of the postal service.

share|improve this answer
Sneakernet4Life! Since the USPS is going TU soon... – Holocryptic Sep 23 '11 at 22:40
That idea had seriously crossed my mind. – ianc1215 Sep 23 '11 at 23:09
Or carrier pigeon. – Shane Madden Sep 24 '11 at 0:17
I can't believe no one's chimed in with this famous quote: – Ward Sep 24 '11 at 4:01
I hear the latency is horrendous, though. – mkomitee Sep 27 '11 at 13:26

Is there duplicate data?

If not, then a dedup will help not at all; if there is, it will only be deduplicated over the wire if you transfer it with zfs send -D - since your destination is Windows, then that's probably a no.

Don't forget good old fashioned compression.

share|improve this answer
There is a lot of data that is redundant, they are a bunch of PDF file and AutoCAD DWG files. We found out that compressing the data in certain ways corrupts the DWG files. If I was going to compression would it be better to use something like bzip2 instead of zip. I was told bzip2 is "lossless" compression whatever that means. – ianc1215 Sep 23 '11 at 23:08
All normal computer compression is lossless (zip, gzip, bzip2, rar, etc). It means the data which goes in is the same when it comes out. It won't corrupt your DWG files unless you have hardware failure or hit a program bug or it gets corrupted while copying. Lossy compression is where some of the data is thrown away on the grounds that we probably won't notice, and it applies to music, pictures and video, typically. (i.e. what goes in isn't what comes out, but it sounds or looks pretty much the same). In other words, you should use compression, but fix why it corrupts things first. – TessellatingHeckler Sep 23 '11 at 23:17

For Directory with lots of redundant data, you can try DZO Compressor. Deduplication makes sense. DZO performs Deduplication as well as Lossless Compression. The product is still in beta phase. As per initial results on Data with duplicates, it performs 2-3 times better than winzip, winrar and others. See their website for details or download the beta from below link.

If the data doesn't contain duplicates, then it gives the same compression as 7zip as it uses lzma for lossless compression.

Above can only help in reducing the size and thus lesser time for transfer, but it may still be in days !!!

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.