2

I need to copy a directory from one of our servers several hundred miles away. Both sites where I am and where the files are have business cable. The cable is 5 Mbps(Down) / 768 Kbps(Up). This poses a small problem considering I need to copy about 85GB of data.

I used a file transfer calculator and it said it would take 11 days and some change.

I was wondering if there is any way to leverage something like DeDup to speed up the transfer.

The server I would be copying from is Windows 2003 R2, my box is Windows 7 Pro x64.

I was going to transfer the data to my iSCSI target which is running Ubuntu Linux, with ZFS and SCST. Would the dedup being performed by ZFS aid in the transfer so does the data need to be DeDup'd on the sending end?

  • Are you enable to employ a human at both ends? Not necessarily at the same time. – mdpc Sep 23 '11 at 22:26
  • I am looking into that, might be the best way. – ianc1215 Sep 23 '11 at 23:25
  • xkcd.com/949 – Yousf Sep 24 '11 at 0:15
19

Buy an external hard drive. Copy data at site A. Mail it to site B. Copy to machine at site B.

Don't underestimate the power of the postal service.

5

Is there duplicate data?

If not, then a dedup will help not at all; if there is, it will only be deduplicated over the wire if you transfer it with zfs send -D - since your destination is Windows, then that's probably a no.

Don't forget good old fashioned compression.

  • There is a lot of data that is redundant, they are a bunch of PDF file and AutoCAD DWG files. We found out that compressing the data in certain ways corrupts the DWG files. If I was going to compression would it be better to use something like bzip2 instead of zip. I was told bzip2 is "lossless" compression whatever that means. – ianc1215 Sep 23 '11 at 23:08
  • 7
    All normal computer compression is lossless (zip, gzip, bzip2, rar, etc). It means the data which goes in is the same when it comes out. It won't corrupt your DWG files unless you have hardware failure or hit a program bug or it gets corrupted while copying. Lossy compression is where some of the data is thrown away on the grounds that we probably won't notice, and it applies to music, pictures and video, typically. (i.e. what goes in isn't what comes out, but it sounds or looks pretty much the same). In other words, you should use compression, but fix why it corrupts things first. – TessellatingHeckler Sep 23 '11 at 23:17
0

For Directory with lots of redundant data, you can try DZO Compressor. Deduplication makes sense. DZO performs Deduplication as well as Lossless Compression. The product is still in beta phase. As per initial results on Data with duplicates, it performs 2-3 times better than winzip, winrar and others. See their website http://essensolabs.com/ for details or download the beta from below link. http://essensolabs.com/invitation-trial.html

If the data doesn't contain duplicates, then it gives the same compression as 7zip as it uses lzma for lossless compression.

Above can only help in reducing the size and thus lesser time for transfer, but it may still be in days !!!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.