Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm transfering the files from Linux to Windows 7 via a mounted share (the share is mounted from Windows on Linux).. I'm copying lots of data (i.e. nearly a TB) from the old to the new machine within my LAN. I'm unfortunate enough already that I only have 100MBit. Naturally I blindly used rsync but already wondered after a day why it feels so slow. Enabling the progress meter showed my a transfer rate of about 2MB/s .

So I took a reasonable big file (800MB) and tracked the transfer timing (1):

cp : 05:33
scp (2): 06:33
rsync : 21:51

1) I deleted the files between each run
2) scp via localhost to the same Linux machine directly onto the share; completely useless but provided a progress meter

The tests were as simple as

(cp|scp|rsync) <source> <destination>

No special arguments except host/port for scp. I even tried the -W switch for rsync but cancelled after ten minutes. rsync is 3.0.3 running on Lenny. To be able to interrupt the copy process anytime and resume lead me to rsync, but now I think I seriously need to reconsider this requirement.

How's such a big difference possible?

Update/Solved:

Thanks to rschuler I was able to solve the matter: use the rsync daemon instead of smb mounting for efficiency reasons. The said DeltaCopy works, but one has to watch out for a few things

  • it's a nice GUI wrapper but if something is going wrong, it's good to know how to fix it. Seems I entered my user credentials, under which to run the rsync service, wrong the first time but the GUI didn't allow me to set a new one. I figured out it was running as a service and was able to set the proper credentials there
  • Needed to manually add the port to the firewall to allow connections
  • Personal taste: ensure the shares are either password protected or if without, maybe ensure the service does not automatically start with windows; just in case
  • The wrapped rsync binary is not a native Windows port but builds on cygwin. However, the contained cygwin DLL does not handle UTF8 properly and mangled non-ascii characters. Get a fixed DLL from http://www.okisoft.co.jp/esc/utf8-cygwin/ .

After that, the transfer rated jumped up from 2MB/s to ~8MB/s ; absolutely fantastic!

share|improve this question
    
Did you delete the destination file between each run of you test? If the destination file already existed and the meta-data wasn't correct then it would need to checksum both the source and destination. –  Zoredache May 28 '10 at 19:46
    
@Zoredache: I deleted the files between the runs; also the -W switch is supposed to ignore that –  mark May 28 '10 at 19:49
add comment

1 Answer

up vote 16 down vote accepted

(the share is mounted from Windows on Linux)

That is your problem. rsync is doing rolling checksums over the destination. The windows share. You are pulling all the data over the network to compute the checksums. (Possibly more than once).

What you need to do is have rsync running on both machines. That way only the differences (and checksums) will be transferred over the network cable. DeltaCopy is a windowized rsync. It has good enough docs to get you going.

See the up voted answer to this question for a better explanation why I think you might be misusing rsync.

share|improve this answer
    
Bingo, your answer couldn't be more to the point. Many thanks, I learned my lesson. –  mark May 28 '10 at 21:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.