2

I try to mirror a share mounted via SMB to a local dir. The share is hosted on a Windows 2016 server on which I have no access nor can I install any tools. All I have therefore is an SMB share.

The share contains around 350GB of data, mostly small files (4MB). Every night a rsync jobs runs and transfers all updates to a local dir. This takes ages to complete. Sending the incremental file list takes around 10 to 15 minutes (I need to verify this) but then rsync needs to transfer a couple of MB (roughly 20 files changed per night) a whopping 12 hours.

Maybe I have a misunderstanding on how rsync works. But I am under the impression that after receiving the incremental file list rsync is able to figure out what files need transferring. So all that is left is to pick the changed files and transfer them which should take no longer than a couple of minutes to max one hour.

I was able to bring down the transfer time from 12 to 9 hours by mounting the SMB share with actimeo=600 option. But it still takes much longer than expected.

So this is the mount (on a Synology NAS):

mount -t cifs //remote.server/share /volume1/sysadmin/share -o vers=3.0,username=john,password='***',domain=EXAMPLE,ro,soft,uid=1026,forceuid,gid=100,forcegid,addr=remote.server,file_mode=0777,dir_mode=0777,iocharset=utf8,nocase,cache=none,noserverino,actimeo=600

And this is the rsync job:

rsyncOptions=(
    --delete
    --human-readable
    --itemize-changes
    "--modify-window=1"
    --partial
    --protect-args
    --recursive
    --stats
    --times
    --verbose
)
rsyncExclusions=(
    --exclude=.DS_Store
    --exclude=.Trash
    --exclude=.TemporaryItems
    --exclude=._*
    --exclude=.AppleDB
    --exclude=.AppleDouble
    --exclude=__MACOSX
    --exclude=.AppleDesktop
    --exclude=.~lock*
    --exclude=.fseventsd
    --exclude=.lock
    --exclude=.afpDeleted*
    --exclude=.smbdelete*
    --exclude=.TEMP_*
    --exclude=@eaDir
    --exclude=Thumbs.db
    --exclude='* '
    --exclude='*.'
)
/usr/bin/rsync "${rsyncOptions[@]}" "${rsyncExclusions[@]}" "/volume1/sysadmin/share/" "/volume1/share/

Here the stats of the last sync:

Number of files: 571170
Number of files transferred: 279
Total file size: 351.95G bytes
Total transferred file size: 387.34M bytes
Literal data: 387.34M bytes
Matched data: 0 bytes
File list size: 25.95M
File list generation time: 0.526 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 413.57M
Total bytes received: 223.89K

sent 413.57M bytes  received 223.89K bytes  13.11K bytes/sec
total size is 351.95G  speedup is 850.56

How could I speed up the transfer? Maybe other tools like lftp are more suited to accomplish this... Any pointers appreciated.

UPDATE 1

  • lftp with mirror --dry-run --no-perms --exclude-glob="*." --exclude-glob="* " --parallel --only-newer option also takes around 9 to 12 hours to complete. So this is no alternative.

  • A simple copy in the GUI or via cp of a 4.7MB file takes around 2-3 seconds to complete.

  • I started rsync with --dry-run option. I will update this post as soon as I have the results.

UPDATE 2

  • rsync with --dry-run is running now for more than 6 hours. I guess it is save to assume that it will take as well 9 hours plus. The one and only entry in the log is sending incremental file list. Today is a public holiday here (therefore no changes of files/dirs) so I'm not sure how long it took to complete this part of the process.

UPDATE 3

With more logging enabled the result looks like this:

2019/10/04 04:00:03 [20771] building file list
2019/10/04 05:42:16 [20771] .d..t...... dir1/
2019/10/04 05:42:17 [20771] >f+++++++++ dir1/file1
2019/10/04 05:42:44 [20771] .d..t...... dir1/subdir1/
2019/10/04 05:42:44 [20771] >f.st...... dir1/subdir1/file1
2019/10/04 05:43:19 [20771] >f..t...... dir1/subdir1/file2
2019/10/04 05:43:20 [20771] >f.st...... dir1/subdir1/file3
2019/10/04 05:43:20 [20771] .d..t...... dir1/subdir2/
2019/10/04 05:43:21 [20771] >f+++++++++ dir1/subdir2/file1
2019/10/04 10:16:38 [20771] .d..t...... dir2/subdir1/

...

Note the huge time gap between dir1 and dir2, either rsync needs a lot of time to traverse the directory tree or I something else is going on.

8
  • How fast is to copy a (relatively) big file from the mounted cifs share to the local filesystem? And vice-versa?
    – shodanshok
    Oct 1, 2019 at 14:21
  • How long does it take to run with the --dry-run option? Oct 1, 2019 at 19:06
  • @shodanshok: I updated my question. The volume is ro so I cannot test on this share. I sync from another local dir to a different share on the remote server and that one is quite fast. Oct 3, 2019 at 13:34
  • @GordonDavisson: I updated my question. So far it is running for 6 hours plus. Since there are no changes (public holiday here) it is hard to tell how long it needed for sending the incremental file list. Oct 3, 2019 at 19:24
  • @TylerDurden That sounds like either it's running its find-the-changes-within-a-file algorithm on everything (it shouldn't, between --times and the implicit --whole-file you should get from the local file paths), or walking the directory to compare file stats is running reeeeely slowly. Can you tell how much network traffic it's generating during the dry run? Oh, another possibility occurs to me: is it running out of memory while comparing trees, and thrashing virtual memory? Oct 3, 2019 at 20:13

1 Answer 1

1

I figured that the only way to solve my issues was to have a daemon running on both sides. Since I didn't want to install Cygwin or WSL on the remove Windows Server (not under my control) I ended up using Unison which works perfectly fine so far. Syncing now takes 10-15 minutes. So use Unison with SSH I downloaded the SSH binary from the PowerShell repository on GitHub. So I had two binaries/portable apps I could just drop on the server and get things working. Please make sure when using Unison to use the same version of Unison and OCAML.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .