Windows rsync that supports long file names, or an good alternative?

Question

I currently rsync on a Linux host to copy things from Windows to my Linux box. But I can't copy files with long names.

I have tried DeltaCopy, cwrsync, and cygwin. From what I have found there all of these tools refuse to copy files when the length of the file becomes long, this seems to be somewhere around 255 characters.

This issue is addressed on the cwrsync forums here and it supposedly it may be fixed sometime in the future whenever cygwin 1.7 comes out and UTF8 is supported. There is even a test build here.

I am not particularly comfortable using the test build on a production system. I am hoping someone knows of another rsync option.

As an alternative to rsync do you know of another tool that I can copy a directory structure on Linux from a windows host that will not have problems with long or unusually named files? The important bit is that I need a tool that can easily work across an SSH tunnel. Some of the systems are beyond firewalls, and I believe SSH is the tunnel I will be allowed to use.

Answer 1

I may be missing the point but have you considered using Robocopy on Windows. It is similar to RSync however you can't schedule it from the application directly.

This can be overcome by writing a batch file for the copy and then creating a Scheduled Task. Robocopy is free and extremely robust. I often use it to copy files between Linux and Windows using Samba and the network, and the resume capability of Robocopy is really powerful.

Answer 2

Aha! You can use subst with rsync.

If you have a deep directory tree d:\very\long\file\n\ame\etc\etc then experiment reveals that you can subst X: to d:\very\long\file\name\etc and then rsync to and from /cygdrive/x/whatever. This works both at the client end and at the server end.

Now while you can use strategic substs to reduce the length of the filenames I don't know if this allows you to bypass the 260 character limit. It may also not be very convenient. Worth a try though.

John Rennie.

----8<----

See http://www.ratsauce.co.uk/notablog/LongFilenames.asp for my ramblings on long filenames.

I don't know of any way to get the \\?\ prefix to Cygwin, and obviously the current version of Cygwin doesn't use the prefix internally. Presumably they are addressing this in the new version and that's why it will support names longer than 260 characters. I use the Cygwin rsync all over the place so like you I eagerly await the release.

There are some other issues with the Cygwin rsync. It makes a mess of ACLs unless you specify cygwin=nontsec, and it tends to hang on very big directories. On my list of things to do before I die is to write a native Windows version of rsync that won't have these problems. I believe this has been done, but only as commercial versions not public domain ones.

JR

Answer 3

I would suggest you try Unison http://www.cis.upenn.edu/~bcpierce/unison/

It is a very interesting alternative to rsync because it can also provides bi-directionnal synchronization (which rsync is not able to do). I have already used it successfully across 2 virtual machines and was very happy with the results.

But as far as long filename are concerned I don't know if it works as you want.

From the official website :

Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

Answer 4

Just a note that I run into the same problem (limitations of pre-1.7 cygwin) and found a working build of cwrsync with cygwin 1.7 here:

http://www.doering-thomas.de/page.php?seite=1&sub=6&lang=en#rsync

(Original link found from cwrsync forums)

Not exactly an official build, but it solved my charset issues :)

Answer 5

http://lists.samba.org/archive/rsync/2009-March/022955.html

http://www.okisoft.co.jp/esc/utf8-cygwin/ Has a UTF8 layer, which has a side effect of increasing the possible path size. According to its author, if you want to use more characters you can up the constant in the patch. That seems even more hackish.

Probably not your best bet for remote sync, but is a windows rsync that claims to support longer file names.

Answer 6

So, if the tools are bad on Windows is there a way to use the Linux tool?

With ssh - sshfs?

Without ssh - use a VPN and mount as SMB?

Either way would allow you to use the more capable Linux rsync client against a filesystem. I haven't done this with 10GB of data so YMMV. :)

Answer 7

This won't do large file structures, but for relatively small data-sets with long names you could look into using something like 7-Zip to create an archive that rsync could then transfer. You said that you need to pull data from Windows servers to a Linux server. If you have shell access, archive the data you need in a container (7-zip) transfer the container, and expand it when it gets to the server. This pushes the long file name problem onto the archiver, for which I believe there is better support and non-Cygwin tools.

Answer 8

In order not to lose portability I would recommend you to stay with rsync and try to solve the 255 limitation. This limitation is no longer inside Windows and is in RSYNC code - in fact the limit on Windows is around 2048 now (if I remember well).

I'm pretty sure that there are no other rsync ports than the ones specified by you and I suggest you to use the test version and report any bugs you may find.

If you don't mind using FTP you could try LFTP - it does have a very good mirroring feature but it doesn't compare with rsync when it comes to backup/sync.

Answer 9

I hacked together a script that can temporarily rename files or directories with shorter names so that rsync (and other programs) can proceed. You can use this as a part of the tool chain that you are using to copy files from Linux to Windows (rsync or otherwise). I don't know what you mean by "unusual", so perhaps this only addresses a part of your requirements. Please let me know if it is helpful for you.