Linux to windows ssh/scp/sftp differential backup solutions?

Question

Here is the setup:

Large amounts of large files (many over 4gb, multiple TBs in total) in a simple tree structure that need to be backed up without compression from a handful of linux boxes to a windows backup server. Every day a few hundred gigs are generated and backups are currently done by hand with WinSCP. My goal is to automate the process and have good logging for monitoring. Write once, read rarely. (offsite will be LTO-4 or 5)

Here is the issue though, almost every solution I have tried has had a showstopping problem.

WinSCP scripting - does not handle symlinks properly, resulting in endless file transfer loops. Manual GUI works ok, (with some ignorable erros)

rdiff-backup for win - does not properly support existing files already on backup server, and doing over 20TB of backups from scratch is not a time/resource feasible option. (tried every stackexchange listed trick to recognize existing data without taking 8 years to generate checksums)

rsync via cygwin - Due to lack of differential, space requirements would quickly outpace our current capacity.

My next thought is to put an ssh server on the backup server (or FTP) and do the automation on the linux side through scp etc...though I would really like checksums and differential support.

Another option could be samba from the nix boxes to the backup server.

I would really appreciate any input or guidance on the matter. I'm open to suggestions, but am trying hard to avoid proprietary and closed source solutions. I'm not used to such mixed environments...

FOLLOW UP EDIT: So after messing around with every option under the sun, mounting windows via cifs and then using the backup program of your choice seems to be the most reliable way to go if you are going from nix to windows. (with a few caveats) That being said, I am only doing so as a stopgap measure until I can implement a ZFS based NAS. With NFS/CIFS/rsync all supported, there will be no problems moving from windows, mac, or linux to the NAS.

Answer 1

I use Backuppc on both Linux and Windows systems for differential incremental backups. For Windows, we just go the CIFS route, but one could do rsync via Cygwin/Gow.

Answer 2

I tried several ways to backup incrementally my Linux box on my Win7 machine, and I concluded that it's practically an impossible task to do it reliably using a non-linux filesystem; hence I ended using with the following combination:

1. VirtualBox: A standard Linux distribution with the bare minimum packages (Debian, text-only) installed as guest on the Windows host (I suggest adding a second virtual network interface, in host mode; in this way, I practically don't use the Linux terminal, I just turn the guest on, and ssh to it locally with Putty)

2. Dirvish, a perl wrapper over rsync, that provides transparent incremental backups using hardlinks.

Works nicely for me.

Answer 3

Maybe rclone (https://rclone.org/commands/rclone_sync/) is worth to try. Written in Go, has several binaries (Windows included) and it permits to sync directories from Linux to Windows using several protocols - including SSH.

You have to first configure it (that is define a remote host alias, protocol and credentials) and then you execute from command line:

rclone sync remoteserveralias:/the/dir/path localpath

WARNING: Since sync deletes files in the destination (it tries to have two identical copies - remote vs local), a safer approach is to use the copy command which differentially copies files from source to destination without any deletion.

See https://rclone.org/commands/rclone_copy/