I am trying to send huge (huge as in approx 5 gb) files between servers .. Which is better, sftp or rsync ?
Could someone point me to a comparision of sftp vs rsync if any available ? Is one better than the other ? If yes why ?
Server Fault is a question and answer site for system and network administrators. It only takes a minute to sign up.Sign up to join this community
For large files like that, use
rsync with the
--partial-dir= options so that if the transfer fails part way through (due to an unexpected connection drop for instance) you can easily resume by just rerunning the same command.
I tend to use
rsync for most general transfers, not just situations where its fuller synchronisation abilities are actually needed, when it is available. It is no less secure than
sftp if run over
ssh (which it usually is) and no less efficient.
I think the main advantage of
sftp (and the related
scp) is that of generally being available anywhere where
ssh is available, so just about any Linux/BSD/similar client or server setup, whereas
rsync doesn't tend to be installed by default.
scp/sftp/rsync are slow when it comes to the actual transfer of files, even on Nehalems as the encryption is quite a heavy burden. Additionally I've got the impression that
scp/ssh set their own magic socket options like TCP window size and are always slow (maxing out at 50-70MB/s on local 10Gbps paths).
Especially for larger files and WANs this is freaking me out, I don't understand why anyone thinks he is more clever then the underlying TCP stack from the OS.
I'd look into the Globus Toolkit's GridFTP (that's the FTP on steroids) which uses parallel/multi-flow TCP sessions for bulk transfers and is also perfectly WAN-tuned. You can back GridFTP with a full-blown PKI or use ssh for credentials and session-initiation. Runs wire speed on 10Gbps and can be scaled out and load-balanced if needed, but that's really something that's needed for TBs of data.
edit: Yes, there are SSH patches that fix the window options and introduce a null cypher for the transfer and only use encryption for key exchange and credentials but you need to have both terminators of the connection to have that SSH build to take full advantage of it.