5

I'm looking for a suitable program to copy files from one linux machine to another one. The program should be able to do authentication but it should not do encryption. The reason behind the latter is the lack of CPU power to do the encryption.

I copy backups from ~70 machines to a single backup server simultaneously. The single server is an HP Proliant DL360 G7, with 10 Gbps ethernet connection and an FC storage backend that can do 4 Gbps. Through FTP I can write ~400MB/sec to the storage (that's about what I want) but through ssh with arcfour I can only do ~100MB/sec while having 100% CPU usage. That's why I want file transfers not to be encrypted.

The alternatives that I found not really suitable:

  • rcp: no authentication, forget it
  • FTP: making the authentication "secure" (at least preventing plain-text password exchange) is possible but not really easy and I haven't found a method to force any FTP daemon to encrypt the control channel (for the authentication) and not to encrypt the data channel (for data transfers)
  • SCP/SFTP: in farely recent ssh(d) implementations you can't turn off encryption. The best you can do is to use the arcfour cypher for the encryption but it sill uses too much CPU power for my needs.
  • rsync over ssh: same problems as with SCP/SFTP.
  • plain rsync: from the documentation of rsyncd: "The authentication protocol used in rsync is a 128 bit MD4 based challenge response system. This is fairly weak protection, though (with at least one brute-force hash-finding algorithm publicly available), so if you want really top-quality security, then I recommend that you run rsync over ssh." It's a no-go.

Is there a protocol/program that can do exactly what I want?

(A big plus would be if it could work on windows as well and/or if it would support rsync-stlye copying/synchronization (e.g. copy only the differences).)

6
  • Well, I copy backups from ~70 machines to a single backup server simultaneously. The single server is an HP Proliant DL360 G7, with 10 Gbps ethernet connection and an FC storage backend that can do 4 Gbps. Through FTP I can write ~400MB/sec to the storage (that's about what I want) but through ssh with arcfour I can only do ~100MB/sec while having 100% CPU usage. That's why I want file transfers not to be encrypted.
    – Zizzencs
    Nov 28, 2012 at 15:51
  • What's wrong with NFS, SMB, or WebDav??
    – Chris S
    Nov 28, 2012 at 15:57
  • In case of these protocols you need to mount the storage at least for the time while file transfers are done. This is not what I want - I want the backup server to establish the connection to the servers that are backed up.
    – Zizzencs
    Nov 28, 2012 at 16:04
  • To be honest, I don't get why this question is not constructive. Could you elaborate on that?
    – Zizzencs
    Nov 28, 2012 at 16:15
  • Shopping Questions are Off-Topic on any of the Stack Exchange sites. See Q&A is hard, lets go Shopping and the FAQ for more details. Someone has a reasonable answer however, so I'll make an exception.
    – Chris S
    Nov 28, 2012 at 16:30

3 Answers 3

9

You tried doing what I normally do, using a lower-weight encryption algorithm (like arcfour). When that becomes the bottleneck, the next approach I use is disabling ssh encryption entirely.

One approach for this is to use hpn-ssh as your transport. My normal application for this is SSH copies across high-bandwidth, long-distance links (e.g. a 10GbE link between Chicago and New York with 16ms latency). hpn-ssh allows tuning of TCP windows there, but also has the option of disabling encryption entirely. That may help in your case.

Also see: Why is my rsync so slow?

2
  • Thanks! hpn-ssh seems to be exactly the solution that I'm looking for. Thanks!
    – Zizzencs
    Nov 28, 2012 at 16:48
  • 2
    Accepted this one as you beat Michael Hampton's answer by... 7 seconds :-D
    – Zizzencs
    Nov 28, 2012 at 16:49
7

A couple of quick-fix options come to mind:

  • Stagger your backups so that they aren't all running at the same time.
  • Replace ssh with hpn-ssh (on the server and all 70 machines) and use its NONE cipher, which turns encryption off after authentication is complete.
1
  • 1
    Staggering is not an option here - the backup window is really small and at 100 MB/sec I simply can't transfer all the data. However this hpn-ssh seems to be exactly what I'm looking for, thanks!
    – Zizzencs
    Nov 28, 2012 at 16:42
0

WebDAV + authentication on Apache2 side sounds like an option :-)

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .