I need to transfer files from one CentOS server to another. Will transfer 5MB files about every 10 minutes. Do not need encryption.
What is an easy was for fast transfer of files?
Is there something simpler than ftp?
Server Fault is a question and answer site for system and network administrators. It only takes a minute to sign up.Sign up to join this community
tar over ssh is okay, but tar over TCP via netcat is about as low-overhead as you can get! If this is a one-time thing, give this a shot:
On the receiver:
nc -l -p 8989 | tar x
On the sender:
tar cf - /source-path | nc (receiving host ip address) 8989
If this is something you're going to do regularly, I'd probably use rsync.
Two people have mentioned tar over ssh, but didn't say how to do it. For the record, the basic procedure is to run:
tar cf - files... | ssh remotehost 'cd /destination && tar xvf -'
Or, if you want to start transfers from the receiving end:
ssh remotehost 'cd /source && tar cf - files' | tar xvf -
The advantage of doing it this way over Evan's netcat solution is that the whole thing can be started from one computer; you don't have to coordinate two netcat invocations. If you need this to run automatically, you can set up an ssh key that lets you make connections without a passphrase, and use that key for these connections.
ssh has a -C option to compress its data stream, or you can use GNU tar's builtin compressing ability:
tar zcf - files... | ssh remotehost 'cd /destination && tar xzvf -'
Rsync is another option, but its strong suit is in updating files that already exist on the receiving end. I've found it to be slower than scp or tar/ssh when using it to transfer files that don't already exist on the other end.
ssh, honestly. The encryption does slow things down, but the ease of setup and use, reliability, and (subjectively, of course) familiarity make me willing to take the hit, unless I really need that speed.
You can speed up the ssh transfer by telling it to use a faster cipher than the default, also. The default is usually
3des and you can usually do
-c des, so that will obviously be faster, and
-c blowfish is represented as fast as well, though I haven't tested it exactingly.
(Back in the days of SSHv1, you could often do
-c none, but I guess somebody decided that was bad juju.)
If you have to go through scp/ssh, my experiments show that the fastest cipher enabled by default these days is RC4. You specify the cipher via '-c arcfour' in your ssh/scp command:
for initial copy:
scp -c arcfour -r foo/ desthost:/destdir
rsync -e 'ssh -c arcfour' -r foo/ desthost:/destdir
Rsync is a good way to go because if you find yourself transferring the same files more than once it will speed up the copy, as shown with this quote from the man page.
rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speed up file transfers when the destination file is being updated. The rsync remote-update protocol allows rsync to transfer just the dif- ferences between two sets of files across the network connection, using an efficient checksum-search algorithm described in the technical report that accompanies this package.
If you want speed you could use netcat and tar. It will be faster than ssh, rsync, or scp on a local network where encryption is not a concern. Google "netcat tar".
nc -l -p 7878 | tar -C /target/dir -xzf -
tar -cz /source/dir | nc DestinationServer 7878
This obviously requires that netcat is actually installed. Google "netcat tar" for more info.
I believe that you already solved your problem but in case your ssh works on another port (not on standard port 22) you can use this
rsync -avz --rsh='ssh -pXXXXX' /local/dir/ email@example.com:/remote/dir
Note: - replace XXXXX with your port number - replace 220.127.116.11 with correct remote server IP
or this one:
Use gist command to upload and download