Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm trying to transfer about 100k files totaling 90gb. Right now I'm using rsync daemon but its slow 3.4mb/s and I need to do this a number of times. I'm wondering what options do I have that would max out a 100mbit connection over the internet and be very reliable.

share|improve this question
1  
You're getting nearly a third of your connection - that's respectable, but not great. How far away as the electron flies are the files being transferred? –  Shane Madden Nov 19 '11 at 1:17
    
50ms latency between the two servers. –  incognito2 Nov 19 '11 at 2:52
4  
I saw an alot of files once hyperboleandahalf.blogspot.com/2010/04/… –  sam Nov 21 '11 at 11:24
    
If you are using rsync daemon, there is no ssh involved, right? Then the explanation is probably the infrastructure in between the hosts. You could try netperf or iperf or flowgrind to test the speed between the hosts. If this test gives you a higher transfer rate, then you should look at how rsync is making things slow: read i/o on server slow, write i/o on client, many small files, filesystem etc.. –  AndreasM Nov 21 '11 at 11:51
    
possible duplicate of How to copy a large number of files quickly between two servers –  Scott Pack Oct 10 '12 at 16:33

6 Answers 6

Have you considered Sneakernet? With large data sets overnight shipping is often going to be faster and cheaper than transferring via the Internet.

share|improve this answer
7  
"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." - AST –  voretaq7 Nov 21 '11 at 21:29
    
well, given the affordability of gigabit LAN hardware, if its a LAN transfer, the time spent writing via eSATA to a single spindle is not all that attractive. –  memnoch_proxy Nov 17 '13 at 0:38

You mention "rsync," so I assume you are using Linux:

Why don't you create a tar or tar.gz file? Network transfer time of one big file is faster than many small ones. You could even compress it if you wish...

Tar with no compression:

On the source server:

tar -cf file.tar /path/to/files/

Then on the receiving end:

cd /path/to/files/
tar -xf /path/to/file.tar

Tar with compression:

On the source server:

tar -czf file.tar.gz /path/to/files/

Then on the receiving end:

cd /path/to/files/
tar -xzf /path/to/file.tar.gz

You would simply use rsync to do the actual transfer of the (tar|tar.gz) files.

share|improve this answer

You could try the tar and ssh trick described here:

tar cvzf - /wwwdata | ssh root@192.168.1.201 "dd of=/backup/wwwdata.tar.gz"

this should be rewritable to the following:

tar cvzf - /wwwdata | ssh root@192.168.1.201 "tar xvf -"

You'd lose the --partial features of rsync in the process, though. If the files don't change very frequently, living with a slow initial rsync could be highly worth-while as it will go much faster in the future.

share|improve this answer

How? Or TL;DR

The fastest method I've found is a combination of tar, mbuffer and ssh.

E.g.:

tar zcf - bigfile.m4p | mbuffer -s 1K -m 512 | ssh otherhost "tar zxf -"

Using this I've achieved sustained local network transfers over 950 Mb/s on 1Gb links. Replace the paths in each tar command to be appropriate for what you're transferring.

Why? mbuffer!

The biggest bottleneck in transferring large files over a network is, by far, disk I/O. The answer to that is mbuffer or buffer. They are largely similar but mbuffer has some advantages. The default buffer size is 2MB for mbuffer and 1MB for buffer. Larger buffers are more likely to never be empty. Choosing a block size which is the lowest common multiple of the native block size on both the target and destination filesystem will give the best performance.

Buffering is the thing that makes all the difference! Use it if you have it! If you don't have it, get it! Using (m}?buffer plus anything is better than anything by itself. it is almost literally a panacea for slow network file transfers.

If you're transferring multiple files use tar to "lump" them together into a single data stream. If it's a single file you can use cat or I/O redirection. The overhead of tar vs. cat is statistically insignificant so I always use tar (or zfs -send where I can) unless it's already a tarball. Neither of these is guaranteed to give you metadata (and in particular cat will not). If you want metadata, I'll leave that as an exercise for you.

Finally, using ssh for a transport mechanism is both secure and carries very little overhead. Again, the overhead of ssh vs. nc is statistically insignificant.

share|improve this answer
    
There's encryption overhead in using SSH as a transport sometimes. See: Copying files between linux machines with strong authentication without encryption –  ewwhite Feb 12 '13 at 12:36
1  
You can use faster encryption mechanisms if you need to. But you don't necessarily need to pipe this thru ssh. I prefer to set the -O and -I ports on mbuffer on both sides. Even tho this is now two commands, you skip the encryption and maximize network bandwidth by buffering both ends. I'm sending a tar stream at 720+Mbps on my local LAN with the equivalent of tar -cf - .|mbuffer -m128k -s 256M -I 9090 & mbuffer -m128k -s 256M -O host:9090 | tar -xf - –  memnoch_proxy Nov 17 '13 at 0:42
1  
@memnoch_proxy: That's a good suggestion (which I up voted) but in this day and age where the NSA is even tapping private data lines between data centers (e.g., Google and Yahoo) using encryption, IMO, is always a good habit to make. Using ssh makes that simple. Using stunnel, socat or openssl works too, but they're more complex to set up for simple transfers. –  bahamat Nov 18 '13 at 19:24
    
@bahamat thank you for making me look at the question again. My suggestion only seems appropriate if the transfer can occur over a VPN then. For an Internet transfer, I would certainly use ssh as well. –  memnoch_proxy Nov 18 '13 at 21:06

You can use various compression options of rsync.

-z, --compress              compress file data during the transfer
     --compress-level=NUM    explicitly set compression level
     --skip-compress=LIST    skip compressing files with suffix in LIST

compression ratio for binary files is very low, so you can skip those files using --skip-compress e.g. iso, already archived and compressed tarballs etc.

share|improve this answer

I'm a big fan of SFTP. I use SFTP to transfer media from my main computer to my server. I get good speeds, over LAN.

SFTP is reliable, I'd give that a shot, as it's easy to set up, and it could be faster in some cases.

share|improve this answer
5  
FTP needs to die. It's unencrypted, it does not handle interruption well, and there are at least half a dozen viable alternatives for it that don't completely suck. –  MDMarra Nov 21 '11 at 11:30
    
Ever hear of SFTP? –  Tillman32 Nov 21 '11 at 21:18
8  
Yes, have you? It is in no way related to the FTP protocol in anything except name and the fact that it moves files around. –  MDMarra Nov 21 '11 at 21:19
4  
FTP is also notoriously unreliable when traversing firewalls (it dates from a time before firewalls when having your client open a random port to accept back-connections was cool, and the hackery of Passive & Extended Passive FTP to work around that limitation is just that: Hackery) –  voretaq7 Nov 21 '11 at 21:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.