How to copy a large number of files quickly between two servers

Question

I need to transfer a huge amount of mp3s between two serves (Ubuntu). By huge I mean about a million files which are on average 300K. I tried with scp but it would have taken about a week. (about 500 KB/s) If I transfer a single file by HTTP, I get 9-10 MB/s, but I don't know how to transfer all of them.

Is there a way to transfer all of them quickly?

Answer 1

I would recommend tar. When the file trees are already similar, rsync performs very well. However, since rsync will do multiple analysis passes on each file, and then copy the changes, it is much slower than tar for the initial copy. This command will likely do what you want. It will copy the files between the machines, as well as preserve both permissions and user/group ownerships.

tar -c /path/to/dir | ssh remote_server 'tar -xvf - -C /absolute/path/to/remotedir'

As per Mackintosh's comment below this is the command you would use for rsync

rsync -avW -e ssh /path/to/dir/ remote_server:/path/to/remotedir

Answer 2

External hard drive and same-day courier delivery.

Answer 3

I'd use rsync.

If you've got them exported via HTTP with directory listings available, you could use wget and the --mirror argument, too.

You're already seeing that HTTP is faster than SCP because SCP is encrypting everything (and thus bottlenecking on the CPU). HTTP and rsync are going to move faster because they're not encrypting.

Here's some docs on setting up rsync on Ubuntu: https://help.ubuntu.com/community/rsync

Those docs talk about tunneling rsync over SSH, but if you're just moving data around on a private LAN you don't need SSH. (I'm assuming you are on a private LAN. If you're getting 9-10MB/sec over the Internet then I want to know what kind of connections you have!)

Here are some other very basic docs that will allow you to setup a relative insecure rsync server (w/ no dependence on SSH): http://transamrit.net/docs/rsync/

Answer 4

Without much discussion, use netcat, network swissarmy knife. No protocol overhead, you're directly copying to the network socket. Example

srv1$ tar cfv - *mp3 | nc -w1 remote.server.net 4321

srv2$ nc -l -p 4321 |tar xfv -

Answer 5

With lots of files if you do go with rsync, I would try to get version 3 or above on both ends. The reason being that a lesser version will enumerate every file before it starts the transfer. The new feature is called incremental-recursion.

A new incremental-recursion algorithm is now used when rsync is talking to another 3.x version. This starts the transfer going more quickly (before all the files have been found), and requires much less memory. See the --recursive option in the manpage for some restrictions.

Answer 6

rsync, like others have already recommended. If the CPU overhead from the encryption is a bottleneck, use another less CPU intensive algorithm, like blowfish. E.g. something like

rsync -ax -e 'ssh -c blowfish' /local/path user@host:/remote/path

Answer 7

Another alternative is Unison. Might be slightly more efficient than Rsync in this case, and it's somewhat easier to set up a listener.

Answer 8

Looks like there may be a couple of typos in the top answer. This may work better:

tar -cf - /path/to/dir | ssh remote_server 'tar -xvf - -C /path/to/remotedir'

Answer 9

I use the tar through netcat approach as well, except I prefer to use socat -- a lot more power to optimize for your situation -- for example, by tweaking mss. (Also, laugh if you want, but I find socat arguments easier to remember because they're consistent.) So for me, this is very very common lately as I've been moving things to new servers:

host1$ tar cvf - filespec | socat stdin tcp4:host2:portnum

host2$ socat tcp4-listen:portnum stdout | tar xvpf -

Aliases optional. :)

R

Answer 10

I don't think you're going to do any better than scp unless you install faster network cards. If you're doing this over the internet, that will not help though.

I would recommend using rsync. It may not be any faster, but at least if it fails (or you shut it down because it's taking too long), you can resume where you left off next time.

If you can connect the 2 machines directly using gigabit ethernet, that will probably be the fastest.

Answer 11

Your biggest constraint will be the 100 meg connection. If there is any possibility of upgrading to a gig connection that will improve your situation.

If my math is correct you are trying to move about 300 gigs of data right?

Answer 12

• Network File System (NFS) and then copy them with whatever you like, e.g. Midnight Commander (mc), Nautilus (from gnome). I have used NFS v3 with good results.
• Samba (CIFS) and then copy the files with whatever you want to, but I have no idea how efficient it is.
• HTTP with wget --mirror as Evan Anderson has suggested or any other http client. Be careful not to have any nasty symlinks or misleading index files. If all you have is MP3s, you should be safe.
• rsync. I have used it with pretty good results and one of its nice features is that you can interrupt and resume the transfer later.

I've noticed that other people have recommended using netcat. Based on my experience with it I can say that it's slow compared with the other solutions.

Answer 13

A simple scp with proper options will easily reach 9-10 MB/s over LAN:

scp -C -c arcfour256 ./local/files.mp3 remoteuser@remoteserver:/opt/remote

With those options it's likely that the throughput became 4x or 5x faster than no options (default)

Answer 14

rsync or you might wish to tar it so its all within one file and then scp. If you lack the diskspace you can pipe the tar directly over ssh while its being made.

Answer 15

For 100Mb/s the theoretical throughput is 12.5 MB/s, so at 10MB/s you are doing pretty well.

I would also echo the suggestion to do rsync, probably through ssh. Something like:

rsync -avW -e ssh $SOURCE $USER@$REMOTE:$DEST

At 100Mb/s your CPUs should be able to handle the encrypt/decrypt without appreciably impacting the data rate. And if you interrupt the data flow, you should be able to resume from where you left off. Beware, with "millions" of files the startup will take a while before it actually transfers anything.

Answer 16

If you're sending over MP3's and other compressed files, you won't gain much from any solution that tries to further compress those files. The solution would be something that can create multiple connections between both servers and thus put more stress on the bandwidth between the two system. Once this maxes out, there's not much that can be gained without improving your hardware. (Faster network cards between those servers, for example.)

Answer 17

Buckets. Lots and lots of buckets!

Answer 18

I've encountered this, except that I was transferring Oracle logs.

Here's the breakdown

• scp

  inefficient and encrypted (encrypted = slower than unencrypted 
  depending on the link and your processor)
  

• rsync

  efficient but typically encrypted (though not necessarily)
  

• FTP/HTTP

  both seem to be efficient, and both are plaintext.
  

I used FTP with great success (where great success is equivalent to ~700Mb/s on a Gb network). If you're getting 10MB (which is equal to 80Mb/s), something is probably wrong.

What can you tell us about the source and destination of the data? Is it single drive to single drive? RAID to USB?

I know this question already has an answer, but if your network is going this slow on a Gb/s crossover cable, something absolutely needs fixed.

Answer 19

You didn't mention if the two machines are on the same LAN, or if a secure channel (i.e. using SSH) is mandatory, but another tool you could use is netcat.

I would use the following on the receiving machine:

cd <destdir>
netcat -l -p <port> | gunzip | cpio -i -d -m

Then on the sending side:

cd <srcdir>
find . -type f | cpio -o | gzip -1 | netcat <desthost> <port>

It has the following advantages:

• No CPU overhead for the encryption that ssh has.
• The gzip -1 provides light compression without saturating a CPU so it makes a good trade-off, giving a bit of compression while maintaining maximum throughput. (Probably not that advantageous for MP3 data, but doesn't hurt.)
• If you can partition the files up into groups, you can run a two or more pipes in parallel and really ensure you're saturating your network bandwidth.

e.g.,

find <dir1> <dir2> -type f | cpio -o | gzip -1 | netcat <desthost> <portone>
find <dir3> <dir4> -type f | cpio -o | gzip -1 | netcat <desthost> <porttwo>

Notes:

• Whatever way you transfer, I would probably run an rsync or unison afterwards to ensure you got everything.
• You could use tar instead of cpio if you prefer.
• Even if you do end up using ssh, I would ensure it is not using any compression itself, and pipe through gzip -1 yourself instead to avoid CPU saturation. (Or at least set the CompressionLevel to 1.)

Answer 20

If you have ftp server in src side, you can use ncftpget from ncftp site. it works prefect with small files as it use tar internaly. one comparison shows this: moving 1.9GB small files (33926 files)

1. using scp takes 11m59s
2. using rsync takes 7m10s
3. using ncftpget takes 1m20s

Ali

Answer 21

I had to copy the BackupPC disk into another machine. I used rsync. The machine had 256 MB of memory.

The procedure i followed was this one: - y executed rsync without -H (took 9 hours) - when rsync finished synchronized the cpool directory and started with the pc directory, i cut the transfer. - then restarted rsync with -H flag, and all the files hard linked in pc directory were correctly transfered (the procedure found all the real files in cpool and then linked to the pc directory) (took 3 hours). In the end i could probe with "df -m" that no extra space were spend.

By this way i elude the problem with the memory and rsync. All time i can verify the perfomance using top and atop and finally i transfered 165GB of information.

Answer 22

In moving 80 TB of data (millions of tiny files) yesterday, switching from rsync to tar proved to be much faster, as we stopped trying

# slow
rsync -av --progress /mnt/backups/section01/ /mnt/destination01/section01

and switched to tar instead...

# fast
cd /mnt/backups/
tar -cf - section01 | tar -xf - -C /mnt/destination01/ 

Since these servers are on the same LAN, the destination is NFS-mounted on the source system, which is doing the push. No make it even faster, we decided not to preserve the atime of files:

mount -o remount,noatime /mnt/backups
mount -o remount,noatime /mnt/destination01

The graphic below depicts the difference the change from rsync to tar made. It was my boss's idea and my colleague both executed it and made the great writeup on his blog. I just like pretty pictures. :)