Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I need to transfer a huge amount of mp3s between two serves (Ubuntu). By huge I mean about a million files which are on average 300K. I tried with scp but it would have taken about a week. (about 500 KB/s) If I transfer a single file by HTTP, I get 9-10 MB/s, but I don't know how to transfer all of them.

Is there a way to transfer all of them quickly?

share|improve this question
1  
What sort of network do you have between the servers. I've used a GB Ethernet crossover between 1 NIC in each machine. I got very good through put in that configuration using SCP –  Jim Blizard Jun 2 '09 at 20:02
    
You may want to investigate why scp is so slow. It is may be slower then things like ftp because of the encryption but it shouldn't be that much slower. –  Zoredache Jun 2 '09 at 20:07
    
I have 100 mbps between them. scp is slower on the small files (most of them are small) –  nicudotro Jun 2 '09 at 20:19

23 Answers 23

up vote 65 down vote accepted

I would recommend tar. When the file trees are already similar, rsync performs very well. However, since rsync will do multiple analysis passes on each file, and then copy the changes, it is much slower than tar for the initial copy. This command will likely do what you want. It will copy the files between the machines, as well as preserve both permissions and user/group ownerships.

tar -c /path/to/dir | ssh remote_server 'tar -xvf - -C /absolute/path/to/remotedir'

As per Mackintosh's comment below this is the command you would use for rsync

rsync -avW -e ssh /path/to/dir/ remote_server:/path/to/remotedir
share|improve this answer
1  
+1 That was I about to write,but I got distracted... –  Zoredache Jun 2 '09 at 20:05
1  
+1 The tar option is far more efficient for large numbers of small files as both scp and rsync will have many more round trips per file across the network. –  Sekenre Jun 2 '09 at 21:16
3  
rsync worked better for me than tar –  nicudotro Jun 2 '09 at 22:24
1  
Also, if you have plenty of CPU available (on both ends), but (at least) a slow link between the hosts, it may be worth enabling compression (gzip or bzip) in the tar command. –  Vatine Oct 14 '10 at 14:03
1  
@Jamie: If you're using ssh-agent, then it should be used. Otherwise just use the '-i' option to specify where to find the private key. See the man page for details. –  Scott Pack Jul 10 '12 at 17:03

I'd use rsync.

If you've got them exported via HTTP with directory listings available, you could use wget and the --mirror argument, too.

You're already seeing that HTTP is faster than SCP because SCP is encrypting everything (and thus bottlenecking on the CPU). HTTP and rsync are going to move faster because they're not encrypting.

Here's some docs on setting up rsync on Ubuntu: https://help.ubuntu.com/community/rsync

Those docs talk about tunneling rsync over SSH, but if you're just moving data around on a private LAN you don't need SSH. (I'm assuming you are on a private LAN. If you're getting 9-10MB/sec over the Internet then I want to know what kind of connections you have!)

Here are some other very basic docs that will allow you to setup a relative insecure rsync server (w/ no dependence on SSH): http://transamrit.net/docs/rsync/

share|improve this answer
    
While SCP uses indeed some CPU for encrypting the data, I don't think that he has a 100% CPU usage, so the CPU is not a bottleneck. I've noticed too a lot of times that SCP is inefficient when it comes to fast transfers. –  Cristian Ciupitu Jun 2 '09 at 20:27
    
Given that he was seeing 300K for SCP and 9MB for HTTP, I assumed that an SCP-related bottleneck (normally CPU) was coming into play. It could certainly be something else, though. W/o knowing the hardware specs of the machines in question it's hard to say. –  Evan Anderson Jun 2 '09 at 20:36
    
rsync will almost definitely be using ssh for transport, as this is default behaviour, so any overhead caused by encryption in scp will also be present in rsync –  Daniel Lawson Jun 3 '09 at 0:35
1  
"You're already seeing that HTTP is faster than SCP because SCP is encrypting everything" → WRONG. Unless he has 10 year old servers, he's not CPU bound on this task. –  niXar May 4 '11 at 9:43
1  
@RamazanPOLAT - You have a command-line that's too long. Specify the file selection differently and it'll work fine for you. Typically you can just specify the source directory w/o a wildcard at the end. You can also use the --include and --exclude arguments to get more nuanced. –  Evan Anderson Feb 19 at 14:49

I don't think you're going to do any better than scp unless you install faster network cards. If you're doing this over the internet, that will not help though.

I would recommend using rsync. It may not be any faster, but at least if it fails (or you shut it down because it's taking too long), you can resume where you left off next time.

If you can connect the 2 machines directly using gigabit ethernet, that will probably be the fastest.

share|improve this answer
    
I have a unused 100mbps link directly between them –  nicudotro Jun 2 '09 at 20:02
1  
Not going to do better than SCP? SCP is pushing all that data thru an encryption step. SCP is going to be one of the slowest ways to copy it! –  Evan Anderson Jun 2 '09 at 20:02
    
True about SCP encrypting the data, but the speed of encryption is orders of magnitude faster than the network connection, and thus negligible. –  Brent Jun 3 '09 at 15:37

External hard drive and same-day courier delivery.

share|improve this answer
3  
Heh heh... no networking technology beats the bandwidth of a station wagon loaded with tapes doing 90 MPH, eh? (snicker) I assumed he was on a LAN because he said he was getting 9-10MB/sec with HTTP. –  Evan Anderson Jun 2 '09 at 20:01
    
I get that sort of speed over the internet, but I'm just lucky in where I live! If it's on a LAN, then cheaper still! –  Adam Jun 2 '09 at 20:15
1  
Ahh-- didn't look at your location. Yeah-- I heard that Internet connectivity in Korea is pretty spectacular. Stuck here in the US, I'm happy to get 900KB/sec over the 'net... –  Evan Anderson Jun 2 '09 at 20:24
    
Yes, but you can get delicious burritos while you're waiting for a download to complete and there are only about three half-decent Mexican restaurants even in Seoul... –  Adam Jun 2 '09 at 20:58

Another alternative is Unison. Might be slightly more efficient than Rsync in this case, and it's somewhat easier to set up a listener.

share|improve this answer

rsync or you might wish to tar it so its all within one file and then scp. If you lack the diskspace you can pipe the tar directly over ssh while its being made.

share|improve this answer

For 100Mb/s the theoretical throughput is 12.5 MB/s, so at 10MB/s you are doing pretty well.

I would also echo the suggestion to do rsync, probably through ssh. Something like:

rsync -avW -e ssh $SOURCE $USER@$REMOTE:$DEST

At 100Mb/s your CPUs should be able to handle the encrypt/decrypt without appreciably impacting the data rate. And if you interrupt the data flow, you should be able to resume from where you left off. Beware, with "millions" of files the startup will take a while before it actually transfers anything.

share|improve this answer

Without much discussion, use netcat, network swissarmy knife. No protocol overhead, you're directly copying to the network socket. Example

srv1$ tar cfv - *mp3 | nc -w1 remote.server.net 4321

srv2$ nc -l -p 4321 |tar xfv -
share|improve this answer
1  
Unfortunately, from what I've noticed netcat is very inefficient even if it shouldn't be. –  Cristian Ciupitu Jun 2 '09 at 20:27
    
I'm downvoting you because this is really, really terrible advice. There is one correct answer: rsync. I could list all the reasons why it's better but it wouldn't fit on this page, let alone this tiny comment box. –  niXar May 4 '11 at 9:45
1  
@niXar: If all you want to do is a single file transfer (no need for further syncing), then tarpipe is really all you need. –  Witiko Jun 14 '13 at 0:03
1  
@niXar netcat is fine if you are doing this in a secure environment like private vlan and/or over VPN. –  Lester Cheung Jun 25 '13 at 2:56

Your biggest constraint will be the 100 meg connection. If there is any possibility of upgrading to a gig connection that will improve your situation.

If my math is correct you are trying to move about 300 gigs of data right?

share|improve this answer
    
you're right around 300 G –  nicudotro Jun 2 '09 at 20:47
1  
ibeast.com/content/tools/band-calc.asp is useful for figuring out how long file transfers might take except it is for cross WAN connections. 300 gigs with a full T1 would still take 463 hours to transfer. –  Codejnki Jun 3 '09 at 13:02
  • Network File System (NFS) and then copy them with whatever you like, e.g. Midnight Commander (mc), Nautilus (from gnome). I have used NFS v3 with good results.
  • Samba (CIFS) and then copy the files with whatever you want to, but I have no idea how efficient it is.
  • HTTP with wget --mirror as Evan Anderson has suggested or any other http client. Be careful not to have any nasty symlinks or misleading index files. If all you have is MP3s, you should be safe.
  • rsync. I have used it with pretty good results and one of its nice features is that you can interrupt and resume the transfer later.

I've noticed that other people have recommended using netcat. Based on my experience with it I can say that it's slow compared with the other solutions.

share|improve this answer

With lots of files if you do go with rsync, I would try to get version 3 or above on both ends. The reason being that a lesser version will enumerate every file before it starts the transfer. The new feature is called incremental-recursion.

A new incremental-recursion algorithm is now used when rsync is talking to another 3.x version. This starts the transfer going more quickly (before all the files have been found), and requires much less memory. See the --recursive option in the manpage for some restrictions.

share|improve this answer

rsync, like others have already recommended. If the CPU overhead from the encryption is a bottleneck, use another less CPU intensive algorithm, like blowfish. E.g. something like

rsync -ax -e 'ssh -c blowfish' /local/path user@host:/remote/path

share|improve this answer
    
+1 for point about changing the cipher –  Daniel Lawson Jun 3 '09 at 0:37
    
The CPU is not going to be a bottleneck, unless you have 10G ethernet and a 10 year old CPU. –  niXar May 4 '11 at 10:46
1  
just comment: cipher "-c arcfour" is faster. –  Arman May 3 '13 at 13:58
    
@niXar: But if you already have a CPU consuming task on your machine, it is a concern. –  Isaac Dec 20 at 11:44

If you're sending over MP3's and other compressed files, you won't gain much from any solution that tries to further compress those files. The solution would be something that can create multiple connections between both servers and thus put more stress on the bandwidth between the two system. Once this maxes out, there's not much that can be gained without improving your hardware. (Faster network cards between those servers, for example.)

share|improve this answer

Looks like there may be a couple of typos in the top answer. This may work better:

tar -cf - /path/to/dir | ssh remote_server 'tar -xvf - -C /path/to/remotedir'
share|improve this answer
    
I found that the command failed when I used the -f option. –  user11749 Jul 20 '12 at 14:22
    
@user11749: There are two -f options in that command, both of which are required. Are you talking about passing -f to ssh to have it go to the background? –  retracile Jul 26 '12 at 13:59

I've encountered this, except that I was transferring Oracle logs.

Here's the breakdown

  • scp

    inefficient and encrypted (encrypted = slower than unencrypted 
    depending on the link and your processor)
    
  • rsync

    efficient but typically encrypted (though not necessarily)
    
  • FTP/HTTP

    both seem to be efficient, and both are plaintext.
    

I used FTP with great success (where great success is equivalent to ~700Mb/s on a Gb network). If you're getting 10MB (which is equal to 80Mb/s), something is probably wrong.

What can you tell us about the source and destination of the data? Is it single drive to single drive? RAID to USB?

I know this question already has an answer, but if your network is going this slow on a Gb/s crossover cable, something absolutely needs fixed.

share|improve this answer

You didn't mention if the two machines are on the same LAN, or if a secure channel (i.e. using SSH) is mandatory, but another tool you could use is netcat.

I would use the following on the receiving machine:

cd <destdir>
netcat -l -p <port> | gunzip | cpio -i -d -m

Then on the sending side:

cd <srcdir>
find . -type f | cpio -o | gzip -1 | netcat <desthost> <port>

It has the following advantages:

  • No CPU overhead for the encryption that ssh has.
  • The gzip -1 provides light compression without saturating a CPU so it makes a good trade-off, giving a bit of compression while maintaining maximum throughput. (Probably not that advantageous for MP3 data, but doesn't hurt.)
  • If you can partition the files up into groups, you can run a two or more pipes in parallel and really ensure you're saturating your network bandwidth.

e.g.,

find <dir1> <dir2> -type f | cpio -o | gzip -1 | netcat <desthost> <portone>
find <dir3> <dir4> -type f | cpio -o | gzip -1 | netcat <desthost> <porttwo>

Notes:

  • Whatever way you transfer, I would probably run an rsync or unison afterwards to ensure you got everything.
  • You could use tar instead of cpio if you prefer.
  • Even if you do end up using ssh, I would ensure it is not using any compression itself, and pipe through gzip -1 yourself instead to avoid CPU saturation. (Or at least set the CompressionLevel to 1.)
share|improve this answer

I use the tar through netcat approach as well, except I prefer to use socat -- a lot more power to optimize for your situation -- for example, by tweaking mss. (Also, laugh if you want, but I find socat arguments easier to remember because they're consistent.) So for me, this is very very common lately as I've been moving things to new servers:

host1$ tar cvf - filespec | socat stdin tcp4:host2:portnum

host2$ socat tcp4-listen:portnum stdout | tar xvpf -

Aliases optional. :)

R

share|improve this answer

A simple scp with proper options will easily reach 9-10 MB/s over LAN:

scp -C -c arcfour256 ./local/files.mp3 remoteuser@remoteserver:/opt/remote

With those options it's likely that the throughput became 4x or 5x faster than no options (default)

share|improve this answer

If you have ftp server in src side, you can use ncftpget from ncftp site. It works prefect with small files as it uses tar internally.

One comparison shows this: moving 1.9GB small files (33926 files)

  1. Using scp takes 11m59s
  2. Using rsync takes 7m10s
  3. Using ncftpget takes 1m20s
share|improve this answer

I had to copy the BackupPC disk into another machine.

I used rsync.

The machine had 256 MB of memory.

The procedure I followed was this one:

  • executed rsync without -H (took 9 hours)
  • when rsync finished, I synchronized the cpool directory and started with the pc directory; I cut the transfer.
  • then restarted rsync with -H flag, and all the files hard linked in pc directory were correctly transfered (the procedure found all the real files in cpool and then linked to the pc directory) (took 3 hours).

In the end I could verify with df -m that no extra space was spent.

By this way I elude the problem with the memory and rsync. All time I can verify the performance using top and atop and finally I transferred 165GB of data.

share|improve this answer

In moving 80 TB of data (millions of tiny files) yesterday, switching from rsync to tar proved to be much faster, as we stopped trying

# slow
rsync -av --progress /mnt/backups/section01/ /mnt/destination01/section01

and switched to tar instead...

# fast
cd /mnt/backups/
tar -cf - section01 | tar -xf - -C /mnt/destination01/ 

Since these servers are on the same LAN, the destination is NFS-mounted on the source system, which is doing the push. No make it even faster, we decided not to preserve the atime of files:

mount -o remount,noatime /mnt/backups
mount -o remount,noatime /mnt/destination01

The graphic below depicts the difference the change from rsync to tar made. It was my boss's idea and my colleague both executed it and made the great writeup on his blog. I just like pretty pictures. :)

rsync_vs_tar

share|improve this answer
    
A hacker I trust tells me "tar over tc instead of nfs might even be faster". i.e. tar cf - directory | ttcp -t dest_machine from ftp.arl.mil/mike/ttcp.html –  Philip Durbin Apr 4 '12 at 13:20
    
Unrelated question, but where is that graph from? –  CyberJacob Jun 11 at 19:48

When copying a large number of files, I found that tools like tar and rsync are more inefficient than they need to be because of the overhead of opening and closing many files. I wrote an open source tool called fast-archiver that is faster than tar for these scenarios: https://github.com/replicon/fast-archiver; it works faster by performing multiple concurrent file operations.

Here's an example of fast-archiver vs. tar on a backup of over two million files; fast-archiver takes 27 minutes to archive, vs. tar taking 1 hour 23 minutes.

$ time fast-archiver -c -o /dev/null /db/data
skipping symbolic link /db/data/pg_xlog
1008.92user 663.00system 27:38.27elapsed 100%CPU (0avgtext+0avgdata 24352maxresident)k
0inputs+0outputs (0major+1732minor)pagefaults 0swaps

$ time tar -cf - /db/data | cat > /dev/null
tar: Removing leading `/' from member names
tar: /db/data/base/16408/12445.2: file changed as we read it
tar: /db/data/base/16408/12464: file changed as we read it
32.68user 375.19system 1:23:23elapsed 8%CPU (0avgtext+0avgdata 81744maxresident)k
0inputs+0outputs (0major+5163minor)pagefaults 0swaps

To transfer files between servers, you can use fast-archiver with ssh, like this:

ssh postgres@10.32.32.32 "cd /db; fast-archive -c data --exclude=data/\*.pid" | fast-archiver -x
share|improve this answer

I tried couple of tools for copying a 1GB file The result is below: HTTP the fastest, with wget -c nc second in line scp slowest, and failed couple of times. No way to resume rsync uses ssh as a backend, thus the same result. In conclusion, I would go for http with wget -bqc and give it some time. Hope that this helps

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.