123

I need to transfer a huge amount of mp3s between two serves (Ubuntu). By huge I mean about a million files which are on average 300K. I tried with scp but it would have taken about a week. (about 500 KB/s) If I transfer a single file by HTTP, I get 9-10 MB/s, but I don't know how to transfer all of them.

Is there a way to transfer all of them quickly?

3
  • 1
    What sort of network do you have between the servers. I've used a GB Ethernet crossover between 1 NIC in each machine. I got very good through put in that configuration using SCP Jun 2, 2009 at 20:02
  • You may want to investigate why scp is so slow. It is may be slower then things like ftp because of the encryption but it shouldn't be that much slower.
    – Zoredache
    Jun 2, 2009 at 20:07
  • I have 100 mbps between them. scp is slower on the small files (most of them are small)
    – nicudotro
    Jun 2, 2009 at 20:19

28 Answers 28

157

I would recommend tar. When the file trees are already similar, rsync performs very well. However, since rsync will do multiple analysis passes on each file, and then copy the changes, it is much slower than tar for the initial copy. This command will likely do what you want. It will copy the files between the machines, as well as preserve both permissions and user/group ownerships.

tar -c /path/to/dir | ssh remote_server 'tar -xvf - -C /absolute/path/to/remotedir'

As per Mackintosh's comment below this is the command you would use for rsync

rsync -avW -e ssh /path/to/dir/ remote_server:/path/to/remotedir
19
  • 6
    +1 The tar option is far more efficient for large numbers of small files as both scp and rsync will have many more round trips per file across the network.
    – Sekenre
    Jun 2, 2009 at 21:16
  • 3
    rsync worked better for me than tar
    – nicudotro
    Jun 2, 2009 at 22:24
  • 4
    Also, if you have plenty of CPU available (on both ends), but (at least) a slow link between the hosts, it may be worth enabling compression (gzip or bzip) in the tar command.
    – Vatine
    Oct 14, 2010 at 14:03
  • 1
    @Jamie: If you're using ssh-agent, then it should be used. Otherwise just use the '-i' option to specify where to find the private key. See the man page for details.
    – Scott Pack
    Jul 10, 2012 at 17:03
  • 4
    @niXar The ~ escape character is only enabled if SSH is using a terminal. This is not the case when you specify a remote command (unless you pass the -t option). So your concern is invalid. Sep 18, 2013 at 14:55
55

External hard drive and same-day courier delivery.

5
  • 15
    Heh heh... no networking technology beats the bandwidth of a station wagon loaded with tapes doing 90 MPH, eh? (snicker) I assumed he was on a LAN because he said he was getting 9-10MB/sec with HTTP. Jun 2, 2009 at 20:01
  • 3
    I get that sort of speed over the internet, but I'm just lucky in where I live! If it's on a LAN, then cheaper still!
    – Adam
    Jun 2, 2009 at 20:15
  • 3
    Ahh-- didn't look at your location. Yeah-- I heard that Internet connectivity in Korea is pretty spectacular. Stuck here in the US, I'm happy to get 900KB/sec over the 'net... Jun 2, 2009 at 20:24
  • 3
    Yes, but you can get delicious burritos while you're waiting for a download to complete and there are only about three half-decent Mexican restaurants even in Seoul...
    – Adam
    Jun 2, 2009 at 20:58
  • 1
    This is technically referred to as 'sneakernet'.
    – John Hunt
    Jun 1, 2022 at 21:43
18

Without much discussion, use netcat, network swissarmy knife. No protocol overhead, you're directly copying to the network socket. Example

srv1$ tar cfv - *mp3 | nc -w1 remote.server.net 4321

srv2$ nc -l -p 4321 |tar xfv -
5
  • 2
    Unfortunately, from what I've noticed netcat is very inefficient even if it shouldn't be. Jun 2, 2009 at 20:27
  • I'm downvoting you because this is really, really terrible advice. There is one correct answer: rsync. I could list all the reasons why it's better but it wouldn't fit on this page, let alone this tiny comment box.
    – niXar
    May 4, 2011 at 9:45
  • 2
    @niXar: If all you want to do is a single file transfer (no need for further syncing), then tarpipe is really all you need.
    – Witiko
    Jun 14, 2013 at 0:03
  • 3
    @niXar netcat is fine if you are doing this in a secure environment like private vlan and/or over VPN. Jun 25, 2013 at 2:56
  • 2
    netcat is great for a secure environment until you get a bit flipped and the whole 1TB stream is bad. I have an elaborate script like this with parallel compression, progress output (via pv) and integrity checking via sha512sum, but once a bit is flipped, the whole stream is bad because there's no way to recover it. What we really need is a lightweight protocol like a streaming torrent for these secure environments when we need low overhead -- something that will check the integrity at the chunk (e.g., 4MB) level and can rexmit a chunk when one fails. TCP crc is not powerful enough. Oct 9, 2019 at 21:36
17

I'd use rsync.

If you've got them exported via HTTP with directory listings available, you could use wget and the --mirror argument, too.

You're already seeing that HTTP is faster than SCP because SCP is encrypting everything (and thus bottlenecking on the CPU). HTTP and rsync are going to move faster because they're not encrypting.

Here's some docs on setting up rsync on Ubuntu: https://help.ubuntu.com/community/rsync

Those docs talk about tunneling rsync over SSH, but if you're just moving data around on a private LAN you don't need SSH. (I'm assuming you are on a private LAN. If you're getting 9-10MB/sec over the Internet then I want to know what kind of connections you have!)

Here are some other very basic docs that will allow you to setup a relative insecure rsync server (w/ no dependence on SSH): http://transamrit.net/docs/rsync/

7
  • While SCP uses indeed some CPU for encrypting the data, I don't think that he has a 100% CPU usage, so the CPU is not a bottleneck. I've noticed too a lot of times that SCP is inefficient when it comes to fast transfers. Jun 2, 2009 at 20:27
  • Given that he was seeing 300K for SCP and 9MB for HTTP, I assumed that an SCP-related bottleneck (normally CPU) was coming into play. It could certainly be something else, though. W/o knowing the hardware specs of the machines in question it's hard to say. Jun 2, 2009 at 20:36
  • 1
    rsync will almost definitely be using ssh for transport, as this is default behaviour, so any overhead caused by encryption in scp will also be present in rsync Jun 3, 2009 at 0:35
  • 3
    "You're already seeing that HTTP is faster than SCP because SCP is encrypting everything" → WRONG. Unless he has 10 year old servers, he's not CPU bound on this task.
    – niXar
    May 4, 2011 at 9:43
  • 2
    @RamazanPOLAT - You have a command-line that's too long. Specify the file selection differently and it'll work fine for you. Typically you can just specify the source directory w/o a wildcard at the end. You can also use the --include and --exclude arguments to get more nuanced. Feb 19, 2014 at 14:49
12

In moving 80 TB of data (millions of tiny files) yesterday, switching from rsync to tar proved to be much faster, as we stopped trying

# slow
rsync -av --progress /mnt/backups/section01/ /mnt/destination01/section01

and switched to tar instead...

# fast
cd /mnt/backups/
tar -cf - section01 | tar -xf - -C /mnt/destination01/ 

Since these servers are on the same LAN, the destination is NFS-mounted on the source system, which is doing the push. No make it even faster, we decided not to preserve the atime of files:

mount -o remount,noatime /mnt/backups
mount -o remount,noatime /mnt/destination01

The graphic below depicts the difference the change from rsync to tar made. It was my boss's idea and my colleague both executed it and made the great writeup on his blog. I just like pretty pictures. :)

rsync_vs_tar

3
  • 1
    A hacker I trust tells me "tar over tc instead of nfs might even be faster". i.e. tar cf - directory | ttcp -t dest_machine from ftp.arl.mil/mike/ttcp.html Apr 4, 2012 at 13:20
  • Unrelated question, but where is that graph from?
    – Foxocube
    Jun 11, 2014 at 19:48
  • This is cool. The graph is likely from a tool called 'Munin' I think. One thing that this doesn't mention is that tar and rsync do totally different things. Rsync is good for pushing just the changes thereby saving potentially huge amounts of bandwidth and time depending on your connection. It isn't perfect or the right tool for everything though (as shown here)
    – John Hunt
    Jun 1, 2022 at 21:48
8

With lots of files if you do go with rsync, I would try to get version 3 or above on both ends. The reason being that a lesser version will enumerate every file before it starts the transfer. The new feature is called incremental-recursion.

A new incremental-recursion algorithm is now used when rsync is talking to another 3.x version. This starts the transfer going more quickly (before all the files have been found), and requires much less memory. See the --recursive option in the manpage for some restrictions.

7

rsync, like others have already recommended. If the CPU overhead from the encryption is a bottleneck, use another less CPU intensive algorithm, like blowfish. E.g. something like

rsync -ax -e 'ssh -c blowfish' /local/path user@host:/remote/path

3
  • The CPU is not going to be a bottleneck, unless you have 10G ethernet and a 10 year old CPU.
    – niXar
    May 4, 2011 at 10:46
  • 1
    just comment: cipher "-c arcfour" is faster.
    – Arman
    May 3, 2013 at 13:58
  • @niXar: But if you already have a CPU consuming task on your machine, it is a concern.
    – Isaac
    Dec 20, 2014 at 11:44
5

When copying a large number of files, I found that tools like tar and rsync are more inefficient than they need to be because of the overhead of opening and closing many files. I wrote an open source tool called fast-archiver that is faster than tar for these scenarios: https://github.com/replicon/fast-archiver; it works faster by performing multiple concurrent file operations.

Here's an example of fast-archiver vs. tar on a backup of over two million files; fast-archiver takes 27 minutes to archive, vs. tar taking 1 hour 23 minutes.

$ time fast-archiver -c -o /dev/null /db/data
skipping symbolic link /db/data/pg_xlog
1008.92user 663.00system 27:38.27elapsed 100%CPU (0avgtext+0avgdata 24352maxresident)k
0inputs+0outputs (0major+1732minor)pagefaults 0swaps

$ time tar -cf - /db/data | cat > /dev/null
tar: Removing leading `/' from member names
tar: /db/data/base/16408/12445.2: file changed as we read it
tar: /db/data/base/16408/12464: file changed as we read it
32.68user 375.19system 1:23:23elapsed 8%CPU (0avgtext+0avgdata 81744maxresident)k
0inputs+0outputs (0major+5163minor)pagefaults 0swaps

To transfer files between servers, you can use fast-archiver with ssh, like this:

ssh [email protected] "cd /db; fast-archive -c data --exclude=data/\*.pid" | fast-archiver -x
3

Another alternative is Unison. Might be slightly more efficient than Rsync in this case, and it's somewhat easier to set up a listener.

1
  • Unison is the best alternative I found to copying large volume of files over the network. Rsync is not multi-threaded so it copies 1 file at a time, there is a multi threaded version of rsync but its very tricky to use, unison has a batch feature and syncs files in the background
    – Arturski
    Mar 30, 2022 at 11:45
3

Looks like there may be a couple of typos in the top answer. This may work better:

tar -cf - /path/to/dir | ssh remote_server 'tar -xvf - -C /path/to/remotedir'
2
  • I found that the command failed when I used the -f option.
    – user11749
    Jul 20, 2012 at 14:22
  • @user11749: There are two -f options in that command, both of which are required. Are you talking about passing -f to ssh to have it go to the background?
    – retracile
    Jul 26, 2012 at 13:59
3

I use the tar through netcat approach as well, except I prefer to use socat -- a lot more power to optimize for your situation -- for example, by tweaking mss. (Also, laugh if you want, but I find socat arguments easier to remember because they're consistent). So for me, this is very very common lately as I've been moving things to new servers:

host1$ tar cvf - filespec | socat stdin tcp4:host2:portnum

host2$ socat tcp4-listen:portnum stdout | tar xvpf -

Aliases are optional.

2
  • Network File System (NFS) and then copy them with whatever you like, e.g. Midnight Commander (mc), Nautilus (from gnome). I have used NFS v3 with good results.
  • Samba (CIFS) and then copy the files with whatever you want to, but I have no idea how efficient it is.
  • HTTP with wget --mirror as Evan Anderson has suggested or any other http client. Be careful not to have any nasty symlinks or misleading index files. If all you have is MP3s, you should be safe.
  • rsync. I have used it with pretty good results and one of its nice features is that you can interrupt and resume the transfer later.

I've noticed that other people have recommended using netcat. Based on my experience with it I can say that it's slow compared with the other solutions.

2

You can also try using the BBCP command to do your transfer. It's a buffered parallel ssh that really screams. We can usually get 90%+ line-rate provided we can keep the pipe fed.

$ bbcp -s 8 -w 64M -N io 'tar -cO srcdirectory' desthostname:'tar -x -C destdir'

Normally, we try real hard to avoid having to move suff around. We use ZFS pools that we can always just "add" more disk space to. But sometimes... you just have to move stuff. If we have a "live" filesystem that may take hours (or days) to copy even when going full-blast.. we do the ole two step zfs send routine:

  1. Make a ZFS snapshot, and transfer to the new pool on the new machine. Let it take as long as it takes.
  2. Make a second snapshot, and send it as an incremental. The incremental snapshot only includes the (much smaller) change-set since the first, so it goes through relatively quick.
  3. Once the incremental snapshot is completed you can turn of the original and cut over to the new copy and your "offline downtime" is kept to a minimum.

We also send our zfs dumps over BBCP as well... it maximizes our network utilization and minimizes the transfer times.

BBCP is freely available, you can google it, and it's a straight-foward compile. Just copy it into your /usr/local/bin on both src and destination machines and it'll pretty much just work.

0
2

Thanks to Scott Pack's wonderful answer (I didn't know how to do this with ssh before), I can offer this improvement (if bash is your shell). This will add parallel compression, a progress indicator and check integrity across the network link:

tar c file_list |
    tee >(sha512sum >&2) |
    pv -prab |
    pigz -9 |
    ssh [user@]remote_host '
        gunzip |
        tee >(sha512sum >&2) |
        tar xC /directory/to/extract/to
    '

pv is a nice progress viewer program for your pipe and pigz is a parallel gzip program that uses as many threads as your CPU has by default (I believe up to 8 max). You can tune the compression level to better fit the ratio of CPU to network bandwith and swap it out with pxz -9e and pxz -d if you have much more CPU than bandwidth. You only have to verify that the two sums match upon completion.

This option is useful for very large amounts of data as well as high latency networks, but not very helpful if the link is unstable and drops. In those cases, rsync is probably the best choice as it can resume.

Sample output:

6c1fe5a75cc0280709a794bdfd23d7b8b655f0bbb4c320e59729c5cd952b4b1f84861b52d1eddb601259e78249d3e6618f8a1edbd20b281d6cd15f80c8593c3e  -                     ]
 176MiB [9.36MiB/s] [9.36MiB/s] [                                            <=>                                                                        ]
6c1fe5a75cc0280709a794bdfd23d7b8b655f0bbb4c320e59729c5cd952b4b1f84861b52d1eddb601259e78249d3e6618f8a1edbd20b281d6cd15f80c8593c3e  -

For block devices:

dd if=/dev/src_device bs=1024k |
    tee >(sha512sum >&2) |
    pv -prab |
    pigz -9 |
    ssh [user@]remote_host '
        gunzip |
        tee >(sha512sum >&2) |
        dd of=/dev/src_device bs=1024k
    '

Obviously, make sure they're the same size or limit with count=, skip=, seek=, etc.

When I copy filesystems this way, I'll often first dd if=/dev/zero of=/thefs/zero.dat bs=64k && sync && rm /thefs/zero.dat && umount /thefs to zero most of the unused space, which speeds up the xfer.

1

I don't think you're going to do any better than scp unless you install faster network cards. If you're doing this over the internet, that will not help though.

I would recommend using rsync. It may not be any faster, but at least if it fails (or you shut it down because it's taking too long), you can resume where you left off next time.

If you can connect the 2 machines directly using gigabit ethernet, that will probably be the fastest.

3
  • I have a unused 100mbps link directly between them
    – nicudotro
    Jun 2, 2009 at 20:02
  • 2
    Not going to do better than SCP? SCP is pushing all that data thru an encryption step. SCP is going to be one of the slowest ways to copy it! Jun 2, 2009 at 20:02
  • True about SCP encrypting the data, but the speed of encryption is orders of magnitude faster than the network connection, and thus negligible.
    – Brent
    Jun 3, 2009 at 15:37
1

For 100Mb/s the theoretical throughput is 12.5 MB/s, so at 10MB/s you are doing pretty well.

I would also echo the suggestion to do rsync, probably through ssh. Something like:

rsync -avW -e ssh $SOURCE $USER@$REMOTE:$DEST

At 100Mb/s your CPUs should be able to handle the encrypt/decrypt without appreciably impacting the data rate. And if you interrupt the data flow, you should be able to resume from where you left off. Beware, with "millions" of files the startup will take a while before it actually transfers anything.

1

I've encountered this, except that I was transferring Oracle logs.

Here's the breakdown

  • scp

    inefficient and encrypted (encrypted = slower than unencrypted 
    depending on the link and your processor) 
    
  • rsync

    efficient but typically encrypted (though not necessarily)
    
  • FTP/HTTP

    both seem to be efficient, and both are plaintext. 
    

I used FTP with great success (where great success is equivalent to ~700Mb/s on a Gb network). If you're getting 10MB (which is equal to 80Mb/s), something is probably wrong.

What can you tell us about the source and destination of the data? Is it single drive to single drive? RAID to USB?

I know this question already has an answer, but if your network is going this slow on a Gb/s crossover cable, something absolutely needs fixed.

1

You didn't mention if the two machines are on the same LAN, or if a secure channel (i.e. using SSH) is mandatory, but another tool you could use is netcat.

I would use the following on the receiving machine:

cd <destdir>
netcat -l -p <port> | gunzip | cpio -i -d -m

Then on the sending side:

cd <srcdir>
find . -type f | cpio -o | gzip -1 | netcat <desthost> <port>

It has the following advantages:

  • No CPU overhead for the encryption that ssh has.
  • The gzip -1 provides light compression without saturating a CPU so it makes a good trade-off, giving a bit of compression while maintaining maximum throughput. (Probably not that advantageous for MP3 data, but doesn't hurt.)
  • If you can partition the files up into groups, you can run a two or more pipes in parallel and really ensure you're saturating your network bandwidth.

e.g.,

find <dir1> <dir2> -type f | cpio -o | gzip -1 | netcat <desthost> <portone>
find <dir3> <dir4> -type f | cpio -o | gzip -1 | netcat <desthost> <porttwo>

Notes:

  • Whatever way you transfer, I would probably run an rsync or unison afterwards to ensure you got everything.
  • You could use tar instead of cpio if you prefer.
  • Even if you do end up using ssh, I would ensure it is not using any compression itself, and pipe through gzip -1 yourself instead to avoid CPU saturation. (Or at least set the CompressionLevel to 1.)
1

A simple scp with proper options will easily reach 9-10 MB/s over LAN:

scp -C -c arcfour256 ./local/files.mp3 remoteuser@remoteserver:/opt/remote

With those options it's likely that the throughput became 4x or 5x faster than no options (default)

1
  • but not for a million small files. did you try your solution yourself?
    – Sajuuk
    Apr 3, 2019 at 6:33
1

If you have ftp server in src side, you can use ncftpget from ncftp site. It works prefect with small files as it uses tar internally.

One comparison shows this: moving 1.9GB small files (33926 files)

  1. Using scp takes 11m59s
  2. Using rsync takes 7m10s
  3. Using ncftpget takes 1m20s
1

I guess my answer is a little late here, but I made good experiences with using mc (Midnight Commander) on one server to connect via SFTP to the other server.

The option to connect via FTP is in the "Left" and "Right" menus, by entering the address like this:

/#ftp:[email protected]/

or

/#ftp:[email protected]/

The you can navigate and do file operations almost like on a local filesystem.

It has a built-in option to do the copying in background, but I prefer using the screen command and detach from the screen while mc is copying (I think it runs faster then too).

1

To @scottpack answer of rSync option

To display progress of the upload use '--progess' as option after -avW in the command as shown below.

rsync -avW --progress -e ssh /path/to/dir/ remote_server:/path/to/remotedir

enter image description here

1

Here is a quick benchmark to compare some techniques,

  • Source is a 4-core Intel(R) Xeon(R) CPU E5-1620 @ 3.60GHz with 250 Mbps and SATA drive
  • Destination is a 6-core Intel(R) Xeon(R) CPU E-2136 @ 3.30GHz with 1 Gbps bandwidth and SSD drive

Number of files : 9632, Total size : 814 MiB, Avg size : 84 KiB

  • RSYNC : 1m40.570s
  • RSYNC + COMPRESSION : 0m26.519s
  • TAR + NETCAT : 1m58.763s
  • TAR + COMPRESSION + NETCAT : 0m28.009s

Command for tar/netcat was :

Source : tar -cf - /sourcedir/ | nc -v 11.22.33.44 5000
Dest : nc -v -l 5000 | tar -xf -
0

rsync or you might wish to tar it so its all within one file and then scp. If you lack the diskspace you can pipe the tar directly over ssh while its being made.

0

If you're sending over MP3's and other compressed files, you won't gain much from any solution that tries to further compress those files. The solution would be something that can create multiple connections between both servers and thus put more stress on the bandwidth between the two system. Once this maxes out, there's not much that can be gained without improving your hardware. (Faster network cards between those servers, for example.)

0
0

I tried couple of tools for copying a 1GB file The result is below: HTTP the fastest, with wget -c nc second in line scp slowest, and failed couple of times. No way to resume rsync uses ssh as a backend, thus the same result. In conclusion, I would go for http with wget -bqc and give it some time. Hope that this helps

1
  • do you provide insight as to why http is the fastest?
    – Sajuuk
    Apr 3, 2019 at 6:34
0

I had to copy the BackupPC disk into another machine.

I used rsync.

The machine had 256 MB of memory.

The procedure I followed was this one:

  • executed rsync without -H (took 9 hours)
  • when rsync finished, I synchronized the cpool directory and started with the pc directory; I cut the transfer.
  • then restarted rsync with -H flag, and all the files hard linked in pc directory were correctly transfered (the procedure found all the real files in cpool and then linked to the pc directory) (took 3 hours).

In the end I could verify with df -m that no extra space was spent.

By this way I elude the problem with the memory and rsync. All time I can verify the performance using top and atop and finally I transferred 165GB of data.

0

if the servers in question are in the same network, simply use \\server IP\d$ to copy files quickly. in this example, server IP refers to the destination server IP, and d is destination hard drive volume.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .