Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a list of files I need to copy on a Linux system - each file ranges from 10 to 100GB in size.

I only want to copy to the local filesystem. Is there a way to do this in parallel - with multiple processes each responsible for copying a file - in a simple manner?

I can easily write a multithreaded program to do this, but I'm interested in finding out if there's a low level Linux method for doing this.

share|improve this question
1  
Parallel file copy won't bring significant speed-up. (At least in theory it shouldn't.) –  Tarnay Kálmán Jun 18 '10 at 1:17
add comment

5 Answers 5

Here's a distributed/parallel and decentralized file copy tool that will chunk up the file and copy all of the chunks in parallel. It'll probably only help you if you have an SSD that supports multiple streams or some sort of setup with multiple disk heads.

https://github.com/hpc/dcp

share|improve this answer
add comment

If you system is not trashed by it (e.g. maybe the files are in cache) then GNU Parallel http://www.gnu.org/software/parallel/ may work for you:

find . -print0 | parallel -0 cp {} destdir

This will run 9 concurrent cp's.

Pro: It is simple to read.

Con: GNU Parallel is not standard on most systems - so you probably have to install it.

Watch the intro video for more info: http://www.youtube.com/watch?v=OpaiGYxkSuQ

share|improve this answer
add comment

The only answer that will not trash your machine's responsiveneess isn't exactly a 'copy', but it is very fast. If you won't be editing the files in the new or old location, then a hard link is effectively like a copy, and (only) if you're on the same filesystem, they are created very very very fast.

Check out cp -l and see if it will work for you.

share|improve this answer
add comment

As mentioned, this is a terrible idea. But I believe everyone should be able to implement their own horrible plans, sooo...

for FILE in *;do cp $FILE <destination> &;done

The asterisk can be replaced with a regular expression of your files, or $(cat <listfile>) if you've got them all in a text document. The ampersand kicks off a command in the background, so the loop will continue, spawning off more copies.

As mentioned, this will completely annihilate your IO. So...I really wouldn't recommend doing it.

--Christopher Karel

share|improve this answer
add comment

There is no low-level mechanism for this for a very simple reason: doing this will destroy your system performance. With platter drives each write will contend for placement of the head, leading to massive I/O wait. With SSDs, this will end up saturating one or more of your system buses, causing other problems.

share|improve this answer
    
Err that doesn't seem to be the case with a single cp at present, I'm sure there's a happy medium for multiple parallel "cp's" at which you're I/O channel doesn't become completely saturated... –  Jon Jun 29 '10 at 3:18
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.