7

I have a list of files I need to copy on a Linux system - each file ranges from 10 to 100GB in size.

I only want to copy to the local filesystem. Is there a way to do this in parallel - with multiple processes each responsible for copying a file - in a simple manner?

I can easily write a multithreaded program to do this, but I'm interested in finding out if there's a low level Linux method for doing this.

8

If you system is not thrashed by it (e.g. maybe the files are in cache) then GNU Parallel http://www.gnu.org/software/parallel/ may work for you:

find . -print0 | parallel -0 -j10 cp {} destdir

This will run 10 concurrent cps.

Pro: It is simple to read.

Con: GNU Parallel is not standard on most systems - so you probably have to install it.

Watch the intro video for more info: http://www.youtube.com/watch?v=OpaiGYxkSuQ

See also https://oletange.wordpress.com/2015/07/04/parallel-disk-io-is-it-faster/ for a discussion of parallel disk I/O.

4

There is no low-level mechanism for this for a very simple reason: doing this will destroy your system performance. With platter drives each write will contend for placement of the head, leading to massive I/O wait. With SSDs, this will end up saturating one or more of your system buses, causing other problems.

  • Err that doesn't seem to be the case with a single cp at present, I'm sure there's a happy medium for multiple parallel "cp's" at which you're I/O channel doesn't become completely saturated... – Jon Jun 29 '10 at 3:18
  • A saturated bus is a happy bus. Idle bandwidth is wasted bandwidth. – rvalue Nov 17 '16 at 2:21
3

As mentioned, this is a terrible idea. But I believe everyone should be able to implement their own horrible plans, sooo...

for FILE in *;do cp $FILE <destination> &;done

The asterisk can be replaced with a regular expression of your files, or $(cat <listfile>) if you've got them all in a text document. The ampersand kicks off a command in the background, so the loop will continue, spawning off more copies.

As mentioned, this will completely annihilate your IO. So...I really wouldn't recommend doing it.

--Christopher Karel

3

The only answer that will not trash your machine's responsiveneess isn't exactly a 'copy', but it is very fast. If you won't be editing the files in the new or old location, then a hard link is effectively like a copy, and (only) if you're on the same filesystem, they are created very very very fast.

Check out cp -l and see if it will work for you.

1

Here's a distributed/parallel and decentralized file copy tool that will chunk up the file and copy all of the chunks in parallel. It'll probably only help you if you have an SSD that supports multiple streams or some sort of setup with multiple disk heads.

https://github.com/hpc/dcp

-1

You should try this:

    $ seq 3 | parallel cp -v /etc/passwd passwd{}

This will copy the file passwd 3 times from /etc/ directory to your $HOME

Or if your file it is in your home directory

    $ seq 3 | parallel cp -v passwd{,{}}

This will copy the file passwd 3 times into your $HOME

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.