Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I recently discovered the xargs --max-procs feature.

How can split the output of the command by proc? Should I just create a mycommand --logfile $LOGFILE, or can I do it from xargs itself?

An example (for womble):

Suppose I have script, and a list of files. They can go in any order, but i want to keep the logging for each separate, then:

find $MY_FILE_TREE --print0 | xargs --null --max-procs 3 --max-args 1 --no-run-if-empty  

might be the parallel job I want to run. If is mouthy, then I'd like to be able to have each invocation print to a different log. Otherwise the stdout for each is the same, and the logs get jumbled.

share|improve this question
Can you give an example of what you want to do? It's not clear to me what you're trying to achieve. – womble Jan 20 '10 at 0:35
womble, I hope my added example is clear. Please request more information if not. – Gregg Lind Jan 20 '10 at 0:50

You could do this by running your xargs command through a shell - this will let you redirect the output - something like this:

find blah -type f | xargs -I{} -P 4 -n 1 sh -c 'yourcommand --input {} > {}.output''ll probably have to tweak it a bit - xargs replaces {} with the item/file it's working on

share|improve this answer
that replacement is a little hairy when the find output is rooted paths, but it's a good idea! – Gregg Lind Jan 20 '10 at 0:51

GNU Parallel seems to be made for you, because it automatically combines the standard output from the processes correctly.

find $MY_FILE_TREE --print0 | parallel --null --max-procs 3 --max-args 1 --no-run-if-empty ">" {}.output

or shorter:

find $MY_FILE_TREE --print0 | parallel -0 -j3 -r ">" {}.output

Watch the intro video:

share|improve this answer

You could change your script so that on startup it'll choose a random number/text, then prefix each line with this number? Then you can later split it using grep.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.