4

I recently discovered the xargs --max-procs feature.

How can split the output of the command by proc? Should I just create a mycommand --logfile $LOGFILE, or can I do it from xargs itself?

An example (for womble):

Suppose I have script myprocessor.sh, and a list of files. They can go in any order, but i want to keep the logging for each separate, then:

find $MY_FILE_TREE --print0 | xargs --null --max-procs 3 --max-args 1 --no-run-if-empty myprocess.sh  

might be the parallel job I want to run. If myprocessor.sh is mouthy, then I'd like to be able to have each invocation print to a different log. Otherwise the stdout for each is the same, and the logs get jumbled.

2
  • Can you give an example of what you want to do? It's not clear to me what you're trying to achieve.
    – womble
    Jan 20, 2010 at 0:35
  • womble, I hope my added example is clear. Please request more information if not.
    – Gregg Lind
    Jan 20, 2010 at 0:50

3 Answers 3

1

You could do this by running your xargs command through a shell - this will let you redirect the output - something like this:

find blah -type f | xargs -I{} -P 4 -n 1 sh -c 'yourcommand --input {} > {}.output'

...you'll probably have to tweak it a bit - xargs replaces {} with the item/file it's working on

1
  • that replacement is a little hairy when the find output is rooted paths, but it's a good idea!
    – Gregg Lind
    Jan 20, 2010 at 0:51
1

GNU Parallel http://www.gnu.org/software/parallel/ seems to be made for you, because it automatically combines the standard output from the processes correctly.

find $MY_FILE_TREE --print0 | parallel --null --max-procs 3 --max-args 1 --no-run-if-empty myprocess.sh ">" {}.output

or shorter:

find $MY_FILE_TREE --print0 | parallel -0 -j3 -r myprocess.sh ">" {}.output

Watch the intro video: http://www.youtube.com/watch?v=OpaiGYxkSuQ

0

You could change your script so that on startup it'll choose a random number/text, then prefix each line with this number? Then you can later split it using grep.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .