Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I recently discovered the xargs --max-procs feature.

How can split the output of the command by proc? Should I just create a mycommand --logfile $LOGFILE, or can I do it from xargs itself?

An example (for womble):

Suppose I have script myprocessor.sh, and a list of files. They can go in any order, but i want to keep the logging for each separate, then:

find $MY_FILE_TREE --print0 | xargs --null --max-procs 3 --max-args 1 --no-run-if-empty myprocess.sh  

might be the parallel job I want to run. If myprocessor.sh is mouthy, then I'd like to be able to have each invocation print to a different log. Otherwise the stdout for each is the same, and the logs get jumbled.

share|improve this question
    
Can you give an example of what you want to do? It's not clear to me what you're trying to achieve. –  womble Jan 20 '10 at 0:35
    
womble, I hope my added example is clear. Please request more information if not. –  Gregg Lind Jan 20 '10 at 0:50
add comment

3 Answers

You could do this by running your xargs command through a shell - this will let you redirect the output - something like this:

find blah -type f | xargs -I{} -P 4 -n 1 sh -c 'yourcommand --input {} > {}.output'

...you'll probably have to tweak it a bit - xargs replaces {} with the item/file it's working on

share|improve this answer
    
that replacement is a little hairy when the find output is rooted paths, but it's a good idea! –  Gregg Lind Jan 20 '10 at 0:51
add comment

GNU Parallel http://www.gnu.org/software/parallel/ seems to be made for you, because it automatically combines the standard output from the processes correctly.

find $MY_FILE_TREE --print0 | parallel --null --max-procs 3 --max-args 1 --no-run-if-empty myprocess.sh ">" {}.output

or shorter:

find $MY_FILE_TREE --print0 | parallel -0 -j3 -r myprocess.sh ">" {}.output

Watch the intro video: http://www.youtube.com/watch?v=OpaiGYxkSuQ

share|improve this answer
add comment

You could change your script so that on startup it'll choose a random number/text, then prefix each line with this number? Then you can later split it using grep.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.