Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am trying to execute a long running process multiple times in parallel. The parameter for each execution of the process is stored in a space separated environment variable. Here is a contrived example of what I am trying to execute:

$ echo -e 1 2 3 4 | xargs --max-procs=3 --max-args=1 --replace=% echo % is the number being processed

Here is the output from that command:

1 2 3 4 is the number being processed

Why is max-args seemingly ignored? I then tried to explicitly set the delimiter which gives better results:

$ echo -e 1 2 3 4 | xargs -d " " --max-procs=3 --max-args=1 --replace=% echo % is the number being processed
1 is the number being processed
2 is the number being processed
3 is the number being processed
4
 is the number being processed

What is xargs doing when it processes the 4th argument?

After some searching, I did manage to almost get what I want. The arguments are processed correctly, but parallelism does not work (verified with another command not shown here):

$ echo -e 1 2 3 4 | xargs -n 1 | xargs --max-procs=3 --max-args=1 --replace=% echo % is the number being processed
1 is the number being processed
2 is the number being processed
3 is the number being processed
4 is the number being processed

What am I missing?

share|improve this question
add comment

1 Answer

up vote 0 down vote accepted

Does

echo -e 1 2 3 4 | sed -e 's/\s\+/\n/g' | xargs --max-procs=3 --max-args=1 --replace=% echo % is the number being processed

accomplish the task? The output seems about right:

1 is the number being processed
2 is the number being processed
3 is the number being processed
4 is the number being processed

I also tried replacing echo with sleep to confirm it executes in parallel, and it does:

echo -e 1 2 3 4 5 6 7 8 9 9 9 9 9 9 9 9 9 9 9 9 | sed -e 's/\s\+/\n/g' | xargs --max-procs=20 --max-args=1 --replace=% sleep 1
share|improve this answer
    
Thanks, that definitely works. For curiosity's sake, do you know why it is necessary to replace white space with a newline? I'm a xargs newbie. –  gordo911 Jun 18 '13 at 14:22
    
Not quite sure about that, could be a bug/feature of xargs. The man page for xargs barely hints on this behaviour, suggesting usage of null character as a separator to solve this problem (-0 option). –  dusty Jun 18 '13 at 15:06
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.