Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have 400ish time consuming web extraction tasks that I want to run, but rather than just running them one after the other I want to run i.e. 6 at a time to speed the overall task time up. What I want to achieve ideally is to create the 400 tasks, select that I want 6 to be running concurrently at all time and then the scheduler software will make sure that 6 are always running, when one finishes the next one in the list is started. Is this possible?

If the above ideal solution is not available, I was thinking about creating 6 separate task lists, each with a 1/6 of the total list in. Then the tasks can be started all at the same time, and each separate task list will start the next in the list when the previous finishes.

Any ideas?

share|improve this question
    
What platform? And you trying to write software, write a script, use an existing program, or are all options open? –  David Schwartz Apr 22 '12 at 22:29
    
I'd recommend he Use the Schwartz, but I would probably have my pun license revoked. –  adaptr Apr 22 '12 at 22:53
add comment

1 Answer

On Linux, this can be achieved with xargs:

cat cmds | xargs -n 1 -P 6 -0 bash -c

where cmds is a list of commands to execute using bash


On Windows I suggest you try to solve this with PowerShell. There are multiple questions on this topic over at stack overflow

share|improve this answer
    
Thanks for the response. Sorry I should have specified that the platform is Windows Server 2003. I don't mind how it is achieved. –  Stephen Apr 25 '12 at 21:46
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.