I have 400ish time consuming web extraction tasks that I want to run, but rather than just running them one after the other I want to run i.e. 6 at a time to speed the overall task time up. What I want to achieve ideally is to create the 400 tasks, select that I want 6 to be running concurrently at all time and then the scheduler software will make sure that 6 are always running, when one finishes the next one in the list is started. Is this possible?

If the above ideal solution is not available, I was thinking about creating 6 separate task lists, each with a 1/6 of the total list in. Then the tasks can be started all at the same time, and each separate task list will start the next in the list when the previous finishes.

Any ideas?

  • What platform? And you trying to write software, write a script, use an existing program, or are all options open? – David Schwartz Apr 22 '12 at 22:29
  • I'd recommend he Use the Schwartz, but I would probably have my pun license revoked. – adaptr Apr 22 '12 at 22:53

On Linux, this can be achieved with xargs:

cat cmds | xargs -n 1 -P 6 -0 bash -c

where cmds is a list of commands to execute using bash

On Windows I suggest you try to solve this with PowerShell. There are multiple questions on this topic over at stack overflow

  • Thanks for the response. Sorry I should have specified that the platform is Windows Server 2003. I don't mind how it is achieved. – Stephen Apr 25 '12 at 21:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.