2

I have 400ish time consuming web extraction tasks that I want to run, but rather than just running them one after the other I want to run i.e. 6 at a time to speed the overall task time up. What I want to achieve ideally is to create the 400 tasks, select that I want 6 to be running concurrently at all time and then the scheduler software will make sure that 6 are always running, when one finishes the next one in the list is started. Is this possible?

If the above ideal solution is not available, I was thinking about creating 6 separate task lists, each with a 1/6 of the total list in. Then the tasks can be started all at the same time, and each separate task list will start the next in the list when the previous finishes.

Any ideas?

Improve this question
2
  • What platform? And you trying to write software, write a script, use an existing program, or are all options open?
    – David Schwartz
    Apr 22, 2012 at 22:29
  • I'd recommend he Use the Schwartz, but I would probably have my pun license revoked.
    – adaptr
    Apr 22, 2012 at 22:53

1 Answer 1

Reset to default
4

On Linux, this can be achieved with xargs:

cat cmds | xargs -n 1 -P 6 -0 bash -c

where cmds is a list of commands to execute using bash

On Windows I suggest you try to solve this with PowerShell. There are multiple questions on this topic over at stack overflow

Improve this answer
1
  • Thanks for the response. Sorry I should have specified that the platform is Windows Server 2003. I don't mind how it is achieved.
    – Stephen
    Apr 25, 2012 at 21:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .