1

I'm using the aws cli and job submission takes a long time , if done one by one. Is there a way to submit more than one job at a time in a single request?

6
  • 1
    Even if there is not a native way to do this (I'm not sure at the moment), it is trivial to write a script to do it for you.
    – EEAA
    Commented Mar 3, 2017 at 13:33
  • I'm not sure.. the request API accepts only one json object per request, so it seems: docs.aws.amazon.com/batch/latest/APIReference/…
    – Protostome
    Commented Mar 3, 2017 at 15:37
  • No, what I'm saying is you write a shell script to run the command repeatedly on your behalf.
    – EEAA
    Commented Mar 3, 2017 at 15:54
  • Of course, that's what i'm doing. :) The problem is, I have 300k jobs and their submission can last several days... (as each submission request lasts ~0.5 second)
    – Protostome
    Commented Mar 3, 2017 at 16:02
  • 1
    Ahh, ok. That's a good start. I guess the next step would be to write a more complex script that can parallelize the job submissions across multiple processes.
    – EEAA
    Commented Mar 3, 2017 at 16:09

1 Answer 1

2

This is handled by "Array Jobs" which let you submit up to 10,000 jobs with a single request, See https://docs.aws.amazon.com/batch/latest/userguide/array_jobs.html

From the docs:

"AWS Batch array jobs are submitted just like regular jobs. However, you specify an array size (between 2 and 10,000) to define how many child jobs should run in the array. If you submit a job with an array size of 1000, a single job runs and spawns 1000 child jobs. The array job is a reference or pointer to manage all the child jobs. This allows you to submit large workloads with a single query."

1
  • Hallelujah! at last they added this feature
    – Protostome
    Commented Feb 6, 2018 at 10:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .