I have a system that needs to deploy hundreds of thousands of short-lived jobs per day. Each job runs anywhere from a few seconds, to a couple of hours. Each job makes HTTP requests to external web servers, writes data to disk (anywhere from a few megabytes to hundreds of gigabytes), and makes a series of connections to databases.

Every job is the same Docker container, running the same single Java process. Each job has a different configuration, passed as a environment variable.

We currently deploy these jobs on a Kubernetes cluster using the "Job" spec. However, the cluster is not immediately available for jobs when a large influx of jobs are to be ran. We also have to constantly query the Kubernetes cluster to determine if the Job has been finished, or was killed (e.g. out of memory).

I'd like to find a solution that would allow us to deploy these jobs as quickly as possible with the least amount of concern about whether resources are available, or requiring us to query a system to determine if the job has completed.

AWS Lambda comes to mind, but I have little experience with it.

As a architectural note, we have our process that serves a scheduler, in that calculates what job should be ran, and when. that process currently submits the job to the Kubernetes cluster.

Given the above description, what architectures should I be evaluating to minimize the amount of concern this system has around 1) if resources are available to handle the job and 2) Whether that job fail for any "non application" reason.

This system currently runs on GCP and AWS. We're open to any solution, even if it means selecting a single (and potentially different) platform.


Presumably your cluster is resource limited. Achieving a higher job volume, if that is a requirement, must involve a more efficient application or more resources.

Large providers like you use will rent you as many instances as your budget allows. Scale up your cluster, possibly automatically. Possibly some spare capacity is needed if you schedule jobs on short notice.

An alternative to polling the Kubernetes job is your code passing a message. At the end of a job, do some kind of call back to your scheduler indicating finished.

Of course, it may have died and will never report back. Eventually this needs to become a failure state. Consider polling the job at intervals after your typical shortest job time, and giving up on it after a hard limit like activeDeadlineSeconds.

  • Thanks. All jobs actually do make several requests to an API to update their status and progress. But when the job completes, we still have "delete" the job, as there is a limit to the number of completed jobs Kubernetes can store. There's a new API in the next release of Kubernetes that allows for Jobs to be automatically cleaned up. However, we still have to check on the Job in case it was terminated for thinks like OOM. We do scale the cluster, but scaling it can take several minutes, during which jobs cannot be reliably submitted to the cluster. – Brett Jul 12 at 11:23
  • Then you have to stay ahead of jobs by maintaining idle capacity 5 minutes before it is needed. At least enough for high priority jobs that need to start now. No way to avoid that if you always want jobs to have resources. – John Mahowald Jul 12 at 13:11
  • Of course, we do that. So the question in the post is what options are available to avoid doing this. AWS Lambda seems like an option, but the amount of data we write to disk might be a limit factor. We're looking to explore other options. – Brett Jul 12 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.