In our company we have a lot of algorithms which need to process large datasets. The time to run these algorithms differs from a few minutes to hours. They also need to be run ad-hoc from multiple times in a week to once a month. We would like to trigger these algorithms with an event such as a file upload in Azure blob storage or and API call.
To solve this I started looking into queued processing of tasks in Azure. At first I thought that Azure functions might be a good solution, because it’s pay as you use. The problem with this is that they are not meant for long running operations. Thus I started looking elsewhere. I found two pretty good alternatives namely Azure WebJobs or Jobs in Azure Kubernetes Services. The problem with both of them is that they still need an active server even though nothing is running on them. This could be quite expensive for tasks that only need to be run once a month.
My question is thus: does there exist a solution in Azure for hosting long running jobs without needing a dedicated server running 24-7?