More or bigger EC2 instances to manage costs [closed]

Question

In an effort to save costs, we are trying to figure out the optimal setup to handle all the worker processes we are running between 3 EC2 instances.

Instead of three 4xlarge EC2's running thousands of queue workers, would it be more cost effective to split them into many smaller instances each handling 100 or so jobs or is it better to continue using larger instances at max capacity?

Typically we consume about the same amount of CPU and memory for each server we've created thus far. If we expect to use 50% of available CPU power, we can often expect to use 50% of the available RAM. For network IO, our biggest costs are communicating with the RDS, reading SQS queue payloads, and movement of data to/from S3.

Jobs could be batch processing of millions of small notifications, to parsing large XML/CSV files, performing complex financial calculations and data modeling, or running ML algorithms.

Answer 1

The best way to do it is create an autoscale group which ties into spot instance bidding and spins up a new instance of any size for a small amount of money that can do the work and you don't mind it being terminated at anytime since its a worker node.. The ASG/Lambda job could just spin up a new instance as needed

I do something similar with kubernetes and save a ton of money on my workload.

Answer 2

You can use an autoscaling group with small SPOT instances, I highly recommend you to use https://spotinst.com/ since it greatly helps you to minimize the cost of your EC2 instances by allowing you to pay the minimum always.