In an effort to save costs, we are trying to figure out the optimal setup to handle all the worker processes we are running between 3 EC2 instances.
Instead of three 4xlarge EC2's running thousands of queue workers, would it be more cost effective to split them into many smaller instances each handling 100 or so jobs or is it better to continue using larger instances at max capacity?
Typically we consume about the same amount of CPU and memory for each server we've created thus far. If we expect to use 50% of available CPU power, we can often expect to use 50% of the available RAM. For network IO, our biggest costs are communicating with the RDS, reading SQS queue payloads, and movement of data to/from S3.
Jobs could be batch processing of millions of small notifications, to parsing large XML/CSV files, performing complex financial calculations and data modeling, or running ML algorithms.