Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

It seems that a lot of people are using AWS nowadays, and I'm really hoping someone can give me a pointer on how, and if, I could design a scalable system within EC2.

I tried searching, but all I can come up with is CloudBuzz, CloudCookie, CloudIceCream and no good documentation about the actual infrastructure.

What I want, is to create a cluster of servers that can do some calculation stuff, and scale up/down. However, the management and their working schedules are dependent on non obvious factors that can't be derived from CPU/RAM/HDD load directly.

So I am wondering if it's possible to call CloudSuperScaleMe function to simply launch and terminate an instance from within, or from outside the cloud programmatically.

From all the buzzwords it seems that closest I can find is CloudWatch and CloudElasticSomething, but it seems they are based around autoscale groups that check CPU/Network/etc. load and, since none of those factors can be trusted in my architecture, I don't see how I could write an alarm and scaleup/scaledown functions without seriously altering and customizing AMIs.

share|improve this question

1 Answer 1

up vote 1 down vote accepted

Cloudwatch is the 'monitoring' service of AWS - it will provide the 'CPU/Network/etc. load' - it is required by autoscaling. You can however define your own custom metrics which can also be used by autoscale.

There are some limitations to the type of data you can log with CloudWatch, as per this article.

It is my understanding (although, I have no personal experience with) that Nagios/Cacti setups may be configured to do the monitoring and then execute your scaling script - unless someone else can provide a good reason for that approach, I cannot.

Another, perhaps simpler approach (a very general version) would be to: Setup a messaging layer (corosync/heartbeat) between the servers It will 'elect' a a master node, and also keep track of the running servers On the 'master node' you track your factors for scaling, and trigger your scaling script Your scaling script would be passed add/remove + number need to be able to:

  1. get a list of instances (ideally use data from your messaging layer, but if not, just use DescribeInstances)
  2. if adding, call RunInstances
  3. if removing, choose which instance(s), and call TerminateInstances on them
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.