Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

So I'm getting a company set up in the Amazon Cloud -- creating IAAS protocol/solutions/standardized implementation, etc while also being the SysAdmin for individual systems, app environments, and day-to-day uptime.

One of the biggest issues I'm having is tracking various system/application logs, as well as logging/monitoring/archiving system metrics like memory usage, cpu usage, etc etc In a centralized fashion. E.g. --> Nagios + Urchin.

The BIGGEST impediment to my endeavors is the following:

The company application is deployed in the form of a Java *.WAR file, uploaded to an Elastic BeanStalk application environment, load balancing and auto-scaling between 3(min) and 10(max) servers, and the EC2's that run the application are fired up and disposed of ad-hoc.

That is to say, I can't monitor the individual EC2's for very long because so many are being terminated then auto-provisioned/auto-scaled on the fly -- so I'd constantly be having to "monitor what I'm monitoring", and continuously remove/add EC2 machine addresses to my monitoring lists.

IS there some sort of way to use monitoring tools like Zabbix or Nagios to monitor the ElasticBeanStalk, and have it automatically add on new EC2's, and remove terminated/failed EC2's from its monitoring list automatically?

Furthermore, is there anything I can do with GrayLog to achieve similar results with the aggregation/centralization of my application logs from multiple EC2 instances into ONE consolidated set of logs/events? If not GrayLog, is there ANYTHING LIKE GrayLog that can automatically detect what EC2 members are being added/removed from the environment, and collect the logs from them automatically?

Any and all advice or direction is appreciated.

Thanks much, and cheers!!

share|improve this question
Zenoss 4 will automatically follow your cloud instances as they get provisioned and terminated. Though I suspect that since this is something of a shopping question, it will get closed. – Michael Hampton Oct 11 '12 at 21:02
I'm not really trying to shop, and I think I might unclear as to what you mean. I was trying to ask a technical question to my fellow technical geniuses, superiors, admins, see if anybody's got any suggestions, or tools, or administrative strategies I might be able to employ. Will look into Zenoss right away...thanks! – A. Avadis Oct 11 '12 at 21:51
It seems to fall under the category of a "product, service or learning material recommendation." Please see the FAQ. :) – Michael Hampton Oct 11 '12 at 21:59

If you are deploying a WAR on elastic beanstalk you can install the metrics by creating a configuration file in the .ebextensions folder under WEB-INF. See the following link for more information on configuring and instance using this: -

To install disk / memory metrics you need to install the "Amazon CloudWatch Monitoring Scripts for Linux" - see

    mode: "000777"
    owner: ec2-user
    group: ec2-user
    command: unzip -d /opt/aws/cwms /opt/aws/cwms/
    ignoreErrors: true
    command: sed -i 's/Key=$/Key=<VALUE OF YOUR SECRET KEY>/;s/KeyId=$/KeyId=<VALUE OF YOUR ACCESS ID>/' /opt/aws/cwms/awscreds.conf
    command: echo "*/1 * * * * /opt/aws/cwms/ --mem-util --disk-path=/ --disk-space-util --from-cron" | crontab - -u ec2-user

Basically what this script does is download the Linux based into a folder such as /opt/aws/cwms (this can be anywhere). The commands then unzip the file, update the access / secret key (using the "sed" command) and finally creating the crontab tab.

Be careful of the crontab tab section, as it could potentially wipe you existing crontab entries.


Here's an updated script which is working for me quite nicely as of Feb 2016 (see


    command: yum install -y perl-Switch perl-Sys-Syslog perl-LWP-Protocol-https

    command: |
      echo '* * * * * root perl /opt/cloudwatch/aws-scripts-mon/ `{"Fn::GetOptionSetting" : { "OptionName" : "CloudWatchMetrics", "DefaultValue" : "--mem-used --memory-units=megabytes --mem-util --disk-space-util --disk-space-used --disk-space-avail --disk-path=/" }}` >> /var/log/cwpump.log 2>&1' > /etc/cron.d/cwpump
    command: chmod 644 /etc/cron.d/cwpump
    command: chmod u+x /opt/cloudwatch/aws-scripts-mon/

  "aws:autoscaling:launchconfiguration" :
    IamInstanceProfile : "MonitorRole"
  "aws:elasticbeanstalk:customoption" :
    CloudWatchMetrics : "--mem-used --memory-units=megabytes --mem-util --disk-space-util --disk-space-used --disk-space-avail --disk-path=/"

NOTE: You must have an IAM role called MonitorRule in place. It's role policy should be as follows (also see

  "Version": "2012-10-17",
  "Statement": [
      "Action": [
      "Effect": "Allow",
      "Resource": [
share|improve this answer
This answer doesn't work with the latest AMI, I needed to install a modules from perl following the instruction in:… The bad news for me, is that I tried to automatize this in app.config file, but is just not work: 00-installperldependencies: command: sudo yum install -y perl-Switch perl-Sys-Syslog perl-core perl-DateTime perl-LWP-Protocol-https – Jose Nobile Mar 26 '15 at 22:00

We are using EC2, S3, etc but not yet ElasticBeanStalk. i can give you some suggestions and ideas...

Cloudwatch: We use CloudWatch from Amazon AWS which gives pretty good details about our EC2 instances. Monitoring setup is very simple and GUI stuff.. No scripting or anything needed. The Cloudwatch getting started guide will give lot of info about CLI, but developerGuide gives the exact info we will need:

I found this link on Amazon ElasticBeanStack forum: -

share|improve this answer

For Elastic beanstack's disk usage, you can enable the RootFileSysmtemUtil cloud watch instance metrics. This is available under Elastic beanstalk's Health configuration section. There is a "Health Reporting" section under "Health" configuration. Please select the RootFileSysmtemUtil option and save the configuration.

Now if you go to CloudWatch's ElasticBeanstack metrics, you can see the new metrics.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.