Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

So I'm getting a company set up in the Amazon Cloud -- creating IAAS protocol/solutions/standardized implementation, etc while also being the SysAdmin for individual systems, app environments, and day-to-day uptime.

One of the biggest issues I'm having is tracking various system/application logs, as well as logging/monitoring/archiving system metrics like memory usage, cpu usage, etc etc In a centralized fashion. E.g. --> Nagios + Urchin.

The BIGGEST impediment to my endeavors is the following:

The company application is deployed in the form of a Java *.WAR file, uploaded to an Elastic BeanStalk application environment, load balancing and auto-scaling between 3(min) and 10(max) servers, and the EC2's that run the application are fired up and disposed of ad-hoc.

That is to say, I can't monitor the individual EC2's for very long because so many are being terminated then auto-provisioned/auto-scaled on the fly -- so I'd constantly be having to "monitor what I'm monitoring", and continuously remove/add EC2 machine addresses to my monitoring lists.

IS there some sort of way to use monitoring tools like Zabbix or Nagios to monitor the ElasticBeanStalk, and have it automatically add on new EC2's, and remove terminated/failed EC2's from its monitoring list automatically?

Furthermore, is there anything I can do with GrayLog to achieve similar results with the aggregation/centralization of my application logs from multiple EC2 instances into ONE consolidated set of logs/events? If not GrayLog, is there ANYTHING LIKE GrayLog that can automatically detect what EC2 members are being added/removed from the environment, and collect the logs from them automatically?

Any and all advice or direction is appreciated.

Thanks much, and cheers!!

share|improve this question
    
Zenoss 4 will automatically follow your cloud instances as they get provisioned and terminated. Though I suspect that since this is something of a shopping question, it will get closed. –  Michael Hampton Oct 11 '12 at 21:02
1  
I'm not really trying to shop, and I think I might unclear as to what you mean. I was trying to ask a technical question to my fellow technical geniuses, superiors, admins, etc...to see if anybody's got any suggestions, or tools, or administrative strategies I might be able to employ. Will look into Zenoss right away...thanks! –  A. Avadis Oct 11 '12 at 21:51
    
It seems to fall under the category of a "product, service or learning material recommendation." Please see the FAQ. :) –  Michael Hampton Oct 11 '12 at 21:59

2 Answers 2

If you are deploying a WAR on elastic beanstalk you can install the metrics by creating a configuration file in the .ebextensions folder under WEB-INF. See the following link for more information on configuring and instance using this: - http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers.html

To install disk / memory metrics you need to install the "Amazon CloudWatch Monitoring Scripts for Linux" - see http://aws.amazon.com/code/8720044071969977

files:
  "/opt/aws/cwms/CloudWatchMonitoringScripts.zip":
    mode: "000777"
    owner: ec2-user
    group: ec2-user
    source:  http://ec2-downloads.s3.amazonaws.com/cloudwatch-samples/CloudWatchMonitoringScripts-v1.1.0.zip
container_commands:
  01_unzip_cloud_watch_zip: 
    command: unzip -d /opt/aws/cwms /opt/aws/cwms/CloudWatchMonitoringScripts.zip
    ignoreErrors: true
  02_update_password_file:
    command: sed -i 's/Key=$/Key=<VALUE OF YOUR SECRET KEY>/;s/KeyId=$/KeyId=<VALUE OF YOUR ACCESS ID>/' /opt/aws/cwms/awscreds.conf
  03_update_crontab:    
    command: echo "*/1 * * * * /opt/aws/cwms/mon-put-instance-data.pl --mem-util --disk-path=/ --disk-space-util --from-cron" | crontab - -u ec2-user

Basically what this script does is download the Linux based CloudWatchMonitoringScripts.zip into a folder such as /opt/aws/cwms (this can be anywhere). The commands then unzip the file, update the access / secret key (using the "sed" command) and finally creating the crontab tab.

Be careful of the crontab tab section, as it could potentially wipe you existing crontab entries.

share|improve this answer

We are using EC2, S3, etc but not yet ElasticBeanStalk. i can give you some suggestions and ideas...

Cloudwatch: We use CloudWatch from Amazon AWS which gives pretty good details about our EC2 instances. Monitoring setup is very simple and GUI stuff.. No scripting or anything needed. The Cloudwatch getting started guide will give lot of info about CLI, but developerGuide gives the exact info we will need: http://docs.amazonwebservices.com/AmazonCloudWatch/latest/DeveloperGuide/AlarmThatSendsEmail.html http://docs.amazonwebservices.com/AmazonCloudWatch/latest/DeveloperGuide/choosing_your_cloudwatch_interface.html

I found this link on Amazon ElasticBeanStack forum: - http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/using-features.healthstatus.html

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.