Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm setting up my first production instance on EC2 and want to make sure I have all necessary monitoring in place. There are three different types of things I want to monitor:

  1. Is the instance running? EC2 instances can be terminated without warning if the underlying hardware fails, and as far as I know they aren't automatically restarted. So if not, start it back up.

  2. Is UNIX running properly? This is the usual stuff about CPU load, disk space, etc.

  3. Is the website responding? If not, restart it.

I initially set up Nagios on a physical server outside the cloud, but it is really only helpful for item 2. It can tell me if the instance is gone or if the website is not responding, but as far as I can tell it can't execute any commands to fix the situation.

My Googling on this subject has yielded a plethora of options - Cacti, Monit, God, Ganglia, and probably more I'm forgetting now. I don't have time to research them all. I am aware of Amazon's Cloudwatch but it doesn't seem to do anything that my Nagios installation doesn't already do.

If you already have something like this in place, can you please share what has worked well for you?

share|improve this question

4 Answers 4

up vote 2 down vote accepted

Monit should do most of what you need. If you want something a bit more advanced but more specifically tailored to EC2, have a look at the services offered by RightScale or Scalr (an open source competitor to RightScale).

share|improve this answer
    
I spent most of today setting up monit; it does indeed do most of what I want, except for the instance restarting thing, because there's no way to have it execute a custom script to check the status of each instance. It's always something... :) –  Janine Ohmer Apr 30 '10 at 5:53
    
I'm accepting this answer because it got me most of the way to what I need. As Jeremy said below, having instances automatically restart is tricky and can easily go wrong, but I can't afford to have one stay down until I can get to a computer, so I'll keep working on that part. –  Janine Ohmer May 3 '10 at 15:50

I've solved this problem with using Puppet... I built my own AMI that has puppet client installed on it and a user-data SysV-init script that sets the hostname before starting and enabling puppet process.

When the EC2 instance connects to the puppetmaster it passes it's current IP address I can then use this to template the EC2 nodes Nagios configuration. When the IP address changes the Nagios config file is updated and Puppet initiates a Nagios restart.

Restarting an EC2 instance would require having the EC2 API tools installed on the monitoring box and setting up something to automate the instance boot process. I've created a ruby script that does this and sets the user-data contents per the host being requested to be started.

share|improve this answer
    
How do you kick off your custom script? The problem I see is that Nagios will run a custom plugin, but that plugin isn't supposed to actually fix the problem, just report it, and there seems to be no way to have Nagios actually fix anything without manual intervention. Monit is exactly the opposite; no problem having it exec a custom script when there's a problem, but it will only check a fixed list of things, none of which allow for running a custom script to see if there is a problem. sigh –  Janine Ohmer Apr 30 '10 at 5:55
    
I don't have Nagios automatically trying to fix issues as the problem may not always be a simple fix, so it sends an SMS to my mobile along with email. The script on the Nagios host though is written to allow me to restart an EC2 instance quickly and easily. I'm not a big fan of auto-fix solutions as they can cause more problems then they solve. Never fun to have your monitoring system DoS your server. –  Jeremy Bouse May 1 '10 at 6:24

Disclaimer, I'm the Zenoss Community Manager.

Zenoss will do exactly what you're asking.

1) There is a Zenoss ZenPack specifically for monitoring EC2 instances: http://community.zenoss.org/docs/DOC-4423

2) You can also monitor the instances by turning on SSH and monitoring them at the OS-level. I've done Debian, Ubuntu, CentOS and OpenSolaris on EC2 this way.

3) You can monitor the website by process, checking the page or a synthetic clickthrough. In response to events in Zenoss, you can call an Event Command, like '/etc/init.d/apache restart' to restart a webserver that's gone down (or anything else you can script).

Check it out at http://zenoss.org

share|improve this answer
    
Looks interesting! I read the docs, but it doesn't say what it will do if an instance just disappears. Will it restart, or just report what happened? –  Janine Ohmer Apr 30 '10 at 5:52
    
I believe the instance remains there decommissioned for reporting purposes. –  mray May 31 '10 at 14:34

I'd just like to add this: in terms of monitoring website uptime, my company uses Pingdom, and I've honestly been thrilled by them.

http://www.pingdom.com/

They even have Android and iPhone apps (which work really well) that will notify you when your site goes down.

And for full disclosure, no, I don't work for them or anything. I am merely an enthusiastic customer :)

share|improve this answer
    
I don't share your enthusiasm. We're a customer as well, but their interface is really lacking in power. In fact, their main selling point for me is the large network of probes, but I'd switch easily enough. –  Martijn Heemels Nov 16 '11 at 23:39
    
Also, Pingdom does just monitoring. There is no way to remediate failures. –  Martijn Heemels Nov 16 '11 at 23:41

protected by Iain Dec 8 '11 at 15:06

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.