Making sense of Ubuntu Server Status Info

Question

I'm trying to create a simple shell script to monitor my server. I plan to set up a CRON job to run it every five or 10 minutes.

Here's how it will work:

1. Run a number of linux commands e.g. iostat, mpstat, top, etc. and output the results to a text file
2. Send the text file via CURL to a URL that will receive the text file and process the data, and then post important metrics to a database

I plan to use this data to determine when I need to have my server upgraded.

However, I don't have much experience with server monitoring so I don't know what kind of thresholds I should be looking out for. For example, when I run something like mpstat -P ALL, what kind of figures should bother me? Or iostat?

I just want to be able to have some kind of point of reference to know when my servers are in a good state i.e. reasonable load, or in a bad state i.e. overloaded, and require upgrade or load-balancing.

Thanks in advance.

Answer 1

I would say that the free Monit would be a more appropriate tool for testing the thresholds you're looking for and giving a simple at-a-glance view of your system's health.

Out of the box, you can setup some basic checks. The syntax is very human-readable, so a barebones setup that checks system load, memory usage, swap utilization, CPU usage and disk space for various mountpoints, and can send an email would look like this:

if loadavg (1min) > 6 then alert
if loadavg (5min) > 5 then alert
if memory usage > 90% then alert
if swap usage > 20% then alert
if cpu usage (user) > 90% then alert
if cpu usage (system) > 75% then alert
if cpu usage (wait) > 75% then alert

check device root with path /
    if SPACE usage > 80% then alert

check device var with path /var
    if SPACE usage > 80% then alert

check device usr with path /usr
    if SPACE usage > 80% then alert

check device tmp with path /tmp
    if SPACE usage > 80% then alert

In addition, I know you're saying that you don't require any graphical tools, but it may make sense to have something that can track trends. Munin is a good tool for this. There are plenty of others, but it's worth considering.

Answer 2

Obi Hill: Well, you ARE reinventing the wheel. Gathering all that data, parsing it and analysing it is a solved problem you should not rewrite Yet Another Time.

SNMP is one very handy way of gathering system information for further processing (for example, graphing trends with MRTG or passing the data to Nagios or similar monitoring program).

Also programs like Cacti or Munin can do all this for you.

Answer 3

How many servers do you have?

Perhaps you should take a look at Puppet, RunDesk or ControlTier.