Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I use a whole bunch of tools to monitor server load (Linux/Apache/MySQL/PHP) of different types - top, iostat and netstat, as well as my own in-script measurements. I collect and summarize the data using scripts and spreadsheets, to get an overall picture of what's keeping my server busy.

Is there a simple tool/daemon I can install under Linux which will sit quietly in the background, collect all this information at appropriate intervals, and then provide simple daily/weekly/monthly summaries for me to read?

share|improve this question

7 Answers 7

up vote 2 down vote accepted

Someone mentioned sar (system activity reporter) and I have to second that recommendation!

It sounds like you've done a boatload of work on your own with various tools. However, you might have found that most of the aggregate system data you're interested in has been collected by sadc (system activity data collector) cron jobs for as long as you've had the sysstat package installed on your systems. ' Oddly enough these don't run as daemons.

The sadc scripts (named sa1 and sa2) collect information from various kernel counters ... some of them are the same that are queried by vmstat and others are pulled from the same sources that ifconfig uses to report on the amount of data transmitted and received on each interface, and so on. By default sa1 runs every 10 minutes.

The sar command can then be used to pull various subsets of this information from the collected data. To become familiar with all available tables run sar -A and redirect the output to some convenient file. Then open the file in your favorite editor, in one window, and bring up the sar man page in another. Now simply walk through the entire file, searching the man page for the various column heading identifiers.

I also like to use sar -A as an early step for troubleshooting unexplained performance and stability. (Often by the time such issues have been escalated to me the system has already been rebooted or the problem is intermittent). This data is often the best available since the user (including most of the junior and intermediate sysadmins I've worked with) don't know what sort of data to collect before rebooting.

The trick to using this data for post-mortem troubleshooting is to scan all those columns of numbers looking for the spikes. (Yes, if you're good with tools like GNUplot you could generate a bunch of graphs from them; but honestly you can often spot the spikes just as well just scanning along the raw numbers). Look at a few similar systems which are not exhibiting the issues, or look at the data on this system during intervals when the things seem to be fine. Once you spot the "spikes" (network traffic, paging activity, disk I/O) try to correlate them (I usually just make a copy of the data and delete all the "nominal" data to leave me with just the stuff that's probably probative). The first correlation to look for is in the timing of different resource spikes ... did the run queue suddenly spike just before paging climbed out of control? ... was there a huge increase in received network traffic just before the disk I/O numbers went through the roof? Then you correlate that to the output from the last command (who was logged in around then) ... and if you've enabled process accounting then you can look at the lastcomm command. And, naturally, you can also look at /var/log/* to focus on the same time interval (and scanning backwards from there, of course).

You'd be amazed at how much you can figure out from this exercise.

"Tom account logged in, a few minutes later that were gradually increasing data flows over eth0 and to the sdb3 disk; perhaps he was copying in a data set, probably from the Netapps over on Lan B ... then we see a huge spike in memory page allocations, but process creation remained nominal until this paging started; that's when the interrupts and run queue went through the room, and brought the system into OOM thrashing. Tom, you ignoramus! You have a memory leak in foobar.pl!" :)

However, for your purposes you can use sar -A just as a starting point. You find the bits that you need and then you call sar with various arguments to capture each table you want to parse, filter, and graph separately. There are options to render the output in a more machine friendly format, as well.

share|improve this answer

There are several tools that let you automatically collect performance metrics for your servers, like for example Nagios, Cacti and Munin.

Most of the monitoring solutions provide a web-based frontend from which you can easily visualize performance data corresponding to a specific time period. Besides, as they usually store the collected information on a database, you can also take advantage of that in case you need to perform further analysis.

share|improve this answer

sar is the system activity reporting command. It can provide realtime data (similar to vmstat), and also saves data in the background for later reporting.

Debian has a few different versions in their repository, look at the atsar and sysstat packages.

You may also find process accounting interesting. When any process finishes, it's usage information is written to a file from which reports can be made. Look at the acct package.

share|improve this answer
1  
Sar gooooooood. –  womble Aug 15 '09 at 8:01

You could use :

1.SeaLion.Its a light weight tool. Since you prefered SAR for usage, this would be further attractive as to me it seemed like an extension of SAR; with all outputs on a timeline(helps debug w.r.t. time) also supporting all possible shell commands that one can think of. Top, iostat, netstat, uptime,etc are default commands provided and it also allows you to add your own commands and configure the interval at which to collect the outputs. However it does not provide alerts yet.

2.New Relic : It tops the list of server monitoring tools and provides reports in the form of graphs and also provides alerts. It is a bit heavy and tough to set up.

These are the only two I have used and can vouch for.

share|improve this answer

I have recently discovered Glances and from ssh is pretty good, more than top and htop. This for a single machine.

For monitoring all your servers (virtual or physical) yes NewRelic is pretty good, and recently Monitis as a backup choice. Also Pingdom paid account is a complete set of utility.

share|improve this answer

I use Zabbix. You can define templates for type of hardware or kinds of usage. You can send reports, alerts, by differents medias like mail, jabber or SMS (if you have a GSM or CDMA modem). Datas are collected by SNMP or by its own zabbix-agent.

share|improve this answer
    
zabbix is nice; you can also insert your measurements directly in its database for later graphing. –  slovon Aug 15 '09 at 9:14

you try following link it has lot of thing for monitoring..

http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.