Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

If a server is experiencing high load, I use top and similar tools to troubleshoot why. However, this is only effective if I can analyze while the server is experiencing the problem.

What are some good tools for finding root cause of high server load in previous times? For example I was planning to put in a cron job to save 'top' output, apache server stats, mysql process list, etc every 5 minutes. But that doesn't seem very elegant, wondering if someone has come up with some utilities to accomplish this already.

share|improve this question
add comment

5 Answers

For ongoing monitoring you could consider installing munin. It will gather information every 5 minutes and generates graphs that will allow you to see where the bottlenecks are. I also use sar which can be run in background mode gathering data to disk. This gives quite detailed infomation on what the bottleneck is. To what processes where running in the past you will need the process accounting package.

share|improve this answer
    
+1 for the sar recommendation. Great tool indeed. –  EEAA Sep 6 '10 at 15:35
    
+1 for sar, although I don't like it too much. –  Cristian Ciupitu Sep 6 '10 at 15:36
    
I don't like it much either, except when I need it. Then I love it. Once it is set up, it disappears into daemon land. Then when something goes wrong you can dig around for the files. Usually they need to be copied and uncompressed. But the data available can be priceless. I have found Munin much friendlier for ongoing monitoring, but it doesn't go to quite the detail of sar. Munin has sar beat on the historical perspective though. –  BillThor Sep 7 '10 at 5:04
    
+1 for sar also. Likewise, you can use a tool like ksar to view the sar output. There's also nmon, if you admin a bunch of different *nix systems that aren't all Linux. –  Matt Sep 7 '10 at 13:18
add comment

Your non-elegant solution is actually a good one without setting up separate monitoring consoles (think SNMP traps). If you're running a RHEL/CentOS style syste make sure you've installed 'sysstat' (and turned it on) to gather ongoing stats about CPU, Memory, Disk I/O and the like. (see /etc/sysconfig/sysstat.* config files to tune).

Once you have that gathering underlying stats for you it can be used to pinpoint when the load trend occurs (so just besides seeing high CPU, is your proc queue backed up? do you see major faults in paging? how's your swap utilization?) which you can then correlate to your 'mysqladmin proc stat' type lists and so forth. If it's a LAMP stack, grab the total httpd processes and then do a quick sum/divide to find out the average process size to record as well. Enable your slow query log in MySQL to then trap those bad boys and look for some tables needing indexes.

Sometimes lo-tech isn't bad tech. :) Why use a chainsaw when a knife will do.

share|improve this answer
    
+1 There is nothing wrong in hacking something together to get this resolved. What you propose will hopefully reveal the issue but if not, it will be easy to modify your script to check other things. Once you have found the issues causing the load you can put in place something "more elegant" to watch for that in future. –  Richard Holloway Sep 6 '10 at 22:05
add comment

I like collectd but I've recently started toying with pcp (performance co-pilot). It has some nice features for historical diagnosis. [1]: http://oss.sgi.com/projects/pcp/

share|improve this answer
add comment

Might also wanna look at collectd, as a munin alternative.

share|improve this answer
add comment

atop.

It complements top/htop, because it can collect stats over time.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.