4

If a server is experiencing high load, I use top and similar tools to troubleshoot why. However, this is only effective if I can analyze while the server is experiencing the problem.

What are some good tools for finding root cause of high server load in previous times? For example I was planning to put in a cron job to save 'top' output, apache server stats, mysql process list, etc every 5 minutes. But that doesn't seem very elegant, wondering if someone has come up with some utilities to accomplish this already.

5 Answers 5

5

For ongoing monitoring you could consider installing munin. It will gather information every 5 minutes and generates graphs that will allow you to see where the bottlenecks are. I also use sar which can be run in background mode gathering data to disk. This gives quite detailed infomation on what the bottleneck is. To what processes where running in the past you will need the process accounting package.

3
  • +1 for sar, although I don't like it too much. Sep 6, 2010 at 15:36
  • I don't like it much either, except when I need it. Then I love it. Once it is set up, it disappears into daemon land. Then when something goes wrong you can dig around for the files. Usually they need to be copied and uncompressed. But the data available can be priceless. I have found Munin much friendlier for ongoing monitoring, but it doesn't go to quite the detail of sar. Munin has sar beat on the historical perspective though.
    – BillThor
    Sep 7, 2010 at 5:04
  • +1 for sar also. Likewise, you can use a tool like ksar to view the sar output. There's also nmon, if you admin a bunch of different *nix systems that aren't all Linux.
    – Matt
    Sep 7, 2010 at 13:18
2

I like collectd but I've recently started toying with pcp (performance co-pilot). It has some nice features for historical diagnosis. [1]: http://oss.sgi.com/projects/pcp/

1

Your non-elegant solution is actually a good one without setting up separate monitoring consoles (think SNMP traps). If you're running a RHEL/CentOS style syste make sure you've installed 'sysstat' (and turned it on) to gather ongoing stats about CPU, Memory, Disk I/O and the like. (see /etc/sysconfig/sysstat.* config files to tune).

Once you have that gathering underlying stats for you it can be used to pinpoint when the load trend occurs (so just besides seeing high CPU, is your proc queue backed up? do you see major faults in paging? how's your swap utilization?) which you can then correlate to your 'mysqladmin proc stat' type lists and so forth. If it's a LAMP stack, grab the total httpd processes and then do a quick sum/divide to find out the average process size to record as well. Enable your slow query log in MySQL to then trap those bad boys and look for some tables needing indexes.

Sometimes lo-tech isn't bad tech. :) Why use a chainsaw when a knife will do.

1
  • +1 There is nothing wrong in hacking something together to get this resolved. What you propose will hopefully reveal the issue but if not, it will be easy to modify your script to check other things. Once you have found the issues causing the load you can put in place something "more elegant" to watch for that in future. Sep 6, 2010 at 22:05
0

Might also wanna look at collectd, as a munin alternative.

0

atop.

It complements top/htop, because it can collect stats over time.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .