Linux: System analysis tool [closed]

Question

I have a server performing multiple tasks over traffic coming to it.
Lately the server has been stuttering. When the system chokes, it reboots - this leaves me with little meaningful information as for the pathology of the choke (this is definitely not a violent crash).
What tool in your experience assisted you in preserving meaningful data regarding a system choke.

This could be memory consumption, "ps", "top" or any other parameter.
Mind you, a script that simply outputs several lengthy commands (ps -a) every so and so - can use massive storage, and be hard to analyse.

Answer 1

Install SAR, it provides snapshot of data every 10 minutes by default, but you can change the rate of infomation using the cron job.

It will provide lots of useful data, such as memory, load cpu usage and disk I/O stats.

Answer 2

you need historical analysis of most common parameters of the system, i would recommend MUNIN, which provides graphing of most common system resources through web, which you can constantly monitor to see Which process/Which service chokes the resources and Which resource.

After that i would recommend to tail -f the system log files, that what causes this problem.

Answer 3

Sometimes I feel SAR to be little inadequate. I have come across instances where you need a complete picture of what is going in your system. Commands like top, ps, vmstat, netstat, iostat, iotop comes handy then. Generally I log outputs these commands into a file. Now if space is a constrain for you, there is a service called SeaLion which execute all above commands and stores it into cloud. You can access these data from you browser.

Also I would like to mention other services like Nagios, Munin, New Relic, Server density which also collects stats from your machine, that might come handy while debugging a problem.