Track resource usage on Ubuntu server while I'm sleeping?

Question

I've got a pretty typical LAMP server running a web site but sometimes, during the night, resource usage skyrockets and the site becomes unresponsive. There are a lot of logged errors about being unable to connect to MySQL, and the MySQL process gets automatically restarted, according to /var/log/messages ..

But I don't know why. It's not getting high traffic, not even from bots crawling the site. It's not happening at the same time every night, so it doesn't seem to be related to any specific cron job that might be firing off. Is there any way to log what's going on with the system when this happens so I can look at it the next morning?

Answer 1

Get Munin installed. It will track your system resources (RAM, CPU, Disk), but can also monitor more detailed apache stats if you enable mod_status. By default, Munin just graphs the data it collects, but you can also configure it to alert on threshold exceeded events.

Answer 2

Like @Erika said, graphs (like munin) are a must to debug this sort of things. You might also consider running atop to log the top informations.

Answer 3

I ran into this same problem about a year ago. My solution was monit

You can set up rules to send off alerts, fire up scripts, whatever. If you're using a hosting provider that provides an API which will allow you to scale up resources when you go down, you could write a script to temporarily increase server resources until load goes back below a certain threshold.

Screenshots here: http://mmonit.com/monit/

Answer 4

Install some monitoring software. I use:

Ganglia

Cacti

MySQL Cacti Templates

Nagios

That ought to give you a good start towards understanding how your system behaves as a baseline. Look for abnormalities and then figure out a way to detect that condition and alert you going forward. Repeat.

Answer 5

Suggestions so far are good, but rather complex if you're just trying to track down the problem you're having. A mixture of sar, atop (with atopsar) and possibly sysshot should get you the information you need without having to configure anything - install them, start the services, sit back and wait while the data is collected.