Application getting VERY slow only in the morning: tools to track the culprit?

Question

I have two Ubuntu Servers: one acting as db server (mysql) and file server and the application server. Using this three service an application has been working flawlessly for months.

Now we've found out that around 7:45 EVERY morning it really slows down. After an hour everything becomes fast again and usable without human intervention. I'm trying to track down the problem...

Is there any kind of tool to monitor and log cpu, ram, disk, network usage? How should I proceed to find the problem quickly?

Answer 1

Quick fix would be to isolate the problem and find the culprit.
Check what cronjobs are running at that time and reschedule them to the different time.
If you can afford it, disable networking during that time.
If issue still persists, shutdown all services that are not needed.
Proper way would be to look at the data that you have already collected as part of the process.As said already, there is a myriad of option on how to do it.We use Cacti graphing system as this gives nice visual representation and you can graph virtually anything.

Answer 2

I find sar is good at pointing out likely causes of slowness. Install the atsar package on both systems. After it has been running for a day you should have data to see which resources are being used heavily when the system is slow. Data will be collected in /var/log/atsar which will eventually have 31 daily files. This is a great tool for examining load issues.

I find munin to be a good tool for overall monitoring of servers. I try not to run the munin server on a system which might get overloaded. In your case you would want a third host to act as the server. munin gathers data from each client every five minutes. This data is then transformed into graphs. It is fairly easy to see from the graphs which resource has the heaviest load. That can help speed up then investigation with sar.

For a quick test, just log on to each server and run top during the problem period. It helps to have run it earlier to know what the normal load looks like. If your load average exceeds the number of cores on the server for a long period of time, you have likely identified the server which is causing the slow down. The top lines can be used to identify if it is CPU (high user%) or some other resource (high system% or high wait%). You also get a view of memory utilization.

The process accounting package acct can tell you which process(es) are completing during the period. You will need to run accton on to enable process accounting. The command dump-acct can be run to dump the data in a readable format. man dump-acct includes information on the format of the output. You likely want processes that start around 7:45.

Answer 3

Is there any kind of tool to monitor and log cpu, ram, disk, network usage?

You would not believe the number of tools available. The ones I use are Nagios, sar, iotop, top, ps, mrtg, and a load of custom scripts. But they are just going to tell you how slow the system is - not why it is slow.

There are all sorts of reasons it could be slow - both stuff happening on the machine (e.g. backups causing a cache flush, batch processing stealing CPU cycles...), stuff happenning elsewhere (network congestion, shift in the majority of traffic from local to overseas) not to mention an absence of stuff which happens when the system is running quickly (low levels of traffic can cause things like loss of caching, early evacuation of connections from badly configured stateful firewalls).