How to track down uncertain httpd/mysqld activity

Question

I'm running a pretty standard LAMP stack on Fedora 15, mostly in support of some Drupal sites. Things generally seem to be running fine, but occasionally the disk on the machine will start to work noticeably harder than it usually does. My limited system debugging skills have found this:

• 'top' reports that httpd and mysqld are indeed active, taking up maybe 3-4% of what is otherwise a mostly idle machine.

• There are no changes in the httpd log files, and nothing out of the ordinary is showing up on other system logs. So the obvious answer of "somebody's hitting your websites, dummy" doesn't seem to be the case.

• resetting httpd with "apachectl graceful" has no effect. However, "apachectl stop" followed by "apachectl start" generally calms the machine down -- the disk churning stops.

• I believe the machine is secure; I have no reason to believe that there's any hackery going on. (Of course, I suppose that's usually the case...)

I'd be surprised (but happy!) if that was enough for somebody to tell me what's going on, but is there any advice out there on how I might dig into the machine a bit more and figure out what's going on? Thanks!

Answer 1

What is occasionally? You are supposed to setup a cron job to hit /cron.php which does Drupal housekeeping.

Check crontab -l for the root user or apache (or www-data) user to see if there is a cron job that you can correlate to the disk activity. The cron job could also be under /etc/cron.d/drupal.cron (or some such file). You should see these /cron.php requests in your Apache access_log so I assume by "no changes in the httpd log files" you mean no external requests and not literally no changes.

In the future you should collect metrics (e.g. Cacti, Munin) on your servers so you can tell what is normal and what is not.

Answer 2

This behaviour can be consistent with running out of physical memory and the kernel using swap as a result. Check the output of:

free -g
free -m

If you run this when the increased activity occurs and it shows you swap is being used (i.e. more than 0 in the used column) then I am pretty sure adding a bit more memory will help.

Although if you have less than 1 GB I would upgrade the memory regardless.

Also consider installing the sysstat package (named that way in debian and redhat), the "sar" command is quite useful in finding out bottlenecks.

Answer 3

Found it -- it was Drupal cron-related after all. The site is new, and had a large amount of stuff to handle via cron (indexing a large number of nodes). For one (unknown) reason or another, the indexing process was going off into the weeds and repeatedly restarting itself. As a result, the indexer was running like crazy (thus the disk activity), but there was no evidence of any problems in the log files, since crontab's call to cron.php would (eventually) finish and show up in the logs.

Anyway, once I smacked the indexing process around a bit and got it under control, cron is running happily and all seems well. Thanks to all for the suggestions!