Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running a pretty standard LAMP stack on Fedora 15, mostly in support of some Drupal sites. Things generally seem to be running fine, but occasionally the disk on the machine will start to work noticeably harder than it usually does. My limited system debugging skills have found this:

  • 'top' reports that httpd and mysqld are indeed active, taking up maybe 3-4% of what is otherwise a mostly idle machine.

  • There are no changes in the httpd log files, and nothing out of the ordinary is showing up on other system logs. So the obvious answer of "somebody's hitting your websites, dummy" doesn't seem to be the case.

  • resetting httpd with "apachectl graceful" has no effect. However, "apachectl stop" followed by "apachectl start" generally calms the machine down -- the disk churning stops.

  • I believe the machine is secure; I have no reason to believe that there's any hackery going on. (Of course, I suppose that's usually the case...)

I'd be surprised (but happy!) if that was enough for somebody to tell me what's going on, but is there any advice out there on how I might dig into the machine a bit more and figure out what's going on? Thanks!

share|improve this question
1  
Do you have any logging for iptables in place? It's possible someone is probing the Drupal installs for vulnerabilities. –  Justin Pearce Apr 24 '12 at 22:08
    
I'll check that out -- thanks. –  Jim Miller Apr 25 '12 at 16:29

3 Answers 3

up vote 0 down vote accepted

What is occasionally? You are supposed to setup a cron job to hit /cron.php which does Drupal housekeeping.

Check crontab -l for the root user or apache (or www-data) user to see if there is a cron job that you can correlate to the disk activity. The cron job could also be under /etc/cron.d/drupal.cron (or some such file). You should see these /cron.php requests in your Apache access_log so I assume by "no changes in the httpd log files" you mean no external requests and not literally no changes.

In the future you should collect metrics (e.g. Cacti, Munin) on your servers so you can tell what is normal and what is not.

share|improve this answer
    
True, and the Drupal cron jobs are in place. But, as you note, those would show up in the website logs, which isn't happening. Thanks, though. –  Jim Miller Apr 25 '12 at 15:25
    
Are you sure you're looking in the right place? Are each of your Drupal sites a VirtualHost with its own logs? You're not looking in the default access_log are you? The websites are servicing client requests right? –  HTTP500 Apr 25 '12 at 20:52

Found it -- it was Drupal cron-related after all. The site is new, and had a large amount of stuff to handle via cron (indexing a large number of nodes). For one (unknown) reason or another, the indexing process was going off into the weeds and repeatedly restarting itself. As a result, the indexer was running like crazy (thus the disk activity), but there was no evidence of any problems in the log files, since crontab's call to cron.php would (eventually) finish and show up in the logs.

Anyway, once I smacked the indexing process around a bit and got it under control, cron is running happily and all seems well. Thanks to all for the suggestions!

share|improve this answer
    
Can you kindly accept my answer if you think it put you on track to determining the issue? Thanks! –  HTTP500 Apr 25 '12 at 22:56
    
Yeah -- I could whine and say it wasn't quite there, but I'm in a good mood...:) –  Jim Miller Apr 26 '12 at 2:30
    
@JimMiller There is also nothing wrong with accepting a self-posted answer. If this isn't a valid answer, it may be better as a comment on HTTP500 's answer. –  jscott Apr 26 '12 at 15:51

This behaviour can be consistent with running out of physical memory and the kernel using swap as a result. Check the output of:

free -g
free -m

If you run this when the increased activity occurs and it shows you swap is being used (i.e. more than 0 in the used column) then I am pretty sure adding a bit more memory will help.

Although if you have less than 1 GB I would upgrade the memory regardless.

Also consider installing the sysstat package (named that way in debian and redhat), the "sar" command is quite useful in finding out bottlenecks.

share|improve this answer
    
Another good idea, but no -- I've got 16 GB in the machine, and "free" is coming in at zero. –  Jim Miller Apr 25 '12 at 15:27
    
The to me it sounds like it's just caused traffic that's hitting your server. Besides iptables loggin you can also use tools such as tcpdump to see what's going on. Though I am not sure if a few % increase in load is that unusual or something to be worried about. –  aseq Apr 25 '12 at 18:22

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.