Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 1 down vote favorite

I have troubles understanding what problems may cause the occasional hangs that the server gets due to sudden high load spikes. I am not a system administrator (I'm a PHP programmer) but since the official sysadmin is quite lacking on effort I'm asked to find a solution myself.

The server runs on a Debian Lenny and serves via apache a wordpress + vbulletin based site with 40-60k visits/day. Having done all the application-side optimization I could, we got to the situation where the site runs smoothly even for weeks, then it trips on something that makes the server load jump up to 80+. Stopping apache to restart it helps, but it usually calms down by itself, if given enough time. It can "crash" twice in a day, or see no problems for weeks. It seems to be totally random.

One weird specific thing happened though. I was warned of a strange behavior and after inspection I found the .htaccess file changed to redirect traffic coming from search engines to some external site. I checked the code and every plugin (all up to date) and finally tried the "hard way" chowning .htaccess to root.root. The weird part is that when another issue came up, I found that file changed back to be owned by the user assigned to the website virtualhost. I understand there is no way for this to happen just via some web exploit, or am I mistaken?

How can I find the cause of this high load spikes?
What can explain a root.root file changing permissions, other that someone with root access doing it?
Could these two things be linked to some kind of attack?

share|improve this question

5 Answers

up vote 2 down vote

Regarding the Apache issue, one possible cause is that your MaxClient/MaxServer setting is too high. When you get spike in traffic you use up all your RAM and cause Apache to start using swap which will very quickly kill your performance. Next time you have the issue check the output of top/free and see if any swap is being used. If it is try reducing the MaxClient/MaxServer values.

I also had an issue with Apache 1.3 where some connections wouldn't ever close and after a few days there would 90% of the connections doing nothing leaving not clients for handling incoming new connections. I solved it by simply restarting Apache each day. From the sounds of it you don't have enough traffic or time between issues for this to be a likely cause.

share|improve this answer
The server starts indeed swapping during the spikes, the swap memory used increases very quickly. The value for MaxClients is 200 under the prefork module configuration, and 150 under the worker worker module configuration (I don't know how to find out which module apache is running). Is that too high? – Matteo Riva Feb 23 '11 at 15:29
I found no MaxServer setting – Matteo Riva Feb 23 '11 at 15:30
My mistake...it was ''ServerLimit''. Basically you want ServerLimit/MaxClients set to a value where you never begin to use the swap. This value depends on the memory usage of each Apache client and how much RAM you have. I usually do it by trial and error. Since you know it is too high try reducing it until you no longer use swap memory any more. If you only have 1GB of RAM you may have to lower it 50 or less. – uesp Feb 23 '11 at 20:23
up vote 0 down vote

It sounds to me as if someone may have compromised your server, more because of the redirection of search engine traffic to a different site than because of the file ownership issue. I'm afraid that some web exploits can give an attacker root access to your system.

I would download and run a rootkit detector such as Rootkit Hunter. If you have been rooted, you'll probably want to get someone experienced to help you to fix it.

share|improve this answer
up vote 0 down vote

It may be an attack, but then why only change the .htaccess file, probably there are more things to check. Maybe a php scripts is generating the file automatically. What are the permissions on the file?

Regarding the load, it may be a lot of things, specially because its random nature. Watch the logs and use tools (webalizer, to see if the load spikes coincide with heavy access times or access to a specific resource that may be the case of it.

Check your server with chkrootkit and similar tools to check for compromise. If it happened, you may need/want to reinstall your server from scratch.

share|improve this answer
up vote 0 down vote

I understand there is no way for this to happen just via some web exploit, or am I mistaken?

On its own this wouldn't account for the permissions of a root:root owned file changing - but a web compromise might enable an attacker to deploy malware on the system which could compromise the system in other ways. And there are other ways to compromise a system.

If you're confident that your code has not been modified since you last checked, then its time to eliminate everythnig else - using rootkit checkers is a good idea on a system you believe to be secure - but if you've got doubts you should reformat/reinstall.

share|improve this answer
up vote 0 down vote

On webserver run # ps -uapache -o wchan=WIDE-WCHAN-COLUMN,cmd try find something similar to flock_lock_file_w /usr/sbin/httpd on first field.

What is your session.save_handler?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.