I have troubles understanding what problems may cause the occasional hangs that the server gets due to sudden high load spikes. I am not a system administrator (I'm a PHP programmer) but since the official sysadmin is quite lacking on effort I'm asked to find a solution myself.
The server runs on a Debian Lenny and serves via apache a wordpress + vbulletin based site with 40-60k visits/day. Having done all the application-side optimization I could, we got to the situation where the site runs smoothly even for weeks, then it trips on something that makes the server load jump up to 80+. Stopping apache to restart it helps, but it usually calms down by itself, if given enough time. It can "crash" twice in a day, or see no problems for weeks. It seems to be totally random.
One weird specific thing happened though. I was warned of a strange behavior and after inspection I found the
.htaccess file changed to redirect traffic coming from search engines to some external site. I checked the code and every plugin (all up to date) and finally tried the "hard way"
root.root. The weird part is that when another issue came up, I found that file changed back to be owned by the user assigned to the website virtualhost. I understand there is no way for this to happen just via some web exploit, or am I mistaken?
How can I find the cause of this high load spikes?
What can explain a
root.root file changing permissions, other that someone with root access doing it?
Could these two things be linked to some kind of attack?