I have a site with low traffic, less than 500 hits a day. It has a 6G of memory and it is way underutilized, on average 5% is in use. But as soon as googlebot establishes a connection to my webserver/apache, the memory and cpu usage spikes in seconds and the server becomes inaccessible - website, ssh and all other services.
When I do lsof for port 80, here is what i see before the site crashes in seconds.
lsof -i:80 mywebsite:http->crawl-66-249-71-200.googlebot.com:43567 (ESTABLISHED)
Google Bot is set to a slow crawl rate.
Apache config is:
ServerLimit 256 MaxClients 150 MaxRequestsPerChild 100 KeepAlive Off KeepAliveTimeout 5 MaxKeepAliveRequests 100
The error log shows:
Cannot allocate memory: couldn't create child process: /opt/suphp/sbin/suphp