0

Over the past year, we had consistently random spikes which caused apache to use all the memory and swap out.

Server Uptime

Server Uptime

I have an example of the latest crash (5:30AM - 12:00PM UTC).

OOM Kernel Dump

OOM Kernel Dump

EC2 Logs

EC2 Logs

RDS Logs

RDS Logs

404 Errors Before Crash

404 Errors Before Crash

I don't want to increase the ec2 instance resources, since under normal circumstances it is enough to handle the traffic.

What can I put in place to log the source of these spikes, and also put in a solution to prevent these spikes from bringing down the website?

Update

I can confirm that the previous two crashes happened immediately after a considerable amount of 404 requests from a group of IP addresses belonging to ChinaNetCenter Ltd.

Update 2:

Solved the issue by reducing the memory needed to serve 404 pages.

0

A look into the Apache logs around the time of the crash would be insightful. You could have something as simple as a cron job executing that is causing the apache service to crash or you could have apache enabled to only accept a specified amount of concurrent connections and your user base influxes during different times of the day causing this limit to be reached.

You could even have a log rotation that is running that is causing the resources to spike. It is not possible to tell for certain without the apache error log but these are some possibilities to check into.

  • Thank you for your reply. I don't have any records in my crontab. I just checked my apache error log and access log. In my error log, there were no events prior to the crash. In my access log I've noticed crawler-like behaviour from a group of IP addresses belonging to ChinaNetCenter Ltd. After these requests happened, my RDS connections stayed at a 150. – ruiyang Oct 10 '18 at 17:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.