Over the past year, we had consistently random spikes which caused apache to use all the memory and swap out.
I have an example of the latest crash (5:30AM - 12:00PM UTC).
OOM Kernel Dump
404 Errors Before Crash
I don't want to increase the ec2 instance resources, since under normal circumstances it is enough to handle the traffic.
What can I put in place to log the source of these spikes, and also put in a solution to prevent these spikes from bringing down the website?
I can confirm that the previous two crashes happened immediately after a considerable amount of 404 requests from a group of IP addresses belonging to ChinaNetCenter Ltd.
Solved the issue by reducing the memory needed to serve 404 pages.