Last night at about 4:30am, both our app servers decided to have a seizure. We can't figure out why. Visitor load was low (100 live on the site) and there are no cron jobs scheduled on the servers.

We run ubuntu with apache, passenger and rails 3.0.3. The apache error log had a "cannot allocate memory" error right when the blue line dropped to the floor. Not sure what this tells me though since, the server runs well 99% of the time.

Any ideas for figuring out the root cause?


CPU Graph over the last 12 hours for our app servers


Some robot indexing your pages? google?

Or an attacking script trying logins or searching for not secured apps or not updated services with known vulnerabilities?

  • Yes, time to hit the logs. The servers were pushed to 60% load for over an hour: there must be footprints left on your system somewhere.
    – DutchUncle
    Feb 18 '11 at 19:21

Notice that you're using EC2. Are both app servers in the same EC2 availability zone? Were there any kernel messages relating to errors (via dmesg)?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.