Last night at about 4:30am, both our app servers decided to have a seizure. We can't figure out why. Visitor load was low (100 live on the site) and there are no cron jobs scheduled on the servers.
We run ubuntu with apache, passenger and rails 3.0.3. The apache error log had a "cannot allocate memory" error right when the blue line dropped to the floor. Not sure what this tells me though since, the server runs well 99% of the time.
Any ideas for figuring out the root cause?