You need to give more information about your server configuration, and post those logs. How do you know that the load average got high ?
The "load average skyrocketing to over 100 before failure" symptom can have many causes, but there are frequent ones.
First, what is the load average ?
I recommend looking at this post to get more information on the subject : http://superuser.com/questions/23498/what-does-load-average-mean-in-unix-linux
The load average is an indicator of how many processes are currently in the "waiting" state. A very high load average indicates ressource exhaustion, and a common cause of this ressource exhaustion is long (or infinite) waiting times for I/O operations to complete.
What could cause this ?
- I/O Waiting can be because of a failing NFS mount, of a failing hard
disk drive. You may have been the victim of an attack.
- Attacks like
Slowloris tend to exhaust the file
handle pool, and can hang I/O operations on the server.
- Pay attention
to fork bombs too. If you suffered a fork bomb, and that your memory
got exhausted, you may see traces about the "OOM Killer", a kernel
worker that sacrifices processes to free up memory for the system
when everything else have failed. On some setups, that OOM Killer can
make the system reboot.
Where to look ?
This really depends on your system configuration and environment. You did not provide enough details in your question to answer to this particular question. Here are some ideas, though :
- First, check all of your hard drives. Run full SMART tests on them.
- If you have hardware RAID, check your RAID controller's log.
- If you are hosting web services, check for bandwidth usage history, and requests count.
- If you have remote network mounts (sshfs, NFS), check their reliability
- If you are hosting user shells on your server, check their .bash_history (This can be faked.)
- If that server is a virtual machine, the problem may have nothing to do with your server but with your virtual services provider.