Currently, I'm the Moodle Admin for my University, and one of my responsibilities is to keep the server running and working all the time. Sometimes, for no apparent reason, Apache or / and MySQL crashes, causing total chaos within the University.
In a broad sense, what are the "basic guidelines" to follow when a server crashes? What should I do at first to find out what happened? How do I know how many users were connected at the time of the crash (or at any given time)? How do I know how much memory or power I need for the current demand?
There are many questions related to each other, but these are the most important. Obviously, I'm far from being an experienced Sysadmin. I know my way around Linux a bit, if that helps.
Our server specs:
- Intel Dual-Core Xeon @2.66 GHz (if I recall correctly)
- 2 GB RAM
- 500 GB HDD
- CentOS 5.4
- MySQL 5.0.45
- PHP 5.3.12
EDIT: Sorry for the lack of information.
I have read both Apache and MySQL logs without any significant data appearing. Apache is the most informative, saying WHEN did it crash, but no other reason. In fact, the error in Apache crashing isn't actually an "error", just the log entry of it reinitiating - worst case scenario is Apache giving "SIGTERM" or "SIGKILL". MySQL logs do not tell absolutely anything.
I usually try to follow up what's happening using "top". When the crashes happen, rarely there's all (or even half) of the system memory consumed. In really dire situations, the CPU usage has reached... 80%?
Disk and Memory usage seems fine (du and free show no problems). SSH access usually is fine. It just seems that MySQL or Apache randomly crashes, because, even when the demand is not that high, it still hangs.
The problem could be reduced to "What logs to check?" "How to check the number of connections?"