I have a computer which keeps becoming unreachable, and a restart fixes the problem. It's under somewhat heavy load (high cpu, and high ram usage), but appears for the most part stable.
What's a good way to determine why it dies on me? I need to do this, while the computer is under load / in production.
E.G:
- What to look for in what logs? (Telltales for common problems? Any good log parser to sort by hardware component or common threads? How to find the timestamp/log entry of restart? Sort by error level? (critical, warning etc.))
- How to determine if ram is good from remote? (memtest, that can run alongside the rest of the system, similar to memtest from Hgi Design for windows)
- How to determine if there are any bad sectors on the harddrive? (How to look for read/write errors at the OS level? s.m.a.r.t. info is inaccurate at best, and not suitable to determine the extent of problems imo. - Looking for something similar to windows event viewer "disk error" category)
- How to find the last error before reboot took place in logs (what's a good way to parse logs) - partially answered by bulleric, but would like a concrete example of how to find the reboot entry/timestamp :)
- How to determine from logs (after any reboot), what caused it to become unresponsive (E.G look for exhausted ram telltales, if network is dropped, kernel panic) - In windows, the unexpected reboot event is logged as "critical", with a high probability of useful log entries just before this event - what to search for to find this in ubuntu?
*Edit:
Based on the answers, I think I should clarify some:
The load is high, as it is a production server, not because a process is leaking ram, or burning cpu in some endless loop. This is normal, and is resulting in a huge logs, which are impractical to read line by line. I am familiar with htop, cacti, nagios, munin - and none of these address what I'm asking as they only show that the system is under heavy load (which I already know).