Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a computer which keeps becoming unreachable, and a restart fixes the problem. It's under somewhat heavy load (high cpu, and high ram usage), but appears for the most part stable.

What's a good way to determine why it dies on me? I need to do this, while the computer is under load / in production.

E.G:

  • What to look for in what logs? (Telltales for common problems? Any good log parser to sort by hardware component or common threads? How to find the timestamp/log entry of restart? Sort by error level? (critical, warning etc.))
  • How to determine if ram is good from remote? (memtest, that can run alongside the rest of the system, similar to memtest from Hgi Design for windows)
  • How to determine if there are any bad sectors on the harddrive? (How to look for read/write errors at the OS level? s.m.a.r.t. info is inaccurate at best, and not suitable to determine the extent of problems imo. - Looking for something similar to windows event viewer "disk error" category)
  • How to find the last error before reboot took place in logs (what's a good way to parse logs) - partially answered by bulleric, but would like a concrete example of how to find the reboot entry/timestamp :)
  • How to determine from logs (after any reboot), what caused it to become unresponsive (E.G look for exhausted ram telltales, if network is dropped, kernel panic) - In windows, the unexpected reboot event is logged as "critical", with a high probability of useful log entries just before this event - what to search for to find this in ubuntu?

*Edit:

Based on the answers, I think I should clarify some:

The load is high, as it is a production server, not because a process is leaking ram, or burning cpu in some endless loop. This is normal, and is resulting in a huge logs, which are impractical to read line by line. I am familiar with htop, cacti, nagios, munin - and none of these address what I'm asking as they only show that the system is under heavy load (which I already know).

share|improve this question
add comment

4 Answers

up vote 2 down vote accepted

you must search the beginning of the problem

What to look for in what logs?

The first thing is open top or if installed htop
and search for overloadet tasks.. Many active store, process load ..

if you find anything look in the logs
tail -f /var/log/syslog
tail -f /var/log/dmesg

and lookup for errors or warnings that named with eventually overloadet tasks you found in top or htop.
In /var/log/ are many logfiles for many programms.

How to determine if ram is good from remote?

For this problem you can use a webbased analytic tool like Munin
http://munin-monitoring.org/

Write a small bash script wich test the ram and when you get problems send a mail. or use
nagios --> its a monitoring tool when you get problems with the ram you get an mail.

How to determine if there are any bad sectors on the harddrive?
find out what are your main harddrive or what are the harddrive you will test
fdisk -l
and check the harddrive(s) with fsck (filesystemcheck: fsck check and repair a Linux file system)

How to find the last error before reboot took place in logs (what's a good way to parse logs)

You can use the tail command with the lines option

tail -200 /var/log/syslog
tail -200 /var/log/dmesg

search for runlevel changes or shutdown sequences init 6 is the reboot runlevel
it can help when you use a pager (less)

tail -200 /var/log/syslog |less

I hope i can help a little bit :)

i wish you success

share|improve this answer
    
Do you have a concrete example on how to find the reboot log entry? –  Jon Skarpeteig Jul 29 '11 at 11:31
add comment

I suggest installing htop which gives a good overview over your system. You might also just use top. If load is very high and the system stops responding, try ps aux and kill not needed suspicious processes or stop services using sudo service $service stop.

  • cd /var/log; ls -lrt gives you the latest changed logs, check these
  • ssh server free -m gives you information about memory and swap usage
  • Bad sectors: use smartctl to check your disks
  • I use less to look into logs and use keys b and f to scroll through them. Typing / lets you search for text. You can also use grep on a file to look for specific dates or text.

On ubuntu systems you might want to add your user to the group adm, so that you don't have to use sudo to read some of the files.

Another good idea is to install logcheck and set it up to send e-mails to you about what is going on. If you have more than one computer that you take care of, you also might want to install a monitoring system like xymon to identify anomalies.

Good luck fixing your system!

share|improve this answer
add comment

I would approach it in different way. Open few sessions - one with tail -f /var/log/syslog, second with tail -f /var/log/kern.log, third to monitor top, and fourth to run utility like stress. Now, simulate loaded system and see what happens. I suggest start with CPU load only and work from there. Also, ping the machine during the tests and monitor how it responds.

Please note - if you exhaust your RAM and hit the swap (unless on SSD) the machine will look exactly as you described it. Good luck!

share|improve this answer
add comment

Have you considered hardware-related problems? I'd do the following in order:

  1. Check for network firewall/config problem.
  2. Check for hardware fault - try to plug in an off-the-shelf NIC and use that instead of the ones in use.
  3. If you have a similar or identical machine around, get the box offline and test the hardware. At the same time see if you have the same problem on the replacement - good way to determine if the problem is hardware/software related.
  4. If you don't have the hardware to spare it's time to make a business case for it.

Good luck!

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.