Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

One of my centos host rebooted recently unfortunately no monitoring was implemented .

How can i find out what caused it to reboot. i only have ssh access to it.

How can determine if the cause is due to Following:

1.power failure.(Will there be any special non maskable interrupt logged some where)

2.Intentional hard reset.(will it be possible to differentiate between power failure n reset also what will be Signal to look for?).

3.manual reboot command(How to differentiate this from hard reset,will inode reveal anythng related to this like "**Cleaning Orphaned node**"  any special signals?)

4.Low disk space.(there is no message in /log/messages 30 minutes before restart,something stopped daemon from writing to log, disk space was ok ).  

5.Overloaded CPU (Not configured to run any cronjob for saving sar o/p,Currently load average is normal Will there be any special signal to look for when CPU is loaded.)

6.Low RAM(currently there is more than 24 GB free space

I am interested in knowing the difference between intentional(hard reset/command issued) and power failure reboot.

share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

For 3, you'll see log messages about processes shutting down before the reboot. You won't see those with 1 and 2. Additionally with non-graceful shutdowns, you may see file system check related messages on boot. Keep in mind, though, that the presence of these does not necessarily indicate a hard shutdown, as some file systems run periodic checks on themselves every N days, or every X mounts.

For 4, 5, and 6, these don't typically cause reboots, and they're all things that your monitoring should catch and alert on.

To expand on 6, under extreme RAM load, the kernel will invoke its out-of-memory killer, which kills off memory-hungry processes in order to preserve the overall stability of the system. When this happens, it will be logged.

share|improve this answer
    
Could please tell me What could be the reason for klogd from not writing to logs. –  kevin Feb 1 '13 at 6:16
    
That's a different topic. Please post that as a separate question, with relevant details. –  EEAA Feb 1 '13 at 6:18
    
in my 4th point i mentioned that,can u? –  kevin Feb 1 '13 at 6:23
    
Who knows...you haven't given us any information. –  EEAA Feb 1 '13 at 6:24
    
i dont have any :( ,nothing got written there 30 minutes b4 reboot.My final question if my HDD has 0% free space can this result in reboot,since it can not save files to /tmp. –  kevin Feb 1 '13 at 6:28
show 1 more comment

For most of hardware failures you may look at System Event Log records, it can reveal such things as:

  1. CPU overheat
  2. ECC memory errors
  3. Power supply failures
  4. IPMI reboots

and much more.

You can access SEL (if it's available on your hardware) via ipmitool or ipmiutil commands, see: ipmiutil sel - show firmware System Event Log records

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.