How to differentiate intentional reboot and power failure from logs

Question

One of my centos host rebooted recently unfortunately no monitoring was implemented .

How can i find out what caused it to reboot. i only have ssh access to it.

How can determine if the cause is due to Following:

1.power failure.(Will there be any special non maskable interrupt logged some where)

2.Intentional hard reset.(will it be possible to differentiate between power failure n reset also what will be Signal to look for?).

3.manual reboot command(How to differentiate this from hard reset,will inode reveal anythng related to this like "**Cleaning Orphaned node**"  any special signals?)

4.Low disk space.(there is no message in /log/messages 30 minutes before restart,something stopped daemon from writing to log, disk space was ok ).  

5.Overloaded CPU (Not configured to run any cronjob for saving sar o/p,Currently load average is normal Will there be any special signal to look for when CPU is loaded.)

6.Low RAM(currently there is more than 24 GB free space

I am interested in knowing the difference between intentional(hard reset/command issued) and power failure reboot.

Answer 1

For 3, you'll see log messages about processes shutting down before the reboot. You won't see those with 1 and 2. Additionally with non-graceful shutdowns, you may see file system check related messages on boot. Keep in mind, though, that the presence of these does not necessarily indicate a hard shutdown, as some file systems run periodic checks on themselves every N days, or every X mounts.

For 4, 5, and 6, these don't typically cause reboots, and they're all things that your monitoring should catch and alert on.

To expand on 6, under extreme RAM load, the kernel will invoke its out-of-memory killer, which kills off memory-hungry processes in order to preserve the overall stability of the system. When this happens, it will be logged.

Answer 2

For most of hardware failures you may look at System Event Log records, it can reveal such things as:

1. CPU overheat
2. ECC memory errors
3. Power supply failures
4. IPMI reboots

and much more.

You can access SEL (if it's available on your hardware) via ipmitool or ipmiutil commands, see: ipmiutil sel - show firmware System Event Log records