How can I log what is causing a server to reboot?

Question

I have three servers from 1and1.

All of them are setup the same running named, apache, mysql, postfix, dovecot and related services.

Two of the servers are running great. The third one - and the one with the least traffic - randomly locks up or reboots.

I've found nothing in my logs and I also asked 1and1 to check and they found no problem in their logs or mine.

Old journal logs are saved and I constantly have top and nethogs open. I never notice any spikes of usage for any resource and there's nothing in any logs as to what's happening.

Sometimes I randomly can't connect and no service responds at all. To remedy this I must log into 1and1 and reboot the server from there.

Other times, it simply reboots itself.

Is there a logging system that performs extensive logging and/or tracking of resources and processes to be able to try to pin this down?

Considering I have three servers configured exactly the same and only this one is having problems I've a few possibilities in mind including possible attacks but I can't be certain.

I feel like I need some EXTREME logging to pin this down.

Each server is running Arch linux. All configuration was done by myself so I know all three are 100% the same configuration-wise.

Answer 1

As usual, logging is indeed your friend! To see quite detailed logs about the system before it went down you can use journalctl. This utility displays the various logs that are handled by systemd. These days (for better or worse) that is a whole lot of different systems on your machine.

$ journalctl -xe -b-1

Here is the meaning of each parameter:

• x: Add explanatory message about each line if it is available.
• e: Jump to the end of the journal so you are first seeing the messages immediately before the reboot.
• b-1: Show information about the boot before the current one.

If you are sure you actually want "EXTREME" logging (you probably don't). You can also append the following option -o verbose.

To display kernel specific log messages you can use dmesg. Something like the following may help you.

$ dmesg -H