I have a problem with a debian system. Today at exactly 04:00 it stopped responding to nagios. I cant login via SSH either. That is not the first time this is happening, but the first time I watched it with nagios.
There is one special, uncommon thing about this box: It boots from an usb stick. That is why I thought it was a smart idea to map /var /tmp to a tmpfs.
I am fairly certain, that if I reboot it, it will work again. But because /var/log is mapped to memory, I cant read the logs after the reboot.
The next problem is that the hardware is offsite, so I can't login locally.
At first the problem might be that the tmpfs is filling up. But nagios did not warn before it could not connect anymore. I have set the warning threshold to 90% free space. So this does not seem to be it.
Other maybe interesting symptoms:
- the openVPN Server is still working
- routing still works
- the SSH port is still open and I am asked for the username. But if i supply the password the connection is dropped
- port 80 is open, but apache does not respond
The question that interests me most is: What could a debian system could do 04:00 in the morning? Some kind of update check?
I am grateful for any ideas or pointers in the right direction. Is there anything worth monitoring with nagios to get a hint? Next time I will add monitoring swap usage.