up vote 1 down vote favorite
share [g+] share [fb]

I have 2 very old Linux servers (1 running RHEL ES release 2.1 and the other FC release 3 - both severely under patched [sorry]) that spontaneously rebooted at exactly [to the second] the same time last Thursday. This happened again yesterday [Tuesday] 5 times! I have lots of Windows & Solaris servers on the same power supply that were not affected - I only have these two Linux servers.

Things I've considered :- There are no hardware issues reported on either server. Only one of them is running the client software needed by the UPS management system [which logs show no recent actions]. There is no local or remote cron/at job also the reboots are at random [AFAIK] times. "last -x" shows nothing [IMHO] useful, which is the same for messages, syslog, secure logs.

I'm currently thinking malicious activity exploiting some [un-patched] Linux vulnerability [more than likely] being called remotely and possibly using some level of broadcast to trip vulnerable nodes - but I am paranoid :)

Its only happening during the day so I'm thinking the source maybe a users workstation [all windows] that is only on during work hours.

My questions are :- 1. Is my paranoid theory viable? 2. How could I trap the source of the reboots?

link|improve this question
feedback

5 Answers

up vote 7 down vote

It could be something as simple as dirty power. Are these the only machines on this power strip/plug?

Unplug the machine from the network if you think it's being rebooted remotely for any reason (if possible), and you can eliminate that.

link|improve this answer
I like your suggeston about unplugging the network cable from one of the machines! – Brent Jul 8 '09 at 17:30
+1 for getting it clearly and before me – nik Jul 8 '09 at 17:34
+1 - Check power first. It may be something network-related as well, but it doesn't have to be malicious. Since both are severely under-patched, you might be getting hit with a bug in the network stack. (Which could be exploitable.. but if so, the first question to answer is where is the bad traffic coming from?) – romandas Jul 8 '09 at 19:18
Thanks - I will definitely unplug one of the servers to at least show if the source is remote or not. – graemeb Jul 9 '09 at 7:36
Forgot to add - the UPS runs 12 power strips and these two servers are on seperate strips – graemeb Jul 9 '09 at 8:01
feedback
up vote 5 down vote

Thanks for all the suggestions. I think its now solved - no malicious intent was found. Unbeknown to me [I'm a remote worker] but our PC Support had plugged my 2 servers into an IP KVM about a month ago. It would appear that through the act of them logging onto their windows servers the CTRL-ALT-DEL signal must leak outside of the intended target and get picked up by other connected nodes. As I'm sure you're aware C-A-D if left in default mode [as mine were] causes Linux servers to reboot. I managed to capture some pertinent info by running & logging a "ps -ef" every second - it showed at the time of reboot the command used was "/sbin/shutdown -t3 -r 0 w" which translates to the trap in /etc/inittab. So mystery solved (yn) but I've found a valuable source of expert knowledge outside of my usual google-world. Thanks again

link|improve this answer
Thanks for sharing your solution. That sure is one of those mysteries which at first will make you want to bang your head against a wall, later makes a nice story to tell your geeky friends. – Marie Fischer Jul 9 '09 at 22:51
feedback
up vote 1 down vote

It sounds like you are on the right track thinking that this could by a compromise - especially if both servers have the same user accounts/passwords.

  • First thing I would do is run chkrootkit or rkhunter and see if they find anything. (not sure if these will work if installed after a compromise has already happened or not)

  • Second thing would be to enable remote logging, and figure out exactly what happened immediately prior to the reboot.

  • A third suggestion might be to install munin or something similar to track your cpu/memory usage prior to the reboot.

link|improve this answer
Thanks Brent - Am interested in your 2nd suggestion. How do I enable remote logging? – graemeb Jul 9 '09 at 7:34
On my Debian machines - I add a line to /etc/syslog.conf "*.notice @remote.log.server" to send the logs to remote.log.server. Then on remote.log.server you can add the "-r" switch to the /etc/init.d/syslogd startup script to cause it to receive the remote logs. In our case we run syslog-ng on our remote.log.server, as it has some better options for separating out the logs into machine-based subdirectories. – Brent Jul 9 '09 at 15:44
Thanks for taking the time to help Brent :) – graemeb Jul 9 '09 at 20:43
feedback
up vote 1 down vote

Have you checked if they are on the same power patch?
If your power problem is between the UPS and the machines, it wont show on the logs.

link|improve this answer
feedback
up vote 0 down vote

I you cannot unplug them from the network, I would sniff the network traffic.

Not sure if running tcpdump on the affected machines itself is a good idea, but you can use either port mirroring on your switch or plug them temporarily into an old hub (not switch) and use a seperate machine to run tcpdump/wireshark/whatever you like.

link|improve this answer
Thanks - I will look into running tcpdump although I suspect the logs will become too onerous given the servers stayed up all day yesterday. I had set a netstat logging at 1sec but this showed nothing. – graemeb Jul 9 '09 at 7:39
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.