Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 1 down vote favorite

I have a computer running tests, sometimes during a test it will freeze. I was thinking of writing a shell script to ping the test machine and if it is hung reboot it.

I ssh into the test machine and was able to reboot it, but during a hang i had no luck! How can i reboot it during a hang through ssh?

Thank you

specs: mac osx

share|improve this question
8  
Might be worthwhile figuring out why it freezes and how to remedy that condition. – lpthnc Dec 21 '09 at 17:18
1  
You will want to start accepting answers if you want people to continue giving them. You're 0 for 4... – David Oneill Dec 21 '09 at 17:22
@rashid, pick the answer that answers your question best, and click the checkmark beside it. That accepts the answer, which gives a reputation reward to both yourself and the person who answers it, but more importantly makes sure that answer appears on the top when people come to the question. – Paul Tomblin Dec 21 '09 at 18:13

migrated from stackoverflow.com Dec 22 '09 at 21:22

8 Answers

up vote 8 down vote

If it's really hung, chances are you won't be able to start a new process, which is what you'd need to be able to do to ssh in or restart it.

Servers that really need to be restarted remotely in this situation often have extra hardware, like another device (I think one is called an RSA card) that allows you to log in and trigger a reboot, or a watchdog timer that looks for something to happen regularly and if it doesn't happen it reboots.

Another possibility is to run your server as a virtual machine under a hypervisor like VMWare, Xen or VirtualBox, and then you can log onto the root machine (called a dom0 in Xen) and reboot the hung virtual machine.

share|improve this answer
+1 What I was typing: you were faster... – David Oneill Dec 21 '09 at 17:20
guess i am gonna have to buy extra hardware then! – rashid Dec 21 '09 at 17:27
You're looking for either a lights out management card (HP = iLO, Dell = DRAC) or an IP KVM – Zypher♦ Jan 7 '10 at 20:49
up vote 4 down vote

Set up another computer close to the test machine. You will need to position it such that the CD-ROM drive is at a 45 degree angle to the test machine. Position it so that the CD-ROM, when ejected, pushes the power button.

Now you can ssh into the new machine, run the eject <device> command and achieve your goal.

share|improve this answer
Funny. Been done, but not really a good solution. You do need to make sure that the remote off machine is heavy enough that the cd-rom tray doesn't push it across the table...... :) – Jim Barrows Dec 21 '09 at 17:23
that was funny!!! – rashid Dec 21 '09 at 17:23
Ghetto solution, a-la thedailywtf.com – Mark Henderson♦ Dec 22 '09 at 21:28
up vote 1 down vote

What do you mean by hang? If you mean that the SSH service is hung up, then you can't reboot. If by hang you mean the CPU is being hammered, then you might have to wait for a while. If by hang you mean some process is hung up that is preventing the system from coming down, then you might have to kill the process first, then issue the shutdown. Also, you might have a security system setting preventing you from issuing a shutdown command via ssh.

share|improve this answer
i can't seem to select anything on the screen. – rashid Dec 21 '09 at 17:25
up vote 0 down vote

Although I would tend to agree with those suggesting you should try to find the cause of the hang, you may be in for more luck if you just keep the ssh connection open all the time.

If you add something like ClientAliveInterval 300 th sshd.conf on the server, you can keep the connection open continiously, which effectively means you do not have to start a new process on the machine you're trying to reboot, since starting a new process may be one of the things that it's having trouble with.

share|improve this answer
my script runs the test figures out which test was hanging, in panic etc. To fix the hang is another team's job – rashid Dec 21 '09 at 17:32
up vote 0 down vote

As the previous answer say, if there is a true hang (kernel crash or so) - you are left with very limited options on rebooting. The functionality you are looking for is Lights-out-management, AKA out-of-band management. Read more about if from Wikipedia

Oh yeah, this question would be more at home at serverfault

share|improve this answer
up vote 0 down vote

You can try a software watchdog.

Linux kernel includes one called "softdog"[1], as part of linux-ha project

http://www.linux-ha.org/softdog

share|improve this answer
up vote 0 down vote

A remote power control unit or a "managed power supply" can be had for under $200. Then you telnet into that and reboot the power supply.

Example

share|improve this answer
up vote 0 down vote

echo b > /proc/sysrq-trigger

better would of course be to echo e i s u b in sequence... but that might not work...

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.