Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a server which I'm administrating remotely over SSH, for various reasons, it needs to be restarted. However, when I try to do so, shutdown goes to uninterruptible sleep almost immediately after sending the wall message.

$ ps aux | grep 'shutdown'
root      4754  0.0  0.1   1888   720 ?        D    07:26   0:00 shutdown -r 0 w

I have no physical access to the machine at the current time. What should I do? The server is still responding, and I can still log in.

share|improve this question
    
Does the server have NFS mounts? Is it connected to SAN via Fibre Channel? Is there some kind of heartbeat system such as Heartbeat v2 installed? –  Janne Pikkarainen Aug 5 '11 at 6:58
    
True, it's probably worth giving background: this server is one of the backend servers for a large LAN event I am administrating. It mostly handles the bots for our our internal IRC network, and we also use it for anything large we need to store on a whim (8TB storage). It is also running rtorrent. –  Matthieu Cartier Aug 5 '11 at 7:05
    
Large LAN event? Assembly? :D Ok, now that I got that said ... what does lsof -n -p 4754 tell you? –  Janne Pikkarainen Aug 5 '11 at 7:10

4 Answers 4

up vote 2 down vote accepted

State D is not "dead", it's blocked waiting on disk IO.

Something like "reboot -n -f" is more forceful and may work. Are you having filesystem trouble? The shutdown command writes a wtmp entry before telling init to shut down.

share|improve this answer
    
I'm not having filesystem trouble as far as I am aware, but it's starting to sound more and more like it... –  Matthieu Cartier Aug 5 '11 at 7:06
    
"reboot -n -f" should avoid the disk IO that shutdown is doing, and it should cause the kernel to reboot. Or crash. I suspect at least something will happen. :) –  akramer Aug 5 '11 at 7:11
    
Here goes nothing! –  Matthieu Cartier Aug 5 '11 at 7:14
    
That did it, thanks. Did seem to wreck the journal on /dev/sdb1, but all is well now. sprunge.us/ZGAS –  Matthieu Cartier Aug 5 '11 at 7:22
    
@neurolysis: A brave, brave man you are. :-) I have also used reboot -nf sometimes but then I've made sure I have physical access to server ... but perhaps you had someone available for that. –  Janne Pikkarainen Aug 5 '11 at 7:29

Enable the Magic SysRq key with:

echo "1" > /proc/sys/kernel/sysrq

and do the clean reboot with following steps:

  1. Alt + SysRq + e
  2. Alt + SysRq + u
  3. Alt + SysRq + i
  4. Alt + SysRq + b
share|improve this answer
    
Computer is now rebooted w/ reboot -n -f, but thanks. What caused it, however, eludes me. –  Matthieu Cartier Aug 5 '11 at 7:23
    
He only had access to the server via ssh, SysRq won't work. –  akramer Aug 5 '11 at 9:03
    
You can send the command key to sysrq-trigger: echo "key" > /proc/sysrq-trigger. –  quanta Aug 5 '11 at 9:13

shutdown is probably waiting to write to something like /dev/initctl -- use strace -p 4754 to see where it's hanging, then fix whatever is broken.

share|improve this answer
    
There is absolutely no output from strace. It states Process 4754 attached - interrupt to quit, but then there is nothing. –  Matthieu Cartier Aug 5 '11 at 6:55
    
Then you've got bigger problems than a machine that won't shut down. D state is only for processes that are waiting for an I/O related syscall to complete. –  womble Aug 5 '11 at 7:00

I have seen shutdown hanging if there are (stale) NFS mounts or server is connected to SAN. Here's a checklist for you:

  • Can you still see all the disk mounts with df -h (or otherwise access the mounts)?
  • Are there other processes than shutdown in D state?
  • Does dmesg return you something about I/O or is there even a nice little kernel panic waiting for you?
  • If there are processes using some SAN/NFS mount, can you stop those processes manually, verify with lsof that no files are open on that mount, and then umount that mount?
  • If btrace is installed, does btrace /your/disk return something?

Should everything else fail, the SysRq stanza provided by @quanta should work, BUT first make sure there's someone who can physically babysit the server if it for some reason does not boot cleanly.

share|improve this answer
    
Already rebooted now, but thanks. Don't think there was anything else in D state. Will see if there is anything interesting in the post-reboot dmesg now. I don't have any SAN/NFS mounts on the machine. –  Matthieu Cartier Aug 5 '11 at 7:30
    
Are disk quotas in use? I've seen them causing hangs with ext3 filesystem, though that was like seven years ago and with a server with tens of thousands of users ... –  Janne Pikkarainen Aug 5 '11 at 7:31
    
Nope. Only interesting stuff I see is here: sprunge.us/YXCf –  Matthieu Cartier Aug 5 '11 at 7:33
    
This drives me nuts. :-) Since you have 8 TB of data, is LVM in use? Perhaps you have LVM snapshots and those caused some kind of hickup during shutdown. Oh well, better stop guessing. –  Janne Pikkarainen Aug 5 '11 at 7:36
    
No LVM. This is troubling. Oh well! I guess I've just got to wait for the next exciting phone call I get to tell me the server room is on fire. :-) –  Matthieu Cartier Aug 5 '11 at 7:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.