I have a server which I'm administrating remotely over SSH, for various reasons, it needs to be restarted. However, when I try to do so, shutdown goes to uninterruptible sleep almost immediately after sending the wall message.

$ ps aux | grep 'shutdown'
root      4754  0.0  0.1   1888   720 ?        D    07:26   0:00 shutdown -r 0 w

I have no physical access to the machine at the current time. What should I do? The server is still responding, and I can still log in.

  • Does the server have NFS mounts? Is it connected to SAN via Fibre Channel? Is there some kind of heartbeat system such as Heartbeat v2 installed? Aug 5 '11 at 6:58
  • True, it's probably worth giving background: this server is one of the backend servers for a large LAN event I am administrating. It mostly handles the bots for our our internal IRC network, and we also use it for anything large we need to store on a whim (8TB storage). It is also running rtorrent. Aug 5 '11 at 7:05
  • Large LAN event? Assembly? :D Ok, now that I got that said ... what does lsof -n -p 4754 tell you? Aug 5 '11 at 7:10
  • I found this question trying to figure out what happened when you have an uninterruptible-sleep process and try to shut down. I expect now I know.
    – Daniel H
    Feb 3 '17 at 22:52

State D is not "dead", it's blocked waiting on disk IO.

Something like "reboot -n -f" is more forceful and may work. Are you having filesystem trouble? The shutdown command writes a wtmp entry before telling init to shut down.

  • I'm not having filesystem trouble as far as I am aware, but it's starting to sound more and more like it... Aug 5 '11 at 7:06
  • "reboot -n -f" should avoid the disk IO that shutdown is doing, and it should cause the kernel to reboot. Or crash. I suspect at least something will happen. :)
    – akramer
    Aug 5 '11 at 7:11
  • Here goes nothing! Aug 5 '11 at 7:14
  • That did it, thanks. Did seem to wreck the journal on /dev/sdb1, but all is well now. sprunge.us/ZGAS Aug 5 '11 at 7:22
  • @neurolysis: A brave, brave man you are. :-) I have also used reboot -nf sometimes but then I've made sure I have physical access to server ... but perhaps you had someone available for that. Aug 5 '11 at 7:29

Enable the Magic SysRq key with:

echo "1" > /proc/sys/kernel/sysrq

and do the clean reboot with following steps:

  1. Alt + SysRq + e
  2. Alt + SysRq + u
  3. Alt + SysRq + i
  4. Alt + SysRq + b
  • Computer is now rebooted w/ reboot -n -f, but thanks. What caused it, however, eludes me. Aug 5 '11 at 7:23
  • 1
    He only had access to the server via ssh, SysRq won't work.
    – akramer
    Aug 5 '11 at 9:03
  • 1
    You can send the command key to sysrq-trigger: echo "key" > /proc/sysrq-trigger.
    – quanta
    Aug 5 '11 at 9:13

shutdown is probably waiting to write to something like /dev/initctl -- use strace -p 4754 to see where it's hanging, then fix whatever is broken.

  • There is absolutely no output from strace. It states Process 4754 attached - interrupt to quit, but then there is nothing. Aug 5 '11 at 6:55
  • Then you've got bigger problems than a machine that won't shut down. D state is only for processes that are waiting for an I/O related syscall to complete.
    – womble
    Aug 5 '11 at 7:00

I have seen shutdown hanging if there are (stale) NFS mounts or server is connected to SAN. Here's a checklist for you:

  • Can you still see all the disk mounts with df -h (or otherwise access the mounts)?
  • Are there other processes than shutdown in D state?
  • Does dmesg return you something about I/O or is there even a nice little kernel panic waiting for you?
  • If there are processes using some SAN/NFS mount, can you stop those processes manually, verify with lsof that no files are open on that mount, and then umount that mount?
  • If btrace is installed, does btrace /your/disk return something?

Should everything else fail, the SysRq stanza provided by @quanta should work, BUT first make sure there's someone who can physically babysit the server if it for some reason does not boot cleanly.

  • Already rebooted now, but thanks. Don't think there was anything else in D state. Will see if there is anything interesting in the post-reboot dmesg now. I don't have any SAN/NFS mounts on the machine. Aug 5 '11 at 7:30
  • Are disk quotas in use? I've seen them causing hangs with ext3 filesystem, though that was like seven years ago and with a server with tens of thousands of users ... Aug 5 '11 at 7:31
  • Nope. Only interesting stuff I see is here: sprunge.us/YXCf Aug 5 '11 at 7:33
  • This drives me nuts. :-) Since you have 8 TB of data, is LVM in use? Perhaps you have LVM snapshots and those caused some kind of hickup during shutdown. Oh well, better stop guessing. Aug 5 '11 at 7:36
  • No LVM. This is troubling. Oh well! I guess I've just got to wait for the next exciting phone call I get to tell me the server room is on fire. :-) Aug 5 '11 at 7:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.