2

I have a server which I'm administrating remotely over SSH, for various reasons, it needs to be restarted. However, when I try to do so, shutdown goes to uninterruptible sleep almost immediately after sending the wall message.

$ ps aux | grep 'shutdown'
root      4754  0.0  0.1   1888   720 ?        D    07:26   0:00 shutdown -r 0 w

I have no physical access to the machine at the current time. What should I do? The server is still responding, and I can still log in.

4
  • Does the server have NFS mounts? Is it connected to SAN via Fibre Channel? Is there some kind of heartbeat system such as Heartbeat v2 installed? Aug 5, 2011 at 6:58
  • True, it's probably worth giving background: this server is one of the backend servers for a large LAN event I am administrating. It mostly handles the bots for our our internal IRC network, and we also use it for anything large we need to store on a whim (8TB storage). It is also running rtorrent. Aug 5, 2011 at 7:05
  • Large LAN event? Assembly? :D Ok, now that I got that said ... what does lsof -n -p 4754 tell you? Aug 5, 2011 at 7:10
  • I found this question trying to figure out what happened when you have an uninterruptible-sleep process and try to shut down. I expect now I know.
    – Daniel H
    Feb 3, 2017 at 22:52

4 Answers 4

3

State D is not "dead", it's blocked waiting on disk IO.

Something like "reboot -n -f" is more forceful and may work. Are you having filesystem trouble? The shutdown command writes a wtmp entry before telling init to shut down.

6
  • I'm not having filesystem trouble as far as I am aware, but it's starting to sound more and more like it... Aug 5, 2011 at 7:06
  • "reboot -n -f" should avoid the disk IO that shutdown is doing, and it should cause the kernel to reboot. Or crash. I suspect at least something will happen. :)
    – akramer
    Aug 5, 2011 at 7:11
  • Here goes nothing! Aug 5, 2011 at 7:14
  • That did it, thanks. Did seem to wreck the journal on /dev/sdb1, but all is well now. sprunge.us/ZGAS Aug 5, 2011 at 7:22
  • @neurolysis: A brave, brave man you are. :-) I have also used reboot -nf sometimes but then I've made sure I have physical access to server ... but perhaps you had someone available for that. Aug 5, 2011 at 7:29
4

Enable the Magic SysRq key with:

echo "1" > /proc/sys/kernel/sysrq

and do the clean reboot with following steps:

  1. Alt + SysRq + e
  2. Alt + SysRq + u
  3. Alt + SysRq + i
  4. Alt + SysRq + b
3
  • Computer is now rebooted w/ reboot -n -f, but thanks. What caused it, however, eludes me. Aug 5, 2011 at 7:23
  • 1
    He only had access to the server via ssh, SysRq won't work.
    – akramer
    Aug 5, 2011 at 9:03
  • 1
    You can send the command key to sysrq-trigger: echo "key" > /proc/sysrq-trigger.
    – quanta
    Aug 5, 2011 at 9:13
1

shutdown is probably waiting to write to something like /dev/initctl -- use strace -p 4754 to see where it's hanging, then fix whatever is broken.

2
  • There is absolutely no output from strace. It states Process 4754 attached - interrupt to quit, but then there is nothing. Aug 5, 2011 at 6:55
  • Then you've got bigger problems than a machine that won't shut down. D state is only for processes that are waiting for an I/O related syscall to complete.
    – womble
    Aug 5, 2011 at 7:00
1

I have seen shutdown hanging if there are (stale) NFS mounts or server is connected to SAN. Here's a checklist for you:

  • Can you still see all the disk mounts with df -h (or otherwise access the mounts)?
  • Are there other processes than shutdown in D state?
  • Does dmesg return you something about I/O or is there even a nice little kernel panic waiting for you?
  • If there are processes using some SAN/NFS mount, can you stop those processes manually, verify with lsof that no files are open on that mount, and then umount that mount?
  • If btrace is installed, does btrace /your/disk return something?

Should everything else fail, the SysRq stanza provided by @quanta should work, BUT first make sure there's someone who can physically babysit the server if it for some reason does not boot cleanly.

5
  • Already rebooted now, but thanks. Don't think there was anything else in D state. Will see if there is anything interesting in the post-reboot dmesg now. I don't have any SAN/NFS mounts on the machine. Aug 5, 2011 at 7:30
  • Are disk quotas in use? I've seen them causing hangs with ext3 filesystem, though that was like seven years ago and with a server with tens of thousands of users ... Aug 5, 2011 at 7:31
  • Nope. Only interesting stuff I see is here: sprunge.us/YXCf Aug 5, 2011 at 7:33
  • This drives me nuts. :-) Since you have 8 TB of data, is LVM in use? Perhaps you have LVM snapshots and those caused some kind of hickup during shutdown. Oh well, better stop guessing. Aug 5, 2011 at 7:36
  • No LVM. This is troubling. Oh well! I guess I've just got to wait for the next exciting phone call I get to tell me the server room is on fire. :-) Aug 5, 2011 at 7:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .