up vote 0 down vote favorite
share [g+] share [fb]

I have a server which I'm administrating remotely over SSH, for various reasons, it needs to be restarted. However, when I try to do so, shutdown goes to uninterruptible sleep almost immediately after sending the wall message.

$ ps aux | grep 'shutdown'
root      4754  0.0  0.1   1888   720 ?        D    07:26   0:00 shutdown -r 0 w

I have no physical access to the machine at the current time. What should I do? The server is still responding, and I can still log in.

link|improve this question
Does the server have NFS mounts? Is it connected to SAN via Fibre Channel? Is there some kind of heartbeat system such as Heartbeat v2 installed? – Janne Pikkarainen Aug 5 '11 at 6:58
True, it's probably worth giving background: this server is one of the backend servers for a large LAN event I am administrating. It mostly handles the bots for our our internal IRC network, and we also use it for anything large we need to store on a whim (8TB storage). It is also running rtorrent. – Matthieu Cartier Aug 5 '11 at 7:05
Large LAN event? Assembly? :D Ok, now that I got that said ... what does lsof -n -p 4754 tell you? – Janne Pikkarainen Aug 5 '11 at 7:10
feedback

4 Answers

up vote 1 down vote accepted

State D is not "dead", it's blocked waiting on disk IO.

Something like "reboot -n -f" is more forceful and may work. Are you having filesystem trouble? The shutdown command writes a wtmp entry before telling init to shut down.

link|improve this answer
I'm not having filesystem trouble as far as I am aware, but it's starting to sound more and more like it... – Matthieu Cartier Aug 5 '11 at 7:06
"reboot -n -f" should avoid the disk IO that shutdown is doing, and it should cause the kernel to reboot. Or crash. I suspect at least something will happen. :) – akramer Aug 5 '11 at 7:11
Here goes nothing! – Matthieu Cartier Aug 5 '11 at 7:14
That did it, thanks. Did seem to wreck the journal on /dev/sdb1, but all is well now. sprunge.us/ZGAS – Matthieu Cartier Aug 5 '11 at 7:22
@neurolysis: A brave, brave man you are. :-) I have also used reboot -nf sometimes but then I've made sure I have physical access to server ... but perhaps you had someone available for that. – Janne Pikkarainen Aug 5 '11 at 7:29
show 1 more comment
feedback
up vote 2 down vote

Enable the Magic SysRq key with:

echo "1" > /proc/sys/kernel/sysrq

and do the clean reboot with following steps:

  1. Alt + SysRq + e
  2. Alt + SysRq + u
  3. Alt + SysRq + i
  4. Alt + SysRq + b
link|improve this answer
Computer is now rebooted w/ reboot -n -f, but thanks. What caused it, however, eludes me. – Matthieu Cartier Aug 5 '11 at 7:23
He only had access to the server via ssh, SysRq won't work. – akramer Aug 5 '11 at 9:03
You can send the command key to sysrq-trigger: echo "key" > /proc/sysrq-trigger. – quanta Aug 5 '11 at 9:13
feedback
up vote 1 down vote

shutdown is probably waiting to write to something like /dev/initctl -- use strace -p 4754 to see where it's hanging, then fix whatever is broken.

link|improve this answer
There is absolutely no output from strace. It states Process 4754 attached - interrupt to quit, but then there is nothing. – Matthieu Cartier Aug 5 '11 at 6:55
Then you've got bigger problems than a machine that won't shut down. D state is only for processes that are waiting for an I/O related syscall to complete. – womble Aug 5 '11 at 7:00
feedback
up vote 1 down vote

I have seen shutdown hanging if there are (stale) NFS mounts or server is connected to SAN. Here's a checklist for you:

  • Can you still see all the disk mounts with df -h (or otherwise access the mounts)?
  • Are there other processes than shutdown in D state?
  • Does dmesg return you something about I/O or is there even a nice little kernel panic waiting for you?
  • If there are processes using some SAN/NFS mount, can you stop those processes manually, verify with lsof that no files are open on that mount, and then umount that mount?
  • If btrace is installed, does btrace /your/disk return something?

Should everything else fail, the SysRq stanza provided by @quanta should work, BUT first make sure there's someone who can physically babysit the server if it for some reason does not boot cleanly.

link|improve this answer
Already rebooted now, but thanks. Don't think there was anything else in D state. Will see if there is anything interesting in the post-reboot dmesg now. I don't have any SAN/NFS mounts on the machine. – Matthieu Cartier Aug 5 '11 at 7:30
Are disk quotas in use? I've seen them causing hangs with ext3 filesystem, though that was like seven years ago and with a server with tens of thousands of users ... – Janne Pikkarainen Aug 5 '11 at 7:31
Nope. Only interesting stuff I see is here: sprunge.us/YXCf – Matthieu Cartier Aug 5 '11 at 7:33
This drives me nuts. :-) Since you have 8 TB of data, is LVM in use? Perhaps you have LVM snapshots and those caused some kind of hickup during shutdown. Oh well, better stop guessing. – Janne Pikkarainen Aug 5 '11 at 7:36
No LVM. This is troubling. Oh well! I guess I've just got to wait for the next exciting phone call I get to tell me the server room is on fire. :-) – Matthieu Cartier Aug 5 '11 at 7:37
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.