5

enter image description here

I have several processes stuck on uninterruptible sleep statuses, all seemingly stemming from auplink /var/lib/docker/aufs/mnt. It's something docker related and it's waiting on an I/O that will never complete -- I get that, but how do I determine the exact cause? How can I know what I/O it is waiting on? Also, is there really no way of making these stuck processes go away without a hard reboot?

8
  • Can you strace the pid and put the output in the question? strace -p <pid> Also, lsof |grep <pid> will show open file handles... also, you could try gdb.
    – Tim
    Nov 20, 2018 at 2:26
  • strace displays nothing after being attached. Similarly, gdb shows nothing. lsof gets stuck (you can see from above screenshot that it also goes into uninterruptible sleep status`. )
    – TtT23
    Nov 20, 2018 at 2:37
  • what about iostat, vmstat, smartctl? Maybe that part of hd is bad. It sounds crazy but it can be. And maybe docker process goes into infinite loop or something.
    – titus
    Nov 20, 2018 at 2:43
  • iostat, vmstat reports nothing out of ordinary.. I doubt that it's the HD, it's a VM on the cloud (if it is, well then I'd need to go buy a lottery ticket!)
    – TtT23
    Nov 20, 2018 at 2:54
  • 1
    AUFS is not a reliable storage backend, which is part of the reason why the kernel developers refused to add it to Linux, and even Ubuntu had to drop it. Switch the Docker storage to overlay2. Nov 20, 2018 at 3:36

1 Answer 1

7

You can see stack of the process:

cat /proc/<process pid>/stack

which will give you information on what it was doing when it ended up in D-state.

echo w > /proc/sysrq-trigger; dmesg

will tell kernel to report all stack traces for D-state processes in dmesg buffer.

Processes in D-state cannot be killed. There are situations where process stays in D-state for long time but occasionally finishes I/O and is interruptible for short period of time and then goes back to the same I/O activity and ends up in D-state again. Then with

while (true); do kill -9 PID; done

there is a little chance of delivering KILL signal while process is interruptible.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.