3

I have a virtual machine running an application. After some time, its load starts increasing and the machine becomes unresponsive to commands.

After monitoring the disks usage, I found out a mounted SSD disk(AWS EBS General SSD) used by application is 100% busy with no read and writes from nmon.

Snapshot of nmon is: enter image description here

Snapshot of top is: enter image description here

I also tried using iotop to find the read and writes but there is no process doing a lot of read/writes.

Output of iostat -x 2 5 is: enter image description here

Besides this, All ps commands are getting stuck and enter into D state.

How to find the cause of 100% disk busy/utilized?

Edit: The mounted partition is using XFS file system.

6
  • 2
    For VM’s with otherwise inexplicable performance issues it may be worthwhile to look at “stealtime” to see if a VM is suffering from noisy neighbours. If that is the case ; simply power the VM down and restart it after a couple of minutes. Usually that will restart the VM on a different hypervisor; mitigating performance issues.
    – HBruijn
    Jul 19, 2018 at 23:47
  • top output shows 0 steal time Jul 19, 2018 at 23:47
  • 2
    In this case I would also stop the instance and copy its storage to a new storage volume, then restart the instance with the new volume. Jul 20, 2018 at 0:39
  • 2
    The instance and/or volume is broken. Nuke it and start again.
    – womble
    Jul 20, 2018 at 1:43
  • There are a number of volume metrics exposed by EBS volumes. One of these may provide some useful insight, as may dmesg. It is possible, even likely, that this condition reflects bad hardware or bad Nitro behavior that can only be mediated by moving to new equipment, but I'm a little disappointed by what feels like premature and speculative throwing up of hands in some of the previous comments. It should not be the instance and the volume, but it may very well be one of them. Trashing the whole thing doesn't provide much of a learning opportunity. Jul 20, 2018 at 10:11

1 Answer 1

1

Use next script for view - who read/wrete to disk:

#!/bin/sh

dmesg -c >/dev/null 2>&1
echo 1 > /proc/sys/vm/block_dump

# Timeout
sleep 60

# Disable block dumping
echo 0 > /proc/sys/vm/block_dump


# Header
printf "%10s %15s %10s %10s %10s\n" CONUT COMMAND PID ACTION DEVICE

# Hide the eyes child! It gets ugly from here on
IFS="
"

for line in $(dmesg | awk '{if ( $3 ~ "READ" || $3 ~ "WRITE" ) { print  $2 " " $3 " " $7}}'| sort |uniq -c |sort -nr );
             do
                num=$(echo $line | awk '{print $1}')
                command=$(echo $line | awk '{ print $2 }' | sed -re 's/\([[:digit:]]+\)://')
                pid=$(echo $line| awk -F'[()]' '{ print $2 }')
                action=$(echo $line | awk '{ print $3 }')
                device=$(echo $line | awk '{ print $4 }')

                printf "%10s %15s %10s %10s %10s\n" "$num" "$command" "$pid" "$action" "$device"
done
# EOF

Note: Depending on OS and dmesg output, you may have to change the fields in the "for line in" line"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .