Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am on amazon ec2 ubuntu 11.04 large instance with a 150GB volume mounted for the database (ext4).

The cpu usage is VERY low but the load average has been consistently at 2.0 for about a day now. I used to have the database partition on a 40GB volume and did not have this problem.

iostat tells me we are spending a lot of time waiting for io:

:~$ iostat 1 2
Linux 2.6.38-11-virtual (flashgroup)    04/05/2012      _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.16    0.09    2.62    1.11    2.09   86.92

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
xvdap1            3.45         0.88        18.59    9137065  192742888
xvdb              4.47         2.84        24.17   29479675  250638760
xvdh             10.62        19.95        88.05  206811124  912892410
xvdf              0.18         0.00         1.93       1378   19971464
xvdg              0.00         0.00         0.00        656          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.22    0.00    1.92   42.58    3.02   47.25

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
xvdap1            0.00         0.00         0.00          0          0
xvdb             43.00         0.00       172.00          0        172
xvdh              0.00         0.00         0.00          0          0
xvdf             49.00         0.00       288.00          0        288
xvdg              0.00         0.00         0.00          0          0

The product is performing just fine and the database is not logging any slow queries...

How should I go about debugging this?

EDIT:

It turns out that none of the volumes are exhibiting high latency and all other aspects of the system seem to be healthy. Wikipedia tells me that linux includes processes in the un-interruptable state in the load average. ps tells me that there are two hung mount commands are in such state:

ps auxww | grep " D"
root     21557  0.0  0.0   9904   760 ?        D    Apr03   0:00 umount db /dev/xvdh
root     26428  0.0  0.0  16456   912 ?        D    Apr03   0:00 mount /dev/xvdh /mnt/db

I am afraid to kill these (probably would not even work if I tried) so I think that this instance is sick and needs a restart. Thanks for your help!

share|improve this question
1  
In theory, any volume could result in the high iowait, however, from the second output - there is a good chance it is xvdb or xvdf. Following traditional naming - xvdb is likely ephemeral storage, while xvdf should be EBS. Run iostat -x to get more details about each volume - and look at the await column to narrow down the problem. EBS volumes do sometimes exhibit strange latencies, and, if that is the case, the best solution is often to simply create a new EBS volume. Another solution may be to look at setting up multiple EBS volume as a RAID. –  cyberx86 Apr 6 '12 at 1:22
    
Thanks for the advice. Followed your advice and updated post. –  Hersheezy Apr 8 '12 at 13:56
    
If restarting the node fixed the issue, and you think it was related to those two processes, you should probably answer your own question so it gets closed. –  EightBitTony May 4 '12 at 9:11

1 Answer 1

up vote 0 down vote accepted

It turns out that none of the volumes are exhibiting high latency and all other aspects of the system seem to be healthy. Wikipedia tells me that linux includes processes in the un-interruptable state in the load average. ps tells me that there are two hung mount commands are in such state:

ps auxww | grep " D"
root     21557  0.0  0.0   9904   760 ?        D    Apr03   0:00 umount db /dev/xvdh
root     26428  0.0  0.0  16456   912 ?        D    Apr03   0:00 mount /dev/xvdh /mnt/db

Restarting the instance got rid of these hung processed and the load average is back to normal.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.