Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

On CentOS 6.5, du -sk /home and sh -c (du -sk \/home) 2>/dev/null run because a scheduled disk usage for /home, but du is causing 100% CPU usage. Plus, before it's even finished, another du gets run and it piles up, bringing the system down to its knees.

I've since disabled the automated checks, but I can't run du -sk /home manually either. du runs fine (fast) on other directories (though /home is the largest - 4912620 - it shouldn't take as long as it does)

I would like to find out why is DU causing high CPU usage on /home


UPDATE 1: A full restart (host+guest) solved the problem.

UPDATE 2: Problem came back. While du is running, CPU usage is split nearly half and half between (us) and (sy). (wa) never creeps over 1%. Also, IOWAIT shows only disk writes?

share|improve this question
    
It's likely to be iowait. Try running top or vmstat or iotop while doing du. –  Eric DANNIELOU Dec 28 '13 at 17:32
    
@EricDANNIELOU: surely IO wait would cause the du to relinquish the cpu ? –  Iain Dec 28 '13 at 17:36
    
@lain I'm not sure about what you mean. 100% cpu can be caused by iowait (wa), userland (us) or kernel (sy). –  Eric DANNIELOU Dec 28 '13 at 17:40
    
Iowait can definitely cause the CPU load to creep up. If the disks are slow, particularly network disks, then this is more common. –  Jason Ilicic Jan 7 at 11:45
    
Disk was an LVM volume for the guest KVM OS. OpCenter crew ran tests and found no physical problems with the disk subsystem. –  Gaia Jan 7 at 19:42
add comment

2 Answers

I can't tell you why exactly, but here's what I'd do to narrow down what is going on:

cd /home
for i in * ; do echo "Starting in ${i}..." ; du -s ${i} ; done

This will run "du -s" for each command inside the /home directory, and show you what it is about to do. This is also assuming you are using sh or bash as your shell.

When you find which directory is causing the problem, move in one directory more, and see what happens there. Also remember this will not check for hidden ("dot") directories at the level you start the for loop from. You may need to check those as well, or modify the glob expression to include them.

share|improve this answer
    
There seem to be no abnormal delays or CPU spikes when I run the above command. If I run it for only /home with --max-depth=1 I also don't get a delay/spike. It's only when I run du -sk /home. –  Gaia Jan 26 at 16:43
    
Do you have any .dot directories in /home? –  Michael Graff Jan 27 at 21:24
    
Directories named ".dot" ? –  Gaia Jan 28 at 17:48
    
No, hidden files or directories inside /home –  Michael Graff Jan 28 at 17:54
    
yes, .htaccess, etc –  Gaia Jan 28 at 19:20
show 5 more comments
up vote 0 down vote accepted

I finally discovered what the problem was when I came across this post.

root> modprobe vhost_net root> echo vhost_net > /etc/modules

For some reason, the virtio interfaces were not loading the driver before. All is good now.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.