Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to track down where all my hard disk space went. I have 1TB total space and, according to my system, 433GB used. However, I know I have more as I don't have any large files or folders that I'm aware of. The following shows the top 10 directories.

du -cks *|sort -rn|head

It says the /var folder is taking up 480GB but when I go into that folder, there is nothing sizeable (i run this from root or '/' w/ su permissions).

What am I missing? Are there any hidden files that that code doesn't pick up? For reference, I am using opensuse 11.3, if that makes a diff.

share|improve this question
    
What does "there is nothing sizeable" mean? Keep checking all the subfolders in /var: # du -s /var/* | sort -rn | cut -f2 | xargs -d '\n' du -sh | head? –  quanta Aug 5 '11 at 2:56
    
How does your total compare with your top n? –  Andrew Aug 5 '11 at 2:59
    
quanta, your answer helped me pin it down...turned out it was the /var/lib/mysql folder...now, I need to figure out what to do w/ that....if you put your solution as an answer below I will accept it. thx! –  user70981 Aug 6 '11 at 0:20
add comment

4 Answers

If you do the recursive du as suggested, and calculate all space for the entire var tree, and it doesn't add up to the total used, you might want to check for an unlinked file handle that has been left open. This can happen with improper logrotate scripts that delete or move a log that is still actively being written to. You unlink the file (delete the inode, therefore ls doesn't report it anymore) but the process is still writing to it.

lsof is not typically a standard install, so you will have to install it through whatever means OpenSuse uses, then run: lsof +L -a /var

You'll see output like

COMMAND   PID      USER     FD    TYPE DEVICE SIZE/OFF NLINK   NODE NAME
httpd     32287    apache   14w   REG  253,7  3020256     1    121  /var/log/httpd/access_log

There will be many files show up, but what you are looking for is one that has a large value in the size column, and a 0 in the NLINK column (and isn't marked deleted). That file is still being written to, but is not linked any more.

You can then correct it by terminating the process that has the unlinked file handle open.

share|improve this answer
add comment

All that means is that you haven't found the files where all the space is being used. It should be trivial to find, because if du is finding it at the top level, then it's sitting on the filesystem.

If a du -s /directory shows space used, but a du -s /directory/* doesn't, then the problem is in hidden ("starts with a .") files or directories directly in /directory. If there is a discrepancy between df and du (larger than can be explained by rounding differences) then you've got deleted but unreleased files consuming space.

share|improve this answer
add comment

quanta, your answer helped me pin it down...turned out it was the /var/lib/mysql folder...now, I need to figure out what to do w/ that...

Have you used MySQL replication? If so, maybe the binary logs is filling up your hard disk. It can be remove automatically by setting the expire_logs_days from mysql> prompt:

mysql> set global expire_logs_days = 7;

(also change the my.cnf to make it permanent)

You should set it no lower than maximum number of days your slaves might lag behind the master.

If not, continue checking for all databases in /var/lib/mysql.

share|improve this answer
add comment

Do you have anything mounted below /var?

There might be data hidden "under" a mountpoint.

Example:

mkdir /var/test
dd if=/dev/zero of=/var/test/data.dat bs=1M count=16

Now mount something at /var/test - you will not see where the missing 16MB are.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.