2

One of our server admins asked me to take a look at this, and I am stumped - our /var partition is full, however I cannot seem to determine where the space has gone.

The following is the output of 'du /var -ah'

...
202M    var/

'df -h' however returns

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/gza-root  268M  108M  147M  43% /
tmpfs                 2.0G     0  2.0G   0% /lib/init/rw
udev                   10M   68K   10M   1% /dev
tmpfs                 2.0G     0  2.0G   0% /dev/shm
/dev/sda1             236M   20M  204M   9% /boot
/dev/mapper/gza-home  115G   15G   95G  14% /home
/dev/mapper/gza-tmp   380M   11M  350M   3% /tmp
/dev/mapper/gza-usr   4.7G  3.9G  610M  87% /usr
/dev/mapper/gza-var   2.9G  2.7G   26M 100% /var

I cannot seem to locate where the other 2.5GB has gone, any ideas or hints?

4 Answers 4

10

Try restarting services. It is possible something had a file open on /var when you deleted (rm unlink things, it ) it. The system will not release the space until everything closes their file handles for that file..

You might need to use lsof to find out what program still has the files open in /var.

Since it is /var, I would guess something still has a log file open.

2
  • 9
    lsof +L1 is your friend
    – derobert
    Jun 1, 2009 at 6:10
  • 1
    I would have prefered to see using lsof (and/or fuser) before just restarting services. With the tools to figure out why it broke you have a better handle on the next similar but not quite problem. Restarting services without knowing which is broken just seems to be a bit scatter shot. It may solve the problem but it may also cause some other issues in the process. Aug 18, 2009 at 19:54
3

There are a couple possible paths to take with this:

  1. Are you out of inodes rather than out of free blocks (actual space)? If there are a lot of small files, they may eat up all the inodes (effectively, entries in the file allocation table.) Once all the inodes are used up, you can't add a new file even though there's plenty of free blocks in the partition. This is a wildcard answer to check if there's no obvious smoking gun. inode exhaustion is a rare occurrence but it still happens (Blackboard 7.0 ...)
  2. du -s /var/* | sort -rn should give you a better idea of what is taking up space in /var as would a graphical utility like KDirStat. I don't recommend KDirStat for this issue since you're dealing with /var. Issues with /var tend to affect system stability compared to space issues with /home. Besides, in a dedicated server situation you shouldn't be running a GUI on the box in the first place making the notion of using a (local) GUI tool moot. In less critical situations, the squarified treemap output of tools like KDirStat, WinDirStat, and SequoiaView make problem identification trivial; the treemap is a great means of visualizing disk use.
  3. Is some service out of control and logging immense amounts of junk to /var/log? If your organizational culture accepts it, it might be valuable to set up a dedicated log host and have all machines write logs to it over the network. At the very least, make sure every log is appropriately rotated, compressed, and deleted according to your site's log retention policy. logrotate is your friend.
  4. You do have a log retention policy, right? ;) Seriously though, if you don't have one, set one. It can be a sentence - "We will keep logs for seven days worth of logs unless alternate arrangements are made with the service owner. In writing."
1
  • df -i will show you how many inodes you have left Jun 26, 2009 at 8:24
2

Do you have permissions to all the folders under /var? du does not report space used for things you don't have permissions to.

1
  • It's best to run du as root, that avoids this problem Jun 26, 2009 at 8:24
0

I wonder what your machine will tell you as soon as you ask it how many inodes it has left. The ability to store data on a partition is not solely a function of the number of free blocks...

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .