Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a problem. I run out of disk space all the time. I don't know exactly what files consume all the space. It seems that there are bunch of little files, so it's hard to find with du command. Every time I find something to get rid of to gain some free space (mails, logs, old archives and rpms), disk is full again after some time. Space is stolen from system partition (/). Results of df command:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              4061540   3848404      3492 100% /
/dev/sda3              4061572   2441348   1410576  64% /home
tmpfs                   517636         0    517636   0% /dev/shm
/dev/sdb1             10317828   1894904   7898808  20% /var/lib/mysql

What would you recommend in this situation?

It would be great if you can also enumerate a list of cases when you encounter excessive or unexpected space consumption. It might help to resolve problem

OS: CentOS 5.

share|improve this question

5 Answers 5

up vote 12 down vote accepted

Here's my standard "find what's eating the space" regime:

  1. du -hx --max-depth=1 / -- look for what's eating the space. Examine the largest subdirectory (say du -hx --max-depth=1 /var) until you find some space hogs. Logs (in /var/log) are common culprits (which you should deal with using logrotate), as are the cruft that yum likes to keep around (repackaged RPMs). If you can identify any repeat offenders, work out a way to automate whatever maintenance work you're doing to clean up repeatedly.
  2. lsof -n |grep deleted -- look for large files that have been deleted. Identify which process is holding them open, and kill it or tell it to recycle it's handles (if they're log files, kill -HUP <pid> will often do the trick) if you can.
  3. My third line of attack is to resize partitions or allocate new partitions to various mounts, although in your case, since you're not using LVM, that's tricky. Consider taking a maintenance to turn /dev/sdb1 into a VG and allocate from there.
  4. Upgrade. Disk space is so cheap, that if you spend an hour of your time hunting down what's causing space consumption, you just "spent" the time required to pay for a new hard drive, so buy new drives rather than spend too much time hunting down problems.
share|improve this answer
    
Great and exhaustive answer. Thanks –  altern Nov 18 '09 at 13:41
    
Whats wrong with simply thanking? :) –  Karolis T. Nov 18 '09 at 14:57
    
ITYM "du", not "df" –  David Mackintosh Nov 18 '09 at 15:07
    
@Karolis: You can save yourself all that typing by just clicking the little arrow... @David: D'oh, thanks for catching. That's what I get for SFing before my first tea of the evening. –  womble Nov 18 '09 at 15:39
    
Wow, what great ideas you have. :P –  kmarsh Nov 18 '09 at 16:35

If its not already, move the /var mount point to another partition. File system /var exists to prevent the root FS from filling up.

Same for /tmp, if not already a seperate partition.

You may have a process that deletes files but holds them open. Space will not be reclaimed until closed/process exit. You can use the lsof command to find out what processes are holding what files open. Command lsof may not be in your standard Centos install, you may have to add the package to get it.

share|improve this answer

One addition to womble's answer - if you find you're getting a lot of large log files, consider using logrotate (which is installed by default on most distros) to keep them down to size. If there is one particula log that's taking up most of the space, you should also check it for repeated errors and make sure that the service which produces the log isn't set to too high a verbosity level.

share|improve this answer
sudo du --all --one-file-system / | xdu -n
share|improve this answer

Example: unmount /home, write 10GB file there, mount /home back again. The space will be consumed, and df shows nothing: mount-point hides files inside that folder.

Check this out: this can happen if mounting errors have ever occurred, and some files were placed there.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.