Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

How do I safely free some space on my root filesystem?

df says:

Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              9.7G  9.2G     0 100% /
tmpfs                 3.9G     0  3.9G   0% /lib/init/rw
udev                   10M  244K  9.8M   3% /dev
tmpfs                 3.9G  620K  3.9G   1% /dev/shm
/dev/md3              1.8T  327G  1.4T  19% /home

======================================================================

du gives me:

root@sbs691:/# ls | xargs du -hs
5.8M    bin
13M     boot
244K    dev
8.0K    dotdeb.gpg
8.1M    etc
281G    home
17M     lib
3.7M    lib32
0       lib64
16K     lost+found
8.0K    media
4.0K    mnt
157M    opt
du: cannot access `proc/31735/task/31735/fd/4': No such file or directory
du: cannot access `proc/31735/task/31735/fdinfo/4': No such file or directory
du: cannot access `proc/31735/fd/4': No such file or directory
du: cannot access `proc/31735/fdinfo/4': No such file or directory
0       proc
41M     root
4.0K    run
14M     sbin
4.0K    selinux
4.0K    srv
0       sys
129M    tmp
2.2G    usr
431M    var

=====================================================

Update after 1st comment, du of /var:

root@sbs691:/var# ls | xargs du -hs
4.8M    backups
149M    cache
4.0K    games
265M    lib
4.0K    local
12K     lock
14M     log
4.0K    mail
4.0K    opt
200K    run
24K     spool
4.0K    tmp
16K     www

Issue is been now resolved temporaryily by restarting nginx

service nginx restart

After restarting

root@sbs691:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              9.7G  6.0G  3.3G  65% /
tmpfs                 3.9G     0  3.9G   0% /lib/init/rw
udev                   10M  244K  9.8M   3% /dev
tmpfs                 3.9G  620K  3.9G   1% /dev/shm
/dev/md3              1.8T  339G  1.4T  20% /home
share|improve this question
    
Somehow these values don't add up. Do you have some large files in /? The du listing shows only about ~3 GB worth of data below / except /home, which is on another FS. –  Sven Jul 19 '13 at 7:24

6 Answers 6

up vote 6 down vote accepted

You've probably got a file that has been deleted but which is being held open (via the fd) by the daemon that is writing to it. Take a look at the output of

lsof +L1

which will list such files. When you know the file, you just need to figure out which daemon and then tell it to restart or reopen it's log file.


From the comments: It turned out that the OP had deleted files in /var and had not restarted the daemons (specifically nginx) that were writing to the files.

share|improve this answer
    
This, or some hidden directory/files in / are the most likely to my mind. Since he's doing all the displays based on the output of ls anything that's a .* file won't show up. I still like large open file better, though. If you have files like that in / it's usually an indication of something else. –  Scott Pack Jul 19 '13 at 13:11
    
link, Output the given command is in that link. I dont understand what to do next, can you guide me futher. ? –  Kunal Panchal Jul 20 '13 at 5:08
    
@KunalPanchal: I would say you need to restart nginx. –  Iain Jul 20 '13 at 5:12
    
@lain Thanks :) that made the space free :D –  Kunal Panchal Jul 20 '13 at 5:22
    
@KunalPanchal: great = now you want to go through that list and look at the lines with (deleted) add the nginx ones to your logrotate configuration. –  Iain Jul 20 '13 at 5:25

Your var folder has about 431M of data, I'd start by clearing out your logs.

share|improve this answer
    
Have added the result of disk-usage inside /var folder –  Kunal Panchal Jul 19 '13 at 6:59

The fastest (and somewhat very hackish) way is to boot the server into a rescue media and move /var and maybe even /usr to the /home file system and symlink it back.

The correct(TM) approach is to resize the partitions in a sensible way, but since you use a software RAID, that's not exactly easy or fast.

share|improve this answer
  • Since it's a Debian system, you have probably accumulated lots of downloaded packages in /var/cache/apt which may fill up your /var. Unless you have a good reason to keep them, issue apt-get clean as root to remove them.

  • You could use parted or similar tools RAID tools to change your partition sizes. Remember to have a current backup ready before that!

  • As a quick resort, which I cannot really recommend, you could also move your /usr folder to the /home partition and symlink from root. Remember to have a current backup ready before that! My work order would be:

    1. If possible, shutdown and reboot from a rescue system.
    2. Copy the folder with all permissions, links etc.: # cp -a /usr /home/root-usr (The paths will differ when you work from a rescue system! Make sure that there is no user named root-usr! ;-))
    3. Compare old and new folder to be sure copying worked: # diff -r /usr /home/root-usr
    4. Remove the old directory and create a symlink: # rm -rf /usr; ln -s /home/root-usr /usr (Be sure the symlink is correct when doing this from a rescue system! You might have to chroot to accomplish that.)
    5. Reboot into your system.

Always be careful. You should know what you do.

share|improve this answer
1  
1. /var/cache is ~150 MB. 2. He uses a software RAID, just toying around with parted doesn't help. –  Sven Jul 19 '13 at 7:26
    
@SvW 1. Okay, I started my answer before the update. apt-get clean won't help much. 2. You're absolutely right, I overlooked the /dev/md*. I will modify my answer. –  Dubu Jul 19 '13 at 7:34

Your numbers don't add up. You might have data on your root fs you don't "see".

I can only count ~3G on the du output of your root. I suspect you have either:

  • at some point copied the data in /home to a different filesystem and changed the mount-point, but failed to delete the original copy - this of course still takes up space. You just can't see it now that you have changed the mount point.

    To find this kind of phantom of the opera data, one does compare totals of du on / (without crossing fs boundaries), and df.

  • Or created some huge files in /, in which case you need to move/delete them - they don't belong there.

share|improve this answer
    
Or he just has some large files directly in /. We can't tell as the du output he gives doesn't include /. –  Sven Jul 19 '13 at 7:54
    
Oops. right. Updating. –  Roman Jul 19 '13 at 8:33

Is /home later added/mounted to the system?

if you have old data in md1 and in his /home directory, and you mounted the new partition (md3) into /home directory with nonempty option the /home can hide the original data which placed and reserv his size in md1, but you can not see becuse hided by this other patition,

If you may interrupt the server operations bring down the system into rescue mode

init 1

umount /home system, and look for files whichs left in the original root partition (md1)

ls -la /home
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.