Linux Debian Primary Partition (/) of Server Full, How to Make space for it

Question

How do I safely free some space on my root filesystem?

df says:

Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              9.7G  9.2G     0 100% /
tmpfs                 3.9G     0  3.9G   0% /lib/init/rw
udev                   10M  244K  9.8M   3% /dev
tmpfs                 3.9G  620K  3.9G   1% /dev/shm
/dev/md3              1.8T  327G  1.4T  19% /home

======================================================================

du gives me:

root@sbs691:/# ls | xargs du -hs
5.8M    bin
13M     boot
244K    dev
8.0K    dotdeb.gpg
8.1M    etc
281G    home
17M     lib
3.7M    lib32
0       lib64
16K     lost+found
8.0K    media
4.0K    mnt
157M    opt
du: cannot access `proc/31735/task/31735/fd/4': No such file or directory
du: cannot access `proc/31735/task/31735/fdinfo/4': No such file or directory
du: cannot access `proc/31735/fd/4': No such file or directory
du: cannot access `proc/31735/fdinfo/4': No such file or directory
0       proc
41M     root
4.0K    run
14M     sbin
4.0K    selinux
4.0K    srv
0       sys
129M    tmp
2.2G    usr
431M    var

=====================================================

Update after 1st comment, du of /var:

root@sbs691:/var# ls | xargs du -hs
4.8M    backups
149M    cache
4.0K    games
265M    lib
4.0K    local
12K     lock
14M     log
4.0K    mail
4.0K    opt
200K    run
24K     spool
4.0K    tmp
16K     www

Issue is been now resolved temporaryily by restarting nginx

service nginx restart

After restarting

root@sbs691:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              9.7G  6.0G  3.3G  65% /
tmpfs                 3.9G     0  3.9G   0% /lib/init/rw
udev                   10M  244K  9.8M   3% /dev
tmpfs                 3.9G  620K  3.9G   1% /dev/shm
/dev/md3              1.8T  339G  1.4T  20% /home

Answer 1

You've probably got a file that has been deleted but which is being held open (via the fd) by the daemon that is writing to it. Take a look at the output of

lsof +L1

which will list such files. When you know the file, you just need to figure out which daemon and then tell it to restart or reopen it's log file.

---

From the comments: It turned out that the OP had deleted files in /var and had not restarted the daemons (specifically nginx) that were writing to the files.

Answer 2

The fastest (and somewhat very hackish) way is to boot the server into a rescue media and move /var and maybe even /usr to the /home file system and symlink it back.

The correct(TM) approach is to resize the partitions in a sensible way, but since you use a software RAID, that's not exactly easy or fast.

Answer 3

• Since it's a Debian system, you have probably accumulated lots of downloaded packages in /var/cache/apt which may fill up your /var. Unless you have a good reason to keep them, issue apt-get clean as root to remove them.

• You could use parted or similar tools RAID tools to change your partition sizes. Remember to have a current backup ready before that!

• As a quick resort, which I cannot really recommend, you could also move your /usr folder to the /home partition and symlink from root. Remember to have a current backup ready before that! My work order would be:

  1. If possible, shutdown and reboot from a rescue system.
  2. Copy the folder with all permissions, links etc.: # cp -a /usr /home/root-usr (The paths will differ when you work from a rescue system! Make sure that there is no user named root-usr! ;-))
  3. Compare old and new folder to be sure copying worked: # diff -r /usr /home/root-usr
  4. Remove the old directory and create a symlink: # rm -rf /usr; ln -s /home/root-usr /usr (Be sure the symlink is correct when doing this from a rescue system! You might have to chroot to accomplish that.)
  5. Reboot into your system.

Always be careful. You should know what you do.

Answer 4

Your var folder has about 431M of data, I'd start by clearing out your logs.

Answer 5

Your numbers don't add up. You might have data on your root fs you don't "see".

I can only count ~3G on the du output of your root. I suspect you have either:

• at some point copied the data in /home to a different filesystem and changed the mount-point, but failed to delete the original copy - this of course still takes up space. You just can't see it now that you have changed the mount point.

  To find this kind of phantom of the opera data, one does compare totals of du on / (without crossing fs boundaries), and df.

• Or created some huge files in /, in which case you need to move/delete them - they don't belong there.

Answer 6

Is /home later added/mounted to the system?

if you have old data in md1 and in his /home directory, and you mounted the new partition (md3) into /home directory with nonempty option the /home can hide the original data which placed and reserv his size in md1, but you can not see becuse hided by this other patition,

If you may interrupt the server operations bring down the system into rescue mode

init 1

umount /home system, and look for files whichs left in the original root partition (md1)

ls -la /home

Answer 7

After having a separate part for system files, to isolate my /home data, then doing a major debian dist release tier upgrade; I completely forgot about...

sudo apt clean

...too and not just...

sudo apt --purge autoremove

...Clean freed up 5.8GB. Thank you !