0

I'm running a Ubuntu server and this evening since rebooting it I am not getting the error message Cannot write to '' (No space left on device).

Earlier I did unzip a large directory in to /root which created around 100 files, I then removed these using rm /root/*. I did check ls prior to doing this and the only files in /root were the actual ones that I wanted to remove. Did I do something wrong here?

Here are some outputs:

df -h

root@localhost:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda        47G   47G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
devtmpfs        494M  8.0K  494M   1% /dev
none             99M  192K   99M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            495M     0  495M   0% /run/shm
none            100M     0  100M   0% /run/user

df -i

root@localhost:~# df -i
Filesystem      Inodes IUsed   IFree IUse% Mounted on
/dev/xvda      3129344 52147 3077197    2% /
none            126527     2  126525    1% /sys/fs/cgroup
devtmpfs        126232  1410  124822    2% /dev
none            126527   792  125735    1% /run
none            126527     4  126523    1% /run/lock
none            126527     1  126526    1% /run/shm
none            126527     2  126525    1% /run/user

du -Pshx /* 2>/dev/null

root@localhost:~# du -Pshx /* 2>/dev/null
9.1M    /bin
4.0K    /boot
4.0K    /dev
6.8M    /etc
31M     /home
29M     /lib
4.0K    /lib64
16K     /lost+found
8.0K    /media
4.0K    /mnt
4.0K    /opt
0       /proc
76K     /root
192K    /run
7.6M    /sbin
4.0K    /srv
0       /sys
12K     /tmp
611M    /usr
541M    /var

Can someone please suggest something here, as I'm out of ideas?

5
  • Would something still have those deleted files in use? Jun 21, 2014 at 19:59
  • @ShaneMadden Hi, I did a wget and then unzip, accidently unzipping in to the current directory /root rather than it's own, so I deleted them. I didn't run any etc. Sequence was wget, unzip and then rm. Jun 21, 2014 at 20:01
  • Anything from find /proc/*/fd -ls | grep '(deleted)'? Jun 21, 2014 at 20:06
  • What does lsof (manpages.ubuntu.com/manpages/hardy/man8/…) say?
    – dartonw
    Jun 22, 2014 at 3:48
  • It doesn't add up: on one hand you've got only one fs, your root partition, which is full at 47Gb, but du reaches 2Gb top. You should try to boot in maintenance mode and do a fs check (fsck).
    – didierc
    Jun 22, 2014 at 14:09

1 Answer 1

2

The magic of deleted files. As long as an application has a file open, it's not actually deleted. When this happens, usually the culprit is a daemon that holds a logfile or temporary open that you (or logrotate) have long since removed.

The command Shane Madden gave will let you find such files:

find /proc/*/fd -ls | grep '(deleted)'

Here's an excerpt of the output on my machine:

348233    0 lrwx------   1 dennis   dennis         64 jun 22 16:15 /proc/5623/fd/25 -> /tmp/vte1D81HX\ (deleted)
348238    0 lrwx------   1 dennis   dennis         64 jun 22 16:15 /proc/5623/fd/30 -> /tmp/vteYAG3HX\ (deleted)
348239    0 lrwx------   1 dennis   dennis         64 jun 22 16:15 /proc/5623/fd/31 -> /tmp/vteP7E3HX\ (deleted)
348240    0 lrwx------   1 dennis   dennis         64 jun 22 16:15 /proc/5623/fd/32 -> /tmp/vte4D91HX\ (deleted)
348243    0 lrwx------   1 dennis   dennis         64 jun 22 16:15 /proc/5623/fd/35 -> /tmp/vteU0VWHX\ (deleted)

So process 5623 has some temporary files open. Let's find out what it is:

$ cat /proc/5623/cmdline | tr '\0' ' ' ; echo
/usr/bin/python /usr/bin/terminator 

Once you find out which process holds lots of deleted files and or big deleted files open, restart it to clear that situation.

Not the answer you're looking for? Browse other questions tagged .