AWS EC2 - No Space Left on Device

Question

I have an EC2 instance and when I log in I get

mktemp: failed to create file via template '/tmp/.colorlsXXX': No space left on device

However when I do df -h I see

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  4.2G  3.5G  55% /
devtmpfs        812M   16K  812M   1% /dev
tmpfs           828M     0  828M   0% /dev/shm

Doing df -i gives me

/dev/xvda1     524288 524288      0  100% /
devtmpfs       207790    414 207376    1% /dev
tmpfs          211891      1 211890    1% /dev/shm

What's going on here?

Answer 1

You have exhausted all of the inodes available on your filesystem. You likely have a directory somewhere with a metric crap-ton of tiny files. You'll need to locate that directory and remove some of the files.

To search which folder of your server is eating the inode limit, you can run this command:

find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -nr

Command taken from here

Answer 2

This error occurred to me while trying to install ubuntu-desktop as instructed by AWS knowledge-center.

Then I found here how to find all those directories with huge number of tiny files:

for i in /*; do count=`sudo find $i | wc -l`; if [ $count -gt 10000 ]; then echo $i $count; fi; done

/lib 18982
/proc 35223
/sys 36490
/usr 253941
/var 14584

Run it again for each suspicious sub-directory (for me it was hidden under /usr/*).

But before you remove those sub-directories with rm -rf (if you're absolutely sure!), try to safely remove Linux packages that were installed by other packages, but are no longer needed:

sudo apt-get -f install
sudo apt-get autoremove

Run df -i again to see if you have more space now.