73

I'm running a Linux instance on EC2 (I have MongoDB and node.js installed) and I'm getting this error:

Cannot write: No space left on device

I think I've tracked it down to this file, here is the df output

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda1             1032088   1032088         0 100% /

The problem is, I don't know what this file is and I also don't know if this file is even the problem.

So my question is: How do I fix the "No space left on device" error?

10 Answers 10

102

That file, / is your root directory. If it's the only filesystem you see in df, then it's everything. You have a 1GB filesystem and it's 100% full. You can start to figure out how it's used like this:

sudo du -x / | sort -n | tail -40

You can then replace / with the paths that are taking up the most space. (They'll be at the end, thanks to the sort. The command may take awhile.)

5
  • 30
    To get the output in a human-readable format, you can use sudo du -x -h / | sort -h | tail -40 (from this answer).
    – mkobit
    Jun 21, 2016 at 14:09
  • 3
    For those on micro AWS AMI instances, this can take a minute or so to run. Be patient! Nov 15, 2018 at 11:31
  • what to do with this:sort: write failed: /tmp/sortGmL8oF: No space left on device
    – dOM
    Dec 3, 2018 at 9:28
  • 1
    @dOM Ouch. Try to clean off some space on /tmp. Or, if you must, narrow it down step by step with commands like du -xhs /*. Dec 3, 2018 at 9:45
  • du -x -h / | sort -h | tail -40 | sort -h -r can be used to sort in descending order when using human-readable output.
    – Vigs
    May 11, 2019 at 17:53
28

I know i am replying in this thread after nearly 5 years but it might help someone, I had the same problem, i had m4.xlarge instance df -h told that the /dev/xvda1 was full, - 100%

Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G     0  7.9G   0% /dev
tmpfs           1.6G  177M  1.4G  12% /run
/dev/xvda1      7.7G  7.7G     0 100% /
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
tmpfs           1.6G     0  1.6G   0% /run/user/1000

i tried to solve it here are the steps

sudo find / -type f -printf '%12s %p\n' 2>/dev/null|awk '{if($1>999999999)print $0;}'

Helped me to know that it was the docker container that was talking all my space so i push all my container to my docker registry then did sudo rm -rf /var/lib/docker/ it cleared up my space :) hope it helps someone :)

1
  • Thanks. How đi you push your container to your docker resgistry? It is not good just to delete the xxx-json.log file right?
    – Freelensia
    Jun 23, 2021 at 13:45
11

If you are running an EBS boot instance (recommended) then you can increase the size of the root (/) volume using the procedure I describe in this article:

Resizing the Root Disk on a Running EBS Boot EC2 Instance
http://alestic.com/2010/02/ec2-resize-running-ebs-root

If you are running an instance-store instance (not recommended) then you cannot change the size of the root disk. You either have to delete files or move files to ephemeral storage (e.g., /mnt) or attach EBS volumes and move files there.

Here's an article I wrote that describes how to move a MySQL database from the root disk to an EBS volume:

Running MySQL on Amazon EC2 with EBS
http://aws.amazon.com/articles/1663

...and consider moving to EBS boot instances. There are many reasons why you'll thank yourself later.

1
  • I'm running on EBS, it's fairly cheap to expand the root disc right? Fortunately I don't have to deal with MySQL, My projects are currently Mongo/Redis. some great material here. +1
    – Chris Biscardi
    Nov 13, 2011 at 6:31
4

I've recently run into this issue on Amazon Linux. My crontab outbound email queue /var/spool/clientmqueue was 4.5GB.

I solved it by:

  1. Locating large files: sudo find / -type f -size +10M -exec ls -lh {} \;
  2. Deleting large files: /bin/rm -f <path-to-large-file>
  3. Restart server instance

Problem solved!

4

Paulo was on the right track for me, but when I tried to run

sudo apt autoremove

it responded:

Reading package lists... Error!
E: Write error - write (28: No space left on device)
E: IO Error saving source cache
E: The package lists or status file could not be parsed or opened.

First, I had to run

sudo apt-get clean

That cleared just enough space for me to run 'sudo apt autoremove', and that took me from 100% full on /dev/xvda1 to 28%.

4

I have just solved that problem by running this command:

sudo apt autoremove 

A lot of old packages were removed, freeing up 5 gigabytes, for instance there were many packages like linux-aws-headers-4.4.0-1028

1

It could be coming from Jenkins or Docker. To solve that, you should clean Jenkings logs and set it's size.

1

Hope this helps to those who are using codedeploy agent and having a similar issue.

I was using Amazon Linux EC2 instance and my directory was 100% full. first, to run the command I deleted all files /var/log/journal/.

then run this command. sudo du -xhc / and found that out of 8GB, codedeploy-agent/deployment-root folder was using 5.1GB space.

By default codedeploy-agent store last 5 archive so i changed :max_revision from 5 to 2 in /etc/codedeploy-agent/conf/codedeployagent.yml

1

Use du -hs * | sort -rh | head -5 to check top 5 usage, then rm -rf name to remove junk like large size log file or archived within logs folder

0
0

If we really have used up the current space, we'll need to extend the virtual disk (EBS) in EC2.

After you increase the EBS volume for EC2 from either the web or CLI, follow this AWS article to finish expanding the disk from the OS level.

In short, only two commands are needed if the virtual technology in your EC2 is Xen. No need to unmount/mount the dev or restart the instance as previous posts suggested.

sudo growpart /dev/xvda 1
sudo xfs_growfs -d /

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .