1

I have ec2 account setup with size of 80GB. I have uploaded ML code of 2gb on this ec2 instance. However for some reasons my instance is 99% occupied in terms of storage space

ubuntu@ip-172-31-41-142:/dev$ df xvda1
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/xvda1      76171508 75318652    836472  99% /

Something is eating up almost 98% of my ec2 instances storage space. What is it and how do I clear this space

3

That something may be log files which have become too large and need to be removed.

Try eliminating the ones above 4MB size:

find / -type f -size +4M -exec ls -lh {} \;

If you attempted file deletions, make sure there are not processes still accessing them and check into deleted ones:

find /proc/*/fd -ls | grep  '(deleted)'
  • I do get a list of files, after executing your first command. But how do I distinguish between a log file and normal files. Do log files have any specific file extentions – Ajinkya May 16 at 7:07
  • For example I am no sure if "-rwxrwxr-x 21 ubuntu ubuntu 11M Mar 22 2018 /home/ubuntu/anaconda3/envs/chainer_p36/lib/libmkl_vml_mc.so -rwxrwxr-x 21 ubuntu ubuntu 11M Mar 22 2018 /home/ubuntu/anaconda3/envs/chainer_p36/lib/libmkl_gf_lp64.so " is a log file or not – Ajinkya May 16 at 7:16
  • 1
    They usually have .log extension. Examples: /var/log/eb-activity.log /var/log/eb-commandprocessor.log /var/log/eb-version-deployment.log . See here some info on where to find other logs: docs.aws.amazon.com/elasticbeanstalk/latest/dg/… – Overmind May 16 at 8:30
2

The version that I took of AWS shipped with pre-installed conda virtual environments. I needed to do

conda env list

and then delete the unnecessary virtual environments

1

Sometimes looking for the largest directories can be a good option. Here's a method I use frequently: du -hx / | sort -hr | head -n 30 - this gives the top 30 directories occupying the most space. Change 30 to whatever number you want - I usually try to make it a few lines less than the number of lines on my terminal.

Once you have an idea of which directories are large, you can research what they're used for and how to clean them up best. e.g. If /var/cache/apt is large, you probably should consider running apt clean; if /var/log is the largest, you should work out which applications are logging the most and look into tuning their log output to be less verbose.

Of course, sometimes it's all legitimate disk space usage, and the simplest and most expedient solution is just to add more disk.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.