0

I can no longer generate new columns in my database. My stack is below. I am new to managing my own server and am unfamiliar with Apache, Ubuntu and AWS. Currently I am receiving an error saying that I have no space left.

It looks like my folder /dev/xvda1 is at 100% capacity. How can I identify how space is being allocated and how can I open up more space?

I will eagerly share more information if you can help me with what to share.

My Error

irb(main):107:0> Survey.generate_for_appointment(a, b)
PG::DiskFull: ERROR:  could not extend file "base/16402/25415": No space left on device
HINT:  Check free disk space.
: INSERT INTO "questions" ("choices", "content", "created_at", "position", "question_type", "survey_group", "survey_group_question_id", "survey_id", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING "id"
log writing failed. No space left on device @ io_write - /etc/projects/myapp/log/production.log
ActiveRecord::StatementInvalid: PG::DiskFull: ERROR:  could not extend file "base/16402/25415": No space left on device
HINT:  Check free disk space.

Storage

ubuntu@ip-172-31-33-140:/etc/projects/myapp$ sudo du -h --max-depth=1 /
4.0K    /media
752K    /tmp
0   /sys
9.6M    /bin
1.3G    /home
908M    /usr
4.0K    /mnt
20M /boot
24K /root
4.0K    /opt
3.7G    /etc
du: cannot access ‘/proc/7489/task/7489/fd/4’: No such file or directory
du: cannot access ‘/proc/7489/task/7489/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/7489/fd/4’: No such file or directory
du: cannot access ‘/proc/7489/fdinfo/4’: No such file or directory
0   /proc
9.4M    /sbin
4.0K    /srv
16K /lost+found
60M /lib
12K /dev
649M    /var
4.0K    /lib64
212K    /run
7.5G    /

ubuntu@ip-172-31-33-140:/etc/projects/myapp$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  7.5G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            284M   12K  284M   1% /dev
tmpfs            59M  212K   59M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            295M     0  295M   0% /run/shm
none            100M     0  100M   0% /run/user

Detailed Storage

ubuntu@ip-172-31-33-140:/etc/projects/myapp$ sudo du -x / | sort -n | tail -40
98880   /home/ubuntu/.rvm/rubies/ruby-2.1.2
98908   /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.4.1
98908   /home/ubuntu/.rvm/rubies/ruby-2.1.5/lib
99436   /home/ubuntu/.rvm/rubies/ruby-2.1.5
107920  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri/tmp/x86_64-linux-gnu/ports
107924  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri/tmp/x86_64-linux-gnu
107928  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri/tmp
108520  /var/lib/apt/lists
108572  /var/lib/apt
111044  /usr/src
119420  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri
119424  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext
145780  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1
146972  /var/cache
155528  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/passenger-4.0.45/buildout
155916  /usr/lib/x86_64-linux-gnu
158396  /etc/projects/myapp/public/assets
163068  /etc/projects/myapp/public
198320  /home/ubuntu/.rvm/rubies
200824  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/passenger-4.0.45
207540  /home/ubuntu/.rvm/gems/ruby-2.1.5/gems
231732  /var/lib
264856  /home/ubuntu/.rvm/gems/ruby-2.1.5
271292  /var/log/apache2
284432  /var/log
292684  /usr/share
403300  /usr/lib
583604  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems
650888  /home/ubuntu/.rvm/gems/ruby-2.1.2
664560  /var
915752  /home/ubuntu/.rvm/gems
929616  /usr
1174228 /home/ubuntu/.rvm
1262052 /home/ubuntu
1262056 /home
3470180 /etc/projects/myapp/log
3782520 /etc/projects/myapp
3782524 /etc/projects
3789104 /etc
7795236 /

Nodes in Use

ubuntu@ip-172-31-33-140:/etc/projects/myapp$ df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/xvda1     524288 153278 371010   30% /
none            75346      2  75344    1% /sys/fs/cgroup
udev            72620    381  72239    1% /dev
tmpfs           75346    266  75080    1% /run
none            75346      3  75343    1% /run/lock
none            75346      1  75345    1% /run/shm
none            75346      2  75344    1% /run/user

Passanger Version

Phusion Passenger version 4.0.55

Apache Version

Server version: Apache/2.4.7 (Ubuntu)

RVM Version

rvm 1.26.4 (latest) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]

Ruby Versions

rvm rubies

   ruby-2.1.2 [ x86_64 ]
=* ruby-2.1.5 [ x86_64 ]

# => - current
# =* - current && default
#  * - default
  • Why is /etc 3.7GB? That is an unusually large size for /etc. – Zoredache Jun 8 '15 at 17:10
  • @zoredache I believe I created a model that generated far too many rows in Postgres. Do you know how I could check that? – MicFin Jun 8 '15 at 17:14
  • I don't know postgres, but if you are using standard packages, I highly doubt it is storing data under /etc. – Zoredache Jun 8 '15 at 17:15
  • @zoredache thanks for the advice. I deleted 3470180 from /etc/projects/myapp/log by using the rake clear command. It looks like my logs were taking up half my storage :/ – MicFin Jun 8 '15 at 17:20
1

You have to explore your HDD and clean up outdated and/or unused files.

The easiest way is to stop all the services like postgres/apache/postfix etc (to prevent data loss) and dig into directory sizes. The command

du -d 1 | sort -nr

list you all the nested directories from the current location and their sizes. After certain iterations you'll find the space-mongers and then you'll need an anothed advice - how to deal with them.

| improve this answer | |
  • thanks! I used that list to remove 3470180 from /etc/projects/myapp/log by using the rake clear command. It looks like my logs were taking up half my storage – MicFin Jun 8 '15 at 17:21
  • You have to setup periodic log rotating. All popular services already have mature tradition of log rotation, as well as well known traps novice can step in. Googling %SERVICENAME% log rotate would be very helpful to you. – Kondybas Jun 8 '15 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.