Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Yesterday I purchased a dedicated server package from 1and1. I installed Rails, Mongo and nginx. Today, when I logged in, it displays this message:

Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-24-generic x86_64)

  • Documentation: https://help.ubuntu.com/

    System information as of Wed May 23 21:45:00 EDT 2012

    System load: 0.82 Processes: 114 Usage of /home: 6.1% of 3.99GB Users logged in: 1 Memory usage: 4%
    IP address for eth0: xx.xx.xxx.xxx Swap usage: 0%

    => /var is using 91.4% of 3.99GB

    Graph this data and manage this system at https://landscape.canonical.com/

21 packages can be updated. 6 updates are security updates.

I am not too fluent at linux or the command line, but isn't 4GB peanuts for a directory?

If I do du -a /var | sort -n -r | head -n 10

3565292 /var
3299164 /var/lib
3145752 /var/lib/mongodb
3145744 /var/lib/mongodb/journal
1048580 /var/lib/mongodb/journal/prealloc.2
1048580 /var/lib/mongodb/journal/prealloc.1
1048580 /var/lib/mongodb/journal/j._0
237272  /var/cache
147260  /var/cache/apt
114844  /var/lib/apt

and if I do df -h

Filesystem             Size  Used Avail Use% Mounted on
/dev/sda1              3.7G  736M  3.0G  20% /
udev                   3.9G  4.0K  3.9G   1% /dev
tmpfs                  1.6G  260K  1.6G   1% /run
none                   5.0M     0  5.0M   0% /run/lock
none                   3.9G     0  3.9G   0% /run/shm
/dev/mapper/vg00-usr   4.0G 1007M  2.9G  26% /usr
/dev/mapper/vg00-var   4.0G  3.7G  146M  97% /var
/dev/mapper/vg00-home  4.0G  251M  3.6G   7% /home

If I do fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204402688 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953524224 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x6c80de21

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              63     7823654     3911796   83  Linux
/dev/sda2         7823655    11743514     1959930   82  Linux swap / Solaris
/dev/sda3        11743515  1953520064   970888275   8e  Linux LVM

Disk /dev/mapper/vg00-usr: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/vg00-usr doesn't contain a valid partition table

Disk /dev/mapper/vg00-var: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/vg00-var doesn't contain a valid partition table

Disk /dev/mapper/vg00-home: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/vg00-home doesn't contain a valid partition table

and mount

/dev/sda1 on / type ext3 (rw)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/mapper/vg00-usr on /usr type ext4 (rw,errors=remount-ro)
/dev/mapper/vg00-var on /var type ext4 (rw,errors=remount-ro)
/dev/mapper/vg00-home on /home type ext4 (rw,errors=remount-ro)

vgdisplay:

  --- Volume group ---
  VG Name               vg00
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               925.91 GiB
  PE Size               4.00 MiB
  Total PE              237033
  Alloc PE / Size       3072 / 12.00 GiB
  Free  PE / Size       233961 / 913.91 GiB
  VG UUID               eI2V0B-oVzj-HiIh-enIh-FnrO-NJKi-KpZb0i

My question: is there any way to overcome this limit?

share|improve this question
    
Can you please provide the output from the following commands separately: fdisk -l and mount so that we can see what other disks and/or partitions there are, and whether they are mounted. –  Richard Keller May 24 '12 at 2:10
    
Updated with your suggestion @RichardKeller –  Duopixel May 24 '12 at 2:15
    
@Duopixel Ok, so it's using LVM. How about vgdisplay? –  Shane Madden May 24 '12 at 2:26
    
@RichardKeller vgdisplay included –  Duopixel May 24 '12 at 2:30

6 Answers 6

What limitation are you talking about? The /var partition is just 4GB and you are using most of its capacity. You need either to remove some unneeded files (such as some logs) or purchase more space from your host (like adding another HD drive). However, you can not modify the /var partition size dynamically without format unless you are using LVM (Logical Volume Manager).

share|improve this answer
    
Sorry, I'm a total newcomer to server administration. I've updated the question, the HDD is 750GB, should I move everything to another directory? –  Duopixel May 24 '12 at 2:22
1  
"However, you can not modify the /var partition size dynamically without format." I'd be careful writing things like this without specifying that it's completely wrong and actually depends on many factors. You can resize some file systems dynamically. –  gparent May 24 '12 at 3:15
    
@gparent correct, I was able to risize it without taking it offline serverfault.com/questions/392102/… –  Duopixel May 24 '12 at 4:36
    
@gparent: You can resize your partitions dynamically if you are using LVM. I neither expected the OP is using LVM nor it was stated in the question. My answer was the first before getting more information like FS type. –  Khaled May 24 '12 at 7:09
1  
Which is exactly why you shouldn't assume things until you know. It's misleading. He could read your advice, not know that you are wrong, and then do something else than what he did to fix the problem, complicating the process. –  gparent May 24 '12 at 14:29

4GB is not peanuts for a filesystem. It all depends on what you're putting in it (for example, your base operating system installs come in under 2GB).

In your case you've got a database there, so there's a good chance that you're going to need more space. You should probably talk to your hosting provider and find out what it costs to obtain additional storage, and then either extend the /var filesystem or create a dedicated volume for /var/lib/mongodb.

share|improve this answer
    
The server is supposed to have 750 GB of storage, I've updated my question with fdisk -l and mount. –  Duopixel May 24 '12 at 2:16

Based on your output from fdisk -l it seems that your /dev/sda disk is actually a 1TB drive. The volume that holds your /var directory, however, is only a 4GB volume, which is way too small to be usable in the long-run.

It is possible to extend a logical volume, but not while it is mounted. This is going to present some difficulty, especially if the server is colocated and you don't have physical access to it. The hosting company may be able to provide you with an IPKVM device, which will allow you to boot up a recovery disk remotely, and adjust the volume size. However, as you've already said that you're not too fluent on the console or with Linux in general, I highly suggest avoiding this route. Rather, I suggest one of the following:

  • Contact the hosting company and find out whether one of the tech support guys can adjust the size of the volume on which the /var directory is mounted to something more reasonable, at least to around 50GB since you do have a 1TB drive available. The server will be taken offline during this change, since the /var directory will have to be unmounted during the process.
  • Alternatively, ask for a reinstall of the entire operating system, and tell them to set the volume size to at least 50GB. This may work out cheaper than having them perform this adjustment on the live system since it's less technical. Given that you don't have much technical knowledge with Linux, an even better solution would be to tell them to reinstall the operating system without adding any special mount points for /usr /var and /home and instead rather just mount everything on the root volume. The only benefit of creating special mount points on separate volumes (which is how you're set up at the moment) is that if one volume fills up, it doesn't bring down your entire system. However, setting a volume size to 4GB as they've done in your case seems very silly, especially given the size of your hard disk.
share|improve this answer
    
Thanks for your help Richard, your answer helped me find a tutorial from my provider with instructions on how to achieve this. I'll update the question with a self answer. –  Duopixel May 24 '12 at 2:47
1  
Great. Also, instead of updating your original question with the answer, rather add an actual answer, that way the question can be marked as answered. –  Richard Keller May 24 '12 at 2:49
    
Done, thanks again. –  Duopixel May 24 '12 at 2:52
up vote 8 down vote accepted

Turns out my system supports extending a logical volume online. Here is how to solve it:

# df -h

Partitions and logical volumes will be listed:

Filesystem             Size  Used Avail Use% Mounted on
/dev/sda1              3.7G  736M  3.0G  20% /
udev                   3.9G  4.0K  3.9G   1% /dev
tmpfs                  1.6G  260K  1.6G   1% /run
none                   5.0M     0  5.0M   0% /run/lock
none                   3.9G     0  3.9G   0% /run/shm
/dev/mapper/vg00-usr   4.0G 1007M  2.9G  26% /usr
/dev/mapper/vg00-var   4.0G  3.7G  145M  97% /var
/dev/mapper/vg00-home  4.0G  251M  3.6G   7% /home

Choose the partition you wish to expand and run it through lvextend

# lvextend -L +46G /dev/mapper/vg00-var

Where +46G is the number is GB you want to add, in my case 46+4 = 50.

After that you need to increase the file system to match that of the logical volume using:

# resize2fs /dev/mapper/vg00-var

And you will have a larger partition:

resize2fs 1.42 (29-Nov-2011)
Filesystem at /dev/mapper/vg00-var is mounted on /var; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 4
Performing an on-line resize of /dev/mapper/vg00-var to 13107200 (4k) blocks.
The filesystem on /dev/mapper/vg00-var is now 13107200 blocks long.

The new df -h output:

Filesystem             Size  Used Avail Use% Mounted on
/dev/sda1              3.7G  737M  3.0G  20% /
udev                   3.9G  4.0K  3.9G   1% /dev
tmpfs                  1.6G  260K  1.6G   1% /run
none                   5.0M     0  5.0M   0% /run/lock
none                   3.9G     0  3.9G   0% /run/shm
/dev/mapper/vg00-usr   4.0G 1008M  2.9G  26% /usr
/dev/mapper/vg00-var    49G  3.7G   43G   8% /var
/dev/mapper/vg00-home  4.0G  251M  3.6G   7% /home
share|improve this answer
1  
You'll be in good shape with the oversized /var. It's relatively common practice to separate special-case storage (like databases that may grow endlessly) on to dedicated volumes. Lots of things break when /var fills.. like the mailer that you might eventually use to send yourself alerts on disk space. –  rjewell May 24 '12 at 7:37
    
Note that you wouldn't have this problem in the first place if they had not set up your partitions in such a goofy way. Specifically, having /var and /usr on their own filesystem, and / on a simple partition outside of LVM. If you fill up your / partition, you won't be able to extend it easily, and now that 50 gb of space is dedicated to /var so you can't use it elsewhere. –  psusi May 24 '12 at 15:02
    
Thanks for posting this. My current server from them is configured in this way. I ended up just moving all my stuff to a /home/var and living with it. Now I know how to extend /var for next time. –  Brian Neal Nov 6 '12 at 16:07

your mongo install will pre-allocate space for the journal which can take up some space.. you can move that to free some up.

share|improve this answer

Start out by clearing the apt cache with apt-get clean. Then investigate the other large users of space on /var. Anything large can be moved elsewhere.

share|improve this answer
    
I was hitting the 10GB in /var and the clean up gave me 1.4 GB in a second, which is nice for now until I resize it later! –  Mahdi Dec 18 at 8:22

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.