# volume group out of space. How do I reclaim it?

So long story there is an SVN VM (vmware server) that has run out of space. I can't commit because it says it's out of space. The VM has one 80GB virtual drive attached (hda), a /boot partition, and a big VG (VolGroup00) for /

I am archiving and deleting old SVN repos from the disk, but I am getting no free space back. What do I need to do to get this space back available for use? I use Linux for almost everything, but have never really played with LVM enough to know what's going on.

vgdisplay output:

--- Volume group ---
VG Name               VolGroup00
System ID
Format                lvm2
VG Status             resizable
MAX LV                0
Cur LV                2
Open LV               2
Max PV                0
Cur PV                1
Act PV                1
VG Size               74.41 GB
PE Size               32.00 MB
Total PE              2381
Alloc PE / Size       2380 / 74.38 GB
Free  PE / Size       1 / 32.00 MB
VG UUID               dPSZpL-kFBn-HpkH-ChfO-dw9q-YGg2-qHOiQF


Since yesterday, space has been freed...

Output of df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
72G   67G  959M  99% /
/dev/hda1              99M   15M   80M  16% /boot
tmpfs                  62M     0   62M   0% /dev/shm


Output of lvs

LV       VG         Attr   LSize  Origin Snap%  Move Log Copy%
LogVol00 VolGroup00 -wi-ao 73.38G
LogVol01 VolGroup00 -wi-ao  1.00G


Something still seems weird, since it says 959M avail, and yet 67G used with a 72G size...

So I am getting space back, now, from deleting repositories...and yet, balance is not yet restored in the universe...

-
Hi. I used to think that LVM was confusing too. Then I figured it out, and I wrote some documentation to help other people: standalone-sysadmin.com/blog/2008/09/… –  Matt Simmons Jul 17 '09 at 11:28
An advice, not necessarily related to your problem: you should try using separate partitions for your (basic) system and your data/services. In your case it would be something like / and /mnt/repos. By doing this you'll avoid taking down the whole system just because a service used all the available space. –  Cristian Ciupitu Jul 17 '09 at 15:57
Good advice, but I just deal with the server, I didn't set it up unfortunately. –  Daniel Huckstep Jul 17 '09 at 16:13

Deleting files doesn't change the size of a logical volume (and, by extension, doesn't affect free space on the volume group).

You can think of a volume group as like a "virtual disk", and the logical volumes on it as being like partitions, but ones that are a lot easier to work with (resize, create, delete) than regular partitions.

As Kamil says, if you're getting "out of space" errors when working, it's not LVM that's directly at fault -- that's a simple filesystem error, and you can determine which filesystem is out of space with df -h, and deleting enough stuff on that filesystem will eventually give you more space. However, with LVM, if you've got a filesystem with heaps of free space and another with none, you can shrink the LV with lots of free space and then allocate that space to the LV that's run out of space, and hence use your allocated disk more efficiently.

The process for shrinking and expanding filesystems is a bit risky (so have backups), and the shrink stage must be done on unmounted filesystems (so dropping to single user mode is good, and you can't shrink the root filesystem; if your filesystems support it (XFS, reiser, recent ext2/3) you can do online expansion). In general, the process is:

1. Shrink the filesystem you want to remove space from to a size a little smaller than what you want it to end up with (with something like resize2fs /dev/mapper/VolGroup00-largeLV xG). The reason why you shrink it to a little smaller is because if you get your math wrong and shrink the LV smaller than the filesystem, you'll stuff it up.
2. Shrink the LV you want to remove space from to the size you want (lvresize -L xG VolGroup00/largeLV)
3. Grow the filesystem on that LV to the new size of the LV: resize2fs /dev/mapper/VolGroup00-largeLV
4. Grow the LV that's too small to it's new, larger size: lvresize -L+nG VolGroup00/smallLV
5. Grow the filesystem on the too-small LV to it's new, larger size: resize2fs /dev/mapper/VolGroup00-smallLV

And you should now have plenty of space everywhere.

A few tips:

• lvs will list all your LVs, along with their sizes
• vgs will give you a quick display of all your VG's sizes and free space
• If your swap is on LVM, that's often a good place to get a bit of temporary free space if the machine isn't working too hard.

Good luck!

-
On Fedora 11, the resize can be done with fsadm (utility to resize or check filesystem on a device). –  Cristian Ciupitu Jul 17 '09 at 15:33

Is /dev/sda local or on the SAN? If it's on the SAN, then there's hope. If it's local, and it's using all of the available space, you're going to have to free up some disk space by removing files.

Update the question and let us know what the status is.

Edit

Ah, it's a VM! You're in luck (if there's free space on the host machine, anyway).

Create another virtual disk in the VM manager that is as large as you want the freespace to be. Then present that disk image to the VM.

Make sure the VM sees it (use dmesg to see if it shows up). Assuming it does, fdisk it, and create one partition.

Issue

 pvcreate /dev/whatever1


Where "whatever" is obviously the device. Probably sdb. 1 is the partition that you just made. It shouldn't have a problem making this physical volume.

Now, run

 vgextend VolumeGroupName /dev/whatever1


You can use vgdisplay to verify that the volumegroup has free space now. Now, grow the logical volume:

 lvextend -l +100%FREE


That should expand the logical volume to fill the volume group.

Now the tricky part. Assuming you've got an ext3 filesystem, you should be able to resize live, on the fly:

 resize2fs /dev/volgroup/logicalvolume


(where volgroup and logicalvolume is the actual path to whatever is mounted on /)

It'll say it's doing a live resize since the volume is mounted, and voila, running df -h should show that you have free space.

-
The disk won't magically present itself. You'll need to force the kernel to scan the scsi bus: echo "- - -" > /sys/class/scsi_host/host<n>/scan n will probably be 0. Then run dmesg and you should see the device details or the error. To find out what n is definitively cat /proc/scsi/scsi It's possible there will be more than one scsi host, though not all that likely on a boring regular vanilla household VM. –  Jason Tan Jul 17 '09 at 12:04
Of course Jason is right. I'm sorry, I assumed you rebooted when you presented the disk, which is unnecessary. –  Matt Simmons Jul 17 '09 at 13:47

The amount of free space on your volume group will not change. It's all allocated by the filesystem you've formatted it with. Deleting files will only remove their inode entries from the filesystem, but the filesystem will still claim the space.

The only way to reclaim space in your volume group is to resize or delete logical volumes.

But, what you're describing sounds simply like you are out of disk space. You should check the output of df -h in your VM and see how much free space there is on the volume where the repository resides.

-
df -h was the first thing I checked, which showed 0 space free. Since there are only two mount points (/boot and /), the svn repositories are clearly on the / mount point. So i go there, and start deleting things but df -h still shows 0 space free... I'll post some more output tomorrow when I get to work. LVM is confusing. –  Daniel Huckstep Jul 17 '09 at 4:46

Try restarting your SVN server. If deleting files isn't freeing up space, they must still be in use somehow. This happens with active logs as well. I once had some log that grew by 10 mb an hour, so pretty soon I had to delete the log, but it didn't free up the space until I restarted the daemon that was spewing into that log.

EDIT: I looked up why this happens. It seems linux/unix reference counts files, so they can be deleted until they are released by all processes that have them opened.

-