volume group out of space. How do I reclaim it?

Question

So long story there is an SVN VM (vmware server) that has run out of space. I can't commit because it says it's out of space. The VM has one 80GB virtual drive attached (hda), a /boot partition, and a big VG (VolGroup00) for /

I am archiving and deleting old SVN repos from the disk, but I am getting no free space back. What do I need to do to get this space back available for use? I use Linux for almost everything, but have never really played with LVM enough to know what's going on.

vgdisplay output:

--- Volume group ---
VG Name               VolGroup00
System ID             
Format                lvm2
Metadata Areas        1
Metadata Sequence No  3
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                2
Open LV               2
Max PV                0
Cur PV                1
Act PV                1
VG Size               74.41 GB
PE Size               32.00 MB
Total PE              2381
Alloc PE / Size       2380 / 74.38 GB
Free  PE / Size       1 / 32.00 MB
VG UUID               dPSZpL-kFBn-HpkH-ChfO-dw9q-YGg2-qHOiQF

Since yesterday, space has been freed...

Output of df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       72G   67G  959M  99% /
/dev/hda1              99M   15M   80M  16% /boot
tmpfs                  62M     0   62M   0% /dev/shm

Output of lvs

LV       VG         Attr   LSize  Origin Snap%  Move Log Copy% 
LogVol00 VolGroup00 -wi-ao 73.38G                              
LogVol01 VolGroup00 -wi-ao  1.00G

Something still seems weird, since it says 959M avail, and yet 67G used with a 72G size...

So I am getting space back, now, from deleting repositories...and yet, balance is not yet restored in the universe...

Answer 1

Deleting files doesn't change the size of a logical volume (and, by extension, doesn't affect free space on the volume group).

You can think of a volume group as like a "virtual disk", and the logical volumes on it as being like partitions, but ones that are a lot easier to work with (resize, create, delete) than regular partitions.

As Kamil says, if you're getting "out of space" errors when working, it's not LVM that's directly at fault -- that's a simple filesystem error, and you can determine which filesystem is out of space with df -h, and deleting enough stuff on that filesystem will eventually give you more space. However, with LVM, if you've got a filesystem with heaps of free space and another with none, you can shrink the LV with lots of free space and then allocate that space to the LV that's run out of space, and hence use your allocated disk more efficiently.

The process for shrinking and expanding filesystems is a bit risky (so have backups), and the shrink stage must be done on unmounted filesystems (so dropping to single user mode is good, and you can't shrink the root filesystem; if your filesystems support it (XFS, reiser, recent ext2/3) you can do online expansion). In general, the process is:

1. Shrink the filesystem you want to remove space from to a size a little smaller than what you want it to end up with (with something like resize2fs /dev/mapper/VolGroup00-largeLV xG). The reason why you shrink it to a little smaller is because if you get your math wrong and shrink the LV smaller than the filesystem, you'll stuff it up.
2. Shrink the LV you want to remove space from to the size you want (lvresize -L xG VolGroup00/largeLV)
3. Grow the filesystem on that LV to the new size of the LV: resize2fs /dev/mapper/VolGroup00-largeLV
4. Grow the LV that's too small to it's new, larger size: lvresize -L+nG VolGroup00/smallLV
5. Grow the filesystem on the too-small LV to it's new, larger size: resize2fs /dev/mapper/VolGroup00-smallLV

And you should now have plenty of space everywhere.

A few tips:

• lvs will list all your LVs, along with their sizes
• vgs will give you a quick display of all your VG's sizes and free space
• If your swap is on LVM, that's often a good place to get a bit of temporary free space if the machine isn't working too hard.

Good luck!

Answer 2

Is /dev/sda local or on the SAN? If it's on the SAN, then there's hope. If it's local, and it's using all of the available space, you're going to have to free up some disk space by removing files.

Update the question and let us know what the status is.

Edit

Ah, it's a VM! You're in luck (if there's free space on the host machine, anyway).

Create another virtual disk in the VM manager that is as large as you want the freespace to be. Then present that disk image to the VM.

Make sure the VM sees it (use dmesg to see if it shows up). Assuming it does, fdisk it, and create one partition.

Issue

 pvcreate /dev/whatever1

Where "whatever" is obviously the device. Probably sdb. 1 is the partition that you just made. It shouldn't have a problem making this physical volume.

Now, run

 vgextend VolumeGroupName /dev/whatever1

You can use vgdisplay to verify that the volumegroup has free space now. Now, grow the logical volume:

 lvextend -l +100%FREE

That should expand the logical volume to fill the volume group.

Now the tricky part. Assuming you've got an ext3 filesystem, you should be able to resize live, on the fly:

 resize2fs /dev/volgroup/logicalvolume

(where volgroup and logicalvolume is the actual path to whatever is mounted on /)

It'll say it's doing a live resize since the volume is mounted, and voila, running df -h should show that you have free space.

Answer 3

The amount of free space on your volume group will not change. It's all allocated by the filesystem you've formatted it with. Deleting files will only remove their inode entries from the filesystem, but the filesystem will still claim the space.

The only way to reclaim space in your volume group is to resize or delete logical volumes.

But, what you're describing sounds simply like you are out of disk space. You should check the output of df -h in your VM and see how much free space there is on the volume where the repository resides.

Answer 4

Try restarting your SVN server. If deleting files isn't freeing up space, they must still be in use somehow. This happens with active logs as well. I once had some log that grew by 10 mb an hour, so pretty soon I had to delete the log, but it didn't free up the space until I restarted the daemon that was spewing into that log.

EDIT: I looked up why this happens. It seems linux/unix reference counts files, so they can be deleted until they are released by all processes that have them opened.