Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

System in question has Debian Lenny installed, running a 2.6.27.38 kernel. System has 16Gb memory, and 8x1Tb drives running behind a 3Ware RAID card.

The storage is managed via LVM, and is exclusively comprised of ext3 filesystems.

Short version:

  • Running a KVM guest which had 1.7Tb storage allocated to it.
  • The guest was reaching a full-disk.
  • So we decided to resize the disk that it was running upon

We're pretty familiar with LVM, and KVM, so we figured this would be a painless operation:

  • Stop the KVM guest.
  • Extend the size of the LVM partition: "lvextend -L+500Gb ..."
  • Check the filesystem : "e2fsck -f /dev/mapper/..."
  • Resize the filesystem: "resize2fs /dev/mapper/"
  • Start the guest.

The guest booted successfully, and running "df" showed the extra space, however a short time later the system decided to remount the filesystem read-only, without any explicit indication of error.

Being paranoid we shut the guest down and ran the filesystem check again, given the new size of the filesystem we expected this to take a while, however it has now been running for > 24 hours and there is no indication of how long it will take.

Using strace I can see the fsck is "doing stuff", similarly running "vmstat 1" I can see that there are a lot of block input/output operations occurring.

So now my question is threefold:

  • Has anybody come across a similar situation? Generally we've done this kind of resize in the past with zero issues.

  • What is the most likely cause? (3Ware card shows the RAID arrays of the backing stores as being A-OK, the host system hasn't rebooted and nothing in dmesg looks important/unusual)

  • Ignoring btrfs + ext3 (not mature enough to trust) should we make our larger partitions in a different filesystem in the future to avoid either this corruption (whatever the cause) or reduce the fsck time? xfs seems like the obvious candidate?

share|improve this question
    
I tried fixing your kernel version number (it was 2.65...), hopefully I didn't mess it up. –  ptman May 13 '10 at 12:38
    
FWIW, I have used both XFS and Ext3 on a SAN + LVM and had had more problems with XFS. XFS formats and grows super fast, but it is much more sensitive to hard umounts, i.e. killing a VM or poweroff. we have had to restore from backup several times because of subtle corruption that fsck didn't pick up. Ext3 has been slower but much more realiable –  Mark Porter May 13 '10 at 15:08

3 Answers 3

In this case it's probably virtio and the 1TB problem.

But for me i came accross similar problems while accessing alternately a device outside a virtual machine (including shutdown this machine) and inside the virtual machine. If you access the block device inside the virtual machine with direct access (e.g. in kvm config), this means without cache/buffers and outside with buffers you can get the following problem:

  • resize device outside vm, the cache/buffers get filled on the kvm host.

  • start vm, recognize (other!) problems and shutdown.

  • fsck device.

If all went very bad you read date from cache but that was changed within the previously run virtual machine which accessed the device without buffers/cache!

I also do a lot of ext3 resizes (since 2.6.18) and i do this all the time ONLINE! AFAIK this uses kernel functions to resize, while offline resize uses userland code.

share|improve this answer

Also check your KVM cache settings; KVM can do no caching, read cache, writeback, and writethrough caching, which can take you rather by surprise.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.