LVM on images very slow

Question

After creating an lvm

dd if=/dev/zero of=1.img bs=1M count=16384
dd if=/dev/zero of=2.img bs=1M count=16384
losetup /dev/loop1 1.img
losetup /dev/loop2 2.img
pvcreate /dev/loop1
pvcreate /dev/loop2
vgcreate LVM1 /dev/loop1 /dev/loop2
vgchange -an LVM1

I allocate 10G volume and format it

lvcreate -L10240 -n lv_34179984_0 LVM1
mkfs.xfs /dev/LVM1/lv_34179984_0
mount -t xfs /dev/LVM1/lv_34179984_0 /home/test/image

When I try to copy a 8G file to /home/test/image it blocks after 215M copied, iostat gives me

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.00   29.82    0.00   70.18

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     1.00    0.00   0.00 100.20

How to solve this?

Answer 1

You solve this by not creating a filesystem on top of volume group on top of files that are already in a filesystem (possibly in a volume group) on a disk. To use an Inception reference, you've gone too deep my friend.

Every time you try to do something on /home/test/image your system has to go through the following steps:

• Filesystem operation on /home/test/image
  • Volume Group operation
  • Virtual disk (loop device) operation on loop1
    • Filesystem operation on 1.img
      • Disk operation on wherever 1.img lives
  • Virtual disk (loop device) operation on loop2
    • Filesystem operation on 2.img
      • Disk operation on wherever 2.img lives

This still leaves out a bunch of intermediate steps that happen in the kernel whenever you initiate disk I/O (physical disk or loop device). Simply put, you're asking your system to do so much work the performance will be lousy.

You can marginally improve the performance by putting 1.img and 2.img on separate physical disks, but that is unlikely to give a substantial benefit. Your only real option for decent performance is to create the volume group on physical disks.