My Goals:
- Convert all my DomUs from image-based to lvm-based
- Minimal downtime
- Minimal performance impact on other DomUs/Dom0
My plan / setup:
All my images are in a big lvm-volume, all lvs for the new setup are created with the filesystem (ext3/ext4). So my approche is, to do this for every single machine:
- make a snapshot of the big image-lv
- mount this snapshot (e.g. on
/tmp/img_snap/
) - loopback-mount the image itself (e.g. on
/tmp/convert_src/
) - mount the new lv (e.g. on
/tmp/convert_dest
) - rsync from
/tmp/convert_src
to/tmp/convert_dest
- unmount everything
- remove the lv-snapshot
- shutdown the DomU
- do steps 1-7 again
- change the disk-settings in the DomU's config
- fire up the domU again
Steps 1-7 are already scripted so in step 9 I just can fire up this script again. This means a bit of overhead because the lv-snapshot is not necessary, but I'm ok with and aware of that.
Because of the fact that the original machine can run until step 8 (and step 9 should only copy the delta) the downtime should be minimal. I also renice
and ionice
the rsync process(es) so the impact on all other systems should be minimal. Or so i thought ...
- Debian lenny
- Xen 3.2 from debian
- HP ML350 G5 with their SmartArray Controller (iirc: e220?)
- drives connected via SATA
Questions / Problems:
- This approach is ok and tested with small images, but I have some big images 400GB+. This takes ages. Are there any better approaches?
- Sometimes - I don't exactly know why - the load of the Dom0 goes up way to much, the machine gets 'stuck' and i get messages like
INFO: task loop27:23352 blocked for more than 120 seconds
,INFO: task pdflush:7329 blocked for more than 120 seconds.
, ... and their call traces and stuff. Why/When? I can't figure out a pattern? Is there too much IO, too CPU? And most of all: Why doesn't get my rsync-processes blocked - i reniced (renice 20 -p $(pidof rsync)
) and ioniced (ionice -c2 -n7 -p$pid
) them - shouldn't these processes be the one to be first blocked? - Generaly any hints and improvement ideas?