3

My Goals:

  • Convert all my DomUs from image-based to lvm-based
  • Minimal downtime
  • Minimal performance impact on other DomUs/Dom0

My plan / setup:

All my images are in a big lvm-volume, all lvs for the new setup are created with the filesystem (ext3/ext4). So my approche is, to do this for every single machine:

  1. make a snapshot of the big image-lv
  2. mount this snapshot (e.g. on /tmp/img_snap/)
  3. loopback-mount the image itself (e.g. on /tmp/convert_src/)
  4. mount the new lv (e.g. on /tmp/convert_dest)
  5. rsync from /tmp/convert_src to /tmp/convert_dest
  6. unmount everything
  7. remove the lv-snapshot
  8. shutdown the DomU
  9. do steps 1-7 again
  10. change the disk-settings in the DomU's config
  11. fire up the domU again

Steps 1-7 are already scripted so in step 9 I just can fire up this script again. This means a bit of overhead because the lv-snapshot is not necessary, but I'm ok with and aware of that.

Because of the fact that the original machine can run until step 8 (and step 9 should only copy the delta) the downtime should be minimal. I also renice and ionice the rsync process(es) so the impact on all other systems should be minimal. Or so i thought ...

  • Debian lenny
  • Xen 3.2 from debian
  • HP ML350 G5 with their SmartArray Controller (iirc: e220?)
  • drives connected via SATA

Questions / Problems:

  • This approach is ok and tested with small images, but I have some big images 400GB+. This takes ages. Are there any better approaches?
  • Sometimes - I don't exactly know why - the load of the Dom0 goes up way to much, the machine gets 'stuck' and i get messages like INFO: task loop27:23352 blocked for more than 120 seconds, INFO: task pdflush:7329 blocked for more than 120 seconds., ... and their call traces and stuff. Why/When? I can't figure out a pattern? Is there too much IO, too CPU? And most of all: Why doesn't get my rsync-processes blocked - i reniced (renice 20 -p $(pidof rsync)) and ioniced (ionice -c2 -n7 -p$pid) them - shouldn't these processes be the one to be first blocked?
  • Generaly any hints and improvement ideas?

1 Answer 1

0

rsync is a very good tool. If you use it, take the options "-aSH --delete" so you will get an amlost identical target (ctime-stamps of soft-links will have the current time).

From your writing I understand that the time needed for the rsync seems to be your main problem, while shutdown/startup of the DomUs is pretty fast (including their applications, I assume).

I see another problem in your snapshot of your big LV holding all your running DomUs. If you have many write-requests your snapshot will fill up pretty fast - and the snapshot will slow down the DomUs during writes.

In this situation I would chosse a different approach: Use a software raid 1 to mirror from your images to your new LVs.

If you want you could combine that with snapshot/rsync to reduce the amount of data that needs to be moved. But IMHO the time neede to get that one sorted out and scripted is not worth the effort.

So here is the plan - in both cases you need block-devices (loopback-mount your images):

A) Using md (set up via mdadm) Write a script to:

  1. Shut down DomU
  2. Loopback-Mount the DomUs image
  3. Create a degraded md-raid-1-device with that image
  4. Boot up your DomU with "phy:mdN" as new disk device
  5. Raise number of raid1-members from 1 to 2
  6. add your target LV
  7. Change the setup of your DomU so it will use the phy:vgX/lvY in the future
  8. If the mirror is complete, restart your DomU with the new setup

Whith plan A you have two downtimes lasting one restart of the DomU. The mirroring will take place in the background as fast as possible.

Drawback: All data will by synced (RAW mirror, not based on file-system)

You could get around that by creating the md-raid1 with (--assume clean) without degradation. But this will be tricky (you will need rsync to get your data to the LV then mirror the delta using md-bitmap).

B) Using drbd (set up via /etc/drbd.conf) If you plan to switch to drbd in the future you should go this way. I did not test this, but can't see a reason why drbd should not mirror between 127.0.0.1 and 127.0.0.2 ;-)

I did not test this, but it should work - and is more complicated than plan A.

Drawback: Complicated, not tested.

But: If you want to switch to drbd anyway your can prepare your DomU this way. Run in disconnected mode until your cluster is ready, then connect, sync via IP and start live-migrating...

4
  • Really nice approches, thanks a lot for your ideas! First of all: sure, i use some (many) options for rsync to be sure, i have all my (meta)data copied. I'm aware of the filling lvm-snapshot, but i have enough place to buffer the changes. I really like the md-approach - next time i'll try this. To comment on your DRBD-idea: if i'd go with DRBD, i wouldn't want to mirror each lv by its own, i'd rather mirror the pv(s) via DRBD - nevertheless: nice idea. Anyways, i've already startet with my approach. All works fine, except the sometimes appearing load-jumps on dom0.
    – m.sr
    Apr 17, 2011 at 8:26
  • BTW: My biggest image took ~36h for the first sync (steps 1-7). Long, but ok, because it ran nicely in background. I wonder how long the second delta-sync will take ...
    – m.sr
    Apr 17, 2011 at 8:29
  • So - how long did it take? I once did a rsync-setup via 4 x 100 MBit. After the initial rsync it took 4h every night to resync the delta. It depends on two things - the amount of data to compare and the change-delta...
    – Nils
    May 1, 2011 at 20:35
  • It was pretty fast. About 36h after the initial rsync-run, a second run took ~1/2h (again with lv-snapshot and in background and with running domU). The final rsync run (directly from image to lv - w/o snapshot and domU shut down) even felt faster. I do not have any real data (how much was copied, how many files, ...) because it was so fast, that i had no time to benchmark it ... ;)
    – m.sr
    May 3, 2011 at 13:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .