Offsite Backup Solution - RAID with LVM for GNU/Linux server

Question

Background

Hello, I am setting up a Ubuntu GNU/Linux server which will combine:
1) Software RAID1 (using mdadm) - To provide data protection against hardware failure
2) Logical Volume Manager (LVM) - Allowing flexibilty in organising my data and the ability to easily add more capacity in the future.

So far I have successfully:
1) Set up RAID1 using mdadm and created /dev/md0
2) Set up LVM making /dev/md0 a Physical Volume attached to a Volume Group called: vg_data. I have a Logical Volume called: lv_shared mounted on /home/shared :

NAME        FSTYPE LABEL UUID                                 MOUNTPOINT
sda
└─sda1      ext4         0xxxxxxx-2xxx-4xxx-8xxx-1xxxxxxxxxxx     /
sdb
└─sdb1     linux_raid_member ubuntu:0        02342342-2333-4444-8888-111111111111
  └─md0    LVM2_member                       57e241ad-aee3-4486-8eaa-222222222222
    └─vg_data-lv_shared ext4                 048b529c-2e39-4f49-83c9-333333333333    /home/shared
sdc
└─sdc1     linux_raid_member ubuntu:0        02342342-2333-4444-8888-111111111111
  └─md0    LVM2_member                       57e241ad-aee3-4486-8eaa-222222222222
    └─vg_data-lv_shared ext4                 048b529c-2e39-4f49-83c9-333333333333    /home/shared

My Question

This is fine if there is a hardware failure on one of the data disks, but how do I go about creating an offsite backup of the above setup?
The backup must be able to restore the entire system completely if something went wrong using just the offsite backup.

After reading up lots on the subject I found the following options but as I am completely new to this and want to ask and see what the community would recommend from their experience.
1) LVM snapshots
2) Duplicate one of the RAID drives. EG: (where sdd is my offsite backup physical disk.)

dd if=/dev/sdb of=/dev/sdd bs=64K conv=noerror,sync status=progress

What would the recovery process look like for the above backup methods?

I am really stuck on this and any pointers/links to relevent articles/suggestions would be really appreciated as I've spent a lot of time trying to figure out what to do, and my comfort zone is nowhere to be seen.

Thank you for reading my question and I hope it makes some sense to someone!

Update 1:
If I were to use rsync wouldn't I loose the Logical Volume Management scheme. A new problem also arises if you imagine that I expand the above setup to contain 3 pairs of RAID1. I then add those to expand total capacity of the logical volume. I would need to rsync the logical volume mount point (to copy all data) which would then contain 3 hard drives worth of data. This would need to be copied to span over multiple offsite drives (as I don't have a single drive that has the capacity of all 3 drives.) Is this possible using rsync. The more I think of this problem the only solution I can think of is making a backup server with the same configuration located offsite?

Answer 1

I think you should use rsync for off-site backup. It's will reduce daily backup traffic. Save files, not disks.

Answer 2

Offsite backup need to be bandwidth-efficient, so forget about using a raw dd to copy the disks/volumes to a remote location. Moreover, it is not generally needed to preserve the precise partition/logical volume layout for remote (read: slow and last effort) backups.

I'll try to show you the various possibilities, ordered by efficiency:

• use rsnapshot (itself based on rsync) to copy the data to a remote location, exploiting hard-links (on the remote side) to have multiple point-in-time backups. The first backup will take some time, but subsequent ones will be much faster. For preserving information about partitions/lvm, simply backup the output of lsblk; fdisk -l /dev/sdX; pvs; vgs; lvs
• use GlusterFS geo-replication, which is again based on rsync
• use dump and/or tar to make full and incremental/differential backups, sending them to the remote location via ssh
• use bdsync or blocksync to backup the raw data disks to a remote location. Pay attention to a) run these utilities against a snapshot of the source volume and b) to really know how they works
• use DRBD + DRBD proxy to have a real-time, block-synchronized remote server. Be sure to understand that a) DRBD proxy is a paid addon and b) this will not protect you from human error/sabotage, as changes are immediately sent to the remote side (think it as a network RAID1)

In short, there is not a silver bullet. However, I strongly suggest you to take the rsnapshot/rsync route.