GlusterFS snapshot backup solution

Question

We're looking for a fast way to take point in time snapshots of our gluster bricks.

• It is not crucial that it is a perfect point in time snapshot (i.e if files change/get added/removed during the time of the backup it's okay for the backup to have/not have these changes).

• Ideally it would be able to run as live (no-downtime) backup, but we can deal with removing a node from the cluster and adding it back again afterwards if we need to.

• The backups will need to be transferred to a remote location, but this can be done after taking the initial backup locally if required.

• We have multiple bricks. All will need to be backed up, and can be done individually or all at once.

• We're currently running glusterfs 3.5.3 at the time of writing, but do tend to upgrade when new versions are released.

We have considered using rsync (as we would do on a normal volume) but it is really slow over gluster as we have many hundreds of thousands of quite small files (100kb - 3mb) and my understanding is that gluster talks to all nodes for each file to ensure it gives the correct version. Can we rsync over the data dir location on one of the server nodes (not over the glusterfs client)? Will this work as expected and be faster as it is not needing to check stuff with all nodes?

Gluster Volume Snapshot sounds like the perfect solution but I don't think it is released yet.

The gluster storage nodes have LVM but I'm not very familiar with the ins and outs of it. Could this be a solution?

Does anybody else have a good suggestion for how to deal with this situation? Or a real life experience? Thank you.

Answer 1

You need to put your gluster bricks on thinly provisioned LVM-Images.

1. Create LVM thinpool
2. Create LVM image for brick within thinpool
3. Creeate brick and setup gluster volume etc.

The easiest solution in my knowledge would be:

1. sudo gluster snapshot create volume_name snapshot_name
2. unmount snapshot from file-system
3. create lz4 image of snapshot with dd and lz4
4. mount image back

The total backup of a imagge with over 2M files and 18G takes about 90s on a dedicated server.

Pseudo-Code:

# create snapshot
echo $(date)" Creating glusterfs snapshot" >> $LOG
gluster snapshot create $SNAP_NAME $GS_VOLUME no-timestamp 2>>$LOG
echo $(date)" [OK]" >> $LOG

# get snapshot volume name
SNAP_VOL_NAME=$(gluster snapshot info $SNAP_NAME | grep "Snap\ Volume\ Name" | sed -e 's/.*S.*:.//g') MOUNT_OBJECT="/dev/"$VG"/"$SNAP_VOL_NAME"_0" 
MOUNT_POINT="/run/gluster/snaps/$SNAP_VOL_NAME/"$BRICK BACKUP_FS=$DIR_BA"/"$SNAP_NAME".ddimg.lz4"

# umount the image
umount $MOUNT_POINT

# create backup
echo $(date)" Creating lz4 of LVM image" >> $LOG
sudo dd if=$MOUNT_OBJECT 2>>$LOG | lz4 > $BACKUP_FS 2>>$LOG
echo $(date)" [OK]" >> $LOG

# mount image back
#mount $MOUNT_OBJECT $MOUNT_POINT

# delete (all) snapshots and umount
yes | gluster snapshot delete volume $GS_VOLUME 2>> $LOG

Answer 2

Bad solution imo.

GlusterNode1 ~ $ df -h | grep gluster

/dev/mapper/vg_12564073a0545f72de3g7835e711459a-33eacc4c9c2c4e6fbec16cdb55dd8b37_0 2.0G 35M 2.0G 2% /run/gluster/snaps/33000c4c9c2c4e6fbec16cdb55dd8b37/brick3

the snapshot is made with lvm and the lv4 image contains only brick of this node.