We're looking for a fast way to take point in time snapshots of our gluster bricks.
It is not crucial that it is a perfect point in time snapshot (i.e if files change/get added/removed during the time of the backup it's okay for the backup to have/not have these changes).
Ideally it would be able to run as live (no-downtime) backup, but we can deal with removing a node from the cluster and adding it back again afterwards if we need to.
The backups will need to be transferred to a remote location, but this can be done after taking the initial backup locally if required.
We have multiple bricks. All will need to be backed up, and can be done individually or all at once.
We're currently running
glusterfs 3.5.3at the time of writing, but do tend to upgrade when new versions are released.
We have considered using rsync (as we would do on a normal volume) but it is really slow over gluster as we have many hundreds of thousands of quite small files (100kb - 3mb) and my understanding is that gluster talks to all nodes for each file to ensure it gives the correct version. Can we rsync over the data dir location on one of the server nodes (not over the glusterfs client)? Will this work as expected and be faster as it is not needing to check stuff with all nodes?
Gluster Volume Snapshot sounds like the perfect solution but I don't think it is released yet.
The gluster storage nodes have LVM but I'm not very familiar with the ins and outs of it. Could this be a solution?
Does anybody else have a good suggestion for how to deal with this situation? Or a real life experience? Thank you.