Backups through distributed filesystems

Question

I hope I can articulate my question as precise as possible.

I am seeking for a way to make backups for my vm-servers as soon and frequent as possible as the data they are handling/producing are valuable. I have a KVM-Host and at least 2 Guests: Webserver (Apache/PHP) and Database server (MySQL/Solr). I don't care much about the host, but the guests. I don't want you to get deep into KVM or virtualization for this topic. This thread should be applicable to all vm based environments and all other environments as well. The vm scenario suites well because it is more tricky and represents one of the most complex situations in my imagination. At least, I need that on this basis.

Currently, I have in-vm backups and LVM-based snapshots which I generate 1-2 times a day. In case of a hardware failure (which I had recently) I miss a whole bunch of data in the best case.

So, one way could be to go down on each application/service and apply the best backup strategy available. Should be considered in each case.

Another interesting way seems to be the use of a distributed file system. The idea is to have a file system that acts slightly like the binary-log of MySQL. Or more general: It captures all write-actions on the file system and replicates it to another machine asynchronously. Depending on the network and the amount of data written, this can end up in seconds or minutes of latency and needless to say it misses all actions that are held back in cache. So I have a vm that is sitting on a distributed file system which is installed on the vm-host. Every write-action gets then asynchronous applied on a (say) backup-server. When it now comes to an hardware failure I can switch to the backup server (in theory) as the new master or simply copy the files back to the restored master-server in case, downtime is more acceptable than data loss. The effect should be that the virtual machine acts like it was turned of immediately seconds to minutes before. But not hours. I do not seek a master-master replication on file-system level, as this is not supported by most applications, especially database servers like MySQL!

So my question is: Is there anyone who has already made some experience with such configurations or has knowledge which is positive as well as negative for this attempt to backup data? I have no deep experience with these file systems. Especially on reliability and performance.

Answer 1

A distributed file system is not a backup, it's redundancy. It'll 'backup' your accidental deletions as well.

That said, DRBD.

Answer 2

The best possible answer to your situation is cluster storage whereby data is redundantly stored at the block level. There are several different way to implement this but the best that i can imagine (at least to your up-time specifications) would be an open stack cluster. Openstack will distribute both storage and computation so that in the event of a hardware failure both execution and storage are redundant and unceasing. In other words the best way to maintain data integrity and uptime is to make sure the application doesn't crash in the first place. As yoonix pointed out this will not protect you from user/logic errors but open stack includes tools for disk imaging/backup as well - loading an image and booting takes minutes if not seconds. Amazon Web Services and Rackspace are examples of openstack deployments. http://www.openstack.org/

A good place to start with openstack is devstack (pretty much a deploy script with various different deployment modes to test with) http://devstack.org/

The weakness of this implementation is lack of hardware, this system doesn't exactly shine in a small office with just two physical servers or the like (works great with blade systems though)

Answer 3

Have you considered migrating to a VMware platform and using their solutions for this?

'Fault Tolerance' (using vLockstep) keeps a second 'standby' copy of any VM up-to-date with all changes made to the primary version of the VM. If something happens to the primary version, the system fails over to the secondary VM immediately. (little to no downtime or impact)

'High Availability' keeps a standby VM prepped, but the standby VM is kept powered off. In the event of the primary VM failing, the system powers on on the standby VM automatically. (several minutes of downtime)

This stuff works great and it's very reliable. But... it's expensive. If you don't have the budget for the licensing needed to run these techs, this suggestion won't help you at all. Hopefully it at least gives you some more ideas of what tools are out there.