3

I'm planning on setting up two Ubuntu Servers as backup solutions where people can store images and documents. What I have in mind is that if a user saves/updates a file on a specific directory(Samba shared directory perhaps?) on one Ubuntu server, it'll synchronize it to another server.

What do you guys think? Is there a better solution for this? Thanks in advance!

Edit: Is it advisable to create the writable directory and set the access of it in Samba? The users have Mac OSX and Windows XP installed as their OSes.

5 Answers 5

7

rdiff-backup

Have you ever taken a look at rdiff-backup?

We've used it to synchronize over 4 million files from one Ubuntu file server to another without any issues. One very nice feature of rdiff-backup is that is keeps both a mirror copy and a differential copy (you can configure set how long). This comes in handy when you're backing up things like source code, images that change, etc as you can always restore the file to a point in time.

Here's the description from their website:

http://rdiff-backup.nongnu.org

rdiff-backup backs up one directory to another, possibly over a network. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, hard links, dev files, permissions, uid/gid ownership, modification times, extended attributes, acls, and resource forks. Also, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted. Finally, rdiff-backup is easy to use and settings have sensical defaults.


rsync

My second choice would be rsync and a cron job set to however frequent you desire.


JungleDisk / Amazon S3

My third choice (depending on your EXACT needs) would be on offsite backup to Amazon's S3 cloud via JungleDisk or some similar program.

http://www.jungledisk.com


We actually backup over 4 million files from our master Ubuntu file server to a slave Ubuntu file server (same location) via rdiff-backup once every hour.

We then backup once every 6 hours via rsync (or Jungledisk) to the Amazon S3 cloud.

This way we can handle the "easy stuff" with just the master/backup but if something catastrophic were to happen we'd still have the Amazon S3 backup in the cloud.

Hope this gives you some ideas...

4
  • Wow thanks for the ideas! That's pretty cool. I just have a few questions though, how big are the files are you transferring and how fast is your connection?Isn't backing up to Amazon S3 slow?
    – user11001
    Aug 12, 2009 at 9:24
  • There are around 4.7 million files (and growing). The files are mainly .PDF and video files ranging from 30k to 30MB in size. The rdiff-backup to the slave server is on a local 10.0.0.x network and takes about 45min to complete (before running again every hour). The Amazon S3 backup IS slow. Originally it took a week to get everything up there! Now it takes only an hour or two (runs every 6 hours) to keep everything synchronized with the master.
    – KPWINC
    Aug 12, 2009 at 15:24
  • cool! btw is your rdiff-backup set up unattended? how did you do it?
    – user11001
    Aug 13, 2009 at 9:53
  • @alimango Yes, rdiff-backup is running in a simple cron job.
    – KPWINC
    Aug 13, 2009 at 14:29
1

If you've got a good connectibity and bandwith between the nodes you may consider using DRDB and heartbeat to block mirroring your datas.

1

I really like glusterfs and have used it for some time to do the same thing you are thinking of doing.

About glusterfs:

GlusterFS is a clustered file-system capable of scaling to several peta-bytes. It aggregates various storage bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. Storage bricks can be made of any commodity hardware such as x86-64 server with SATA-II RAID and Infiniband HBA).

This tutorial describes how to get things up and running with server-side replication. Note that a "server" can also be a client...

//rkthkr

1

Take a look at the open source project BackupPC. I use it for backing up servers for a small business; some benefits:

  • A nice web interface to allow remote and headless control/monitoring and even restoring files
  • Uses a data pool for storage so any identical files only gets stored once; saving disk space
  • Allows pool to compress; saving disk space
  • Allowing several different protocols/tools for copying/backing up; rsync, tar, etc.
  • Keeps a history of files not just the latest (configurable)
  • Possible to send automatic reports by email
0

There are lots of different solutions to this. They all have advantages, but unfortunately they all have disadvantages. It depends on what you want to do. The easiest is a one way mirror for backups or warm-swap file server. Also easy is a shared datastore allowing load-balancing across your file servers, but you'd still have a single datastore. A lot more complex is two way syncing and replicated data. You then get into problems with what happens when one node can't see the other node. What happens if they get out of sync?

Simplest to complex

  • scheduled rsync
  • SAN
  • NBD and mirroring
  • cluster filesystems like lustre, glustrefs, GFS, OCFS2.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .