How to set up a linux box for easy restoration?

Question

At my work, we use a linux box to store our source code and host our revision control software (svn). We also have some other products on there like "trac" for project management, fisheye and crucible for code reviews. If or when this box goes belly up, I would like to be able to keep all the services, software, user accounts, etc. up and running with near zero downtime. What solution am I looking for?

Some useful tips:
- Cost for the solution is not an issue. I would rather have a one time cost than a subscription though.
- I want minimal admin work for both maintaining the back up and restoring.
- The box is idle at night and on the weekends.
- We have another facility a couple of miles away but a relatively slow connection between the two buildings (faster at night though). I would like this restore option off site in case of fire, etc.
- I want the backup purchased, running, and ready before I ever call on it. Not "after crash, buy a new box, ..."
- The box is nothing fancy, just a standard desktop with ubuntu linux on it. Nothing we use it for is high performance.

Does anyone know of a solution for me? I am not well versed in anything linux or server related, so please give basic explanations with your answers.

Thanks!

Answer 1

You are actually talking about three interrelated, but diferent things:

1. Fault tolerance (how do I keep running, or get backup up with minimal downtime)
2. Data Backup ( what do i do when someone rm -rf's my repository )
3. Disaster Recovery (What do I do if my office is wiped from the face of the earth)

You should really think of them as three distinct but interrelated processes. I'll go into the most detail with fault tolerance as that seems to be what you are really looking for with the max 1 hour downtime.

Some things to consider for fault tolerance:

• How long will it take me to get new equipment?
• How long will it take me to rebuild the box?
• How long will it take me to verify and restore the data?

Take the sum of those times, multiply but 30% (nothing every goes as smooth as you think in an emergency) and if that sum is greater than your acceptable downtime, you need to start looking at some high availability setups. If it's less, it's your call on taking the risk your estimates are off and people may be down longer than you expected.

As far as some possible solutions well there are alot of things you can do. But in every case I would highly reccomend replacing the desktop with a server class machine. The quality of the components is higher, and they are built to be run 24x7x365 so there is a decent amount of redundency already built into the hardware (good RAID cards, redundent PSU's etc.)

• You could setup a standby server at your second site, then rsync your data across every x amount of time - where x is the amount of data you are willing to lose if the server goes down between replications. rsync is very small data pipe friendly after the first sync as it only sends delta's and changed files. Also setup your servers so they are accessed via CNAME so you can just swap where it is pointed and off you go.
• Do the same as above, except have the standby server at your primary location.
• Get a SAN/NAS and two servers. Then setup them up in an Active/Active cluster, or an Active/Passive cluster

Backups are a very important part, of the scenario too. You should remember there is no replacement for a point in time backup stored off site. Personally I still think backup to tape, and then having that stored off site by a company like Iron Mountain is the best option. For your size enviroment any of the "big" backup solutions - ArcServ, BackupExec, NetBackup should do just fine. Also make sure you TEST your backups at least quarterly. Nothing sucks more than finding out that backup you need is bad.

Disaster recover is really just sitting down and planning out where you will work from, where you will get the replacement equipment from, making sure that you have good offsite backups. I view DR as bringing all the components mentioned above into a cohesive plan of action for when the worst happens.

Answer 2

You could virtualize the environment then all you would need to do is restore the image.

Answer 3

seems like something as simple as rsync + cron might be sufficient here.

Answer 4

There are a lot of options here depending on the amount of data, complexity of the main system and how much management you want to do.

I like XenServer for this if the virtualized box is relatively small in size (few GB). For example, an internal app server we run is just 3GB in size. I can easily stop it make a backup and transfer the backup to another system. However, if you are not up to speed on XenServer this could be a steep learning curve.

I also use R1Soft's CDP server backup software but it is not really suited for a rapid recovery. It is great for doing a full bare-metal restore of a failed server, but for sub-hour backup and recovery.

I have done something like this for clients: Use the CDP backup software to clone a primary system to a cold spare. This assures the spare is identical to the primary system. Then we have hourly snapshots stored in the CDP server. CDP server uses a very efficient backups algo so there's little impact on the live server.

In case of a failure, you can restore the data from the CDP server to your cold spare.

The problem with this or an rsync-based approach is that you need to be sure to managed both the hot and cold spare so that their software stays in sync. You would not want to run OS updates on one and forget to do them on the other.

One recommendation is try as best as possible to use standardized configuration on your server, this will reduce the impact of config/update changes on restoring/rsyncing data over to the cold standby system.

Also, I like to keep my data - that is stuff I add - well isolated from the system. If you use LVM, LVM snapshot methods may work as well.

There are many options to consider but the best one will depend on your in-house expertise, time to management the system and data usage patterns.

Also if the amount of data is very light, you may want to look into desktop level backup/recovery tools. I am not as familiar with those.

http://www.r1soft.com/ CDP Server Software

http://www.citrix.com/ XenServer

http://samba.anu.edu.au/rsync/ rsync

Answer 5

Do not run production software on desktop hardware. At the very least, put 2 disks in the box and configure software raid for it, but that is suboptimal in case of disk failure: you still have to power down the server to replace the disk. With server hardware you will not have this problem: hot swap.

Monitor the hardware health (lots of free monitoring tools out there, my choice would be opsview because it's nagios with a brilliant web interface. You still get all the nagios plugins out there to work in it). They also have an enterprise edition for all the support you may need.

For the backups I would go with bacula, if you need support you can also get it from them (enterprise edition).

If you really do not have sysadmins in your company able to accomplish these things, outsource the operations to another company or fire your present ones and get yourself some good ones.

I aggree with Ignacio Vazquez-Abran that configuration management software is at the end of the day the most important thing. I would go with cfengine, though, puppet is a resource pig and it is more hyped than it should be. If your need another svn server, just netboot it, have it itself installed and configured in a matter of minutes. You can buy all the support you want from them as well.

Answer 6

Puppet can be used to get a system up and running, for when you need it. Set up a minimal install, add the Puppet client, and let the Puppetmaster do its work configuring the machine.