8

We are embarrassed to ask this question. Apologies for our lack of UNIX expertise. We have inherited 40+ centos 5.5 servers, and don't know how to back them up. We need low level clone type images so that we could restore the servers from scratch if we had to replace the HDs etc.

We have used the "dd" command, but we assume this only works if you want to back up one local disk to another, not 40 servers to one server with an external USB HD attached.

All 40 servers have a pair of mirrored disks (dont know if its HW or SW raid). Most only have 100MB used. SErvers are running apache, zend, tomcat, mysql etc. Ideally we dont want to have to shut them down to backup (but could).

We assume that standard unix commands like tar, cpio, rsync, scp etc. are of no use as they only copy files, not partitions, all attributes, groups etc. i.e. do not produce a result which can simply be re-imaged to a new HD to get the serer back from dead.

We have a large SAN, a spare windows box and spare unix boxes, but these are only visible to one layer in the network. We have an unused Dell DL2000 monster tape unit, but no sw or documentation for it. WE have a copy of symantec backup exec, but we have no budget for unix client licenses. (The company has negative amounts of money).

We need to be able to initiate the backup remotely, as we can only access the servers in person in an emergency (i.e. to restore)

Googling returns some applications to do this, e.g. clonezilla - looks difficult to install and invasive. Mondo, only seems to support backup if you are local to the machine. Amanda might be an option, but looks like days/weeks of work to learn and setup?

Is there anything built into Centos, or do we have to go the route of installing, learning and configuring a set of backup softwares?

Any ideas? This must be a pretty standard problem which goggling doesnt give an obvious answer.

1
  • 1
    Why do you think you need an full disk backup? If you have a good backup of the data/config then rebuilding should be trivial. Particularly if you have something like puppet in place.
    – Zoredache
    Nov 16, 2012 at 16:31

3 Answers 3

3

Considering that this is an "emergency" situation, you should do the following only for so long as it takes you to get competent staff in place and a more reliable long-term backup strategy. Do not do this forever.


You can use dd to make images of the hard drives in these systems (or the individual partitions, if you want). dd can be used to read or write data to a plain file, as well, and you'll take advantage of this. Since the hard drives are likely much larger than the actual space used, I recommend compressing them as well.

So the general idea would be something like this:

  1. Plug USB hard drive into USB port.
  2. Mount USB hard drive, e.g. at /media/backup.
  3. Copy the hard drive images. For example:

    dd if=/dev/sda of=/media/backup/$(hostname)-sda.img
    

    Better yet, compress the image while you take it:

    dd if=/dev/sda | gzip -c > /media/backup/$(hostname)-sda.img.gz
    
  4. Unmount the USB hard drive, and move to the next machine.

You can use a tool such as kpartx to work with the backup images as if they were actual hard drives (see its man page) or just restore them directly by reversing if= and of= in the dd command.

4
  • Would LVM be an issue with the dd? We don't know how the disks are actually arranged.
    – cjc
    Nov 16, 2012 at 18:00
  • Nice guide, very clear. Could this be done to an NFS or iSCSI mount, thus allowing us to do the image remotely? How do running process affect it? Obviously we would stop databases at least.
    – eos
    Nov 16, 2012 at 18:01
  • df -k shows: /dev/sda6 /, /dev/sda8 /usr, /dev/sda3 /home etc. So doing if=/dev/sda will get all of them?
    – eos
    Nov 16, 2012 at 18:10
  • Sure, you can create the image(s) wherever you like, as long as it's mounted somewhere that you can write to it. Copying /dev/sda will get the entire disk; if you need to access individual partitions, you can use kpartx on the image. Nov 16, 2012 at 18:11
3
  1. Everything reliable will take a lot of time to learn and implement (and test) in such an environment, there is no easy way around this.

  2. You can use standard tools like rsync etc. to get complete backups, you just need to make sure that all the informations you need for recovery is backed up as well (e.g. the partition tables and boot records).

  3. The most important point: You have a very complex environment which you obviously don't understand fully and you lack the necessary knowledge and experience. This way, you will never get a disaster-proof backup. Your best (I would even say only) option is to hire a consultant to create a viable backup solution for you. Or better yet, for an environment of 40+ servers, hire a competent sysadmin to manage the systems.

1
  • Thanks for the help. For now any backup would be better than nothing. There is no money for consultants, its us or nothing, unfortunately. Looks like you are recommending rsync. I will try and research how to use this to do backups without missing info. I have visualized most of environment to make a staging version, so we are not completely without hope, just without backup knowlege. WE have networking experts and programmers.
    – eos
    Nov 16, 2012 at 16:52
2

G4L or Clonezilla will give you restorable image backups.

You'll need to schedule downtime for each server, have enough connected storage for all the compressed images. Once you're done, you'll need to set up some kind of ongoing incremental backup.

6
  • Thanks for the help. How long might it take an average admin to learn, setup and get one of these two working for a single first backup? where does the connnected storage need to be connected? IF the server is down, how does one back it up? Thanks again.
    – eos
    Nov 16, 2012 at 16:55
  • Connected storage can be USB or Ethernet. Image backup consists of a server shutdown, booting the G4L or Clonezilla USB or CD, identifying the source disks (or partitions) and identifying the destination (USB Drive or file server), and go. The learning curve is pretty short.
    – kmarsh
    Nov 16, 2012 at 17:29
  • Restore consists of replacing bad disks, booting the G4L or Clonezille and restoring the image. Then you have to catch up on any updates made since then, but at that point you have a working system, with a working backup/restore client installed, such as NetBackup.
    – kmarsh
    Nov 16, 2012 at 17:31
  • Great, I see, this is like using a copy of norton Ghost on a local PC. The crux is we really need to do 40 machines remotely, but if we have to, we may be able to send an engineer to the DC and have him stand there all day with boot disks.
    – eos
    Nov 16, 2012 at 17:49
  • This might be possible with Clonezilla, a PXE server and some creative scripting. This link has some details, but is not exactly the process you are looking for. The key to good boot image creation is that it can't be live, it has to be static. openlogic.com/wazi/bid/188051/…
    – kmarsh
    Nov 16, 2012 at 18:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .