Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a standalone linux IDS system that I have been putting together. It runs MySQL locally, as well ntop, nagios, base, snort, apache etc. I want to be able to make a backup of the system that has everything from the old system so that I can do a quick dd from a static image of the system and then restore it to the last state.

To my knowledge I cannot DD a live system, which leaves me at a loss as to how to get everything from the server backed up.

I have never used rsync in the past, and I think it may be the solution, but I am not certain that I can use that on a live mysql DB.

I am pretty sure this is going to need a layered approach, but any input would be helpful.

edit:

The systems being monitored are low traffic most of the time, and apache is only for the web UIs for the applications, so its not as overloaded as it may seem

share|improve this question

9 Answers 9

up vote 2 down vote accepted

For mysql you can use mysqldump to dump the database contents to a file. Other than that, you usually only need to back up the config files (/etc and any other directories you modified), any other data directories, log files, and the list of installed packages. You should be able to restore the system by doing a fresh install of the OS, installing the packages from the list, and replacing the data and configuration files.

share|improve this answer
    
the problem with this is ntop doesn't interface with mysql to my knowledge, so while the mysqldump would catch most of it, it would miss the ntop data. –  Jimsmithkka Aug 26 '09 at 16:34
    
That's why I also mentioned copying other data directories. You just have to figure out where ntop stores its files. –  Kamil Kisiel Aug 26 '09 at 17:07

You know, I'm a REALLY big fan of shutting OFF the server and using something like Clonezilla or Norton Ghost to image that sucker.

Even if its 11pm on a Friday night, one a month... you get the idea.

A couple hours of "scheduled downtime" is worth HOURS and HOURS of crash recovery.

Like I tell my assistant, "I'm not saying this because I'm smart. I'm saying this because I WAS stupid at one point and learned my lesson the HARD WAY!"


Thank you for your update. Okay, so no downtime for this server.

Is it possible to build a "warm standby" you can use for the 2-3 hours it will take to clone the drive?

Is taking the network down at 3am on a Sunday morning an option?

Surely there has to be SOME sort of way to schedule some downtime?

I'm just trying to give you a "simple" solution here. :-)

share|improve this answer
    
there is no off time for this system, its currently the only IDS online. most of the bad traffic is sees comes in during usual "down time". I use dd of a linux live cd, usually parted magic, for exact duplication (similar to clonezilla or ghost). but that can take hours of possible break in time. –  Jimsmithkka Aug 26 '09 at 16:33
    
That's A LOT of stuff to be running on a server. I hope its pretty beefy. Do the HDDs ever take a break? :-) –  KPWINC Aug 26 '09 at 16:39
    
its decently beefy, I am probably going to have to reconfigure at a later point to better drive configuration. Its also not a "vital" system (according to management). The only intense apps are snort and ntop, nagios really only polls a few system every so often. The apache is just for the web interfaces, and its not a public server. –  Jimsmithkka Aug 26 '09 at 16:46
    
also note, the downtime has less to do with angry users and more to do with threat assessment. –  Jimsmithkka Aug 26 '09 at 16:48

I see two options in your situation:

  1. Run the system in a virtual container. You can take live disk images this way; even if it is the only VM on the system. Since it is analyzing network traffic, don't know how well that would work for you; but I see this as the only way to do a complete image of the disk without taking it down.
  2. Back up the individual components. Figure out all of the customized components, back them up, and test your backups by restoring to a new system. Maybe write out a plan or script that can automatically restore from a backup.

You can use rsync on a live DB but you could get data corruption. Stop the db, run the rsync, and start it again. You can also dump the DB and rsync that file.

share|improve this answer

I am really a big fan of using dirvish for backups. It is basically just a perl script that makes taking backups with rsync easy.

For the mysql database you really do need to use the mysqldump utility or something specifically designed for that purpose. Copying database files with rsync is not a good idea.

share|improve this answer

Usually one can just rsync the file system contents to another location and that's that. The incremental backup should be pretty fast (a few minutes? depending on the bandwidth available to rsync and how often you backup) so you can even take problematic services down for the duration with almost no effect (I did note the "no down time" requirement).

The usual offender for this practice is the MySQL database that doesn't handle snapshotting well. You can tackle this problem by either using Kamil's suggestion of dumping to SQL file and backing up that, or doing some sort of "hot backup". The advantage with the latter approach is that rsync likes it much better - creating a new SQL dump for each backup tends to cause rsync to copy the entire database everytime which is a lengthy process for large databases. using "hot backup", you can take advantage of rsync's ability to copy only changes to the data. Innobase (creators of InnoDB) sell a commercial product to do hot backup to the entire database. I found that as long as you do not have a lot of DB writes, you can get away with LOCK TABLES, rsync the binary files, and UNLOCK TABLES. when you restore MySQL thinks you had a server crash and recovers nicely (you do want to run mysqlcheck after recovery).

share|improve this answer

Another method for backing up your server, slightly like Dave's first suggestion, is to backup a standby raid device. This is a very different approach then my first answer so I'm putting it in another answer.

What you can do is to setup your server using software raid (mdraid) with mirroring. Then when you want to take a snapshot, disconnect one raid device (mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1), dd the contents of the disconnected device of the server, and reattach it to the raid (mdadm --re-add /dev/sdb1). The raid sync will copy all the data that was changed while the device was off.

When you restore you can dd the same data to both raid devices (its a mirror anyway). The restored image would look, at worst, as if the server lost power during a transaction so you should use a journaling file system and transactional databases anyway.

You'd probably want to use a 3 drive system in such a case so your system can handle a disaster during backup, in which case you'd either want to have a 3-way mirror and disconnect the 3rd device for backup, or even have it mounted on a hot-switch bay and just take it away physically as described in this thread (please read the thread through if you want to do it this way - the original poster has some issues with the specific commands and their order)

share|improve this answer
    
i really like this idea, but i rather use hardware raids for redundancy. A possible expansion on this, using HW raid is to use raid 1 with more 3+ discs and just physically pop a drive out and dd that instead, although this is not a solution for my current setup, i only have 2 drives, and after looking into their formatting, i need to do a total system rebuild to accomplich this. –  Jimsmithkka Aug 26 '09 at 18:03
    
The obvious benefit of the software raid solution is that you can completely automate the backup procedure as it can be implemented entirely in software. And so it can simply run periodically at night. working with 2 drives is also feasible, but you will be poorly protected against a drive failing during backup. Try the LVM solution offered by Rajat. –  Guss Aug 26 '09 at 18:33

i'm using

Blockquote

rsync --delete --times --recursive --perms --owner --group --verbose --progress --stats -e ssh root@192.168.0.100:/backup1/ /backup1/

Blockquote

on 2nd server on that server i have lvm config so there is very easy to run backup just taking a snapshot you can follow link

http://www.howtoforge.com/linux%5Flvm%5Fsnapshots

share|improve this answer
    
LVM backup is a great idea and its probably safer then my mdraid solution, although it is slower - lets assume that you want to LOCK TABLES on MySQL before you take the snapshot and UNLOCK TABLES after. With LVM backup you'd need to wait the second or so it takes the snapshot to complete, while with mdraid you'd wait only the time it takes to remove a drive from the raid which is basically instantaneous. –  Guss Aug 26 '09 at 17:38

Take a look at backupEdge at www.microlite.com

(Too bad they don't have a windows version)

share|improve this answer

Cloning isn't a viable backup scheme for a production system.

Look for the book "Unix Backup and Recovery" by Curtis Preston. Backing up a live system is possible with lots of commercial applications, as well as open-source applications like Amanda. There's going to be a learning curve, though.

Typically the most challenging aspect of backing something up is recovery. Start planning for recovery first. If you cannot afford to take the system down for backup, you can't afford to restore it eihter.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.