I'm trying to test and document a backup and restore procedure for Centos 6. Here's where I'm up to, but there are a few areas where I need a bit of clarity. CentOS backup/restore documentation on the 'net is a bit hit and miss.
General backup and restore plan
Backup your systems each day using your favourite backup software. I'm not going to go into this in great depth, but let's say you have a proper backup system that allows you to make a backup of one system and restore it to another.
Smoke and flame engulfs one of your servers! After dealing with the immediate danger you realise that an important system is irreparably damaged. You need to restore it to different hardware.
Check the backup files. Look at the backup of the failed system's
/etc/redhat-release
file. Use this to establish which version (patch level) of CentOS the failed system was using? Grab the install media for this version.Using the install media, do a minimal install of the Operating System to your replacement hardware, partitioning the disks as appropriate for the system's end use.
After the minimal system is installed, temporarily disable selinux,
echo ‘0’> /selinux/enforce
stop iptables,service iptables stop
and install your backup client.Recover from the backup, excluding the following files from recovery:
/proc
/sys
/tmp
/dev
/var/lock<- don't exclude from restore - see answer
/var/run<- don't exclude from restore - see answer
/var/tmp
/etc/fstab
/etc/mdadm.conf
/etc/mtab
/etc/resolv.conf
/etc/networks
/etc/sysconfig/network*
/etc/sysconfig/kernel
/etc/hosts
/etc/modprobe*
/etc/networkmanager <- to ensure that IP isn't restored - see answer
/etc/udev
/lib/modules
/boot
When the restore has completed, reboot and watch for errors
Check that the network configuration is correct. You may need to use
system-config-network
to make changes to your network settings.Some applications like Apache and MySQL may not start correctly after the restore. BecauseThis shouldn't be a problem as long as you don't exclude /var/run and /var/lock from the restore - see answer./var/run
was excluded from the restore, subfolders like/var/run/httpd
won't exist, and so applications won't be able to create PID files properly. You need to restore folders like/var/run/httpd/
and/var/run/mysqld/
and give them the correct permissions.After completing remedial action ensure applications come up correctly.
If you're running a MySQL database it may still be OK without you having to restore it from any flatfile backup you may have made. You can check the state of the database by running
mysqlcheck -c -u root –p******** --all-databases
. If you see any errors runmysqlcheck -c -u root –p******** --all-databases --auto-repair
to repair them. You should always ensure that you have a proper backup your database as indicated in the answer below. I personally use mysqldump.Patch the system up to the latest level using
yum update
.After rebooting to ensure the system comes back up clearly and thoroughly checking /var/log/messages for any errors, test the system's functionality to ensure it is operating correctly. When this is the case, use
system-config-network
to change the IP address to that of the original faulty system.
Issues/Questions
Excluding/var/run/*
from the restore causes the subfolders used to contain PIDs for some apps not to not be created when you restore. Is it really necessary to exclude/var/run/*
from restore? Is a better way to simply not restore the PID files?When the system was restored, the IP address of the 'faulty system' was also restored. I didn't want this. I must have missed a file off of my 'exclude from recovery' list. Any ideas where it is?
When updating I get lots of messages like
/sbin/ldconfig: /usr/lib64/libblah.so is not a symbolic link
. When I reboot the system after updating some services don't come up correctly. I wonder if this is something to do with the backup system restoring the files that the symbolic links point to instead of the symbolic links themselves. If I run ldconfig and look at one of the shared objects it complains about, the shared object is an actual file rather than a symlink. Anybody else seen this?