Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

On an RHEL5.4 system I setup a script to backup a drive with dd copy every night by cron. I spit the dd output into a log file and email. It is in /etc/crontab and /var/spool/cron/root when I figured out it wouldn't even run under cron.

The script is supposed to copy /dev/sda to /mnt/backup/sda.img (/mnt/backup is a mounted 250gb external).

When I run it as root at the terminal it works fine, I can see data being written to the disk and sda.img is getting bigger.

However when run as cron, I get the output from dd saying it copied 147gb, but cannot find where it spat that 147gb to - it didn't put it in sda.img. Its not on the filesystem anywhere as there is only 50gb left on it.

Where did it go? And how can I make sure the same thing happens in cron that happens in terminal.

I do stop crond and start it before and after the backup, however I am under the impression that cron kicks the job off, I shut it down, it backs up, starts again and is on its merry way.

Thanks.

EDIT: Sorry, the dd line is
dd if=/dev/sda of=/mnt/backup/sda.img bs=400K

And the cron line is
01 0 * * * 2-6 /root/applog_backup.sh

I can access the files when it works with
mount -o loop,offset=32256 sda.img /mnt/restore

I shut down cron to prevent hourly jobs from modifying the disk during backup. I have also shutdown other services and the production database to minimize disk writing in the important places.

share|improve this question
1  
A dd image copy is not a backup, far from it... are you aware? Also, you didn't mention the command you are using, and the line in /etc/crontab. –  Juliano Nov 28 '09 at 14:18
2  
I'd also like to see the actual dd command and the cron job entry –  Matt Simmons Nov 28 '09 at 14:32
3  
Though I'd say that a dd image can definitely be a backup –  Matt Simmons Nov 28 '09 at 14:32
2  
Why are you shutting down cron? You should never need to do that. –  Paul Tomblin Nov 28 '09 at 14:52
3  
Matt: Backups need a few important properties in order to be called "backups". In this specific case, the image copy is not resilient to media errors (just like the actual partition...), you can't do a partial restore, you can't restore the backup to a disk with different geometry, and more importantly it looks like the OP is doing a live copy of an online read-write mounted disk. There is no guarantee at all that the disk contains all the files (some may be in cache) and that the filesystem will remain in consistent state from the beginning to the end of the copy. –  Juliano Nov 28 '09 at 16:17

4 Answers 4

up vote 10 down vote accepted

You have your "backup" script being executed by cron... and you shut down cron in the script in order to prevent cron jobs from running during the "backup". You really can't see where is the problem here? Your script shuts down crond, but crond is running your script, so, shutting down crond will close the descriptors connected to your script, which will then die, either with a broken pipe or by a interruption signal from crond itself.

Since the script died, crond won't be restarted anymore. That is what we call "shot yourself on the foot".

Even after restarting crond, it won't have registered that the job completed, since it was shutdown during its execution and/or had to signal its termination. Either crond itself or anacron (depends on what cron scheduler you are using), it will have to run the job again, potentially going into an infinite loop.

Your problem is an excellent example of everything that is wrong with inventing your own "backup" solution if you have no real-life experience with reliability management and disaster recovery. Worse, a lack of knowledge of how the system works.

First, and most important, you do not make a raw disk dump on a live filesystem. Filesystems were invented so that you do not touch the raw disk contents directly. You want to save the files stored in the filesystem, that is what matters for you. So you have to access them through the filesystem, not the raw bytes stored on the disk. If the partition is mounted, there is absolute no guarantee that your data is actually stored on the disk and that the disk will stay in a consistent state during the copy.

Even if you could snapshot the state of the disk in a recoverable manner (like a sudden power failure, which could be quickly recoverable with a journaled filesystem like ext3), that is never true with a hot disk dump. A disk dump takes a long time to complete, there are virtually infinite intermediate states between the beginning and the end of the dump, and the dump will contain a mixture of these states, which is potentially unrecoverable even with a journaled filesystem.

And I still didn't mention everything else that is wrong with raw disk dump backups:

  • There is no difference between used and free space. It doesn't matter if you have a single 100 kB file or 250 GB in tens of thousand files, everything will be copied. It is extremely inefficient. You use this approach only if you need an identical clone of your disk, and with the disk unmounted.
  • You can't do differential or incremental backups. All your backups must be full backups. All kinds of inefficiencies:
    1. Since this takes a lot of space, you usually will keep only a single copy of all the data. If your files are damaged or deleted before the backup and you don't notice, the damaged or deleted data is copied over the previous backup, making it useless.
    2. Since you do this over the previous data, if your system fails in the middle of the dump (which takes a longer time since you are copying the whole disk), both your original system and the backup are lost on a single shot.
    3. If 100 kB of data changed since the previous backup, you will still dump the whole disk. In your case, this is at least a million times less efficient.
  • You can't restore this dump to a disk with a different geometry. If your replacement disk is smaller there is no discussion; if your replacement disk is bigger, you may be able to restore either losing the extra space or doing some manual (and dangerous for the uninitiated) changes to the partition table and partition superblocks. Do you want to trust your files, your work, to such a hack?
  • Even if you mount the raw image using a loop device and copy the files manually... you end up copying your files manually!! So what you earned from doing a raw disk dump?? Just copy your damn files!

Many people have been there and have a lot of experience to share regarding disaster recovery. Don't try to invent your own backup solution, you will end messing things up. Use proper backup tooks, like dump, tar or rsync. If you need something more robust, use Amanda or Bacula, or one of the other hundreds of solutions ready to use.

Probably not the answer you were expecting, but had to be said.

share|improve this answer
    
Wow thank you for the grilling... what a dickhead I am... I am going to switch backup methods now. Since we have many backup disks and lots of space on them I think I will just do a cp -dpRx of the root. Thanks again. –  hamstar Nov 29 '09 at 10:39
    
*rsync -auEHAXx --stats –  hamstar Nov 29 '09 at 11:35

I would suggest that you take the time, and look into a product called Bacula. It's open source (free!), and does a great job. It's complex to setup, but once you get going, you'll forget to worry about backups ever again.

share|improve this answer

I suspect that your cron line is not correct. Please edit your crontab file with the below entries:

1 0 * * 2-6 /root/applog_backup.sh

Do let me know if this resolves your issue.

share|improve this answer

Please, please, please do not use dd to take backups of your physical volume, for all of the reasons stated in the comments to your question.

At the very least if you want to do disk-to-disk backups use something like rsync (although even that isn't without its problems), and come back here if you can't get that to work from cron.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.