Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I want to make incremental backups on a Linux machine, on the next way:

BACKUP1                BACKUP2
|                      |
|                      |
|--- file1             |--- file1 (symlink to file1 in backup1 because it hasn't changed)
|                      |
|                      |
|--- file2             |--- file2 (is copied again because it has changed)
|                      |
|                      |
|--- file3             |--- file 3 (same as file1, a symlink)

Is there any simple way of doing this? I was using this script:

#!/bin/sh

date=`date "+%Y-%m-%dT%H:%M:%S"`
rsync -aP --link-dest=~/Backups/current ~/Documents ~/Backups/back-$date
rm -f ~/Backups/current
ln -s back-$date ~/Backups/current

But that just copies everything again. Thanks :D

share|improve this question
1  
FYI: --link-dest produces hard links, not symlinks. –  andol Aug 22 '11 at 17:37
    
I prefer using something like time capsule or windows 2008 backup system –  pmerino Aug 22 '11 at 17:39
    
@andol: how would I porduce symlinks instead? –  pmerino Aug 22 '11 at 17:41
    
So, do you want to use hard links or symlinks? You example mentions symlink, but then you mention Time capsule, which assuming OSX and Time Machine, implies hard links. –  andol Aug 22 '11 at 17:43
    
Any particular reason you want to go with symlinks instead of hard links? –  andol Aug 22 '11 at 17:45

3 Answers 3

up vote 3 down vote accepted

I use a self written bash script for it using rsync and cpio: http://pastebin.com/uRdH2uQf

So, the first thing I have done is create a directory structure. I work like this: create a backup every day, the 7th day (sunday) I take the last backup (previous week sunday) and put it in weekly. Every 4 weeks I keep a monthly backup.

All these backups are incremental and based on 1 full backup.

my directory structure is based in /mnt/backups and looks like this:

--- SERVER1
   |--- daily
        |  --- 0
        |  --- 1
        |  --- 2
        |  --- 3
        |  --- 4
        |  --- 5
        |  --- 6
    |--- weekly
        |--- 0
        |--- 1
        |--- 2
        |--- 3
    |--- monthly
        |  --- 0
        |  --- 1
        |  --- 10
        |  --- 2
        |  --- 3
        |  --- 4
        |  --- 5
        |  --- 6
        |  --- 7
        |  --- 8
        |  --- 9

I also use a script to quickly create this structure : http://pastebin.com/LyFLBZGx

So, all my scripts are located in /root/backup_tools. The backup.sh script is placed into crontab to run every day. I have key-exchange from my backup server to all of my servers I need to backup. In my tools dir I place my exclude files (folders / files I do not want to backup) in this format:

rsync.exclude.server1

These files contain the not to backup dirs :

/proc
/sys
/tmp

I also use my /.ssh/config file to add the hosts (f.e.: server1.example.com is defined as server1 with ssh port xxxx and username foo). This makes it a lot easier to add the servers to backup in the first line of the script.

Host server1
        User root
        Port 31337
        Hostname server1.example.com

The script will check the rule SERVERS="" and for every server defined there (space seperated) it will start an incremental backup (and excluding all the dirs in the exclude files).

It will use cpio for the rotating of the dirs (cpio allows a copy with link to the actual block on the disk, so the file will show up twice on your hard drive, and only use space once. Its not a symlink either, because when you delete the original file, the duplicate will still be readable)

I hope this was somewhat clear. The bash script is not perfect, but it does its job. I use it to backup 4 servers every night. I have backups of a couple of months now and they are not big. It is really space saving.

share|improve this answer
    
This is a very well explained answer! –  Lucas Kauffman Aug 22 '11 at 18:22
    
Thanks! This was pretty useful :D –  pmerino Aug 22 '11 at 20:21
    
But why go through all this work instead of using rsnapshot? –  Phil Hollenback Aug 26 '11 at 19:50
    
it was a challenge :-) –  Goez Aug 30 '11 at 5:57

In your example, you mention symlinks, however rsync deals in hard links. You mention in the comments that you're unsure what links are, so the Reader's Digest version is:

  • A symlink is like a "shortcut" in Windows -- it quite simply tells you "the file you want is over there"
  • A hard link has no direct correlation to Windows -- at least not in common usage. A hard link is quite literally another "entry point" to the same file; it appears to the file system to be an exact duplicate of the linked file, however on the physical disk there is only one copy of the file, no matter how many hard links there are.

rsync's '--link-dest' option creates hard links for files that don't change. This makes it somewhat confusing when trying to determine if your script is working as intended, because if you were to check the size of all the files in your backup directory (e.g. using du -sh [directory] or by checking the properties in the GUI), it would look to be the same size as your original directory, regardless of how many of those files are actually hard links and thus not taking up any additional space.

Check the space on disk, using either df or via GUI tools that look at actual disk space. Then, run your backup script, and check again -- if no files changed, disk usage should not change at all (well, okay, a tiny bit -- it takes a small amount of space for the hard link itself); if files did change, disk usage will increase by only what files changed.

In either case, rsync's output will list the files it is checking, whether it's actually copying them or not. Look at the end, at the "speed-up" value -- if that's a number less than 1, that indicates you got at least some hard links, as that represents the percentage difference from what rsync estimates it would have taken to copy all the files.

share|improve this answer

You actually want to do this with hardlinks. The best tool to do these sorts of backups on linux machines is rsnapshot. It does exactly what you describe and is quite simple to set up.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.