Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I need to backup data and config files on this server, daily. I need to keep:

  • daily backups for a week
  • weekly backups for a month
  • monthly backups for a year
  • yearly backups after that

All of this accomplished via a shell script run daily from cron.

This is how the backup files should look after 10 years of running:

blog-20050103.tar.bz2
blog-20060102.tar.bz2
blog-20070101.tar.bz2
blog-20080107.tar.bz2
blog-20090105.tar.bz2
blog-20100104.tar.bz2
blog-20110103.tar.bz2
blog-20120102.tar.bz2
blog-20130107.tar.bz2
blog-20130902.tar.bz2
blog-20131007.tar.bz2
blog-20131104.tar.bz2
blog-20131202.tar.bz2
blog-20140106.tar.bz2
blog-20140203.tar.bz2
blog-20140303.tar.bz2
blog-20140407.tar.bz2
blog-20140505.tar.bz2
blog-20140602.tar.bz2
blog-20140707.tar.bz2
blog-20140728.tar.bz2
blog-20140804.tar.bz2
blog-20140811.tar.bz2
blog-20140816.tar.bz2
blog-20140817.tar.bz2
blog-20140818.tar.bz2
blog-20140819.tar.bz2
blog-20140820.tar.bz2
blog-20140821.tar.bz2
blog-20140822.tar.bz2
share|improve this question
5  
...my normal suggestion would be "Use Bacula" (or some other backup software that can handle retention and rotation for you) :-) –  voretaq7 Feb 12 at 21:46

2 Answers 2

You are seriously over-engineering this. Badly.

Here's some pseudocode:

  • Every day:
    • make a backup, put into daily directory
    • remove everything but the last 7 daily backups
  • Every week:
    • make a backup, put into weekly directory
    • remove everything but the last 5 weekly backups
  • Every month:
    • make a backup, put into monthly directory
    • remove everything but the last 12 monthly backups
  • Every year:
    • make a backup, put into yearly directory

The amount of logic you have to implement is about the same, eh? KISS.

This looks easier:

s3cmd ls s3://backup-bucket/daily/ | \
    awk '$1 < "'$(date +%F -d '1 week ago')'" {print $4;}' | \
    xargs --no-run-if-empty s3cmd del

Or, by file count instead of age:

s3cmd ls s3://backup-bucket/daily/ | \
    awk '$1 != "DIR"' | \
    sort -r | \
    awk 'NR > 7 {print $4;}' | \
    xargs --no-run-if-empty s3cmd del
share|improve this answer
    
I actually don't have separate directories. It was written to dump files into an S3 bucket. Once everything is in one place, the total amount of logic that you need to implement is about the same, no matter how you go about it. –  Florin Andrei Feb 12 at 21:23
3  
Evidently it isn't. –  MadHatter Apr 28 at 8:19

This is a sample script, feel free to edit and adapt it to your needs.

#!/bin/bash

# local backup scratchpad
bdir="/root/backup"
# hostname
hname="someserver"
# date, now
now=`date +%Y%m%d`
# final resting place for backups
rembdir="/share/backups"

cd $bdir || exit

# backup various software such as DB
mysqldump -u root -pXXXXXXXX foobar > ${bdir}/foobar-db-${now}.sql

# make one big tarball for backup
# hostname-YYYYMMDD.tar.bz2
tar -cjf ${bdir}/${hname}-${now}.tar.bz2 \
    /etc \
    /var/spool/cron /usr/local/bin \
    ${bdir}/foobar-db-${now}.sql

# copy it to it's final destination
# you could replace this with a cloud upload command,
# or whatever
cp ${bdir}/${hname}-${now}.tar.bz2 ${rembdir} || exit
# clean up
rm -f ${bdir}/*

# list all archive files in the backup bucket
# replace this with a cloud filelist command,
# or whatever
filelist=`ls ${rembdir}`

# culling daily archives older than 1 week
# unless they are weekly archives (Monday)
#
# day of week, 1 week ago
wk1day=`date -d '1 week ago' +%u`
# full date, 1 week ago
wk1date=`date -d '1 week ago' +%Y%m%d`
# if not Monday...
if [ 1 != $wk1day ]; then
    fn=${hname}-${wk1date}.tar.bz2
    count=`echo "${filelist}" | grep ${fn} | wc -l`
    # ...and file exists...
    if [ 1 -eq $count ]; then
        # ...then delete
        # (or use the appropriate cloud delete command)
        rm -f ${rembdir}/${fn}
    fi
fi

# culling weekly archives older than 4 weeks
# unless they are monthly archives
#
# day of week, 4 weeks ago
wk4day=`date -d '4 weeks ago' +%u`
# full date, 4 weeks ago
wk4date=`date -d '4 weeks ago' +%Y%m%d`
# day of month, 4 weeks ago
dayofmonth4=`date -d '4 weeks ago' +%d`
# if it's Monday, but it's not first Monday of the month...
if [ 1 == $wk4day ] && [ $dayofmonth4 -gt 7 ]; then
    fn=${hname}-${wk4date}.tar.bz2
    count=`echo "${filelist}" | grep ${fn} | wc -l`
    # ...and file exists...
    if [ 1 -eq $count ]; then
        # ...then delete
        # (or use the appropriate cloud delete command)
        rm -f ${rembdir}/${fn}
    fi
fi

# culling monthly archives older than 1 year
# unless they are in January
yr1day=`date -d '1 year ago' +%u`
yr1date=`date -d '1 year ago' +%Y%m%d`
yr1month=`date -d '1 year ago' +%m`
# if it's Monday, but not in January...
if [ 1 == $yr1day ] && [ $yr1month -gt 1 ]; then
    fn=${hname}-${yr1date}.tar.bz2
    count=`echo "${filelist}" | grep ${fn} | wc -l`
    # ...and file exists...
    if [ 1 -eq $count ]; then
        # ...then delete
        # (or use the appropriate cloud delete command)
        rm -f ${rembdir}/${fn}
    fi
fi

How to test it:

Launch a Linux virtual instance. Create the $bdir and $rembdir directories. Edit the backup script, comment out the tar -cjf statement, and replace it with (this will create a mock-up zero-size "backup file"):

touch ${bdir}/${hname}-${now}.tar.bz2

Turn off time synchronization. Set time back 10 years.

service ntpd stop
date --set='10 years ago'

Run the script in an infinite loop, advancing 1 day between steps:

while [ 1 ]; do ./backup ; date --set='1 day'; done

Hit CTRL-C when it reaches present time.

Now investigate $rembdir. It should contain a list of "backup" files like the one above.

share|improve this answer
6  
This is a scary, scary script. I'd vote down if I had the rep. The combination of unquoted variables and running as root is a dangerous trap that could wreck someone's system. The lack of error checking and reporting means someone's not going to get the backups they think they're getting (we used to have a backup script that ran for 2 years, happily reporting success every day and just as happily not doing anything useful). As a quick-and-dirty script on a single system it's okay, but if you're searching the Internet for a way to do backups, please do not use this script. –  drewbenn Feb 13 at 6:09

protected by Tom O'Connor Aug 7 at 13:41

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.