Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I need to backup data and config files on this server, daily. I need to keep:

  • daily backups for a week
  • weekly backups for a month
  • monthly backups for a year
  • yearly backups after that

All of this accomplished via a shell script run daily from cron.

This is how the backup files should look after 10 years of running:

blog-20050103.tar.bz2
blog-20060102.tar.bz2
blog-20070101.tar.bz2
blog-20080107.tar.bz2
blog-20090105.tar.bz2
blog-20100104.tar.bz2
blog-20110103.tar.bz2
blog-20120102.tar.bz2
blog-20130107.tar.bz2
blog-20130902.tar.bz2
blog-20131007.tar.bz2
blog-20131104.tar.bz2
blog-20131202.tar.bz2
blog-20140106.tar.bz2
blog-20140203.tar.bz2
blog-20140303.tar.bz2
blog-20140407.tar.bz2
blog-20140505.tar.bz2
blog-20140602.tar.bz2
blog-20140707.tar.bz2
blog-20140728.tar.bz2
blog-20140804.tar.bz2
blog-20140811.tar.bz2
blog-20140816.tar.bz2
blog-20140817.tar.bz2
blog-20140818.tar.bz2
blog-20140819.tar.bz2
blog-20140820.tar.bz2
blog-20140821.tar.bz2
blog-20140822.tar.bz2
share|improve this question
9  
...my normal suggestion would be "Use Bacula" (or some other backup software that can handle retention and rotation for you) :-) –  voretaq7 Feb 12 at 21:46

2 Answers 2

up vote 13 down vote accepted

You are seriously over-engineering this. Badly.

Here's some pseudocode:

  • Every day:
    • make a backup, put into daily directory
    • remove everything but the last 7 daily backups
  • Every week:
    • make a backup, put into weekly directory
    • remove everything but the last 5 weekly backups
  • Every month:
    • make a backup, put into monthly directory
    • remove everything but the last 12 monthly backups
  • Every year:
    • make a backup, put into yearly directory

The amount of logic you have to implement is about the same, eh? KISS.

This looks easier:

s3cmd ls s3://backup-bucket/daily/ | \
    awk '$1 < "'$(date +%F -d '1 week ago')'" {print $4;}' | \
    xargs --no-run-if-empty s3cmd del

Or, by file count instead of age:

s3cmd ls s3://backup-bucket/daily/ | \
    awk '$1 != "DIR"' | \
    sort -r | \
    awk 'NR > 7 {print $4;}' | \
    xargs --no-run-if-empty s3cmd del
share|improve this answer
    
I actually don't have separate directories. It was written to dump files into an S3 bucket. Once everything is in one place, the total amount of logic that you need to implement is about the same, no matter how you go about it. –  Florin Andrei Feb 12 at 21:23
4  
Evidently it isn't. –  MadHatter Apr 28 at 8:19

If you just want to keep, for example, 8 daily backups and 5 weekly (every sunday) backups, it works like this:

for i in {0..7}; do ((keep[$(date +%Y%m%d -d "-$i day")]++)); done
for i in {0..4}; do ((keep[$(date +%Y%m%d -d "sunday-$((i+1)) week")]++)); done
echo ${!keep[@]}

As of today (2014-11-10), this will output:

20141012 20141019 20141026 20141102 20141103 20141104
20141105 20141106 20141107 20141108 20141109 20141110

As an exercise left for you, you just have to delete all backup files whose names do not appear in the keep-array.

If you want to keep 13 monthly backups (first sunday of every month) and 6 yearly backups (first sunday of every year) as well, things get a little bit more complicated:

for i in {0..7}; do ((keep[$(date +%Y%m%d -d "-$i day")]++)); done
for i in {0..4}; do ((keep[$(date +%Y%m%d -d "sunday-$((i+1)) week")]++)); done
for i in {0..12}; do
        DW=$(($(date +%-W)-$(date -d $(date -d "$(date +%Y-%m-15) -$i month" +%Y-%m-01) +%-W)))
        for (( AY=$(date -d "$(date +%Y-%m-15) -$i month" +%Y); AY < $(date +%Y); AY++ )); do
                ((DW+=$(date -d $AY-12-31 +%W)))
        done
        ((keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++))
done
for i in {0..5}; do
        DW=$(date +%-W)
        for (( AY=$(($(date +%Y)-i)); AY < $(date +%Y); AY++ )); do
                ((DW+=$(date -d $AY-12-31 +%W)))
        done
        ((keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++))
done
echo ${!keep[@]}

As of today (2014-11-10), this will output:

20090104 20100103 20110102 20120101 20130106 20131103
20131201 20140105 20140202 20140302 20140406 20140504
20140601 20140706 20140803 20140907 20141005 20141012
20141019 20141026 20141102 20141103 20141104 20141105
20141106 20141107 20141108 20141109 20141110

Same as above, just delete all backup files not found in this array.

share|improve this answer

protected by Tom O'Connor Aug 7 at 13:41

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.