17

I need to backup data and config files on this server, daily. I need to keep:

  • daily backups for a week
  • weekly backups for a month
  • monthly backups for a year
  • yearly backups after that

All of this accomplished via a shell script run daily from cron.

This is how the backup files should look after 10 years of running:

blog-20050103.tar.bz2
blog-20060102.tar.bz2
blog-20070101.tar.bz2
blog-20080107.tar.bz2
blog-20090105.tar.bz2
blog-20100104.tar.bz2
blog-20110103.tar.bz2
blog-20120102.tar.bz2
blog-20130107.tar.bz2
blog-20130902.tar.bz2
blog-20131007.tar.bz2
blog-20131104.tar.bz2
blog-20131202.tar.bz2
blog-20140106.tar.bz2
blog-20140203.tar.bz2
blog-20140303.tar.bz2
blog-20140407.tar.bz2
blog-20140505.tar.bz2
blog-20140602.tar.bz2
blog-20140707.tar.bz2
blog-20140728.tar.bz2
blog-20140804.tar.bz2
blog-20140811.tar.bz2
blog-20140816.tar.bz2
blog-20140817.tar.bz2
blog-20140818.tar.bz2
blog-20140819.tar.bz2
blog-20140820.tar.bz2
blog-20140821.tar.bz2
blog-20140822.tar.bz2
  • 9
    ...my normal suggestion would be "Use Bacula" (or some other backup software that can handle retention and rotation for you) :-) – voretaq7 Feb 12 '14 at 21:46
  • 1
    this question made me write cronicle <github.com/Kraymer/cronicle> because the accepted answer has the obvious defect of duplicating backups into the daily/weekly/etc folders. cronicle relies on symlinks and takes care of the rotation, deleting underlying files when no folders contain symlinks pointing to it. – kraymer Jan 6 '18 at 18:06
30

You are seriously over-engineering this. Badly.

Here's some pseudocode:

  • Every day:
    • make a backup, put into daily directory
    • remove everything but the last 7 daily backups
  • Every week:
    • make a backup, put into weekly directory
    • remove everything but the last 5 weekly backups
  • Every month:
    • make a backup, put into monthly directory
    • remove everything but the last 12 monthly backups
  • Every year:
    • make a backup, put into yearly directory

The amount of logic you have to implement is about the same, eh? KISS.

This looks easier:

s3cmd ls s3://backup-bucket/daily/ | \
    awk '$1 < "'$(date +%F -d '1 week ago')'" {print $4;}' | \
    xargs --no-run-if-empty s3cmd del

Or, by file count instead of age:

s3cmd ls s3://backup-bucket/daily/ | \
    awk '$1 != "DIR"' | \
    sort -r | \
    awk 'NR > 7 {print $4;}' | \
    xargs --no-run-if-empty s3cmd del
| improve this answer | |
  • I actually don't have separate directories. It was written to dump files into an S3 bucket. Once everything is in one place, the total amount of logic that you need to implement is about the same, no matter how you go about it. – Florin Andrei Feb 12 '14 at 21:23
  • 5
    Evidently it isn't. – MadHatter Apr 28 '14 at 8:19
8

If you just want to keep, for example, 8 daily backups and 5 weekly (every sunday) backups, it works like this:

for i in {0..7}; do ((keep[$(date +%Y%m%d -d "-$i day")]++)); done
for i in {0..4}; do ((keep[$(date +%Y%m%d -d "sunday-$((i+1)) week")]++)); done
echo ${!keep[@]}

As of today (2014-11-10), this will output:

20141012 20141019 20141026 20141102 20141103 20141104
20141105 20141106 20141107 20141108 20141109 20141110

As an exercise left for you, you just have to delete all backup files whose names do not appear in the keep-array.

If you want to keep 13 monthly backups (first sunday of every month) and 6 yearly backups (first sunday of every year) as well, things get a little bit more complicated:

for i in {0..7}; do ((keep[$(date +%Y%m%d -d "-$i day")]++)); done
for i in {0..4}; do ((keep[$(date +%Y%m%d -d "sunday-$((i+1)) week")]++)); done
for i in {0..12}; do
        DW=$(($(date +%-W)-$(date -d $(date -d "$(date +%Y-%m-15) -$i month" +%Y-%m-01) +%-W)))
        for (( AY=$(date -d "$(date +%Y-%m-15) -$i month" +%Y); AY < $(date +%Y); AY++ )); do
                ((DW+=$(date -d $AY-12-31 +%W)))
        done
        ((keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++))
done
for i in {0..5}; do
        DW=$(date +%-W)
        for (( AY=$(($(date +%Y)-i)); AY < $(date +%Y); AY++ )); do
                ((DW+=$(date -d $AY-12-31 +%W)))
        done
        ((keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++))
done
echo ${!keep[@]}

As of today (2014-11-10), this will output:

20090104 20100103 20110102 20120101 20130106 20131103
20131201 20140105 20140202 20140302 20140406 20140504
20140601 20140706 20140803 20140907 20141005 20141012
20141019 20141026 20141102 20141103 20141104 20141105
20141106 20141107 20141108 20141109 20141110

Same as above, just delete all backup files not found in this array.

| improve this answer | |
  • Nice, and how do I do rm /dir/*.* except keep[@]? – takeshin Mar 6 '15 at 13:52
  • @takeshin If backups are named like 'auto_20200912', you can do something like this: for backup in "${backups[@]}"; do if [[ " ${!keep[*]} " != *"$(echo "$backup" | cut -d'_' -f 2"* ]]; then echo "Deleting: $backup" fi done – gbyte Sep 12 at 21:17
0

As mentioned in a comment, it is normally best to delegate the task of managing backups to a backup management software.

But here is the logic in bash for deleting older backups as per you requirement

#!/bin/sh

delete() {
    echo "Deleting $1"
}

DOW=$(date +%u)

if [ $DOW -eq 1 ]; then
    DATE_DAY=$(date -d "-28 days" +"%d")
    if [ $DATE_DAY -gt 7 ]; then
        DATE=$(date -d "-28 days" +"%Y-%m-%d")
        delete $DATE
    fi

    DATE_DAY=$(date -d "-364 days" +"%d")
    DATE_MONTH=$(date -d "-364 days" +"%m")
    if [ $DATE_DAY -le 7 ] && [ $DATE_MONTH -gt 1 ]; then
        DATE=$(date -d "-364 days" +"%Y-%m-%d")
        delete $DATE
    fi
else
    DATE=$(date -d "-7 days" +"%Y-%m-%d")
    delete $DATE
fi

PHP Code showing which files will remain after running for 3520 days

https://ideone.com/n2ymQy

Array
(
    [0] => 2005-01-03
    [1] => 2006-01-02
    [2] => 2007-01-01
    [3] => 2008-01-07
    [4] => 2009-01-05
    [5] => 2010-01-04
    [6] => 2011-01-03
    [7] => 2012-01-02
    [8] => 2013-01-07
    [9] => 2013-09-02
    [10] => 2013-10-07
    [11] => 2013-11-04
    [12] => 2013-12-02
    [13] => 2014-01-06
    [14] => 2014-02-03
    [15] => 2014-03-03
    [16] => 2014-04-07
    [17] => 2014-05-05
    [18] => 2014-06-02
    [19] => 2014-07-07
    [20] => 2014-07-28
    [21] => 2014-08-04
    [22] => 2014-08-11
    [23] => 2014-08-16
    [24] => 2014-08-17
    [25] => 2014-08-18
    [26] => 2014-08-19
    [27] => 2014-08-20
    [28] => 2014-08-21
    [29] => 2014-08-22
)
| improve this answer | |
0

I recently had the same problem. IMHO, trying to write a shell script to do it is painful, and it is much easier to write some reusable logic using a higher-level language with builtins like sets, dictionaries, etc. The general idea is to take configuration indicating how many files of each period you want to keep, and then decide for each file if it should be kept.

There is a fairly popular python-based script that looks really nice and has some easy-to-understand source. Plus being python-based instead of shell-based gives it a cross-platform advantage: https://github.com/xolox/python-rotate-backups

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.