How to keep: daily backups for a week, weekly for a month, monthly for a year, and yearly after that

Question

I need to backup data and config files on this server, daily. I need to keep:

• daily backups for a week
• weekly backups for a month
• monthly backups for a year
• yearly backups after that

All of this accomplished via a shell script run daily from cron.

This is how the backup files should look after 10 years of running:

blog-20050103.tar.bz2
blog-20060102.tar.bz2
blog-20070101.tar.bz2
blog-20080107.tar.bz2
blog-20090105.tar.bz2
blog-20100104.tar.bz2
blog-20110103.tar.bz2
blog-20120102.tar.bz2
blog-20130107.tar.bz2
blog-20130902.tar.bz2
blog-20131007.tar.bz2
blog-20131104.tar.bz2
blog-20131202.tar.bz2
blog-20140106.tar.bz2
blog-20140203.tar.bz2
blog-20140303.tar.bz2
blog-20140407.tar.bz2
blog-20140505.tar.bz2
blog-20140602.tar.bz2
blog-20140707.tar.bz2
blog-20140728.tar.bz2
blog-20140804.tar.bz2
blog-20140811.tar.bz2
blog-20140816.tar.bz2
blog-20140817.tar.bz2
blog-20140818.tar.bz2
blog-20140819.tar.bz2
blog-20140820.tar.bz2
blog-20140821.tar.bz2
blog-20140822.tar.bz2

Answer 1

You are seriously over-engineering this. Badly.

Here's some pseudocode:

• Every day:
  • make a backup, put into daily directory
  • remove everything but the last 7 daily backups
• Every week:
  • make a backup, put into weekly directory
  • remove everything but the last 5 weekly backups
• Every month:
  • make a backup, put into monthly directory
  • remove everything but the last 12 monthly backups
• Every year:
  • make a backup, put into yearly directory

---

The amount of logic you have to implement is about the same, eh? KISS.

This looks easier:

s3cmd ls s3://backup-bucket/daily/ | \
    awk '$1 < "'$(date +%F -d '1 week ago')'" {print $4;}' | \
    xargs --no-run-if-empty s3cmd del

Or, by file count instead of age:

s3cmd ls s3://backup-bucket/daily/ | \
    awk '$1 != "DIR"' | \
    sort -r | \
    awk 'NR > 7 {print $4;}' | \
    xargs --no-run-if-empty s3cmd del

Answer 2

If you just want to keep, for example, 8 daily backups and 5 weekly (every sunday) backups, it works like this:

for i in {0..7}; do ((keep[$(date +%Y%m%d -d "-$i day")]++)); done
for i in {0..4}; do ((keep[$(date +%Y%m%d -d "sunday-$((i+1)) week")]++)); done
echo ${!keep[@]}

As of today (2014-11-10), this will output:

20141012 20141019 20141026 20141102 20141103 20141104
20141105 20141106 20141107 20141108 20141109 20141110

As an exercise left for you, you just have to delete all backup files whose names do not appear in the keep-array.

If you want to keep 13 monthly backups (first sunday of every month) and 6 yearly backups (first sunday of every year) as well, things get a little bit more complicated:

for i in {0..7}; do ((keep[$(date +%Y%m%d -d "-$i day")]++)); done
for i in {0..4}; do ((keep[$(date +%Y%m%d -d "sunday-$((i+1)) week")]++)); done
for i in {0..12}; do
        DW=$(($(date +%-W)-$(date -d $(date -d "$(date +%Y-%m-15) -$i month" +%Y-%m-01) +%-W)))
        for (( AY=$(date -d "$(date +%Y-%m-15) -$i month" +%Y); AY < $(date +%Y); AY++ )); do
                ((DW+=$(date -d $AY-12-31 +%W)))
        done
        ((keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++))
done
for i in {0..5}; do
        DW=$(date +%-W)
        for (( AY=$(($(date +%Y)-i)); AY < $(date +%Y); AY++ )); do
                ((DW+=$(date -d $AY-12-31 +%W)))
        done
        ((keep[$(date +%Y%m%d -d "sunday-$DW weeks")]++))
done
echo ${!keep[@]}

As of today (2014-11-10), this will output:

20090104 20100103 20110102 20120101 20130106 20131103
20131201 20140105 20140202 20140302 20140406 20140504
20140601 20140706 20140803 20140907 20141005 20141012
20141019 20141026 20141102 20141103 20141104 20141105
20141106 20141107 20141108 20141109 20141110

Same as above, just delete all backup files not found in this array.

Answer 3

I recently had the same problem. IMHO, trying to write a shell script to do it is painful, and it is much easier to write some reusable logic using a higher-level language with builtins like sets, dictionaries, etc. The general idea is to take configuration indicating how many files of each period you want to keep, and then decide for each file if it should be kept.

There is a fairly popular python-based script that looks really nice and has some easy-to-understand source. Plus being python-based instead of shell-based gives it a cross-platform advantage: https://github.com/xolox/python-rotate-backups

Answer 4

As mentioned in a comment, it is normally best to delegate the task of managing backups to a backup management software.

But here is the logic in bash for deleting older backups as per you requirement

#!/bin/sh

delete() {
    echo "Deleting $1"
}

DOW=$(date +%u)

if [ $DOW -eq 1 ]; then
    DATE_DAY=$(date -d "-28 days" +"%d")
    if [ $DATE_DAY -gt 7 ]; then
        DATE=$(date -d "-28 days" +"%Y-%m-%d")
        delete $DATE
    fi

    DATE_DAY=$(date -d "-364 days" +"%d")
    DATE_MONTH=$(date -d "-364 days" +"%m")
    if [ $DATE_DAY -le 7 ] && [ $DATE_MONTH -gt 1 ]; then
        DATE=$(date -d "-364 days" +"%Y-%m-%d")
        delete $DATE
    fi
else
    DATE=$(date -d "-7 days" +"%Y-%m-%d")
    delete $DATE
fi

PHP Code showing which files will remain after running for 3520 days

https://ideone.com/n2ymQy

Array
(
    [0] => 2005-01-03
    [1] => 2006-01-02
    [2] => 2007-01-01
    [3] => 2008-01-07
    [4] => 2009-01-05
    [5] => 2010-01-04
    [6] => 2011-01-03
    [7] => 2012-01-02
    [8] => 2013-01-07
    [9] => 2013-09-02
    [10] => 2013-10-07
    [11] => 2013-11-04
    [12] => 2013-12-02
    [13] => 2014-01-06
    [14] => 2014-02-03
    [15] => 2014-03-03
    [16] => 2014-04-07
    [17] => 2014-05-05
    [18] => 2014-06-02
    [19] => 2014-07-07
    [20] => 2014-07-28
    [21] => 2014-08-04
    [22] => 2014-08-11
    [23] => 2014-08-16
    [24] => 2014-08-17
    [25] => 2014-08-18
    [26] => 2014-08-19
    [27] => 2014-08-20
    [28] => 2014-08-21
    [29] => 2014-08-22
)

Answer 5

I've been through the same problem. Basically needing space on a filesystem where daily full backups were running.

I wrote a script that just prints the rm -f commands.

Basic but good enough for me. Easy to adapt/improve.

Benefits:

• Super simple (based on find)
• easy to pipe into a file or just copy/paste the relevant output sections for interractive use
• possibly review, execute, and possibly keeping traces of the delete (cleanup) actions

Limitations:

• You need to adapt the regex if your file formats are different
• Since it's just based on a count of files, there could be a shift for a month when backups did not run or started during the month.
• Needs some rewrites since it will delete the weekly backups on the second run. So to be used only once. Might work if run only once per month, at the exact same day.

#!/bin/bash
# Build for this file format: 
# ./YYYYmmdd-HHhMM_container_name.tar.xz
# example: ./20210923-06h00_petrussewp.tar.xz

DELETE_DAILY_AFTER=30   # days
DELETE_WEEKLY_AFTER=180 # days (so keep only one monthly)

# List the backup sources to group them:
sourcename=$( ls -1 20*.tar.?z | sed -E 's/^[-h0-9]+_(.+)\.tar\.[gbx]z$/\1/' | sort | uniq )

for ctname in ${sourcename}
do
    echo "## ${ctname} backups :"
    for month in {01..12}
    do
        # Delete daily backups older than DELETE_DAILY_AFTER days.
        echo "# Obsolete daily backups for ${ctname} for the month ${month}:"
        find ./ -maxdepth 1 -type f -name "20??${month}*${ctname}*" -mtime +${DELETE_DAILY_AFTER} -printf "rm -f %p\n" | sort -n | awk 'NR % 7 != 1'
        # Delete all but the first matching backup of the month after DELETE_WEEKLY_AFTER days.
        echo "# Obsolete weekly backups for ${ctname} for the month ${month}:"
        find ./ -maxdepth 1 -type f -name "20??${month}*${ctname}*" -mtime +${DELETE_WEEKLY_AFTER} -printf "rm -f %p\n" | sort -n | awk 'NR % 7 == 1' | tail -n+2
    done
done

Script output example:

## wekan-db backups :
# Obsolete daily backups for wekan-db for the month 01:
# Obsolete weekly backups for wekan-db for the month 01:
# Obsolete daily backups for wekan-db for the month 02:
rm -f ./20210206-06h00_wekan-db.tar.xz
rm -f ./20210207-06h00_wekan-db.tar.xz
rm -f ./20210208-06h00_wekan-db.tar.xz
rm -f ./20210209-06h00_wekan-db.tar.xz
rm -f ./20210210-06h00_wekan-db.tar.xz
rm -f ./20210211-06h00_wekan-db.tar.xz
rm -f ./20210213-06h00_wekan-db.tar.xz
rm -f ./20210214-06h00_wekan-db.tar.xz
rm -f ./20210215-06h00_wekan-db.tar.xz
rm -f ./20210216-06h00_wekan-db.tar.xz
rm -f ./20210217-06h00_wekan-db.tar.xz
rm -f ./20210218-06h00_wekan-db.tar.xz
rm -f ./20210220-06h00_wekan-db.tar.xz
rm -f ./20210221-06h00_wekan-db.tar.xz
rm -f ./20210222-06h00_wekan-db.tar.xz
rm -f ./20210223-06h00_wekan-db.tar.xz
rm -f ./20210224-06h00_wekan-db.tar.xz
rm -f ./20210225-06h00_wekan-db.tar.xz
rm -f ./20210227-06h00_wekan-db.tar.xz
rm -f ./20210228-06h00_wekan-db.tar.xz
# Obsolete weekly backups for wekan-db for the month 02:
rm -f ./20210212-06h00_wekan-db.tar.xz
rm -f ./20210219-06h00_wekan-db.tar.xz
rm -f ./20210226-06h00_wekan-db.tar.xz