4

I'm looking for a tool (for Linux) that will prune my backup files, not like the standard logrotate that completely deletes them after defined number of rotations, but where the files are basically kept permanently, except they are pruned as they get older and older. For example:

  • for the first month I want to keep every single daily backup file
  • after that I only want to keep a weekly backup for the next 6 months
  • after that I only want to keep a monthly backup for another 6 months
  • after that I only want to keep a quarterly backup

Does a tool to do such a time-based cleanup of files exist?

It could either be looking at a date in the file name, or the timestamp of the file (not ideal though).

Improve this question
1
  • This seems like a pretty basic thing for any backup software to handle transparently. How are you generating these backups?
    – GregL
    Commented Feb 3, 2017 at 13:39

4 Answers 4

Reset to default
2

There isn't such a tool I know of, usually this is inculded in your backup software. What backup tool do you use?

I would recommend rsnapshot This can be configured to do these smart things you ask for, and keeps backup size small by extensive use of hardlinks.

If however you have your own cronjobs/system for creating backups, you could just create a few more cronjobs.

  • make backups to a 'daily folder'
  • have a weekly cronjob that moves files older then 30days from your monthly folder to your 'weekly folder' find /path/to/daily -maxdepth 1 -mtime +30 -type f -exec mv "{}" /path/to/weekly/ \;

because the below cronjob will remove older files this will only be one day's worth of backup files.

  • have a daily cronjob that removes files older then 30 days from your daily folder (make sure it runs after the weekly cron above) find /path/to/daily/* -mtime +30 -exec rm {} \;

If your find supports it (e.g. recent versions of gnu findutils) you can also use -delete instead of `-exec rm {} ; Be sure to put it at the end of the file command

repeat for weekly to monthly and monthly to quarterly

Improve this answer
2
  • Sneaky solution. It's only solid though if the cron job runs every day - it can't do it after-the-fact. But that could be okay.
    – TheStoryCoder
    Commented Feb 16, 2022 at 12:09
  • @theStoryCoder yes it does work after the fact, find will return all files older then 30 days if you use -mtime +30; you can manualy run it once if you want to cleanup now.You might have to play with the amount of days...
    – Jens Timmerman
    Commented Feb 17, 2022 at 14:36
1

This question is rather old, but since it's the first result for this type of search, I would recommend Borg as your backup option, as it supports exactly the kind of backup schedule you're looking for via its Prune command.

From one of the examples in their documentation:

# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The '{hostname}-' prefix is very important to
# limit prune's operation to this machine's archives and not apply to
# other machines' archives also:

borg prune                      \
--list                          \
--prefix '{hostname}-'          \
--show-rc                       \
--keep-daily    7               \
--keep-weekly   4               \
--keep-monthly  6               \
Improve this answer
1

You should look into the "duplicacy" software, too.

It has very good reputation; and (independently) I am using it, and I am satisifed.

Improve this answer
4
  • Payware though.
    – TheStoryCoder
    Commented Mar 21, 2022 at 11:44
  • Not if you are using the CLI version on Linux
    – Helen Craigman
    Commented Mar 24, 2022 at 6:14
  • @HelenCraigman that would be duplicity?
    – Jens Timmerman
    Commented Sep 11 at 17:53
  • @Jens Timmerman: no, duplicacy, not: duplicity
    – Helen Craigman
    Commented yesterday
1

This was a problem I had and could not find a solution for. I was generating a lot of backup files gzip'ing configs on my servers. I ended up coding a solution.

https://github.com/BinaryPatrick/Prune

It uses the same pruning logic as Proxmox's backup server, and runs in the command line. I compiled it for linux-x64 with the following command and then I run it on the server from a cron job.

dotnet publish --runtime linux-x64 --self-contained -p:PublishSingleFile=true -c Release -o ./out/linux-x64

prune --path /mnt/backup/images \
      --ext img                 \
      --keep-last 3             \
      --keep-daily 3            \
      --keep-weekly 2           \
      --keep-monthly 2
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .