10

I am currently using a duplicity script to backup my 110G/2T CentOS server to a 2T sftp server.

Since 4 days passed, duplicity backed up just 90G. This is not a problem. The main problem is; I have got nearly 600G cache that duplicity generated at "/user/.cache/duplicity". This size is not normal, so what should I do? Will duplicity shrink or remove these cache files and folders when finished the task? Will duplicity backup it's cache too (I did not exclude the /user folder)?

Additional info: I am using Hetzner server and this is the backup script that I am using: https://wiki.hetzner.de/index.php/Duplicity_Script/en

In addition, I just excluded directories proc, sys and dev from the root (and backed up everything else starting from the root, because I wanted a full server backup).

Improve this question
2
  • I have seen this before. I have never seen a solution before. But the cache is safe to delete, as long as duplicity is not running.
    – Michael Hampton
    Apr 6, 2016 at 19:34
  • Local cache is safe to delete. But will re-download, as far as I know.
    – user18099
    Jul 5, 2018 at 13:05

3 Answers 3

Reset to default
8

According to the mailing list

you will have to manually exclude it ..

it holds your backup chains index files (table of contents of the backup repository). Caching them locally accelerates options like status and incremental backup and others. These operations need to know what is already backed up to work. If they are cached they do not need to be transferred and decrypted every time again and again.

.. ede

For the rest it seems to be a long standing bug.

On the Debian bug tracker, they recommend to

duplicity cleanup --extra-clean --force ....

Warning: The suggested --extra-clean option is dangerous and can bite you very hard. It makes backups unrestorable by usual means.

Improve this answer
2
  • 1
    Thank you very much for your answer, so I must exclude those folders. What I must ask now is this command that you gave me "duplicity cleanup --extra-clean --force ....". Must I use this after the backup process ends or every time that I need?
    – Bahadir Tasdemir
    Apr 7, 2016 at 6:58
  • 1
    About that recommendation: You should be VERY sure about handling backups older than the current active backup chain (1 week?). Before PURGING metadata with --extra-clean. You will need that old metadata for the restore of old backups.
    – user18099
    Jun 26, 2018 at 12:34
1

Better to set the proper parameters in your config file to control the validity of your backup and the purge it:

# Time frame for old backups to keep, Used for the "purge" command.
# see duplicity man page, chapter TIME_FORMATS)
MAX_AGE=6M

# Number of full backups to keep. Used for the "purge-full" command.
# See duplicity man page, action "remove-all-but-n-full".
MAX_FULL_BACKUPS=2

# Number of full backups for which incrementals will be kept for.
# Used for the "purge-incr" command.
# See duplicity man page, action "remove-all-inc-of-but-n-full".
MAX_FULLS_WITH_INCRS=2

# activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
# forces a full backup if last full backup reaches a specified age, for the
# format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
# Uncomment the following two lines to enable this setting.
MAX_FULLBKP_AGE=2M

And then run the purge commands to see what would be deleted and then rerun it with the --force option to delete them:

duply your_config  purgeFull --force
Start duply v2.2, time is 2023-08-05 23:52:50.


--- Start running command PURGEFULL at 23:52:50.544 ---
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Fri Jul 21 18:13:20 2023
Deleting backup chains at times:
Sun Feb 26 10:32:51 2023
Sun Mar 19 08:07:44 2023
Mon Apr 17 13:46:08 2023
Fri May 19 16:40:16 2023
Sat Jun 10 09:24:03 2023
Deleting complete signature chain Sat Jun 10 09:24:03 2023
Deleting complete signature chain Sat Jun 10 09:24:03 2023
Deleting complete signature chain Fri May 19 16:40:16 2023
Deleting complete signature chain Fri May 19 16:40:16 2023
Deleting complete signature chain Mon Apr 17 13:46:08 2023
Deleting complete signature chain Mon Apr 17 13:46:08 2023
Deleting complete signature chain Sun Mar 19 08:07:44 2023
Deleting complete signature chain Sun Mar 19 08:07:44 2023
Deleting complete signature chain Sun Feb 26 10:32:51 2023
Deleting complete signature chain Sun Feb 26 10:32:51 2023
Deleting complete backup chain Sat Jun 10 09:24:03 2023
Deleting complete backup chain Fri May 19 16:40:16 2023
Deleting complete backup chain Mon Apr 17 13:46:08 2023
Deleting complete backup chain Sun Mar 19 08:07:44 2023
Deleting complete backup chain Sun Feb 26 10:32:51 2023
--- Finished state OK at 23:52:57.431 - Runtime 00:00:06.887 ---

If you put this is a monthly cron the cleanup will happen accordingly.

Improve this answer
0

When we started cleaning up very old backups remotely (S3), then duply commands started to delete very old local metadata automatically.

That is, we are keeping backups for only x months now. And the local metadata cache size did shrink accordingly.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .