4

How can I configure dovecot to compress/archive old emails (say 6 months old) with the users still being able to read them, to save some precious space on the server?

1 Answer 1

5

The configuration to compress emails is a two step process:

  1. We need to configure dovecot to be able to read compressed emails
  2. We need to setup a cronjob to compress emails

Dovecot configuration

Create a configuration file for dovecot to enable the zlib plugin

Dovecot configuration files are usually found in /etc/dovecot/conf.d/

You can create a file named 35-zlib.conf in this directory with this content

# Enable Zlib for imap
protocol imap {
  mail_plugins = $mail_plugins zlib
}
# Enable Zlib for pop3
protocol pop3 {
  mail_plugins = $mail_plugins zlib
}
# Increase memory allowed for imap as it costs more to read compressed files
service imap {
  vsz_limit = 1024MB
}

Make sure your configuration is correct with doveconf -n, then restart dovecot using service dovecot restart

Daily cronjob

Next we need to run a script every day to compress emails that are older than 6 months

Hopefully this can be done with a [verbose] find command

In /etc/cron.daily/ create a file, let's call it archive-email

Write this script in it

#!/bin/bash
find /var/qmail/mailnames -mtime +182 -type f -regex ".*\.$HOSTNAME,.*S=.*,[a-yA-Y]*" ! -name "*\.gz*" -exec cp -a "{}" /tmp/ \; -exec sh -c "basename \"{}\" | xargs -L1 -I[] gzip \"/tmp/[]\"" \; -exec sh -c "basename \"{}\" | xargs -L1 -I[] sh -c '[ -f \"{}\" ] && mv \"/tmp/[].gz\" \"{}Z\" && rm \"{}\" && (echo \"{}\" >> /var/log/archived-emails.log) || rm \"/tmp/[].gz\" || rm \"{}Z\"'" \;

Don't forget to run chmod +x on the created file to make it executable

/var/qmail/mailnames is usually the root folder in which emails are stored, it may depend on your configuration

The -mtime +182 modifier allows us to find only files created more than 182 days ago (basically 6 months), modify it accordingly

With the-type f -regex ".*\.$HOSTNAME,.*S=.*,[a-yA-Y]*" modifier, we want to find only email files that don't have the Z flag in them, which are usually named in this format 1505740244.M351559P24632.my.domain.com,S=38204,W=40910:2,S More information on the maildir format here

Finally we copy the emails to compress to the \tmp dir to avoid conflict while compressing, we run the gzip command on the file and move them again if the file still exists in their original location adding a Z flag in their name to indicate they are compressed so we don't gzip them again and finally remove the original file

This process has been implemented according to the guidelines of the dovecot wiki but the maildirlock utility has a bug since v2 that has never been fixed, so I'm not using it, however it's unlikely it will cause issues

We also log all the successfully archived files in /var/log/archived-emails.log

You can run the command in your terminal first without exec to make sure you match the correct files

find /var/qmail/mailnames -mtime +182 -type f -regex ".*\.$HOSTNAME,.*S=.*,[a-yA-Y]*"

5
  • Script works great thanks. I did get some duplicate file entry errors in dovecot log though, probably due to not using maildirlock. Apr 21, 2021 at 9:51
  • The bug still hasn't been fixed unfortunately...
    – Tofandel
    Apr 21, 2021 at 11:36
  • Any idea how to avoid the e-mail client (specifically Thunderbird), from re-downloading and re-indexing all the messages that were compressed? Apr 21, 2021 at 17:57
  • I don't have this issue, make sure the name and mtime of before/after are the same (check /var/log/archived-emails.log and ls -la), since only the flag is changed, they shouldn't be considered as a different email by dovecot (which is what's happening in your case), if it's not make sure your email storage is actually maildir
    – Tofandel
    Apr 21, 2021 at 21:40
  • 1
    The error message and reindexing are part of the same problem, which I think is this dovecot-uidlist is updated lazily to optimize for disk I/O. If a message is expunged, it may not be removed from dovecot-uidlist until sometimes later. This means that if you create a new file using the same file name as what already exists in dovecot-uidlist, Dovecot thinks you “unexpunged” message by restoring a message from backup. This causes a warning to be logged and the file to be renamed.
    – Tofandel
    Apr 21, 2021 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.