I'm looking for an email server that only stores duplicate emails once. We are currently using exchange and forward emails and replies create duplicate copies for every recipient.

I'm sure that there are smarter email servers out there that would hash the content and then store references to duplicate content, thus reducing the size of the mail store.

I have used Exim and Courier before but can't remember if they hashed email content or not.

  • 1
    Are you running out of space? Why don't you just throw hardware at the problem like the rest of us? JK, good question. +1 – beatgammit Apr 20 '11 at 2:57
  • An MTA is responsible for sending messages, which implies you're asking only about storage of messages queued to be sent out. Is that the case? If not, you might instead be talking about mailbox storage; please change the question if so. – bignose Apr 20 '11 at 5:39
  • Thanks bignose, yes I am referring to the server storage and have now revised the question to reflect this. – Brett Ryan Apr 20 '11 at 8:27
  • 1
    Brett, are you actually having capacity issues or is there another reason you seem so desperate for single instance storage? I feel you have an underlying problem you're not telling us about, and you think single instancing everything will make it magically go away. – Ben Pilbrow Apr 20 '11 at 10:33
  • 2
    Having had a lot of experience with Exchange's use of single instance storage in the past, I can tell you it might be a fair way to slow down the growth of data but it isn't a magical fix for an inadequate amount of storage or poor utilisation. The cost of moving to a new platform is likely to be much larger than the cost saving on storage that SIS gives you. Really. – Rob Moir Apr 20 '11 at 11:21

Exchange 5.5, 2003, and 2007 all are capable of using Single Instance Storage, which is pretty much exactly what you're asking for. It's per-database, but it's better than nothing.

2007 changed it to only work on attachments, and 2010 did away with SIS altogether due to some of the scaling issues and lack of overall effectiveness over time. Here's a blog entry on Technet explaining it a bit.

If you're looking for a new mail server, then I believe that both Dovecot and Cyrus support single instance storage.

| improve this answer | |
  • Sounds like a plan Hyppy, thanks for the answer. I'll recommend it to the admins (I'm just a programmer). Unfortunately they are running 2007, what's worse is the admins didn't understand this fully and split everyone's mailbox into 26 separate databases (1 for each letter of the alphabet for their last name) as a result their finding it harder to manage now and there would be very minimal SIS going on. – Brett Ryan Apr 20 '11 at 4:01
  • And they are too stupid to move the mailboxes into a better structure? Mailboxes CAN be moved, you know ;) – TomTom Apr 20 '11 at 8:43
  • Indeed, mailbox moves are quite easy. We maintain two databases right now: one active and one passive on each of our two mailbox servers. SIS does us no good though, since we're on 2010. – Hyppy Apr 20 '11 at 12:43
  • Thanks for your answer, I will make my recommendation to the admins, however I think they might just look at me funny and just reduce us down to 50MB mailbox sizes. – Brett Ryan Apr 21 '11 at 3:56

I originally held off answering this question, but I feel I have to share my opinion, even though it may not be the most popular. For a start though, it irks me a little that you have come to Server Fault after an "answer" and subsequently reveal you are not an admin and presumably can't even make this change happen. It also winds me up a little bit that your question and subsequent comments seem to be a big whinge about your admins. I think your only saving grace is that this question may be valuable to others on the internet searching for such a solution. By all means direct your admins here for some sensible discussion and we'd certainly welcome it and try and give them some good advice.

I sympathise with your predicament of ridiculously small mailboxes, I really do, however I don't for a second believe your admins are being draconian about it for no good reason. It is often the case that money simply won't allow it and you just have to work with what little you've got. Unfortunately it may take a very bad situation such as someone deleting a very important email because their mailbox was full before management realise the extent of the problem. In the mean time your admins will be forced to enable mailbox quotas, because if they don't and the disk totally fills up they are going to have a much bigger problem and email will be down for everyone.

Single instance storage (SIS) is not the magical solution you think it will be. SIS will certainly slow down the rate at which disk space is consumed, but I can almost guarantee not to the level you think it will. In actual fact, Microsoft have long recommended you totally ignore the disk space savings you can achieve with SIS when sizing mailbox servers, because there are quite a few ways to totally blow some of your SIS away. The only real solution is adding more mailbox servers with more capacity, but again this is very much constrained by budgetary limits.

I understand you're on Exchange 2007 but I'm going off on a little tangent here to give you some more info on SIS. One of the reasons the IO requirements in Exchange 2010 decreased was by removing SIS. That among other reasons means Microsoft now think it's acceptable to use slower disks (which will be cheaper) without seeing a massive performance hit. I'm in the UK so I can't comment on your situation, but certainly over here enterprise grade disks aren't massively expensive. The bottom line being that SIS incurs a performance penalty, and to get around that you might be forced into buying more/quicker disks to offset this.

As for a total mail sever migration, that is definitely not something to be taken lightly and I'd go as far as not recommending that, and I'm a guy who'll recommend what's right for the job, not what will make me most money. Heck, even Exchange upgrades need to be carefully thought out so they go as smooth as possible with as little downtime as is practical. You've made a significant investment in Microsoft Exchange, and I dare say user productivity will take a hit (even if only in the short term while users get used to the new system) and there will be complaints that $NewMailServer doesn't do all the shiny things Exchange did.

If you did go for a total mail server migration, there is an awful lot of cost involved in that too. You've got admin training on how to manage the thing, user training because things will work differently, then you've got the actual migration to do. The migration itself may involve downtime, it may involve clunky imports (that might not SIS your mail and may actually undo any single instancing that Exchange 2007 has done) and a few other gotcha's you'll only find out about after encountering them.

You've also got to think about your admins. If they're all Windows people, and someone suggests you move them over to (for example) a Linux mail server they may well freak out. If I was told we were moving to a Linux based system and there was no option, I would actually leave. I am a Windows guy through and through and while I use Linux where I think it's appropriate, I don't particularly enjoy dealing with it or even want to if I'm perfectly honest.

Sorry for my wall of text, make what you will of it. Sorry for the little rant at the beginning, but I just felt that I needed to say it. Ultimately though, I'd say you need to sort out your underlying problem of lack of storage rather than trying to paper over the cracks. If your admins do have any questions, please get them to ask a question on Server Fault and we'll be more than happy to help.

| improve this answer | |
  • 1
    I wish I could upvote this several times. SIS in this sort of case is just papering over the cracks rather than actually solving the problem. @Brett - when you say "I thought more servers would act this way" you're missing the fact that they used to, but there is a cost to doing SIS as well as a cost to not doing it, and the cost of not doing it is typically less on a modern system. It sounds like you've already decided that SIS is the answer to your prayers and you're not interested in checking your assumptions in that. – Rob Moir Apr 21 '11 at 11:57
  • Thanks Ben for your post, I'm not having a go at our admins by all means, however whenever there is any reasoning for new hardware they always say no. They are not proactive people that will willingly try to find other professionals on places like serverfault, I've recommended they come here many times of which they never feel interested to do so and usually ask me to solve their problems. – Brett Ryan Apr 22 '11 at 18:11

As an alternative idea - you could use storage that does de-duplication. I think NetApp can do something like this. Although, it may still be cheaper just to buy more storage. Just my 2 cents.

| improve this answer | |
  • 1
    For this to work you need to have a storage format that stores the message body separate from the message header. – adamo Apr 20 '11 at 8:38
  • Totally with you there Dmitri, NetApp does indeed dedupe but I prefer your second comment, local storage is super cheap! – Chopper3 Apr 20 '11 at 11:05
  • Other platforms such as Nexxenta also do dedupe; it may be worth the internal admin cost for your site. – MikeyB Apr 21 '11 at 12:26

Having previously run OpenMail systems which do exactly that I would very strongly recommend you find a different solution to the problem. There aren't many mailservers capable of doing this, they are niche products and they are very complicated.

Spend your time and effort on adding storage, providing an easy to use file repository for your users, implement mail quotas and educate your users - it'll be much cheaper and you'll save yourself a world of pain.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.