I run several mail servers of varying sizes ranging from my own for two users to hundreds of IMAP mailboxes. My opinion of email can be summed up by telling you that I am planning to decommission my own private mail server and move to Gmail for my domain.
The main reason why I want rid of this responsibility is spam. It is compute- and resource-expensive to filter inbound spam with any kind of effectiveness. It takes time and effort on my part to maintain the spam filtering to ensure that we are as up-to-date as possible with the techniques being used by the spammers. And then there are times when your tools seem to be actively mis-maintained by the maintainers, such as when SpamAssassin started marking up everything with a date in 2010 or later because it was impossibly far in the future.
Greylisting works much of the time too, but some relay systems just can't deal with it properly -- and even though greylisting is legal, dealing with the broken systems is your problem.
Using black-lists can skim much of it off, but inevitably someone finds a blacklisted host that they want to receive mail from.
If you run a mail server, blacklisting is always your problem. You get blacklisted so your users can't mail out? That's your problem. Especially when the blacklist is some penny-ante ISP in Southern Wisconsin which is blacklisting you because ten years ago your IP block was used by some fly-by-night DSL provider and not the backbone provider it is today. Or they insist that they have to run a "relay test" on your server before they'll de-list you, even though the IP that is in their list is an outbound-only IP and doesn't accept email from the internet at large.
Someone trying to email one of your users gets blacklisted so they can't mail you? That's your problem. The email is always of earth-shattering importance and it is up to you to create an exception to let their email in.
Secondary-MXing is broken. Spammers just beat up on that, and your system gets to accept, then scan and possibly bounce, drop, or false-negative it into your users mailbox. Frankly I never secondary-MX anymore because if my primaries are offline for longer than it takes email to die then I've got bigger problems (probably headed by the need for finding a new job).
Then there are the RFC-nazis. You'll get blacklisted if you are not strictly RFC compliant. And then you'll get shouted down by people who hate the fact that your anti-spam choses to bounce rather than just drop, meaning the innocent people used as header-forging get buried in the back-scatter.
Email used to be interesting and fun. Now it's just one long, slow, hard kick in the nuts (pardon my colloquialism).