Question is really simple. If I want to setup a mail server, do i have to have it or is it just a security issue? What kind of problems can it cause?
Do you want to prevent people from sniffing your password? This is especially easy on WiFi. Then use encryption (SSL).
Do you care about having a self-signed certificate (getting a warning about it)? If not, just self sign. But beware there are apps (outlook) that do not allow you to ignore that warning after having seen it once.
The thing with SSL is, that it incorporates encryption and identity checks in the same protocol. You don't need a certificate for the former, but you do for the latter. If you don't suspect someone is going to redirect your login attempts to another server, you don't need the identity check: so self-sign. And even then it is not really a problem. Because in most clients once you've accepted your own certificate, it will warn again when it changes.
If you do want a certificate (I like to have them), use Startcom SSL. They allow you to have a simple SSL certificate for free.
Edit: Startcom can now no longer be recommended, because they have been blacklisted by major browsers, because their new owner, WoSign, violated the rules.
For mail retrieval (POP/IMAP) and MSA usage, although it's not necessary, it's definitely a good idea. Nowadays, many people access the e-mails from networks that cannot be trusted, typically from a mobile phone on an open WiFi network. Making your users connect via SSL/TLS is a good idea.
For MTA usage, you probably shouldn't expect SSL/TLS. It would be a good idea in principle, but few MTAs support SSL/TLS connections between themselves. (See this question for details.) There are multiple options here:
If you enable SSL/TLS only on your MTA (without any possibility of interaction without it enabled), you'll effectively cut yourself out of a number of other MTAs that wouldn't support it.
If you enable both SSL/TLS and plain text communications:
- It's fine for other MTAs that know that your server supports SSL/TLS and will only connect to it in secure mode. This can be useful in a few cases where they expect the connection between them and you to need to be secure, but they'd have to be explicitly configured to do so.
- If the other MTAs don't know that you support SSL/TLS, even if they try SSL/TLS in an opportunistic fashion, the fact that they would fall back to plain-text connections when SSL/TLS isn't available leaves them open to active MITM attacks. There's little point in using SSL/TLS at all in this case.
Remember that checking that SSL/TLS is used (and used correctly, including certificate verification) is solely the responsibility of the client, i.e. the other MTA here (nothing can be done on the server side in case of downgrading MITM attacks, unless client-certificates are also used, which is unlikely between MTAs).
If your server acts both as an MTA and an MSA (i.e. it expects direct connections from the user), I'd suggest enabling both with and without SSL/TLS, but tell your users to use the SSL/TLS option.