On server to server communications there's no problem, because SMTP servers accepts any certificate and even falls back to unencrypted connection. In that sense, there's no public key infrastructure for MX servers. Against downgrade attacks there's DNS-Based Authentication of Named
Entities (DANE) (see RFC 7672 on SMTP Security via Opportunistic DANE TLS), utilizing public keys published in DNS over public keys signed by a CA. Self-signed certificates are ok for that.
For client to server communications it's at least more convenient to have a trusted CA signed certificate. If users learn to accept an exception, it's way more likely they do so even when there's a MitM attack in place. Moreover, e.g. Android mail client only allows requiring a trusted certificate or accepting any, making the MitM attack even easier. That's why I would recommend using a CA signed certificate for client communications.
It's even possible to get a free Let's Encrypt certificate for your SMTP/IMAP server as it's not limited to use the same certificate for multiple protocols. This pretty much removes any need for self-signed certificates with publicly-referenced mail servers. (Example configuration for Postfix.)