This is actually not explicitly defined anywhere, and whether or not the server should be "trusted" depends on the client (which could of course be another mail server) connecting to it; quoting from the relevant RFC (RFC 2487):
If the SMTP client decides that the level of authentication or
privacy is not high enough for it to continue, it SHOULD issue an
SMTP QUIT command immediately after the TLS negotiation is complete.
If the SMTP server decides that the level of authentication or
privacy is not high enough for it to continue, it SHOULD reply to
every SMTP command from the client (other than a QUIT command) with
the 554 reply code (with a possible text string such as "Command
refused due to lack of security").
The decision of whether or not to believe the authenticity of the
other party in a TLS negotiation is a local matter. However, some
general rules for the decisions are:
- A SMTP client would probably only want to authenticate an SMTP
server whose server certificate has a domain name that is the
domain name that the client thought it was connecting to.
What this basically means is, when the server offers TLS encryption using a given certificate, the decision about accepting or refusing it depends entirely on the other part, which will probably want the name on the certificate to be the same it connected to, but could very well accept it even if it doesn't match.
But wait, there's more. Quoting again from the same RFC:
Upon completion of the TLS handshake, the SMTP protocol is reset to
the initial state (the state in SMTP after a server issues a 220
service ready greeting). The server MUST discard any knowledge
obtained from the client, such as the argument to the EHLO command,
which was not obtained from the TLS negotiation itself. The client
MUST discard any knowledge obtained from the server, such as the list
of SMTP service extensions, which was not obtained from the TLS
negotiation itself. The client SHOULD send an EHLO command as the
first command after a successful TLS negotiation.
So, what the server is saying in response to HELO/EHLO before the TLS handshake doesn't seem to actually matter at all.
In my experience, self-signed certificates work pretty well on Internet-facing mail servers, which means the other mail servers aren't even bothering to validate them, they'll just happily accept anything that can provide TLS encryption, regardless of the issuing authority or subject name.