I set up a web and email server on a VPS that uses apache2 with multiple virtual hosts and postfix, dovecot, and roundcube for the email server. I am getting an authentication failure when trying to send mail using roundcube, and I am thinking it might have to do with a mismatch between /etc/mailname (host.vpshoster.com) and the SSL certificate which was registered for one of the virtual hosts (www.example.com). Do they have to match?
The authentication error was not a direct result of the mismatch between /etc/mailname and the common name (CN) on the SSL certificate, it was due to misconfiguration of the mail server; however, once I got the mail server working the CN mismatch did cause security warnings when I sent an email to a gmail account. I created a new SSL certificate so the CN matched /etc/mailname and the security warnings related to this issue went away (although I am still receiving security warnings related to the message not being encrypted).
Lesson learned: the CN on the SSL certificate used by the email server should match the name of the email server as opposed to the name of a virtual host hosted on the same machine as the email server (I included the email server and all of my virtual hosts on the same SSL certificate, but listed the email server first to get the CN set correctly).
In addition to the lesson learned about the original question, I also learned the following during the process of setting up my mail server:
"Additionally for some mail services like gmail - you need to have reverse DNS set up on your mail server's IP and that name also needs to match the mailname" -ivanivan
I had a reverse DNS PTR record for my IPv4 address, but my IPv6 PTR record was wrong and I had to update it. In fact, without the proper PTR record, the emails I sent to gmail were not being delivered at all. They were returning to sender with information about IPv6 sending guidelines.
The mail was still unencrypted. See Outgoing email not encrypted for my postfix configuration and how I resolved the encryption issue.