It appears from your message that you may be an administrator. If so, you have the rights because at times you will need them. Accept, but don't abuse the privilege granted you. A trustworthy administrator will only use the rights when required to do your duties. Those duties may include scanning emails for specific content, tracing email sources, and other activities which require access to other user's email.
If you are making someone an administrator, consider the following.
Anyone with administrative rights more or less has the keys to the kingdom. If you can't trust them, don't make them administrators. They should be in a position to undo pretty well anything you do to remove access.
I would expect that most solutions would be best implemented by the administrator. The mail server would need to be able to decrypt the mailboxes. This would give the administrator access to the mailboxes.
Encrypting the mail messages at both ends could be done. However, this would severely limit who you can exchange email with. It is likely that you would want the administrator to be able to install and debug the encryption software.
The administrator is likely to be in a position to access the mail by packet capture as well. This is more difficult, but not extremely difficult.
A trustworthy administrator will not abuse their authority. Where they do need to access mail folders, they will limit their access as much as possible.