Is it somehow possible to block/abort unencrypted smtp connections with iptables?
effectively rejecting all connections which dont do a start tls.
Iptables is mostly designed to filter on the metadata of a network packet, such as IP headers, TCP options, and so forth. TLS is handled at the application level: you need to watch what happens inside the TCP stream.
You can inspect the contents of packets with the
What you're looking for here is an SMTP proxy that forwards TLS connections and blocks non-TLS connections. Iptable's contribution would be only to ensure that SMTP traffic does go through the proxy.
This is unavoidably tricky. The nature of TLS is such that a plaintext connection to your MTA has to be established before TLS can be negotiated, so iptables (operating as it does at the transport layer) is ill-designed to influence issues at the application layer.
You could write another target module and direct your traffic through that, but unless you're a networking God, this is probably no more feasible for you than it is for me. And I definitely don't know how to do it.
The upshot is that application-layer stuff is much easier to enforce inside the application. You don't say what MTA you're using, but I suspect that most MTAs that are bright enough to do TLS are bright enough to mandate it.
I use sendmail. There's a nice piece on mandating TLS from various providers at http://www.brandonhutchinson.com/Using_TLS_with_Sendmail.html , which directs me to the access database entry
which requires a particular communication partner, presumably identified by IP address, to both authenticate with a key of at least 112 bits length, and have a properly-signed certificate. The sendmail config page at http://www.sendmail.org/documentation/configurationReadme , in the ANTI-SPAM CONFIGURATION CONTROL section, says that access db entries involving IPv4 addresses can take the form of a single octet, which then apply to all addresses beginning with that octet. So I speculate, and I stress it's just speculation, that sendmail would allow me to have a series of entries
Mandating encryption (though not verifiably-signed certificates; self-signed TLS certs are very common, and I'd be inclined not to bar them) from all IP addresses. I would also not have an entry for
I repeat that I've not tested any of the above, and if your MTA is something other than sendmail, the above won't be specifically helpful; but I wanted to show that my MTA (at least) seems to have hooks for doing what you want. Good luck with your investigations.