I'm not a huge fan of fail2ban, as it's only effective against attackers who don't have hundreds / thousands of IP addresses to use in a distributed brute-force attack. So it only catches the attackers who do all of their brute-forcing from a single IP address.
Are the hosts caught by it evil? Well, they're at least infected by bot code so that they can be used in attacks.
Over the past decade, I've found it more profitable to focus on the things that you've already done, and move my energy to preventing other avenues of attack. I find it better to take a whitelist approach to security, rather then a retroactive / reactionary blacklist approach.
Required for securing SSH:
- Disallowing "root" account login
- Using public-key authentication for accounts
- Limiting password-based accounts to specific IP ranges (I still consider this very risky)
- Limiting access to the server and/or port to specific parts of the world using a firewall box.
- Moving the SSH port off of the default, which will greatly reduce the amount of log file cruft that you have to sift through (automated or by hand).
I realize this is not the technical answer that you're looking for.