I would like to pick the community's brain regarding linux server security, specifically regarding brute-force attacks and using fail2ban vs custom iptables.
There are a few similar questions out there but none of them address the topic to my satisfaction. In short I am trying to determine the best solution to secure linux servers exposed to the internet (running the usual services, ssh, web, mail), from brute-force attacks.
I have a decent handle on server security, i.e. locking down ssh by not allowing root or password logins, changing the default port, ensuring software is up to date, checking log files, only allowing certain hosts to access the server and making use of security auditing tools such as Lynis (https://cisofy.com/lynis/), for general security compliance, so this question is not necessarily regarding that although input and advice is always welcome.
My question is which solution should I use (fail2ban or iptables), and how should I configure it, or should I use a combination of both to secure against brute-force attacks?
There is a interesting response regarding the topic (Denyhosts vs fail2ban vs iptables- best way to prevent brute force logons?). The most interesting answer for me personally was (https://serverfault.com/a/128964), and that iptables routing occurs in the kernel as opposed to fail2ban which makes use of user mode tools to parse log files. Fail2ban uses iptables of course, but it still has to parse log files and match a pattern until it performs an action.
Does it make sense then to use iptables and use rate-limiting (https://www.rackaid.com/blog/how-to-block-ssh-brute-force-attacks/) to drop requests from an IP for a period of time that makes too many connection attempts during a specific period regardless of what protocol it was attempting to connect to? If so, then there are some interesting thoughts about using drop vs reject for those packets here (http://www.chiark.greenend.org.uk/~peterb/network/drop-vs-reject), any thoughts on that?
Fail2ban allows for custom configuration in the form of being able to write custom 'rules' for services that might not be addressed in the default configuration. It is easy to install and setup and is powerful, but could it be an overkill if all I am trying to achieve is to 'block' an IP from the server if they make 2 failed access attempts on any service/protocol over a x amount of time?
The goal here is to open daily logwatch reports and not have to scroll through pages of attempted failed connections to the server.
Thanks for taking the time.