In general, security is an onion thing, as already mentioned. There are reasons firewalls exist, and it's not just all the other lemmings being stupid morons.
This answer comes, as searching for 'fail2ban' on this page did not give me any results. So if I double other content, bear with me. And excuse me, if I rant a little, I provide plain experience as this might come in handy for others. :)
Networking considerations, local vs. external
This is rather Linux specific and concentrates on host based firewalling, which is usually the use case. External firewalling goes hand in hand with a proper network structure and other security considerations usually go into that. Either you know what is implied here, then you will likely not need this posting. Or you don't, then just read on.
Running firewalls externally and locally may seem counter-intuitive and double work. But this also gives the possibility turn of the rules on the external one, without compromising security on all other hosts being behind it. The need could arise from either debugging reasons or because somebody has just fucked up. Another use case will come down there in the 'adaptive global firewalling' section, for which you will also need both global and local firewalls.
Cost and availability and the same story all the time:
Firewalling is just one aspect of a proper secured system. Not installing a firewall as it 'costs' money, introduces a SPOF or whatever is just bullshit, pardon my French here. Just setup a cluster. Oh, but what if the fire cell has an outage? Then setup your cluster on spanning two or more fire compartments.
But what if the whole datacenter is unreachable, as both of the external carriers are out of business (excavator killed your fiber)? Just make your cluster spanning several datacenters, then.
That's expensive? Clusters are too complex? Well, paranoia has to be paid for.
Just whining about SPOFs, but not wanting to pay more money or creating little more complex setups is a clear case of double standards or just a small wallet on company or customer side.
This pattern applies to ALL these discussions, no matter which service is the current matter of the day. No matter if it's a VPN gateway, a Cisco ASA used just for firewalling, a MySQL or PostgreSQL database, a virtual system or server hardware, a storage backend, switches/routers, ...
By now you should get the idea.
Why bother with firewalling?
In theory your reasoning is sound. (Only running services can be exploited.)
But this is only half the truth. Firewalling, especially stateful firewalling can do much more for you. Stateless firewalls are only important if you experience performance issues like others already mentioned.
Easy, central, discrete access control
You mentioned TCP wrappers - these implement basically the same functionality for securing your environment. Because ACLs, by the way, are also just feature-incomplete host-based stateless firewalls.
For the sake of the argument, let's assume someone doesn't know of
tcpd and likes using a mouse?
fwbuilder might come into mind.
Having full access from your management network is something you should have enabled, which is something of the first use cases of your host based firewall.
How about a multi-server setup, where the database runs somewhere else and you cannot put both/all the machines within a shared (private) subnet for whatever reason? Use the firewall to allow MySQL access on port 3306 only for the single given IP address of the other server, done, simple.
And that also works flawlessly for UDP. Or whatever protocol. Firewalls can be so damn flexible. ;)
Further, with firewalling, general portscans can be detected and mitigated as the amount of connections per timespan can be monitored via the kernel and its networking stack, and the firewall can act upon that.
Also invalid or obscure packets can be handled prior to them ever reaching your application.
Outbound traffic limiting
Filtering outbound traffic is usually a pain in the ass. But it can be a must, depending on contract.
Another thing that a firewall can give you, is statistics. (Think
watch -n1 -d iptables -vnxL INPUT with having added some rules for special IP addresses right at the top to see if packets are coming through.)
You can see in plain daylight if things work or they do not. Which is VERY useful when troubleshooting connections and being able to tell the other person on the telephone you do not get packets, without having to resort to chatty
tcpdump's. Networking is fun, most people just do now know what they are doing, and all to often it's just simple routing errors. Hell, even I don't always know what I am doing. Although I have worked with literally dozens of complex systems and appliances, often with tunneling, too, by now.
Layer7 firewalling is usually snake-oil (IPS/IDS), if not attended properly and updated regularly. Plus the licenses are damn expensive, so I'd spare getting one if you have no real need for getting everything money can buy you.
Easy, just try this with your wrappers. :D
Local port forwarding
Securing password access channels with dynamic IP addresses
What about if customers have dynamic IP addresses and there is not a VPN setup deployed? Or other dynamic approaches to firewalling? This is already hinted at in the question, and here will come a use case for the Unfortunately, I can't find any firewalls that do that. part.
Having disabled the root account access via a password is a must. Even if access is limited to certain IP addresses.
Also, still having another blanko account ready with a password login if ssh keys get lost or deployment fails is very handy if something goes really wrong (a user has administrative access to the machine and 'things happened'?). It is the same idea for network access as it is having single-user mode on Linux or using
grub for local access really really bad and cannot use a live disk for whatever reason. Don't laugh, there are virtualization products prohibiting that. Even if the functionality exists, what if an outdated software version is run lacking the functionality?
Anyway, even if you run your ssh daemon on some esoteric port and not on 22, if not having implemented things like port knocking (to open even another port and thus mitigating portscans, slowly bordering on being too impractical), port scans will detect your service eventually.
Usually you also set up all servers with the same configuration, with the same ports and services for efficiency reasons. You cannot set up ssh to a different port on every machine. Also you cannot change it on all machines every time you consider it being 'public' information, because it already is after a scan. The question of
nmap being legal or not is not an issue when having a hacked Wi-Fi connection at your disposal.
If this account is not named 'root', people may not be able to guess the user account name of your 'backdoor'. But they will know, if they get another server from your company, or just buy some webspace, and have an unchrooted/unjailed/uncontainered look at
For purely theoretical illustration now, they could use a hackable website there to gain access to your server and look up how things are usually run at your place. Your hack search tools might not run 24/7 (usually they do at night for disk performance reasons for the filesystem scans?) and your virus scanners are not updated the second a new zero-day sees the light of the day, so it will not detect these happenings at once, and without other protection measures you may never even know what happened. To get back to reality, if someone has access to zero-day exploits, it's very likely he will not target your servers anyway as these are just expensive. This is just for illustrating that there is always a way into a system if the 'need' arises.
But on the topic again, just don't use an extra passworded account, and don't bother? Please read on.
Even if attackers to get the name and port of this extra account, a
iptables combination will stop them short, even if you used only an eight-letter password. Plus fail2ban can be implemented for other services, too, widening the monitoring horizon!
For your own services, if the need ever arose: Basically every service logging failures to a file can get fail2ban support via providing a file, what regex to match and how many failures are allowed, and the firewall will just happily ban every IP address it is told to.
I am not telling to use 8-digit passwords! But if they get banned for 24 hours for five wrong password tries, you can guess how long they will have to try if they do not have a botnet at their disposal even with such lousy security. And you'd be astonished what passwords customers tend to use, not just for
ssh. Having a look at peoples' mail passwords via Plesk tells you everything you'd rather not want to know, if you'd ever do, but what I am not trying to imply here of course. :)
Adaptive global firewalling
fail2ban is just one application which uses something along the lines of
iptables -I <chain_name> 1 -s <IP> -j DROP, but you can easily build such stuff yourself with some Bash magic quite fast.
To further expand something like this, aggregate all fail2ban IP addresses from servers within your network on an extra server, that curates all the lists and passes it in turn to your core firewalls blocking all the traffic already on the edge of your network.
Such functionality cannot be sold (of course it can, but it will just be a brittle system and suck), but has to be interweaved into your infrastructure.
While at it, you can also use blacklist IP addresses or lists from other sources, be it aggregated by yourself or external ones.