I would definitely recommend using something like Fail2ban or DenyHosts to do this. It's exactly what they're made for, and like Zoredache says, I can't imagine why you would have a problem with using a third-party program.
That being said, you could use the
recent module for IPtables, which exists in newer versions of the software. (I'm not sure how new exactly, but if you've kept up to date, I think you should have it.) It's not as easy to configure or as well-featured as the third-party programs, but it is a possibility. A set of rules like this:
iptables -A INPUT -m recent --name nobruteforce --rcheck -j DROP
iptables -A INPUT -m recent <other options> --name nobruteforce --set -j DROP
will add all packets which match the
<other options> to a blacklist, and will block any further packets coming from the same source IP address. You would replace
<other options> with whatever IPtables options you normally use to identify brute-force attacks; for example, it might be something like
iptables -A INPUT -m recent -p tcp --dport 22 --name nobruteforce --set -j DROP
although do note that every packet which comes to port 22 (that hasn't matched an earlier rule) would trigger this. Be careful with the
recent module, because it could seriously mess things up if your rules ever produce a false positive. If you must use it, to limit the consequences, I'd suggest adding a time limit and/or a minimum hit requirement to the second rule:
iptables -A INPUT -m recent --name nobruteforce --rcheck --seconds 7200 --hitcount 5 -j DROP
This would block packets only after 5 brute-force packets are received, and at the end of 2 hours (7200 seconds) it will remove the source address from the blacklist.
More information about
ipt_recent is available at http://snowman.net/projects/ipt_recent/