There are two stages to dealing with automated bots:
- Detecting and identifying the bots.
- Deciding what to do about them.
If you have any kind of moderately popular website, the numbers of bots will make it impractical to do this manually. You will need an automated response to the automated bots.
There's a good chance that a vulnerability scanning bot will produce a lot of 404s (I'm presuming you're talking about web application vulnerabilities and not SSH or Apache vulnerabilities. If you are more concerned about port scanners looking for MySQL exposed to the internet, then a firewall and ignoring them is the best bet.) so you might want to start by creating a script that scans your Apache error log for 404 errors and groups the results by IP address. Something like this:
grep "File does not exist:" /var/log/httpd/error_log | cut -d' ' -f1 | sort | uniq -c | sort -n | tail
Will give you the top 10 highest 404-requesting IP addresses. However, this isn't evidence of any wrongdoing. You also need to look at some of the actual requests from the access log to verify that these really are attempts to probe for vulnerabilities. This script will give you a list of "likely suspects" from which you can start narrowing down the real culprits. Run it from a cron job and use the output in the next step.
The next thing I would do is to start assigning scores, a bit like the way SpamAssassin scores spam emails, to IP addresses based on the requests that it has made. For instance, certain strings crop up a lot in automated attacks.
- URLs that end in ".txt???????"
- URLs that include "?var1=http://evil.com/foo.txt"
- URLs that include "../../../"
Any of these (and I'm sure you will find your own strings) should increase the score of an IP address. Of course, attackers can adapt to a certain extent to evade your detection, but the chances are good that they will not do this to you until you become a much higher value target. This strategy will work for a long time to come.
Once you have scored each IP address, find a score threshold that all the bots are higher than and all your real users are lower than and add all the IP addresses that are higher to a list to do something about.
Now we get to the second stage: deciding what to do about them.
- Send them an HTTP 403 (Forbidden) message.
- Send them an ICMP reject packet (This is -j REJECT in your iptables config)
- Drop their packets and don't send them anything at all. (This is -j DROP in your iptables config)
- Rate limit their connection to 60 packets per minute or 512 bytes per second.
- Log everything and contact their ISP.
- None of the above... there are plenty more options.
All of the above options involve modifying config files and restarting services. I would recommend auto-generating an entire file and using an Include statement in your Apache config or appending it to your iptables config and then restarting Apache or iptables. Be very careful with iptables, it's easy to lock yourself or your users out of your box. Make sure you build a failsafe into the script or you have another method of accessing it such as a console port or a KVM.