You could try using moblock (google it - can't add links yet, new user). Disable all downloaded blocklists, and use only a local blocklist that you generate. You may need to add NFQUEUE (netlink queue) support to your kernel, but it may already be there by default.
The general setup is: for all SYN packets on the ports you want to filter, use netfilter's NFQUEUE action to push them to moblock, sitting in userspace. Moblock does efficient matching and sends back either an ACCEPT or DROP response to netlink.
The moblock config file format is quite simple: on each line, give a name and an IP range, in the form 18.104.22.168-22.214.171.124. When moblock loads the ranges, it builds an efficient data structure to match against those ranges. When a packet is dropped because of a match, the range name and actual source IP is logged (or not, if you disable logging of matches).
I've used moblock in its default configuration (downloaded blocklists) with about 230000 IP ranges, and have observed no discernible performance hit (filtering only the SYN packet is important to keep kernel/userspace traffic down though).
One caveat: if moblock is not running, I believe NFQUEUE's default action is to DROP, resulting in a denial of service of your application. That said, I've had moblock running continuously without any problem for over 6 months. Still, you may want to set up a monitoring probe that alerts you if a known-good IP can no longer connect to :80 on your server. You definitely don't want to use moblock to filter ssh, unless you have explicitely whitelisted some trusted IPs in netfilter to recover.