I don't think that you're actually looking for an alternative to iptables. The Linux kernel only has one packet filtering framework, and iptables is it. That said, there are a number of packages out there that provide enhanced management of iptables, including features such as you've described.
One tool along these lines is fail2ban, which watches log files for certain triggers and will then populate your firewall with rules to fan the offending system, and then remote those rules after a certain amount of time has elapsed. This is often used to block systems that are attempting brute-force attempts an ssh server, but it can be used for other application, as well.
Shorewall is a high-level tool for configuring your iptables firewall, and may have some of these features. I'm not particularly familiar with it myself.
You could also fairly easily roll your own. The combination of the iptables
comment target and
cron would make this fairly easy to automate (place a comment on a rule indicating when it should expire, and have a cron job running periodically that removes rules that are past their expiration date).