My server occasionally get massively hit by (probably bots) on a small group of IPs. Is there a good way (probably in iptables) to slow down or temporarily block/tarpit connections coming from an IP that has connected many times recently?
Yes, iptables supports this kind of dynamic rate limiting.
See here for details: http://codingfreak.blogspot.com/2010/01/iptables-rate-limit-incoming.html
Maybe denyhosts would be a useful tool for you.