I have some web crawlers, and a specific website seems to be blocking traffic temporarily after some time. The thing is, even though all clients have the same external IP address (they access the internet via the same gateway) it blocks specific machines from my network. That is, it's not a simple IP address block.
How can this happen? What kind of rule can be made (on a web server, or firewall, etc.) to get this behavior? Can the rules be based on the MAC address or another machine-specific data?