Does a forward web proxy exist that checks and obeys robots.txt on remote domains? I agree when you look at it like that, but arguably it's an issue of terminology. So, as different model, define "crawler" to be a system of components - a forward web proxy and numerous agents behind it. The whole thing can be dubbed "a crawler" because that is what it looks like to the outside world. But now we can have multiple, probably heterogeneous requesting agents at the back and a single point at the front responsible for obeying remote policy (i.e. robots.txt). It's interesting!
Does a forward web proxy exist that checks and obeys robots.txt on remote domains? Thanks for answering! My thought is that the proxy can act as a central, single point of obedience to any robots.txt rules encountered - it means that multiple crawlers can start from one network, go through a proxy, and all, in aggregate obey any robots.txt rules encountered.
meta user
network profile
