I'm in a difficult situation, the Baidu spider is hitting my site causing about 3Gb a day worth of bandwidth. At the same time I do business in China so don't want to just block it.

Has anyone else been in a similar situation (any spider)? Did you come across a magical solution? Or did you just accept it and either block or not block the bot?


As long as it follows robots.txt you can throttle the requests

User-agent: *
Crawl-delay: 10
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.