Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We have a Rails 2.3.8 app running on a nginx/passenger server. Recently the crawlers generate a huge load on us. We'd like to turn on page caching ONLY for crawlers (specific IP addresses), i.e., crawlers get static pages which expire every day. Questions are:

  • shall we do it in Nginx or Rails?

  • how to achieve this IP-wise caching strategy

I googled for nginx and rails, but no luck so far. Appreciate any inputs!

share|improve this question

I would instead look at the User-Agent string to detect crawlers. You can then use either rewrites in Nginx or view logic in Rails to display the desired content. Either way works, it's just a matter of what technology you are most familiar with.

share|improve this answer
thanks tylerl. I played with nginx rewrite, and finally got it to work with Rails page caching. However, I'm forced to give this up because there are over 100 million pages, and the cache size, ext3 file number limits and cache expiration are headaches. Need to find a more advanced way to do this. – Will Jul 22 '13 at 7:45

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.