Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

My website is getting badly hit by spambots and scrappers. I've used Cloudflare but the problem still remains there. The problem is spambots accessing non-existing urls causing a lot of load to my drupal backend which goes all the way and bootstraps db just to serve a 404 error doc.

I cant simply dish out non-drupal 404's for all page not found errors, as I need to have drupal catch them. Since, varnish is in front it can check if the bot is acting nice and asking for valid url - if not it servers them a 404 or 403. These bots are causing errors using this pattern :

Now, pls. suggest a regex varnbisg VCL directive which catches this URL pattern and serves a 404 error from varnish, preventing it from reaching apache/drupal ?

share|improve this question
up vote 3 down vote accepted

Have you tried looking for url paths where that begin with /http ?

if (req.url ~ "^/https?:") {
  error 404 "Not found" 
share|improve this answer
This should do the trick. Thanks :) – iTech Nov 4 '12 at 3:58
Also, can you suggest a regex expression to detect usage of repeated words in same url for example "" in the case above, your suggestion blocked the hits from above query but there are others which dont use http twice but only repeated or nested paths ? – iTech Nov 4 '12 at 13:43

Have you considered implementing caching of your 404s? In order to block these with regexes, you'd need to keep your VCL and your backend in sync to avoid blocking valid URLs. If you just cache 404s for a significant amount of time, you will still need to load from the backend once per URL but subsequent requests will get a cached 404 page. You can do a manual purge if you ever add real content to a URL with a cached 404 response.

share|improve this answer
Pax, I am not sure how valid url's will get blocked. What I was thinking was to implement a varnish filter which says : "If the request URL has two http:/ (instead of normal single occurrence) serve a 404 or 503 as this is a bad bot". – iTech Oct 28 '12 at 4:12

"My website is getting badly hit by spambots and scrappers,"

Are they coming from the same IP(s)? Have you tried blocking the IP addresses?

share|improve this answer
My website seem to have made into some sort of list for those automated comment spammers, and hits come from various IP's. I guess they are using dynamic IP's. – iTech Nov 6 '12 at 12:27

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.