My website is getting badly hit by spambots and scrappers. I've used Cloudflare but the problem still remains there. The problem is spambots accessing non-existing urls causing a lot of load to my drupal backend which goes all the way and bootstraps db just to serve a 404 error doc.

I cant simply dish out non-drupal 404's for all page not found errors, as I need to have drupal catch them. Since, varnish is in front it can check if the bot is acting nice and asking for valid url - if not it servers them a 404 or 403. These bots are causing errors using this pattern :


Now, pls. suggest a regex varnbisg VCL directive which catches this URL pattern and serves a 404 error from varnish, preventing it from reaching apache/drupal ?


Have you tried looking for url paths where that begin with /http ?

if (req.url ~ "^/https?:") {
  error 404 "Not found" 
  • Also, can you suggest a regex expression to detect usage of repeated words in same url for example "megaleecher.net" in the case above, your suggestion blocked the hits from above query but there are others which dont use http twice but only repeated or nested paths ? – iTech Nov 4 '12 at 13:43

Have you considered implementing caching of your 404s? In order to block these with regexes, you'd need to keep your VCL and your backend in sync to avoid blocking valid URLs. If you just cache 404s for a significant amount of time, you will still need to load from the backend once per URL but subsequent requests will get a cached 404 page. You can do a manual purge if you ever add real content to a URL with a cached 404 response.

  • Pax, I am not sure how valid url's will get blocked. What I was thinking was to implement a varnish filter which says : "If the request URL has two http:/ (instead of normal single occurrence) serve a 404 or 503 as this is a bad bot". – iTech Oct 28 '12 at 4:12

"My website is getting badly hit by spambots and scrappers,"

Are they coming from the same IP(s)? Have you tried blocking the IP addresses?

  • My website seem to have made into some sort of list for those automated comment spammers, and hits come from various IP's. I guess they are using dynamic IP's. – iTech Nov 6 '12 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.