Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Recently one of the sites I work on started seeing some strange traffic to URL's that don't exist and never have.

Pages like:

  • /imprss/123go.php
  • /imprss/24cpv.php
  • /bnnrs1002/728x90/4b4254ab5edd4.html
  • /bnnrs1002/728x90/4b4254ab5f59e.html
  • /bnnrs1002/728x90/4b4254ab5e1f8.html

In fact there are over 300 different URL's so far... none of which even resemble anything that's ever been on the site.

So naturally all of these are redirected to our 404 page, which happens to be a modified version of our site map. The trouble is that we're seeing thousands of these 404's a day and I'm afraid this number could grow. What's more, I've looked at server logs and I haven't found any common IP addresses for all of this traffic.

So my first idea is to trim down and simplify the 404 page. That's step one to minimize the server load and reduce bandwidth.

I'm also convinced that this traffic is either coming from bots, some sort of malware, or completely uninterested users because I've followed the sessions with a couple tools we use and they never actually use the site after hitting the 404 page.

So what else can I do to better detect these traffic patterns? It'd be nice if I can get to the bottom of this.

share|improve this question
    
I also see requests to check for open proxy servers too. Interesting what people probe for these days... –  Michael Graff Jan 19 '10 at 22:49

3 Answers 3

up vote 2 down vote accepted

It is common to see this kind of "probing" where someone is guessing whether specific directories exist. It's a kind of profiling where the existence of certain pages indicates that certain software is installed on the server, or that certain usernames likely exist on the server, and can be a precursor to other kinds of hacking.

I run a number of websites, and all of them see this kind of probing almost daily.

The fact that they are coming from different IP addresses could be an indication that the probe is being conducted via a botnet - especially if there appears to be a pattern to the probing across various IP addresses.

There is nothing you can do to stop this; however, you could definitely minimize the impact on your system by serving up a very simple 404 page. It could also be a reminder to ensure all your software is patched and up-to-date, and you have strong passwords on any internet-facing servers.

share|improve this answer
    
Thanks Brent. I've seen this kind of behavior on some of my other sites as well, but never to this magnitude. Thousands of hits a day from rogue sources is a bit worrying. But I will build a simple 404 page in pure HTML (no server-side code, no database queries). If I make the page so lightweight that we could handle millions of hits a day without missing a beat, then I suppose it won't really matter. As for the security side of things, we are using strong passwords and IP filters for all the important bits so I feel pretty confident to that end. –  Steve Wortham Jan 19 '10 at 18:53

If you have enough of a common denominator (such as /impress/* and /bnnrs1002/*) then you could put a rewrite module in place to send those requests elsewhere. Say, for example, to a very minimal custom error page or some other "Bugger Off" page. This would help save your bandwidth while you try to put something more robust in place.

share|improve this answer
    
I was thinking about doing something like that. Most of them do start with /imprss/ or /bnnrs1002/ so it would work, at least for a little while. –  Steve Wortham Jan 19 '10 at 17:29

Well if they're really only hitting you once per ip address, for all different pages, then there really isn't anything you can do but trim down the 404 page...Or you could just redirect 'em to the front page and declare it as a page view...;)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.