Recently one of the sites I work on started seeing some strange traffic to URL's that don't exist and never have.
In fact there are over 300 different URL's so far... none of which even resemble anything that's ever been on the site.
So naturally all of these are redirected to our 404 page, which happens to be a modified version of our site map. The trouble is that we're seeing thousands of these 404's a day and I'm afraid this number could grow. What's more, I've looked at server logs and I haven't found any common IP addresses for all of this traffic.
So my first idea is to trim down and simplify the 404 page. That's step one to minimize the server load and reduce bandwidth.
I'm also convinced that this traffic is either coming from bots, some sort of malware, or completely uninterested users because I've followed the sessions with a couple tools we use and they never actually use the site after hitting the 404 page.
So what else can I do to better detect these traffic patterns? It'd be nice if I can get to the bottom of this.