Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Since 02.05.2013 some sort of bot is visiting my website, every 5 minutes.

Most of the time it calls this URL, which doesn't exist:

/viewtopic.php?f=3&t=849

Always with this user agent

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; TencentTraveler ; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727)

(= IE 6, Windows XP)

Not only it distorts the visitors statistic significantly, it doesn't even identify as a bot, so I guess it is a spammer of some sort. My question here is, what does this client do and why is it calling this page continuously? - And how can I prevent this?

I'm using a vServer with Linux Debian.

share|improve this question
    
If it really bothers you block it thesitewizard.com/apache/block-bots-with-htaccess.shtml –  Iain Aug 28 '13 at 12:27
    
The problem with that is I don't really have a 'filter' in order to block it. I can't block it by the user agent and I also can't block the IP because it changes. –  Devils Child Aug 28 '13 at 12:29
    
Would browsermatchnocase on TencentTraveler not do the trick ? –  Iain Aug 28 '13 at 12:32
    
I guess that would work. If you post it as an answer, I will accept it. –  Devils Child Aug 28 '13 at 12:34
    
Can't you block it by the name of the non-existent resource it's requesting? If you don't have /viewtopic.php on your site why not create a filter for it? –  squillman Aug 28 '13 at 12:38

3 Answers 3

up vote 6 down vote accepted

What does this client do?

Difficult to know, but just for curiosity I'd set up a PHP script at that url to play with your pal:

  • get the client IP address and scan it immediately (open ports, reverse address, whois information on the netblock)
  • dump HTTP headers to look into them. If it sends a referrer maybe it's coming from somewhere else. Look for other interesting things.
  • reply with an HTTP redirect to some URL you manage and see if it follows it. Just to know...
  • reply with some Javascript and see if it executes it
  • if you have lots of bandwidth try replying with some gigabytes of garbage :)
    • (and while you're at it, measure how fast he can digest data)
  • when you're bored of playing with him then go tarpitting him (if you have enough resources): everytime it connects enter some long sleep() before completing the HTTP reply. Make him consume more resources.

why is it calling this page continuously?

Did you try to ask Google? Seems like Tencent Traveler is one of the most popular browser in China. Maybe someone with it visited your site, or tried to visit invalid URLs on your site, maybe there's a spider run by the software house trying to index / scan / cache the sites visited by its browser users. Or maybe it's just a chinese bored boy, or maybe an infected PC, or... who knows, you should also ask him directly, but you need to know who he is.

And how can I prevent this?

To actually prevent him for trying, you should report it as a spammer to someone/something that has authority over him. Usually this is the Technical Contact (or tech-c, or abuse, or whatever they call it) of the AS responsible for its IP range. You can find that piece of information from whois.

If prevention does not succeed and it keeps doing those requests, you can block them in various ways:

  • if they come from a single subnet and you're not interested in that public, block the whole subnet at the IP level (eg. if it comes from a Chinese /8 subnet and you don't mind blocking chinese clients, simply drop traffic from the whole subnet)

  • block all requests matching that User-Agent at the HTTP level if you have no known legitimate users with the very same User-Agent.

Why do these aggressive bots always come from China or Russia?

Ahhh, life would be easier if that was actually true ;)

it distorts the visitors statistic significantly

You're talking about one hit per 5 minutes, or roughly 0.003hit/second - should be well below the noise floor.. Please rephrase your question and explain what you mean by "distorts".

share|improve this answer
    
Tencent Traveler is just a web browser. According to wikipedia, third most used in China in 2008. –  Kwaio Aug 28 '13 at 13:05
    
When I wrote "block the user-agent" I did actually mean the user-agent, not just the "Tencent Traveler" substring. But the agent is not the point here, the originating client / IP is. –  Luke404 Aug 28 '13 at 13:15
    
I just wrote that to correct your sentence "Seems like Tencent Traveler is some sort of browser plugin and a web spider." which i thought wrong... but maybe you have more insights about that than me. –  Kwaio Aug 28 '13 at 13:23
    
@Kwaio, yeah sorry for me being imprecise and thanks for your correction. I'm editing the answer to incorporate your fix. –  Luke404 Aug 28 '13 at 13:30

You could use mod_setenvif to match a known portion of the UserAgent (TencentTraveler for example) and then block it

BrowserMatchNoCase TencentTraveler bad_bot
Order Deny,Allow
Deny from env=bad_bot
share|improve this answer
    
Assuming he uses Apache httpd. but great solution –  Kwaio Aug 28 '13 at 12:46
    
Yes, I'm using apache. Thank you :) –  Devils Child Aug 28 '13 at 12:52
    
Note that Tencent Traveler seems to be a very used chinese web browser. According to Wikipedia, it was the third most used browser in China in 2008. If your website may concern chinese, you would by this way block some of these users. –  Kwaio Aug 28 '13 at 12:59
1  
Please also note it is a great solution... to another problem. The question is about preventing those requests, he does not talk about blocking them. But it's good to know anyway how to do the latter :) –  Luke404 Aug 28 '13 at 13:07

Install fail2ban and configure jail.conf to allow apache jailing. Then add a failregex match in the apache section something like this:

failregex = [[]client (?P<host>\S*)[]] File does not exist: .*\.phpf=3&t=849
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.