Since 02.05.2013 some sort of bot is visiting my website, every 5 minutes.

Most of the time it calls this URL, which doesn't exist:


Always with this user agent

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; TencentTraveler ; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727)

(= IE 6, Windows XP)

Not only it distorts the visitors statistic significantly, it doesn't even identify as a bot, so I guess it is a spammer of some sort.

My question here is, what does this client do and why is it calling this page continuously? - And how can I prevent this?

I'm using a vServer with Linux Debian.

  • If it really bothers you block it thesitewizard.com/apache/block-bots-with-htaccess.shtml – user9517 Aug 28 '13 at 12:27
  • The problem with that is I don't really have a 'filter' in order to block it. I can't block it by the user agent and I also can't block the IP because it changes. – bytecode77 Aug 28 '13 at 12:29
  • Would browsermatchnocase on TencentTraveler not do the trick ? – user9517 Aug 28 '13 at 12:32
  • Can't you block it by the name of the non-existent resource it's requesting? If you don't have /viewtopic.php on your site why not create a filter for it? – squillman Aug 28 '13 at 12:38
  • That is also an option, but it also calls other URLs related to phpBB and WordPress. I think I will just block this so called "TencentTraveler". Why do these aggressive bots always come from China or Russia? – bytecode77 Aug 28 '13 at 12:40

What does this client do?

Difficult to know, but just for curiosity I'd set up a PHP script at that url to play with your pal:

  • get the client IP address and scan it immediately (open ports, reverse address, whois information on the netblock)
  • dump HTTP headers to look into them. If it sends a referrer maybe it's coming from somewhere else. Look for other interesting things.
  • reply with an HTTP redirect to some URL you manage and see if it follows it. Just to know...
  • reply with some Javascript and see if it executes it
  • if you have lots of bandwidth try replying with some gigabytes of garbage :)
    • (and while you're at it, measure how fast he can digest data)
  • when you're bored of playing with him then go tarpitting him (if you have enough resources): everytime it connects enter some long sleep() before completing the HTTP reply. Make him consume more resources.

why is it calling this page continuously?

Did you try to ask Google? Seems like Tencent Traveler is one of the most popular browser in China. Maybe someone with it visited your site, or tried to visit invalid URLs on your site, maybe there's a spider run by the software house trying to index / scan / cache the sites visited by its browser users. Or maybe it's just a chinese bored boy, or maybe an infected PC, or... who knows, you should also ask him directly, but you need to know who he is.

And how can I prevent this?

To actually prevent him for trying, you should report it as a spammer to someone/something that has authority over him. Usually this is the Technical Contact (or tech-c, or abuse, or whatever they call it) of the AS responsible for its IP range. You can find that piece of information from whois.

If prevention does not succeed and it keeps doing those requests, you can block them in various ways:

  • if they come from a single subnet and you're not interested in that public, block the whole subnet at the IP level (eg. if it comes from a Chinese /8 subnet and you don't mind blocking chinese clients, simply drop traffic from the whole subnet)

  • block all requests matching that User-Agent at the HTTP level if you have no known legitimate users with the very same User-Agent.

Why do these aggressive bots always come from China or Russia?

Ahhh, life would be easier if that was actually true ;)

it distorts the visitors statistic significantly

You're talking about one hit per 5 minutes, or roughly 0.003hit/second - should be well below the noise floor.. Please rephrase your question and explain what you mean by "distorts".

| improve this answer | |
  • Tencent Traveler is just a web browser. According to wikipedia, third most used in China in 2008. – mveroone Aug 28 '13 at 13:05
  • When I wrote "block the user-agent" I did actually mean the user-agent, not just the "Tencent Traveler" substring. But the agent is not the point here, the originating client / IP is. – Luke404 Aug 28 '13 at 13:15
  • I just wrote that to correct your sentence "Seems like Tencent Traveler is some sort of browser plugin and a web spider." which i thought wrong... but maybe you have more insights about that than me. – mveroone Aug 28 '13 at 13:23
  • @Kwaio, yeah sorry for me being imprecise and thanks for your correction. I'm editing the answer to incorporate your fix. – Luke404 Aug 28 '13 at 13:30

You could use mod_setenvif to match a known portion of the UserAgent (TencentTraveler for example) and then block it

BrowserMatchNoCase TencentTraveler bad_bot
Order Deny,Allow
Deny from env=bad_bot
| improve this answer | |
  • Assuming he uses Apache httpd. but great solution – mveroone Aug 28 '13 at 12:46
  • Note that Tencent Traveler seems to be a very used chinese web browser. According to Wikipedia, it was the third most used browser in China in 2008. If your website may concern chinese, you would by this way block some of these users. – mveroone Aug 28 '13 at 12:59
  • 1
    Please also note it is a great solution... to another problem. The question is about preventing those requests, he does not talk about blocking them. But it's good to know anyway how to do the latter :) – Luke404 Aug 28 '13 at 13:07

Install fail2ban and configure jail.conf to allow apache jailing. Then add a failregex match in the apache section something like this:

failregex = [[]client (?P<host>\S*)[]] File does not exist: .*\.phpf=3&t=849
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.