Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Since about two months, a software package (~20 MiB, .tar.gz) we host on AWS is downloaded ~600 times a day by somebody using the bogus UA string from the title. The HTTP referrer is set to a legitimate page that contains a link to that download.

I have yet to run a more detailed analysis, but the IP always seems to be from China, with no reverse lookup record set up. And interestingly, the connection is invariably closed after approximately ~1MB.

The whole thing amounts to several tens of gigabytes of traffic per month – a small amount in the grand scheme of things, obviously, but still enough that I wonder what might be going on, assuming that we are not the only ones "crawled" by that bot.

Have you ever seen something similar on your systems? An idea what kind of bot might be running amok here?

share|improve this question
add comment

2 Answers 2

up vote 1 down vote accepted

This string belongs to Thunder Downloader (aka Xunlei), a Chinese download manager.

One of the numerous nasty things it does is that it provides direct download link to resources.
For example, Anna visits your page and downloads justin-bieber-baby.mp4 with Xunlei. Xunlei saves the file url and hash value on its server.
Next day, Betty downloads B~A~B~Y.mp4 from another site using Xunlei, Xunlei also saves the file url and hash value. Now Xunlei knows these two files are identical.
On the third day, when Cathy downloads B~A~B~Y.mp4 from that site, Xunlei also downloads part of this file from your site (using your site as an accelerator).
That means, Cathy doesn't visit your website, yet you'll have to provide bandwidth for her.
Imagine a popular file (~100MB) which millions of people*1 want to download. It could cost you ~100GB bandwidth but brought to you only one visitor.

So imho it's better to block this user agent.

*1: Xunlei has ~400 million users by 2012.

share|improve this answer
    
Accepting this answer, as other results on Google seem to corroborate it. Still, this doesn't make much sense, as the piece of software in question was an open source compiler project, and I somehow doubt it gained massive adoption in a certain part of China all of a sudden. –  klickverbot Apr 28 at 19:55
add comment

Windows 2000? I'd just ban and forget about it. Nobody has any business running Windows 2000 anymore anyway. (Though I'm sure many of us know of one still running somewhere...)

share|improve this answer
    
Note the "Mozilla/5.0 (compatible; MSIE 6.0; …" part – the UA is clearly fake, IE 6 would be Mozilla 4.0. Banning is not so easy (download page is static, downloads itself served via CloudFront), but it isn't really a huge problem in the first place. –  klickverbot Aug 8 '13 at 6:54
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.