0

It seems that the referrer on the following log is a folder.

112.200.208.5 - - [29/Jul/2013:20:43:14 +0800] "GET /sites/default/files/download/argie/pos-code.zip HTTP/1.1" 206 294677 "http://www.mysite.com/sites/default/files/download/argie/" "Mozilla/5.0 (Windows NT 6.2; rv:22.0) Gecko/20100101 Firefox/22.0"
61.3.158.113 - - [29/Jul/2013:20:43:14 +0800] "GET /sites/default/files/download/lnosKHEN/payroll_system_-_lnoskhen_0.zip HTTP/1.1" 206 10806 "http://www.mysite.com/download-code" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.25 Safari/534.3"
112.200.208.5 - - [29/Jul/2013:20:43:15 +0800] "GET /sites/default/files/download/argie/pos-code.zip HTTP/1.1" 206 21465 "http://www.mysite.com/sites/default/files/download/argie/" "Mozilla/5.0 (Windows NT 6.2; rv:22.0) Gecko/20100101 Firefox/22.0"
112.200.208.5 - - [29/Jul/2013:20:43:16 +0800] "GET /sites/default/files/download/argie/pos-code.zip HTTP/1.1" 206 469304 "http://www.mysite.com/sites/default/files/download/argie/" "Mozilla/5.0 (Windows NT 6.2; rv:22.0) Gecko/20100101 Firefox/22.0"
112.200.208.5 - - [29/Jul/2013:20:43:17 +0800] "GET /sites/default/files/download/argie/pos-code.zip HTTP/1.1" 206 238639 "http://www.mysite.com/sites/default/files/download/argie/" "Mozilla/5.0 (Windows NT 6.2; rv:22.0) Gecko/20100101 Firefox/22.0"
112.200.208.5 - - [29/Jul/2013:20:43:18 +0800] "GET /sites/default/files/download/argie/pos-code.zip HTTP/1.1" 206 267724 "http://www.mysite.com/sites/default/files/download/argie/" "Mozilla/5.0 (Windows NT 6.2; rv:22.0) Gecko/20100101 Firefox/22.0"
39.41.211.234 - - [29/Jul/2013:20:43:22 +0800] "GET /sites/default/files/download/john.lemar/zest-project.zip HTTP/1.1" 206 23361 "http://www.mysite.com/sites/default/files/download/john.lemar/" "Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0.1"
39.41.211.234 - - [29/Jul/2013:20:43:23 +0800] "GET /sites/default/files/download/john.lemar/zest-project.zip HTTP/1.1" 200 632601 "http://www.mysite.com/sites/default/files/download/john.lemar/" "Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0.1"
39.41.211.234 - - [29/Jul/2013:20:43:24 +0800] "GET /sites/default/files/download/john.lemar/zest-project.zip HTTP/1.1" 206 285171 "http://www.mysite.com/sites/default/files/download/john.lemar/" "Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0.1"
39.41.211.234 - - [29/Jul/2013:20:43:24 +0800] "GET /sites/default/files/download/john.lemar/zest-project.zip HTTP/1.1" 206 138366 "http://www.mysite.com/sites/default/files/download/john.lemar/" "Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0.1"
39.41.211.234 - - [29/Jul/2013:20:43:25 +0800] "GET /sites/default/files/download/john.lemar/zest-project.zip HTTP/1.1" 206 104108 "http://www.mysite.com/sites/default/files/download/john.lemar/" "Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0.1"
39.41.211.234 - - [29/Jul/2013:20:43:25 +0800] "GET /sites/default/files/download/john.lemar/zest-project.zip HTTP/1.1" 206 52055 "http://www.mysite.com/sites/default/files/download/john.lemar/" "Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0.1"
39.41.211.234 - - [29/Jul/2013:20:43:25 +0800] "GET /sites/default/files/download/john.lemar/zest-project.zip HTTP/1.1" 206 63038 "http://www.mysite.com/sites/default/files/download/john.lemar/" "Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0.1"
39.41.211.234 - - [29/Jul/2013:20:43:27 +0800] "GET /sites/default/files/download/john.lemar/zest-project.zip HTTP/1.1" 206 32452 "http://www.mysite.com/sites/default/files/download/john.lemar/" "Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0.1"
112.200.208.5 - - [29/Jul/2013:20:43:33 +0800] "GET /sites/default/files/download/argie/pos-code.zip HTTP/1.1" 206 215059 "http://www.mysite.com/sites/default/files/download/argie/" "Mozilla/5.0 (Windows NT 6.2; rv:22.0) Gecko/20100101 Firefox/22.0"

As I believe, the only valid download is this line is:

61.3.158.113 - - [29/Jul/2013:20:43:14 +0800] "GET /sites/default/files/download/lnosKHEN/payroll_system_-_lnoskhen_0.zip HTTP/1.1" 206 10806 "http://www.mysite.com/download-code" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) 

Because I set all download to be coming from this URL:

http://www.mysite.com/download-code

So, how come the referrer seems to come from a folder?

Just like this line:

112.200.208.5 - - [29/Jul/2013:20:43:33 +0800] "GET /sites/default/files/download/argie/pos-code.zip HTTP/1.1" 206 215059 "http://www.mysite.com/sites/default/files/download/argie/" "Mozilla/5.0 (Windows NT 6.2; rv:22.0) Gecko/20100101 Firefox/22.0"

The refferer is:

http://www.mysite.com/sites/default/files/download/argie/

This line:

/sites/default/files/download/argie/

is a folder.

Even if this is a web crawler, is it possible that it can access the folder on my site?

When I manually type the following:

http://www.mysite.com/sites/default/files/download/argie/

It will simply return a "Page not found". That's why I'm wondering how it becomes the referrer.

BTW, I'm using nginx.

1

You shouldn't put too much attention on the referer. The referer can be set to anything by the client. It is just a header in the request.

For example

GET /sites/default/files/download/argie/pos-code.zip HTTP/1.1
Host: www.mysite.com
Referer: http://example.org/JUST/SOME/REFERRER

So I guess the crawler just cuts off the end of the path and sets it as a referrer. I wouldn't worry.

  • Some of the User-Agents look faked too. Firefox 15.0.1 and Chrome 6 are both quite old. – Ladadadada Jul 29 '13 at 13:32
  • Believe it or not but the crawler doesn't cut off the URL. That's the folder on some of the files in my server. – jaYPabs Jul 29 '13 at 13:39
  • At lest in the logs you provided every referrer is the requested URL without the filename – Christopher Perrin Jul 29 '13 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.