Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am running a little apache2 webserver for a department at my university, and I noticed the access logs are filled with messages like the following:

"GET /path/on/my/site/7z|aac|arc|arj|asf|asx|avi|bin|csv|doc|exe|flv|gif|gz|gzip|hqx|jar|jpe?g|js|mp(2|3|4|e?g)|mov(ie)?|msi|msp|pdf|phps|png|ppt|qtm?|ra(m|r)?|sea|sit|tar|tgz|torrent|txt|wav|wma|wmv|wpd|xls|xml|z|zip HTTP/1.0" 200 22932

This seems to be occurring literally every few seconds all day every day, each time with a different path on our website, and it all comes from one IP address within my university. Sometimes it only has the path and no "7z|aac|arc|arj..." appended to the end. I am new to running a webserver, so this may be obvious to professionals — am I simply being crawled or is this something sinister that I should block?

share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

Take a look at the HTTP agent string. (this should also be in your access log) That may help you figure out /what/ it's coming from.

It does NOT look like a normal webcrawler request, it looks like an attempt to find downloadable material.

Take a look at the reverse DNS lookup of the host that's doing this probe. It might also help.

And lastly, whenever you see traffic that isn't what you want/expect to your own website, feel free to block it. If you don't see any legitimate requests coming from this IP, then it's effectively overloading your logs with junk that you don't want. My advice, block it.

share|improve this answer
    
Thanks for the advice. The log actually doesn't have the agent string — it is newly set up and didn't have that in there by default (which I will fix for the future). However, after investigating based on the IP, it turns out the crawls are coming from my university's IT department. I wonder if they might be trolling the network for illegal filesharing... –  Animadvert Mar 11 '12 at 3:14
    
Ask your Univeritys IT Department before blocking this. Remember, you're on their network! –  juwi Mar 11 '12 at 6:51
    
juwi: You are absolutely right and I will indeed check with them first, especially now that I know it is the IT dept. –  Animadvert Mar 11 '12 at 13:29
add comment

Why don't you ask within the University then. Someone within must have assigned that IP and will know, who or what it belongs to.

From what you say you're on the University Network and this traffic does, too. So there are two options: 1. A student trolling - unlikely, they usually have IDS and stuff nowadays and would've probably already shut this down 2. Its some check the university does.

In either case. ASK! Blocking may be a very bad idea, because if it is something legitimate they run for security or whatever and you block it - they will probably do the same with you.

share|improve this answer
    
This should probably be a comment rather than an answer as worded... –  Bart Silverstrim Mar 10 '12 at 21:29
    
In my opinion this technically is an answer though, even though it is absolutely non-technical, as it points him towards where he will get a definitive answer instead of guesses on this. –  juwi Mar 11 '12 at 6:46
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.