Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have been getting a lot of web hits in my logs that crawl most top level pages of my site and show a referrer as a Java version.

I see different variants of the Java versions in the referrer, i.e. Java/1.6.0_04, Java/1.4.1_04, Java/1.7.0_25, etc.

And sometimes, but not always, I get a 404 for /contact/ but none of the other pages below.

The IPs are usually always spam harvesters and bots, according to Project Honeypot

78.129.252.190 - - [24/Jan/2014:01:28:52 -0800] "GET / HTTP/1.1" 200 6728 "-" "Java/1.6.0_04" 198 7082
78.129.252.190 - - [24/Jan/2014:01:28:55 -0800] "GET /about HTTP/1.1" 301 - "-" "Java/1.6.0_04" 203 352
78.129.252.190 - - [24/Jan/2014:01:28:55 -0800] "GET /about/ HTTP/1.1" 200 29933 "-" "Java/1.6.0_04" 204 30330
78.129.252.190 - - [24/Jan/2014:01:28:56 -0800] "GET /articles-columns HTTP/1.1" 301 - "-" "Java/1.6.0_04" 214 363
78.129.252.190 - - [24/Jan/2014:01:28:57 -0800] "GET /articles-columns/ HTTP/1.1" 200 29973 "-" "Java/1.6.0_04" 215 30370
78.129.252.190 - - [24/Jan/2014:01:28:58 -0800] "GET /contact HTTP/1.1" 301 - "-" "Java/1.6.0_04" 205 354
78.129.252.190 - - [24/Jan/2014:01:28:58 -0800] "GET /contact/ HTTP/1.1" 200 47424 "-" "Java/1.6.0_04" 206 47827

What are they looking for? A vulnerability?

Can I block these visits by their Java referrer? If so, how? Or with a php function?

I know how to block IPs in .htaccess, but blocking by User-agent is a more proactive method for me).

Update 2/04/14 I'm not able to block a Java User-agent with either of these two rules.

RewriteCond %{HTTP_USER_AGENT} Java/1.6.0_04
RewriteRule ^.*$ - [F]

RewriteCond %{HTTP_USER_AGENT} ^Java
RewriteRule ^.*$ - [F]

Note: I'm on shared hosting and have limited access to apache configs.

share|improve this question
    
I think you mean User-Agent string, not referrer. And, are you asking for us to code this for you?! –  Michael Hampton Jan 27 at 15:54
2  
I'm not asking for someone to code it for me; it's a question, like any other. And it's multi-part: what's the vulnerability, if there is one? How would this work? Should I block them with the more labor-intensive .htaccess method? –  songdogtech Jan 27 at 17:06
    
RewriteLog and RewriteLogLevel. Gather some information about the rewrite processing and share it. Your condition might need to be less specific. Try matching on "^Java.*" for example to block any version. (I doubt the User-Agent is just "Java".) –  Aaron Copley Feb 10 at 15:40
    
@AaronCopley, thanks, but I'm restricted from using RewriteLog. But I am trying the match for Java as you suggested. –  songdogtech Feb 10 at 17:07

2 Answers 2

User Agent string matching is not reliable method, as anyone can change it on headers.

From my experience, every internet facing webserver is akin to be crawled and surfed (that's THE point, right? :).

If anything, they're just crawling your webserver for indexing of some sort. If you want to frustrate or limit the frequency for those requests, I'd suggest apache mod_evasive, or mod_dosevasive, or mod_qos, to limit the number of concurrent connections per IP per second, and more.

Keep in mind that this solution could lead to your webserver blocking legitimate requests from NAT routed requests and so on.

Then, you'll need to code the 403 forbidden yourself defining a set of rules from crawling behaviour into your php app when bots learn your apache mods evasive frequency setup.

share|improve this answer
    
Thanks, I know that all sites get crawled by all manner of spiders, but after watching the bots with the java user-agent, I think they all area malicious. I'm now blocking them in .htaccess to see how it works. –  songdogtech Feb 2 at 15:37
    
Added the two rules above in my question that don't work; any ideas? Thanks... –  songdogtech Feb 7 at 2:19
    
is RewriteEngine on? –  Marcel Feb 7 at 21:29
    
Yes, it's on... –  songdogtech Feb 8 at 14:52

Is AllowOverride set to All?

As a more proper solution, I would recommend using mod_evasive[1] to block excessive scanning by any client. Requires iptables though.

  1. http://www.zdziarski.com/blog/?page_id=442
share|improve this answer
    
Thanks, but I'm on shared hosting and have limited to no access to apache configs and other server settings. –  songdogtech Feb 7 at 14:37
    
Then, you have to ask your hosting provider whether .htaccess is allowed and enabled? –  antimatter Feb 7 at 14:53
    
I am using .htaccess. I can't use ip tables. –  songdogtech Feb 7 at 15:07
    
Do you mind posting your entire .htaccess file? –  antimatter Feb 7 at 18:14

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.