I have been getting a lot of web hits in my logs that crawl most top level pages of my site and show a referrer as a Java version.

I see different variants of the Java versions in the referrer, i.e. Java/1.6.0_04, Java/1.4.1_04, Java/1.7.0_25, etc.

And sometimes, but not always, I get a 404 for /contact/ but none of the other pages below.

The IPs are usually always spam harvesters and bots, according to Project Honeypot - - [24/Jan/2014:01:28:52 -0800] "GET / HTTP/1.1" 200 6728 "-" "Java/1.6.0_04" 198 7082 - - [24/Jan/2014:01:28:55 -0800] "GET /about HTTP/1.1" 301 - "-" "Java/1.6.0_04" 203 352 - - [24/Jan/2014:01:28:55 -0800] "GET /about/ HTTP/1.1" 200 29933 "-" "Java/1.6.0_04" 204 30330 - - [24/Jan/2014:01:28:56 -0800] "GET /articles-columns HTTP/1.1" 301 - "-" "Java/1.6.0_04" 214 363 - - [24/Jan/2014:01:28:57 -0800] "GET /articles-columns/ HTTP/1.1" 200 29973 "-" "Java/1.6.0_04" 215 30370 - - [24/Jan/2014:01:28:58 -0800] "GET /contact HTTP/1.1" 301 - "-" "Java/1.6.0_04" 205 354 - - [24/Jan/2014:01:28:58 -0800] "GET /contact/ HTTP/1.1" 200 47424 "-" "Java/1.6.0_04" 206 47827

What are they looking for? A vulnerability?

Can I block these visits by their Java referrer? If so, how? Or with a php function?

I know how to block IPs in .htaccess, but blocking by User-agent is a more proactive method for me).

Update 2/04/14 I'm not able to block a Java User-agent with either of these two rules.

RewriteCond %{HTTP_USER_AGENT} Java/1.6.0_04
RewriteRule ^.*$ - [F]

RewriteCond %{HTTP_USER_AGENT} ^Java
RewriteRule ^.*$ - [F]

Note: I'm on shared hosting and have limited access to apache configs.

  • I think you mean User-Agent string, not referrer. And, are you asking for us to code this for you?! Jan 27, 2014 at 15:54
  • 2
    I'm not asking for someone to code it for me; it's a question, like any other. And it's multi-part: what's the vulnerability, if there is one? How would this work? Should I block them with the more labor-intensive .htaccess method? Jan 27, 2014 at 17:06
  • RewriteLog and RewriteLogLevel. Gather some information about the rewrite processing and share it. Your condition might need to be less specific. Try matching on "^Java.*" for example to block any version. (I doubt the User-Agent is just "Java".) Feb 10, 2014 at 15:40
  • @AaronCopley, thanks, but I'm restricted from using RewriteLog. But I am trying the match for Java as you suggested. Feb 10, 2014 at 17:07

2 Answers 2


User Agent string matching is not reliable method, as anyone can change it on headers.

From my experience, every internet facing webserver is akin to be crawled and surfed (that's THE point, right? :).

If anything, they're just crawling your webserver for indexing of some sort. If you want to frustrate or limit the frequency for those requests, I'd suggest apache mod_evasive, or mod_dosevasive, or mod_qos, to limit the number of concurrent connections per IP per second, and more.

Keep in mind that this solution could lead to your webserver blocking legitimate requests from NAT routed requests and so on.

Then, you'll need to code the 403 forbidden yourself defining a set of rules from crawling behaviour into your php app when bots learn your apache mods evasive frequency setup.

  • Thanks, I know that all sites get crawled by all manner of spiders, but after watching the bots with the java user-agent, I think they all area malicious. I'm now blocking them in .htaccess to see how it works. Feb 2, 2014 at 15:37
  • Added the two rules above in my question that don't work; any ideas? Thanks... Feb 7, 2014 at 2:19
  • is RewriteEngine on?
    – Marcel
    Feb 7, 2014 at 21:29
  • Yes, it's on... Feb 8, 2014 at 14:52

Is AllowOverride set to All?

As a more proper solution, I would recommend using mod_evasive[1] to block excessive scanning by any client. Requires iptables though.

  1. http://www.zdziarski.com/blog/?page_id=442
  • Thanks, but I'm on shared hosting and have limited to no access to apache configs and other server settings. Feb 7, 2014 at 14:37
  • Then, you have to ask your hosting provider whether .htaccess is allowed and enabled?
    – antimatter
    Feb 7, 2014 at 14:53
  • I am using .htaccess. I can't use ip tables. Feb 7, 2014 at 15:07
  • Do you mind posting your entire .htaccess file?
    – antimatter
    Feb 7, 2014 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.