Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

So I just opened up my Nginx error log as all my pages are throwing error code 500.

It's enormous, my site is still in development and their are well over 100k errors for resources Nginx was unable to find all with spammy names, it's like someone is using some kind of dictionary to try to connect to masses of random pages on my website.

Is this a DDOs attack or am I missing something and how can I prevent this person from connecting again? They are all coming from an IP in the range of 66.249.73.*

share|improve this question

2 Answers 2

up vote 1 down vote accepted

Isn't that one of the ranges the Googlebot comes from?

Sounds like someone has set up a link farm pointing at your domain for some reason. (Probably spam but the details elude me.)

You should be able to control the rate of scanning and which pages get crawled by using the Google webmasters tools and robots.txt.

My first concern is why are you returning 500 responses when you should be returning 404s?

Why is your still-under-development site available to the public?

share|improve this answer
    
How can I test the site if it is not available to the public? The pages do throw 404, the pages that actually exist are throwing 500 which was the reason I checked the error logs. It does indeed seem to be a googlebot. The rate isn't more than my server can handle it just seems strange, I don't understand what benefit could be had from it. –  George Reith Jun 5 '12 at 23:41

Knowing what the actual error message is would probably help; also, check the access.log and see which requests match up with the errors - then try visiting those URLs by hand, if you get an error yourself, that suggests that your site has bugs :P

share|improve this answer
    
You misunderstood the question, I am not having a problem with the error codes, they are just the reason I checked the logs in the first place, I can handle that. I am asking about the massive number of attempted GET requests for tons of random non-existent pages. –  George Reith Jun 6 '12 at 10:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.