Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This is the best I can describe my problem...

I am running a search engine scraper essentially on Windows 2008 Server. Constantly accessing millions of random domain names and following their links basically.

After a few hours of serious scraping... the server will become non responsive in terms of HTTP requests. The only solution is the restart the server.

The URL http://google.com does not respond. Even an http://localhost/ request to a running WAMP instance does not respond. Although I am still able to connect remotely to the machine somehow... so internet services are indeed working.

I have tried disabling the firewall already. I have tried clearing the DNS cache as well. The only solution is a restart, and then HTTP requests work immediately.

Is there some setting I can alter or error list I can view for these types of errors on Windows 2008 Server?

EDIT: Upon further inspection, it looks like my maximum available ports are being maxed out. cports tells me I have 15,000 "System" ports (every single one from 49,158 to 65,535) is stuck in a "Fin Wait 1" state. Any ideas?

A script of some sort that forcibly removes all fin wait 1 connections regularly via task scheduler perhaps?

share|improve this question

1 Answer 1

up vote 0 down vote accepted

It sounds like you have managed to seriously overload IIS.

Have you adjusted the settings for your application pool to account for this huge throughput ?

Of course, you don't say whether IIS itself is used in making external web requests - I do hope you understand that these have nothing to do with one another in principle.

share|improve this answer
    
the OP says he is running WAMP - which might be part of the problem. –  Jim B Dec 22 '11 at 15:32
    
It's an obscure reference at best. –  adaptr Dec 22 '11 at 15:41
    
WAMP is an obscure reference? –  Jim B Dec 22 '11 at 20:56
    
Upon further inspection, it looks like my maximum available ports are being maxed out. cports tells me I have 15,000 "System" ports (every single one from 49,158 to 65,535) is stuck in a "Fin Wait 1" state. Any ideas? –  darkAsPitch Dec 22 '11 at 22:34
    
@JimB: it is if it is referenced halfway down the end of a sentence that talks about HTTP client services, and doesn't have anything to do with a web server at all. He could still be using IIS for the client part. –  adaptr Dec 23 '11 at 8:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.