up vote 0 down vote favorite
share [g+] share [fb]

This is the best I can describe my problem...

I am running a search engine scraper essentially on Windows 2008 Server. Constantly accessing millions of random domain names and following their links basically.

After a few hours of serious scraping... the server will become non responsive in terms of HTTP requests. The only solution is the restart the server.

The URL http://google.com does not respond. Even an http://localhost/ request to a running WAMP instance does not respond. Although I am still able to connect remotely to the machine somehow... so internet services are indeed working.

I have tried disabling the firewall already. I have tried clearing the DNS cache as well. The only solution is a restart, and then HTTP requests work immediately.

Is there some setting I can alter or error list I can view for these types of errors on Windows 2008 Server?

EDIT: Upon further inspection, it looks like my maximum available ports are being maxed out. cports tells me I have 15,000 "System" ports (every single one from 49,158 to 65,535) is stuck in a "Fin Wait 1" state. Any ideas?

A script of some sort that forcibly removes all fin wait 1 connections regularly via task scheduler perhaps?

link|improve this question
100% accept rate
feedback

1 Answer

up vote 0 down vote accepted

It sounds like you have managed to seriously overload IIS.

Have you adjusted the settings for your application pool to account for this huge throughput ?

Of course, you don't say whether IIS itself is used in making external web requests - I do hope you understand that these have nothing to do with one another in principle.

link|improve this answer
the OP says he is running WAMP - which might be part of the problem. – Jim B Dec 22 '11 at 15:32
It's an obscure reference at best. – adaptr Dec 22 '11 at 15:41
WAMP is an obscure reference? – Jim B Dec 22 '11 at 20:56
Upon further inspection, it looks like my maximum available ports are being maxed out. cports tells me I have 15,000 "System" ports (every single one from 49,158 to 65,535) is stuck in a "Fin Wait 1" state. Any ideas? – darkAsPitch Dec 22 '11 at 22:34
@JimB: it is if it is referenced halfway down the end of a sentence that talks about HTTP client services, and doesn't have anything to do with a web server at all. He could still be using IIS for the client part. – adaptr Dec 23 '11 at 8:15
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.