This is the best I can describe my problem...
I am running a search engine scraper essentially on Windows 2008 Server. Constantly accessing millions of random domain names and following their links basically.
After a few hours of serious scraping... the server will become non responsive in terms of HTTP requests. The only solution is the restart the server.
The URL http://google.com does not respond. Even an http://localhost/ request to a running WAMP instance does not respond. Although I am still able to connect remotely to the machine somehow... so internet services are indeed working.
I have tried disabling the firewall already. I have tried clearing the DNS cache as well. The only solution is a restart, and then HTTP requests work immediately.
Is there some setting I can alter or error list I can view for these types of errors on Windows 2008 Server?
EDIT: Upon further inspection, it looks like my maximum available ports are being maxed out. cports tells me I have 15,000 "System" ports (every single one from 49,158 to 65,535) is stuck in a "Fin Wait 1" state. Any ideas?
A script of some sort that forcibly removes all fin wait 1 connections regularly via task scheduler perhaps?