Our .NET web app makes web service calls to Amazon S3 and to a partner system.
Out of nowhere, we've had two recent episodes where our web server starts to intermittently timeout on these calls. Logged error:
System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond x.x.x.x:443 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context) at System.Net.HttpWebRequest.GetRequestStream() ...
We get a few timeout errors an hour (sometimes more; sometimes less - but each represents a failure to our users). In our first episode, the errors got gradually worse until we rebooted.
When the problem is in-effect, the Event Log shows absolutely nothing out of the ordinary. Task Manager > Networking peaks at about 25% utilization. From everything we can tell, the server is not overtaxed (CPU, files etc.); and everything looks normal.
The problem gets cleared up when we reboot (but apparently can start again at any time).
Any thoughts or suggestions would be most appreciated - including thoughts on logs we could check ... tools that we could run to monitor ... etc. Thanks!
Notes on our environment:
- Win 2008 R2 Standard - OS Updates performed regularly
- IIS 7.5 - App Pool is restarted nightly (but this hasn't helped)
- .NET 4 web app - connects to a separate dedicated SQL Server machine
- Dedicated firewall device (Win Firewall disabled)
- Besides IIS, the only other installed software running is hMailServer and Symantec Endpoint Security