Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

On sporadic occasions we receive the following error when attempting to call an .asmx web service from a .Net client application:

"The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host."

By sporadic I mean that it might occur zero, once every few days, or a half-dozen times a day for some users. It will never occur for the first web service call of a user. And the subsequent (usually the same) call will always work immediately after the failure. The failures happen across a variety of methods in the service and usually happens between 15-20 seconds (according to the log) from the time of the request.

Looking in the IIS site log for the particular call will show one or the other of the following windows error codes:

121: The semaphore timeout period has elapsed.

1236: The network connection was aborted by the local system.

Some additional environment details:

  • Running on internal network web farm consisting of two servers running IIS7 on Windows Server 2008 OS. These problems did not occur when running in an older IIS6 web farm of three servers running on Windows Server 2003 (and we use a single IIS6/2003 instance for our development and staging environments with no issues). EDIT: Also, all of these server instances are VMWare virtual machines, not sure if that is a surprise anymore or not.

  • The web service is a .Net 2.0/3.5 compiled .asmx web service that has its own application pool (.Net 2.0, integrated pipeline). Only has Windows Authentication enabled.

  • We have another web service on the farm that uses the same physical path as the primary service, the only difference being that Basic Authentication is enabled. This is used for a portion of our ERP system. Have tried using the same and different application pool - no effect on the error. This site isn't hit as often as the primary site and has never had an error.

  • As mentioned, the error will only happen when called from the .Net client - not from other applications. The client application is always creating a new web service object for each request and setting the service credentials to System.Net.CredentialCache.DefaultCredentials.

    The application is either deployed locally to a client or run in a Citrix server session. Those users running in Citrix doesn't seem to experience the issue, only locally deployed clients. The Citrix servers and the web farm are located in the same physical location and are located in the same IP range (10.67.xx.xx). Locally deployed clients experiencing the error are located elsewhere (10.105.xx.xx, 10.31.xx.xx).

I've checked the OS logs to see if I can see any problems but nothing really sticks out.

EDIT: Actually, I myself just ran into the error a little bit ago. I decided to check out the logs again and saw that there was a Security log entry of "Audit Failure" at the 'same' time (IIS log entry at 1:39:59, event log entry at 1:39:50). Not sure if this is a coincidence or not, I'll have to check out the logs of previous errors. I'm probably grasping for straws but the details:

Log Name: Security Source: Microsoft-Windows-Security-Auditing Date: 7/8/2009 1:39:50 PM Event ID: 5159 Task Category: Filtering Platform Connection Level: Information Keywords: Audit Failure User: N/A Computer: is071019.<**>.net Description: The Windows Filtering Platform has blocked a bind to a local port.

Application Information: Process ID: 1260 Application Name: \device\harddiskvolume1\windows\system32\svchost.exe

Network Information: Source Address: 0.0.0.0 Source Port: 54802 Protocol: 17

Filter Information: Filter Run-Time ID: 0 Layer Name: Resource Assignment Layer Run-Time ID: 36

I've also tried to use Failed Request Tracing in IIS7 but the service call never actually gets to where FRT can capture it (even though the failure is logged in the web service log).

The network infrastructure group said they checked out the DNS and any NIC settings are correct so there is no 'flapping'. Everything pans out. I'm not sure that they checked out any domain controller servers though to see if that could be an issue.

Any ideas? Or any other debugging strategies to get to the bottom of this? I'm just the developer in charge of the software and don't really have the knowledge on what to investigate from the networking side of things - although it does sound like a networking issue to me based on what is happening.

Thanks in advance for any help.

share|improve this question

3 Answers 3

You could make a page that will fail with an error when that occurs (try catch), then use http://www.iis.net/downloads/default.aspx?tabid=34&i=1466&g=6">WCAT to simulate various load conditions. Hopefully then you can see a pattern or at least see if it is load related. Otherwise, I would just built something into the .Net client that catches this issue and simply retries the request, so it is transparent to the user.

share|improve this answer
    
I don't think it is load related as I've seen instances of the error occur with ony 1 or 2 people hitting the server. I don't think it is related to any period of inactivity either as the error has occurred with high activity on the site. Good idea on the 'retry' idea, although I would really like to know why this is happening. –  user11764 Jul 8 '09 at 19:00
    
Not sure if I can comment on my comment, but I just edited the question above to include information on a recent failure and a security event that may be related. –  user11764 Jul 8 '09 at 19:40

Are you required to enable Windows Filtering Platform? If you are allowed to disable it, that should avoid this audit error; if you must enable it perhaps they can make an exception to let you disable the audit category -- see: http://msdn.microsoft.com/en-us/library/bb309058%28VS.85%29.aspx

If you're required to leave WFP enabled and intact, this won't help.

share|improve this answer
    
I'm not sure on how to disable WPF or if it is allowed. Is is part of the Windows Firewall? The WF is turned off on these machines (but the underlying service is running since IPSec I think needs to be running). I think the auditing error is a herring: I have seen other posts about this error and MS has released a KB article to correct it: support.microsoft.com/kb/969257. So I'm guessing that this isn't the cause of the root problem. I still think it is somehow network/domain related but since the error doesn't pop up for 95% of the users it isn't a big concern to the network group –  user11764 Sep 18 '09 at 15:25

I am also experiencing the same sporadic situation in a production environment only. Some of the suggestions I have found but not yet validated is to either turn Http Keep-Alive off on the server, or turn it off on the web request. See http://support.microsoft.com/kb/819450

I am planning to test this in a test environment.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.