1

I don't even know where to begin with this, mostly because I can't even determine the root of the problem -- I can only find symptoms.

I'm just going to describe what it occurring, details of diagnostics, and what I've already done to try to fix it in the hopes someone can deliver me from this hell, or at least point me towards the exit.

Occurs When

  • POSTing any type of file larger than ~6MB using a web form. (like <form>)
  • Happens independently of the type of site being run -- problems persists across a PHP and ASP.NET site hosted on the same server.
  • Does not occur when tried against a load-balanced version of the same sites -- two servers using ELB
  • Not consistently reproducable when the file size is ~6MB -- the same test succeeded when failed request tracing was disabled, and then failed when it was enabled.

Symptoms

Using IIS's Failed Request Tracing I have been able to pick up some clues.

When the file POSTs successfully results are immediate. The trace records all reads by the server of the request sent from the client. At the end of the file I get this entry

GENERAL_READ_ENTITY_START
Duration
0ms

GENERAL_REQUEST_ENTITY_END
BytesReceived
7870 
ErrorCode
0 
ErrorCode
The operation completed successfully. (0x0)
Duration 
468ms

After this entry the rest of the process runs, the final entry in the trace is a flush and the end of the response.

GENERAL_FLUSH_RESPONSE_END 
BytesSent
12245 
ErrorCode
0 
ErrorCode
The operation completed successfully. (0x0) 

GENERAL_REQUEST_END 
BytesSent
12245 
BytesReceived
2598298 
HttpStatus
200 
HttpSubStatus
0 

When the file POST fails there is a long delay in reading, followed by I/O abort and loss of connection

The last entries in the request read is as follows:

GENERAL_READ_ENTITY_START
Duration
133443 ms

GENERAL_REQUEST_ENTITY_END
BytesReceived
0 
ErrorCode
2147943395 
ErrorCode
The I/O operation has been aborted because of either a thread exit or an application request. (0x800703e3) 
Duration
138248ms

Note the huge duration times. Usually the END entry is 3000-4000ms and the START is 0 or close to 0.

At the end of the trace

GENERAL_FLUSH_RESPONSE_END
BytesSent
0 
ErrorCode
2147943629 
ErrorCode
An operation was attempted on a nonexistent network connection. (0x800704cd)

GENERAL_REQUEST_END
BytesSent
0 
BytesReceived
7210715 
HttpStatus
200 
HttpSubStatus
0 

Other Symptoms

The only time when POSTing works for these files is when doing the upload from the server itself!

By this I mean navigating to http://localhost and performing the upload process. This works 100% of the time.

Fixes Already Tried

  • executionTimeout under httpRuntime in web.config is set to 6 minutes. reference
  • maxRequestLength under httpRuntime in web.config is set to 50MB.
  • maxAllowedContentLength in requestLimits in web.config is set to 50MB.

I am at wit's end, can anyone shed some light on this?

  • What kind of network bits are between your server and the client where it fails? I find your comment about the ELB very weird indeed. Do you mean Amazon's ELB? Are your other tests 100% without the ELB being involved? If not, remove it from the equation and see. Or set up apache and see if you can see the same behaviour. If so, you can perhaps stop looking at IIS... – Nik Jun 12 '14 at 22:01
  • Yes I meant Amazon's ELB. All failing tests are 100% without the ELB. I can't setup apache on the server to test it out because it's our "live" instance with old infrastructure(no ELB). The infrastructure using ELB is our new infrastructure and is not currently public-facing -- The tests done on those are on two different environments (QA and to-be-live) and both work 100%. – Matt Foxx Duncan Jun 17 '14 at 15:13
  • Is the IIS7 instance on the latest service pack? Check if the default site has some low max concurrent connection setting possibly too, this case lead to some confusion when it happens. – Schrute Jun 23 '14 at 5:13
  • The server is running 2008 R2 SP1 (Build 7601). IIS is 7.5.7600. I believe the default max concurrent connection is pretty high for IIS, and our site does not have a high load (almost ever, and certainly not when I am doing these tests). The queue limit for the app pool is 9000 so it's not that either.. – Matt Foxx Duncan Jun 23 '14 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.