I don't even know where to begin with this, mostly because I can't even determine the root of the problem -- I can only find symptoms.
I'm just going to describe what it occurring, details of diagnostics, and what I've already done to try to fix it in the hopes someone can deliver me from this hell, or at least point me towards the exit.
- POSTing any type of file larger than ~6MB using a web form. (like
- Happens independently of the type of site being run -- problems persists across a PHP and ASP.NET site hosted on the same server.
- Does not occur when tried against a load-balanced version of the same sites -- two servers using ELB
- Not consistently reproducable when the file size is ~6MB -- the same test succeeded when failed request tracing was disabled, and then failed when it was enabled.
Using IIS's Failed Request Tracing I have been able to pick up some clues.
When the file POSTs successfully results are immediate. The trace records all reads by the server of the request sent from the client. At the end of the file I get this entry
GENERAL_READ_ENTITY_START Duration 0ms GENERAL_REQUEST_ENTITY_END BytesReceived 7870 ErrorCode 0 ErrorCode The operation completed successfully. (0x0) Duration 468ms
After this entry the rest of the process runs, the final entry in the trace is a flush and the end of the response.
GENERAL_FLUSH_RESPONSE_END BytesSent 12245 ErrorCode 0 ErrorCode The operation completed successfully. (0x0) GENERAL_REQUEST_END BytesSent 12245 BytesReceived 2598298 HttpStatus 200 HttpSubStatus 0
When the file POST fails there is a long delay in reading, followed by I/O abort and loss of connection
The last entries in the request read is as follows:
GENERAL_READ_ENTITY_START Duration 133443 ms GENERAL_REQUEST_ENTITY_END BytesReceived 0 ErrorCode 2147943395 ErrorCode The I/O operation has been aborted because of either a thread exit or an application request. (0x800703e3) Duration 138248ms
Note the huge duration times. Usually the
END entry is 3000-4000ms and the
START is 0 or close to 0.
At the end of the trace
GENERAL_FLUSH_RESPONSE_END BytesSent 0 ErrorCode 2147943629 ErrorCode An operation was attempted on a nonexistent network connection. (0x800704cd) GENERAL_REQUEST_END BytesSent 0 BytesReceived 7210715 HttpStatus 200 HttpSubStatus 0
- The client eventually shows a 504 (Gateway Timeout) when the POST fails
- Global error catching in our application does not log any errors
- There are no errors recorded in Windows Event Viewer for IIS or ASP
The only time when POSTing works for these files is when doing the upload from the server itself!
By this I mean navigating to
http://localhost and performing the upload process. This works 100% of the time.
Fixes Already Tried
web.configis set to 6 minutes. reference
web.configis set to 50MB.
web.configis set to 50MB.
I am at wit's end, can anyone shed some light on this?