Apache 2.4 sends 502 errors when backend sends 401 on large file uploads

Question

Using Apache 2.4.25 (Windows) and backend server Tomcat 8 (Windows).

We have some client software that uploads files using HTTPS on an authenticated session that is proxied through Apache to Tomcat. However, if an authenticated session is not used (or the session times out), the backend Tomcat server sends a HTTP 401 response. However, Apache returns a 502 error to the client instead.

Tomcat access log:

127.0.0.1 - - [21/Apr/2017:18:30:38 +0000] "POST /upload/10507.wav HTTP/1.1" 401 39

Apache access log:

10.8.21.23 - - [21/Apr/2017:18:30:27 +0000] "POST /upload/10507.wav HTTP/1.1" 502 232 14030

The Apache error log:

[Fri Apr 21 18:30:39.770715 2017] [proxy:error] [pid 33652:tid 120896] (OS 10053)An established connection was aborted by the software in your host machine.  : [client 10.8.21.23:62863] AH01084: pass request body failed to 127.0.0.1:8888 (127.0.0.1)
[Fri Apr 21 18:30:39.770715 2017] [proxy_http:error] [pid 33652:tid 120896] [client 10.8.21.23:62863] AH01097: pass request body failed to 127.0.0.1:8888 (127.0.0.1) from 10.8.21.23 ()

This question has a similar symptom to the following case and bug but with a different error in the logs:

• Intermittent error when using mod_proxy to do reverse proxy to SOAP service
• https://bz.apache.org/bugzilla/show_bug.cgi?id=37770#c88
• https://bz.apache.org/bugzilla/show_bug.cgi?id=48037

I've tried various combination of the following environment variables with no luck:

• SetEnv force-proxy-request-1.0 1
• SetEnv proxy-nokeepalive 1
• SetEnv proxy-initial-not-pooled 1
• RequestHeader unset Expect early
• SetEnv proxy-sendcl 1

On requests with a smaller body, it proxies the 401 as expected. On larger requests (> 1MB or so, but not consistent) it sends the 502 instead. The backend processing does not have to read the entire body before deciding whether the session exists or not, so it can fire the 401 as soon as it reads the header. With files over 10MB, it can be reproduced 100% of the time, even if proxying to a local backend server.

Speculation is that this occurs when Apache has not fully read the whole request before receiving the 401 response. It sees this as a 'proxy error' rather than pass on the 'correct' 401 response.

We've tried with a different load balancer (AWS ELB) and it sends back the 401 in all circumstances so I doubt sending a 502 is expected behavior.

Is there any Apache environment variables I'm missing to try? Is this an Apache bug?

Answer 1

The fix was to modify Tomcat config for 'maxSwallowSize' which has Tomcat swallow bytes before the response is sent.

https://tomcat.apache.org/tomcat-8.0-doc/config/http.html

maxSwallowSize

The maximum number of request body bytes (excluding transfer encoding overhead) that will be swallowed by Tomcat for an aborted upload. An aborted upload is when Tomcat knows that the request body is going to be ignored but the client still sends it. If Tomcat does not swallow the body the client is unlikely to see the response. If not specified the default of 2097152 (2 megabytes) will be used. A value of less than zero indicates that no limit should be enforced.

Answer 2

Bug in apache https://bz.apache.org/bugzilla/show_bug.cgi?id=60330 Hopefully corrected in version 2.4.40