Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We are suddenly seeing an intermittent, but very frequent, error while POSTing files to a Tomcat servlet over https via an Apache Server reverse proxy. The error only seems to occur while POSTing files of 20 MB or more (videos). We have not seen the problem with smaller files of 2 to 5 MB (JPEGs).

The error occurs on two of the five servers we've tried it on.

On the server side of the connection, i.e., a Tomcat servlet built using Jersey, we get:

java.net.SocketException: Connection reset

The Apache Server on the same machine, acting as a reverse proxy, gives this error message:

[Thu May 15 17:08:58 2014] [error] proxy: pass request body failed to 127.0.0.1:8080 (localhost) from 192.168.16.xx ()

Setting Apache Server's logging level to debug and reproducing the problem produces no additional information - we still get that same error message without any associated messages.

A little less often, we don't get an exception on the Tomcat side, but checking the number of bytes transferred against the Content-Length header reveals that not everything made it through. The error in Apache Server in this second scenario is the same as in the first "proxy: pass request body failed..."

The Apache Server version is 2.2.15.29 on one server and 2.2.15.30 on the other, running under CentOS 6.2 in all cases. The reverse proxy rules are set up like so:

<IfModule mod_proxy.c>
        ProxyRequests Off

    # Case Manager Tomcat web service
        ProxyPass /casemanager http://localhost:8080/casemanager
        ProxyPassReverse /casemanager http://localhost:8080/casemanager

    # Matcher images directories
        ProxyPass /matcher-images http://x.x.x.108:80/matcher-images
</IfModule>

Note that the proxy pass-through is just going over http, not https.

We are using self-signed certificates for the SSL configuration. OpenSSL version is 1.0.1e-fips on one of the failing servers and 1.0.0-fips on the other.

On the Tomcat side, we are running 7.0.26 and using Jersey 1.8.

I doubt that it matters, but the browser connection from which the POST originates is either Firefox 27 or Chrome 34.

In one case, our servlet was recently updated, although the code for handling files uploads has not recently changed. On the other server experiencing the problem, we are running a servlet build from months ago. In fact, on that second machine, we aren't aware of any code or configuration changes in the last few months -- it had been sitting idle from mid-February until today.

What should I do next to troubleshoot this issue? Where should I be looking?

-- Update --

Further testing shows that the connection is sometimes dropped even if I bypass Apache Server and POST directly to Tomcat. So it doesn't appear to be a proxy issue at all.

-- Further Updates --

We are intermittently seeing problems copy the same large files via scp. It looks like the underlying issue is the firewall between our office/development subnet and the production/staging network. The error message when an scp copy fails is "Broken pipe."

Here is the stack trace from the servlet when the connection drops:

2014-05-16 13:20:44,566 [http-bio-8080-exec-7] ERROR com.objectvideo.wx.casemanager.service.resources.QueryFileService [null] - Failed to upload file(s). javax.ws.rs.WebApplicationException: java.net.SocketException: Connection reset at com.objectvideo.wx.casemanager.service.resources.QueryFileService.uploadRawFile(QueryFileService.java:342) at com.objectvideo.wx.casemanager.service.resources.QueryFileService.uploadFile(QueryFileService.java:607) at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:189) at java.net.SocketInputStream.read(SocketInputStream.java:121) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:532) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501) at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:563) at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:118) at org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:326) at org.apache.coyote.Request.doRead(Request.java:422) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:431) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:167) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1719) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1696) at org.apache.commons.io.IOUtils.copy(IOUtils.java:1671) at org.apache.commons.io.FileUtils.copyInputStreamToFile(FileUtils.java:1444) at com.objectvideo.wx.casemanager.service.resources.QueryFileService.writeTempFile(QueryFileService.java:535) at com.objectvideo.wx.casemanager.service.resources.QueryFileService.uploadRawFile(QueryFileService.java:320) ... 35 more

share|improve this question

1 Answer 1

Try appending keepalive=on to your ProxyPass directives.

This may require KeepAlive to be set to on on the so global Apache server side as well.

share|improve this answer
    
Good suggestion, thanks. It does not seem to have an effect, though. –  dgvid May 16 at 13:53
    
Any chance you can post the complete java stack trace? –  alxgomz May 16 at 20:43
    
This seems like the perfect solution for a crappy network, “This parameter should be used when you have a firewall between your Apache and the backend server, who tend to drop inactive connections. This flag will tell the Operating System to send KEEP_ALIVE messages on inactive connections and thus prevent the firewall to drop the connection. To enable keep alive set this property value to On.” httpd.apache.org/docs/2.2/mod/mod_proxy.html –  JakeGould May 16 at 21:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.