Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Apache is receiving requests at port :80 and proxying them to Jetty at port :8080

The proxy server received an invalid response from an upstream server
The proxy server could not handle the request GET /.

My dilemma: Everything works fine normally (fast requests, few seconds or few tens of seconds long requests are processed ok). Problems occur when request processing takes long (few minutes?).

If I issue request instead directly to Jetty at port :8080 the request is processed OK. So problem is likely to sit somewhere between Apache and Jetty where I am using mod_proxy. How to solve this?

I have already tried some "tricks" related to KeepAlive settings, without luck. Here is my current configuration, any suggestions?

#keepalive Off                     ## I have tried this, does not help
#SetEnv force-proxy-request-1.0 1  ## I have tried this, does not help
#SetEnv proxy-nokeepalive 1        ## I have tried this, does not help
#SetEnv proxy-initial-not-pooled 1 ## I have tried this, does not help
KeepAlive 20                       ## I have tried this, does not help
KeepAliveTimeout 600               ## I have tried this, does not help
ProxyTimeout 600                   ## I have tried this, does not help

NameVirtualHost *:80
<VirtualHost _default_:80>
    ServerAdmin webmaster@mydomain.fi

    ServerName www.mydomain.fi

    ServerAlias mydomain.fi mydomain.com mydomain www.mydomain.com

    ProxyRequests On
    ProxyVia On
    <Proxy *>
            Order deny,allow
            Allow from all
    </Proxy>

    ProxyRequests Off
    ProxyPass / http://www.mydomain.fi:8080/ retry=1 acquire=3000 timeout=600
    ProxyPassReverse / http://www.mydomain.fi:8080/

    RewriteEngine On
    RewriteCond %{SERVER_NAME} !^www\.mydomain\.fi
    RewriteRule /(.*) http://www.mydomain.fi/$1 [redirect=301L]

    ErrorLog /var/log/apache2/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog /var/log/apache2/access.log combined
    ServerSignature On

</VirtualHost>

Here is also the debug log from a failing request:

74.125.43.99 - - [29/Sep/2010:20:15:40 +0300] "GET /?wicket:bookmarkablePage=newWindow:com.mydomain.view.application.reports.SaveReportPage HTTP/1.1" 502 355 "https://www.mydomain.fi/?wicket:interface=:0:2:::" "Mozilla/5.0 (Windows; U; Windows NT 6.1; fi; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10"
[Wed Sep 29 20:20:40 2010] [error] [client 74.125.43.99] proxy: error reading status line from remote server www.mydomain.fi, referer: https://www.mydomain.fi/?wicket:interface=:0:2:::
[Wed Sep 29 20:20:40 2010] [error] [client 74.125.43.99] proxy: Error reading from remote server returned by /, referer: https://www.mydomain.fi/?wicket:interface=:0:2:::
share|improve this question
    
Hi.. I am still stuck with this one. All above settings tried and also increasing jetty maxIdleTime did not help. Any pointers what to try next? –  Martin Feb 12 '11 at 7:49

3 Answers 3

I have solved the problem. The KeepAlive=On should be inserted into ProxyPass config line:

ProxyPass / http://www.dom.fi:8080/ retry=1 acquire=3000 timeout=600 Keepalive=On

See that

Keepalive=On

there? It is critical ;)

share|improve this answer
2  
I believe you can mark your own answers as Accepted. It marks the question as resolved in the system for other people to find. –  sysadmin1138 Feb 18 '11 at 22:50

Have you tried setting setenv proxy-initial-not-pooled 1?

Reference here

share|improve this answer
    
No, this didn't help. Then it's not about race condition, it's the long delay and something happens in between (some kind of misunderstanding between Jetty and mod_proxy). –  Martin Sep 29 '10 at 19:47

Looking at the log, there's something that times out at 5 minutes (=300 seconds). That's a pretty long time to wait for a response. When you access the Jetty server directly, does this resource really take that long to produce a response?

If the five minutes really is within possible response times, you might try tweaking the ProxyTimeout configuration directive.

Depending on your network set-up, it might well be that there's no reason to even try to use any keepalive system (is there a firewall between the app server and proxy which might be configured to drop sessions that are idle for too long?), but the ProxyTimeout would affect the behaviour of the proxy itself.

If the same proxy also serves other backends, it would be better to keep the current ProxyTimeout, and configure the timeout in the ProxyPass directive (see mod_proxy documentation).

If, however, the responses without the proxy are consistently something much less than the five minutes see here as the cut-off limit, then there might really be some odd interference between the proxy and app server, but you're not providing anything of value for identifying what it might be.

share|improve this answer
    
"When you access the Jetty server directly, does this resource really take that long to produce a response?" -- YES. I also tried setting this ProxyTimeout to 600. Does not help. There is no firewall between proxy and jetty. Timeout is configured also in ProxyPass. –  Martin Oct 11 '10 at 12:39
    
"you're not providing anything of value for identifying what it might be": I don't know what that could be. All i am getting is an error message from the server: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /. Reason: Error reading from remote server –  Martin Oct 11 '10 at 12:40
    
As for increasing the proxy timeout, did this also change the time your browser did spin before getting the 502 error? –  user56707 Oct 11 '10 at 17:01
    
Then to ways how you could find out what is happening in the application server: you could take a couple of thread dumps, and look from them what is executing when the browser is waiting. Alternatively, add enough debug logging statements to be able to trace your code to pinpoint the cause of slowness. –  user56707 Oct 11 '10 at 17:04
    
I know why it is slow. I don't know why apache proxy refuses the response when it is finally ready. –  Martin Oct 12 '10 at 3:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.