Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Issue with apache mod proxy, it stops proxying requests after a graceful restart but not all the time. It seems to happen only on a Sunday when a graceful restart is triggered by logrotate.

    [Sun Sep 9 05:25:06 2012] [notice] SIGUSR1 received.  Doing graceful restart
    [Sun Sep 9 05:25:06 2012] [notice] Apache/2.2.22 (Ubuntu) Phusion_Passenger/3.0.11 configured -- resuming normal operations
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(492) failed in child 26153 for worker proxy:reverse
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(486) failed in child 26153 for worker http://api.myservice.org/api
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(487) failed in child 26153 for worker http://api.myservice.org/editor/$1
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(489) failed in child 26153 for worker http://api.myservice.org/build
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(490) failed in child 26153 for worker http://api.myservice.org/help
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(491) failed in child 26153 for worker http://api.myservice.org/motd.html
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(480) failed in child 26153 for worker http://api.myservice.org/api
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(481) failed in child 26153 for worker http://api.myservice.org/editor/$1
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(483) failed in child 26153 for worker http://api.myservice.org/build
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(484) failed in child 26153 for worker http://api.myservice.org/help
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(485) failed in child 26153 for worker http://api.myservice.org/motd.html
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(479) failed in child 26153 for worker http://api.myservice.org/motd.html

After these lines, the logs are flooded with 404's because the requests are not being proxied. It's worth noting that the destination is just another vhost on the same apache instance, but the vhost (http://api.myservice.org) is serving passenger (mod_rails)

I was thinking that maybe there's some startup issues with the passenger workers not being ready during a graceful restart?

After a full restart resolves it and everything returns to normal.

//Edit Here's the vhost config, thanks :)

 <VirtualHost *:80>
      UseCanonicalName Off

      LogFormat "%V %h %l %u %t \"%r\" %s %b" vcommon

      <Directory /var/www/vhosts>
              RewriteEngine on
              AllowOverride All
      </Directory>

      RewriteEngine on
      RewriteCond /var/www/vhosts/%{SERVER_NAME} !-d
      RewriteCond /var/www/vhosts/%{SERVER_NAME} !-l
      RewriteRule ^ http://sitenotfound.myservice.org/ [R=302,L]

      VirtualDocumentRoot /var/www/vhosts/%0/current

      # Rewrite requests to /assets to map to the /var/file-store/<SERVER_NAME>/
      RewriteMap lowercase int:tolower
      RewriteCond %{REQUEST_URI} ^/assets/
      RewriteRule  ^/assets/(.*)$ /var/file-store/${lowercase:%{SERVER_NAME}}/$1

      # Map /login to /editor.html as it's far friendlier.
      RewriteCond %{REQUEST_URI} ^/login
      RewriteRule .* /editor.html [PT]

      # Forward some requests to the API
      ProxyPass /api http://api.myservice.org/api
      ProxyPass /site.json http://api.myservice.org/api/editor/site
      ProxyPassMatch ^/editor/(.*)$ http://api.myservice.org/editor/$1
      ProxyPassMatch ^/api/(.*) http://api.myservice.org/api/$1
      ProxyPass /build http://api.myservice.org/build
      ProxyPass /help http://api.myservice.org/help
      ProxyPass /motd.html http://api.myservice.org/motd.html

      <Proxy *>
              Order allow,deny
              Allow from all
      </Proxy>

      # TODO generate slightly more specific Error Documents for 401/403/500's, 
      # but for now the 404 page is good enough
      ErrorDocument 401 /404.html
      ErrorDocument 403 /404.html
      ErrorDocument 404 /404.html
      ErrorDocument 500 /404.html
 </VirtualHost>
share|improve this question
    
welcome to ServerFault. It would be helpful if you posted your relevant httpd confs as part of your question. –  KM. Sep 10 '12 at 13:46
    
That would help, Added, Thanks! –  Rob Sep 11 '12 at 9:37

1 Answer 1

I can't answer your question about the proxying other than it appears correct that the workers don't get setup properly with a graceful restart. But I'd suggest that there's no need to restart apache at all, graceful or otherwise, just for rotating the logs. You just need to use the option copytruncatein your logrotate config; instead of moving the file, requiring a restart to get the new filehandle, it'll copy the contents of the file and empty the old one, so apache can keep writing to the same filehandle.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.