Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

The setup details:

I have Apache 2.2.14 that runs under ubuntu 10.04. The document root directory for a virtualhost is set up as a symlink.

The requested script in a glance looks like:

<?php

// other useful stuff here
echo 'v1.2.42';

So apart from doing useful job the script outputs its version in the footer. The version is hardcoded in the script and is modified by a build script.

When the new version is deployed the document root is switched to a new version using pretty standard approach:

ln -s /path/to/app-1.2.43 app-new
mv -Tf app-new app # where `app` is the document root

After that I don't do any reload or restart for apache.

Today right after another release that's what I've experienced:

  1. I upgraded 1.2.42 version to 1.2.43
  2. Opened the application in the browser
  3. It was 42 in the bottom
  4. F5 - still 42.
  5. Then 42 and 43 appeared randomly. And after about 30 seconds the expected 43 started showing consistently.

Another server that has the similar configuration behaves exactly the same. And seems like there is some constant timeout until it starts working fine :-S

More configuration details:

apache is configured to run via mpm-prefork. php's version is 5.3.2 and it's installed as mod_php. There is NO any opcode cache installed.

Any ideas on where it could cache it? Would apache2 reload guarantee this won't happen?

PS: I've proven (to myself) that apache reload doesn't reset that state and the undefined behaviour continues.

UPDATE:

I've found that every apache worker expands the symlink to a document root into a real path.

And when you perform a graceful reload command - only spare workers are restarted, while the busy ones, including the ones who are serving Keep-Alive connections are kept until they are spare from work. I could reproduce it using 1 worker and it perfectly explains my observations.

Now the question is: how to interrupt keepalive connections gracefully?

Interesting article to read on topic: https://blogs.oracle.com/oswald/entry/apache_s_graceful_restart_reprise

share|improve this question
add comment

3 Answers 3

It definitely seems as though the apache process is caching the result of the symlink. Even an apache reload may not solve your issue, because a reload simply re-reads the config file and re-applies the new settings. It doesn't actually terminate the running processes.

I'm thinking the best way to handle the situation is an apache2 restart. This is probably safer than you may be thinking, because it will block new connections, and allow old connections to complete cleanly before restarting. On a moderate web server this should happen in under one second. And since HTTP is stateless, your users won't even notice. Other than a possible half second hang if they attempt to initiate a new connection during the restart.

Another possibility is to actually change the document root in your config and the name of your symlink, then do an apache2 reload. It should then detect the new document root and force a re-resolution of the symlink's referenced directory.

The last thing I can think of is to experiment with hard links. This will require a different workflow though to prevent over-writing your new app version.

share|improve this answer
    
"This is probably safer than you may be thinking, because it will block new connections, and allow old connections to complete cleanly before restarting" -- just tried it - it interrupts all current requests. –  zerkms Aug 8 '13 at 20:55
    
My apologies, I must have been recalling how some other daemon operates. Good to know anyway. Does your web app do long polling or something? I'm trying to understand why your keep-alive connections last so long as I believe apache's default keep-alive is 2 seconds. –  TheGunner Aug 9 '13 at 1:51
    
Man I'm 0 for 2... you can't hard link a directory... Perhaps you can do something like mount --bind app-1.2.3 app. Then have a simple shell script that would quickly unmount and remount to a new app version. This would allow the directory to never be resolved in a sense, so you may never have to do an apache reload. I'm unsure whether this could make a worse interruption, but maybe worth an experiment. –  TheGunner Aug 9 '13 at 2:01
    
Actually I've found a solution that suits me and allows to serve content seamlessly. –  zerkms Aug 9 '13 at 3:05
add comment

Apache caches symlinks (lstat system call) in path components when Options FollowSymlinks is set (by design).

To avoid caching you can try Option SymLinksIfOwnerMatch that guarantees to not cache lstat call.

However that needs to be properly configured and performance impact needs to be measured. See here

share|improve this answer
add comment
up vote 0 down vote accepted

Well, I have found a solution that suites me. It is a tiny script that does the following:

  1. Turn off KeepAlive in apache config
  2. Get current workers PIDs
  3. Run apache reload
  4. Wait for all workers from #2 to quit
  5. Switch symlink
  6. Turn on KeepAlive back and reload apache once again
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.