Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We recently migrated our site to a load balanced apache cluster behind varnish. Since that time a very small subset of users is reporting they cannot view any pages. I have narrowed the issue down quite a bit. This issue was not present before the move, the old infrastructure was a single large box.

We are on Rackspace Cloud running 8 apache2 instances behind varnish 3.0 all load balanced using Rackspace Cloud load balancers (Zeus) and 2 mysql instances for a total of 10 servers, all linux.

User can view a static html file. User can view a static asset such as an image. User cannot view any php file, even a simple one which only includes phpinfo(); User cannot view any php file when the load balancer is taken out of the picture.

The apache logs show nothing of note, other than in the access logs. PHP error reporting is set to log, and not display, although I set it to display for a short time, and the user still gets a blank page without an error. Apache/Varnish/PHP error logs show nothing of note.

Servers are:

  • Ubuntu Maverick 10.10
  • Apache 2.2.16-1ubuntu3.1 (mpm-worker)
  • PHP 5.3.3-1ubuntu9.5 (used in fcgi)
  • PHP APC is in use
  • Application is on Code Igniter
  • Varnish was 2.1.3, now 3.0.0 - issue was present with both versions
  • MySQL is the database backend in a master-master setup but due to the client access issues with a file only containing phpinfo(); I am sure the database is not the issue.

Snapshots of some configurations:

I can provide any/all logs needed to further debug, but there is nothing of note in them for the users having this issue, typical access from apache, no errors from php.

I have a feeling it might somehow be relate to php session storage although I cannot confirm this.

Any insight into the problem is greatly appreciated. Just to reiterate one final time, this issue only affects a very small handful of users. 5-10 have contacted us about the issue but I assume the number is larger than that of people who have not bothered to report the issue. These 5-10 users who have contacted us span various continents/countries/isps.

share|improve this question
    
For some reason I'm thinking APC is the issue. We used it by default at Yahoo and I remember getting some blank pages with it. IIRC there's a way to clear APC's cache programmatically. I'd give that a shot. –  Ryan Doherty Jul 27 '11 at 21:13
    
I can't imagine why specific users would see repeatable failures - aer you sure its not just on a per-request basis? You might want to install mod_security (which can create a log entry at the start of a request as well as the default completion entry - and add %P in the logs to se if the request is completing, also enable core dumps. –  symcbean Jul 28 '11 at 11:56
add comment

2 Answers

Maybe this will help you: Have you KeepAlive to set on?

We had the same problem in varnish with mpm-itk, the problem was, that when mpm-itk was accessed with a different vhost then the actuall keep-alive session was, he just killed the connection. Every normal Browser would then try to reconnect, but varnish doesn't in a default configuration. With removing KeepAlive this behaviour could not happen, so it fixed this problem.

I know, that you are not using mpm-itk, but maybe it is worth a try.

Another Idea: Can you access the site with bypassing the varnish and try to force the blank page? So you would maybe find out if the varnish is the problem.

share|improve this answer
    
+1 for accessing the site directly, that will help narrow down the cause. –  Ryan Doherty Jul 27 '11 at 21:13
add comment
up vote 3 down vote accepted

This turned out to be an issue where PHP was unable to log to the specified log file due to a permissions problem, and display of errors was disabled.

As such, the PHP error was not displayed, or able to be logged - but the root cause of the blank pages was due to a strange PHP fatal error.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.