Backup Plan for Server Down Time

Question

I am looking for a way to avoid as much actual downtime as possible (aren't we all). We have experienced a good deal server overload/problems lately. Our existing setup is as follows (they're Linux servers):

2 Load Balancers - one for HTTP requests, the other for HTTPS 3 Web Servers (4GB of memory per drive) - one master head which is synced to two slave drives using lSync 2 Database Servers connected to the sites via private IP We use Varnish for caching, but primarily to uploads and admin connections to web01 for proper syncing

Almost all of our sites are WordPress installations. Our traffic is largely driven by email blasts so we have concentrated periods of time with heavy traffic.

The problem: when one site is hit with a lot of traffic, or when something goes wrong with our system configurations, all sites go to a crawl if not complete halt.

Question #1: are there server configurations that could be set in place that when one server cluster goes down, the other kicks in, etc.?

Question #2: can anybody recommended good resources on this topic possibly including diagrams of various server configurations? How do the really big guys do it? :-)

P.S. My server management knowledge is limited, but I'm looking forward to learning and attempting to understand your responses!

Answer 1

Without knowing a bit more about how you have WordPress set up, it's hard to give more than general advice.

The varnish wiki has a great set of suggestions for preparing for traffic spikes on a WordPress site.

As for me, I run all my WordPress sites on nginx and php-fpm. WP has W3 Total Cache plugin installed to generate static content, which nginx then serves directly. WP and PHP never get touched in these cases and nginx just blasts the static data out the door. So far it has not been necessary to cache further. I presume something similar could be done with varnish.