I had to design something similar recently. Here's my conceptual back of the envelope design
Load distribution
I'm assuming that the network ops guys have setup a redundant and highly
available routing, firewalling and switching fabric that delivers requests to load
balancers.
(If not, I'd go with a stateful HA setup of either PF or IPtables with automatic failover
using carp or keepalived. )
The load balancers specifications would depend on whether the web application
load distribution methodology and cost among other metrics.
Depending on the budget, the load balancing could be implemented using
- Hardware based load balancers which tend to be pricy
- Software based proxies such as HAProxy
The load balancers have to be highly available so I'd go for a couple of active load
with standby backups (say 2 HAProxy instances with 2 in standby mode).
I'd have routing layer send the requests to the load balancers. In case one of the
load balancers failed, a solution based on keepalived would be used to seamlessly
replace the faulty box.
Once the load balancers accept requests, they'd pass them on to the caching layer.
The caching layer would handle:
- Requests for static content.
- Content that has an HTTP header that states that it has not yet expired.
- Compressing static text content such as CSS and JS files.
The caching layer can be implemented using a solution such as SQUID or NGINX in reverse proxy. By doing this, we'll reduce the load on the application server by sending only dynamic requests to the Apache/PHP servers.
To keep costs at a minimum, I'd have HAProxy and NGINX sitting on the same box.
An easy and scalable way to do this would be to have CSS, JS and static images
served by a subdomain of the website (say http://cdn.myservice.com/static). Using
such a setup, we could in future install caching instances globally and have DNS
send static requests to the closest CDN instance. Initially though, the CDN work can
be handled by these NGINX instances to keep costs low.
Processing Layer
The processing layer consists of a pool of servers optimised for Apache/PHP. They
would load their configuration files from an NFS or distributed filesystem share and
would serve their requests by processing PHP scripts from another remote share
(NFS or DFS). Using these remote share eases configuration overheads of
maintaining and syncing server configurations.
Apache and PHP could be further optimised for example by:
- Removing unneeded modules in PHP and Apache.
- Using a PHP Opcode cache such as APC to reduce the PHP compile overhead.
- Optimising Apache's settings such as MPM and keep-alive settings.
A memcache server pool can also be configured to store results of common and
expensive database queries. Read queries would typically be sent to the slaves if
they are not in the memcache layer and their results would then be cached. Writes
would be sent to the master and may involve invalidating data in the memcache
layer. PHP session data can also be shared over memcached so that if any single
Apache/PHP server fails, remaining servers can pick up the session data from
memcache.
Scaling for load in the processing pool would be a matter of adding more servers
and updating the reverse proxies. The server pool may be partitioned into a
number of logical groups. A logical group would then use a common configuration
shared over NFS and can be upgraded as block.
The upgrade can then be monitored and if issues are detected, fixes can be
implemented or a rollback implemented. The logical group could be distributed
over racks that share nothing (Power, network switches etc) and consist of
disparate members (say server models A, B and C from Dell) so that block
migration tests are comprehensive.
Database
For the database, I'd have a MySQL server running in a master/multi slaves setup.
The master would be optimised for writes with binary logging enabled for
replication. This typically means that we'd use the usual optimisation for MySQL
such as:
- Using RAID 10 and high RPM drives.
- Disabling atime/mtime on the mysql data directories.
- Tuning innodb settings for CPU and RAM.
- Proper indexing and partitioning of tables.
- Monitoring the slow query log and using explain to profile slow queries.
- Monitoring the database performance.
Slaves would be configured for reads and would need to be constantly monitored
for replication lag using a utility such as maatkit's mk-heartbeat . A lagging server
may be removed from the PHP read set until it catches up.
In case of a master failure:
- A slave will be promoted to master.
- Changes will be made to DNS to point to the new master.
- Resolvers in the network are reloaded to reflect DNS changes.
- Other slaves automatically pick from the new master(We can make the binlog
position and file to be the same on master and slaves to make this migration easy )
- The Apache/PHP servers would automatically write to the new master since the
DNS resolvers will return the new master. Reads will also be sent to the proper
servers by using DNS round robin with A/AAAAA records for the slave set.
An alternative to DNS may be storing the list of good slaves in a cache such as
memcache and updating it appropriately.
To top this off, I'd have a workstation or two for network monitoring and report
aggregation. I'd use Munin/Zenoss for trend analysis, A syslog server for aggregation of the servers logs, custom scripts for log analysis and alerting. Nagios
may also be used to provide a global overview of the infrastructure and alerting.
Scaling
Upgrading the infrastructure for more load would be handled by:
- Increasing the load balancers reverse proxy servers to handle more static content.
- Geographical distribution of the static content. Depending on cost, CDN work may
be outsourced to a service such as Amazon's cloudfront.
- Increasing the number of Apache/PHP servers to handle more load.
- Increasing the number of MySQL slaves.
- Sharding the master and MySQL federation to present a unified view of tables
spread over a a cluster of database servers