2

I'm looking for advice on the best way (mainly in terms of hardware) to set up a high speed load balancing server cluster on a budget (say £2,000 - £ 3,000) to host web services behind a single ip address, supported by db servers and a common file system. All on linux.

For the web servers, I know I would want to set up IPVS with apache, but I don't know the best way to spend on hardware. I would envisage having a single machine (idealy with a backup) taking requests from the internet and load balancing these amongst an array of apache servers. Each of the servers in the array have shared access to a common file system. In time I would then add more severs to the array to increase the capacity.

  1. The system is always bottlenecked at the load balancer. what kind of machine would I need to be able to support very high / not so high volumes of traffic? which is more important - processor / ram

  2. For the machines in the apache array, what do i go for - more processors, faster processors, more ram, etc - which are the most important, and does it even matter a lot if I intended to add more machines

  3. What is the best way to implement the shared file system in order to give scalability (easy to add more disk space) and performance (since it is also a bottleneck). Here I want software and hardware advice.

  4. Cost/performance estimates for machines for each of the various tasks.

  5. Any ideas on the sorts of volumes of traffic you could serve for a given number of machines with this setup.

2
  • What do you call very high traffic? Is your application bound by CPU or RAM? Why use a shared file system? May 23, 2011 at 6:39
  • Your budget is too low, especially with the shared filesystem requirement.
    – sciurus
    May 23, 2011 at 14:44

1 Answer 1

5

I had to design something similar recently. Here's my conceptual back of the envelope design

Load distribution

I'm assuming that the network ops guys have setup a redundant and highly available routing, firewalling and switching fabric that delivers requests to load balancers.

(If not, I'd go with a stateful HA setup of either PF or IPtables with automatic failover using carp or keepalived. ) The load balancers specifications would depend on whether the web application load distribution methodology and cost among other metrics.

Depending on the budget, the load balancing could be implemented using - Hardware based load balancers which tend to be pricy - Software based proxies such as HAProxy

The load balancers have to be highly available so I'd go for a couple of active load with standby backups (say 2 HAProxy instances with 2 in standby mode).

I'd have routing layer send the requests to the load balancers. In case one of the load balancers failed, a solution based on keepalived would be used to seamlessly replace the faulty box.

Once the load balancers accept requests, they'd pass them on to the caching layer. The caching layer would handle:

  • Requests for static content.
  • Content that has an HTTP header that states that it has not yet expired.
  • Compressing static text content such as CSS and JS files.

The caching layer can be implemented using a solution such as SQUID or NGINX in reverse proxy. By doing this, we'll reduce the load on the application server by sending only dynamic requests to the Apache/PHP servers.

To keep costs at a minimum, I'd have HAProxy and NGINX sitting on the same box.

An easy and scalable way to do this would be to have CSS, JS and static images served by a subdomain of the website (say http://cdn.myservice.com/static). Using such a setup, we could in future install caching instances globally and have DNS send static requests to the closest CDN instance. Initially though, the CDN work can be handled by these NGINX instances to keep costs low.

Processing Layer

The processing layer consists of a pool of servers optimised for Apache/PHP. They would load their configuration files from an NFS or distributed filesystem share and would serve their requests by processing PHP scripts from another remote share (NFS or DFS). Using these remote share eases configuration overheads of maintaining and syncing server configurations.

Apache and PHP could be further optimised for example by:

  • Removing unneeded modules in PHP and Apache.
  • Using a PHP Opcode cache such as APC to reduce the PHP compile overhead.
  • Optimising Apache's settings such as MPM and keep-alive settings.

A memcache server pool can also be configured to store results of common and expensive database queries. Read queries would typically be sent to the slaves if they are not in the memcache layer and their results would then be cached. Writes would be sent to the master and may involve invalidating data in the memcache layer. PHP session data can also be shared over memcached so that if any single Apache/PHP server fails, remaining servers can pick up the session data from memcache.

Scaling for load in the processing pool would be a matter of adding more servers and updating the reverse proxies. The server pool may be partitioned into a number of logical groups. A logical group would then use a common configuration shared over NFS and can be upgraded as block.

The upgrade can then be monitored and if issues are detected, fixes can be implemented or a rollback implemented. The logical group could be distributed over racks that share nothing (Power, network switches etc) and consist of disparate members (say server models A, B and C from Dell) so that block migration tests are comprehensive.

Database

For the database, I'd have a MySQL server running in a master/multi slaves setup. The master would be optimised for writes with binary logging enabled for replication. This typically means that we'd use the usual optimisation for MySQL such as:

  • Using RAID 10 and high RPM drives.
  • Disabling atime/mtime on the mysql data directories.
  • Tuning innodb settings for CPU and RAM.
  • Proper indexing and partitioning of tables.
  • Monitoring the slow query log and using explain to profile slow queries.
  • Monitoring the database performance.

Slaves would be configured for reads and would need to be constantly monitored for replication lag using a utility such as maatkit's mk-heartbeat . A lagging server may be removed from the PHP read set until it catches up.

In case of a master failure:

  • A slave will be promoted to master.
  • Changes will be made to DNS to point to the new master.
  • Resolvers in the network are reloaded to reflect DNS changes.
  • Other slaves automatically pick from the new master(We can make the binlog position and file to be the same on master and slaves to make this migration easy )
  • The Apache/PHP servers would automatically write to the new master since the DNS resolvers will return the new master. Reads will also be sent to the proper servers by using DNS round robin with A/AAAAA records for the slave set.

An alternative to DNS may be storing the list of good slaves in a cache such as memcache and updating it appropriately.

To top this off, I'd have a workstation or two for network monitoring and report aggregation. I'd use Munin/Zenoss for trend analysis, A syslog server for aggregation of the servers logs, custom scripts for log analysis and alerting. Nagios may also be used to provide a global overview of the infrastructure and alerting.

Scaling

Upgrading the infrastructure for more load would be handled by:

  • Increasing the load balancers reverse proxy servers to handle more static content.
  • Geographical distribution of the static content. Depending on cost, CDN work may be outsourced to a service such as Amazon's cloudfront.
  • Increasing the number of Apache/PHP servers to handle more load.
  • Increasing the number of MySQL slaves.
  • Sharding the master and MySQL federation to present a unified view of tables spread over a a cluster of database servers

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .