0

We currently have a simple 3 server setup:

  1. Cloud NGinx/Apache web server (NGinx serves static, dynamic is proxied to Apache/PHP backend)
  2. Dedicated Database server
  3. Cloud Outgoing email + monitoring server

We're at the point where we need to scale up/out our web server. We're looking to acheive the following:

  1. Minimise downtime if a web server fails
  2. Offer additional capacity during peak times
  3. Allow us to upgrade one web node without taking down website
  4. Allow SSL serving also

I would like to add an additional web node, with equal capacity to the original one, rather than doubling the capacity of the existing web node, as this will give us a failover/mirror node (when load balanced) to improve capacity and offer redundancy.

I've looked into the options in depth, at the moment HAProxy seems to be out of the question as I want to handle SSL (and don't want to terminate at the Load Balancer, all SSL traffic must be encrypted end-to-end).

My options seem to be:

A: Add a software load balancer (eg Nginx) on it's own node, in front of the two web nodes. Issue here is when UGC is uploaded by a user, it won't be available on the other node. With sticky sessions this is less of an issue but ultimately we need to sync uploaded content otherwise the nodes will become out of sync. Can use Rsync but as we're looking at master-master web config, need bidirectional..tricky!

B: As above, but use the load balancer to also serve static content. This would then offload some of the processing from the web nodes. We could rsync all uploads from the web nodes to the load balancing node, this way it's only a one-way rsync from web->load balancer. Any non-static requests are load balanced to the web nodes. This keeps all static content in one place but I was aiming to have a small load balancing node and this may end up growing quite big?

c: As A, but use NGinx to direct any visitor traffic where uploading is likely (this is easy to segment) to one 'master' server to do their uploads, and fairly load balance between the other nodes for 'read only' traffic. This makes syncing easy as it will be one-directional (master web node->slave web node), allows us to use the disk space on the web nodes better and keeps static content off the load balancer. However it means that if the master node goes down, we'd need to manually switch traffic for our 'upload' areas to the remaining server (nginx may be able to do this switch, we just have a different load balance config for our upload areas)

B seems like a good config, and I could probably combine the Load balancer node with our existing email/monitoring server as it sits idle. There is a single point of failure with the load balancing node, but in an emergency we could bring up a new cloud instance to replace this. However I don't like the idea of the static images only being on one node as we'd have no inherent backup. I also like the idea of C as it means it's kept very simple. If the load balancer fails, we could direct traffic direct to one of the web nodes without any issues (as they would both have a full set of static files).

Any input on the solutions above would be very much appreciated

1 Answer 1

1

I would chose B, because the advantages you name are correct and compelling.

When you have two web servers you already have to solve the problem of keeping content in sync on multiple machines; adding a third is trivial, so the cost of syncing static content from whatever source to the load balancer is easy.

Disk space is, generally speaking, insanely cheap - and if you grow significantly you probably want to push that static content to a CDN (eg: S3, Akami) anyway, so having the load balance know where that is and direct the client appropriately is a reasonably middle step.

Given that the problem of syncing content between two master nodes is hard, in most cases, I would start with directing all upload traffic to one machine and doing read-only replication to the other servers. That keeps the changes smaller, and you can work on the master/master model after the load balancer is in place, working, and tested.

2
  • As were using cloud based setup disk space is relative to instance size. therefore we may end up using a bigget server just for disk space even if cpu etc is low. I think option A is out so need to balance up between B and C now!
    – Ben
    Jan 17, 2012 at 21:19
  • In an Amazon style scenario, I would look to something like S3 that offered public URL support for serving content, and was specialized to the purpose. You could also introduce that as step one, before you touch any of the other changes, to reduce the overall risk. Jan 17, 2012 at 21:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .