Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm in middle of re-building a website that currently gets about 4mil+ visitors a month (and that's going straight up lately). It's currently being run and hosted by an external company, but we're dumping them so I need to design hosting.

I'm thinking about building a small cluster (probably on Linode):

  • One Linode NodeBalancer to balance the load among app servers. It can keep all traffic from a specific client going to one app server, but WP handles sessions via cookies so that's not so important.

  • Two (or more) app servers - Linode (512?) VPS's running Debian6/Apache2/PHP5/Wordpress, but nginx for caching.

  • One MySQL (or MariaDB?) database server (again, a VPS), and maybe a slave with HyperDB.

  • Development is done in-house on a plain old FBSD/Apache2/MySQL/PHP5, deployment would just push new code to all the app servers one at a time and any DB changes to the DB servers.

  • Backups would be stored locally. We could back up one app server (they should be identical?) at a low-traffic time to keep load down.

  • E-mail is handled via MailChimp. Easy.

WP itself is running W3 Total Cache with Xcache, I'm considering a CDN for images and other static files, cache headers are already being used for those static files...

The plan is that as we expand, I can just add more app and/or db servers as needed.

In a nutshell: would this setup work? Would it be efficient? I've never built something like this before so I'd like to make sure I'm not missing something.

Just for reference: it's a news website. We run articles in several sections, some media, and visitors can comment on articles, sign up for our email list, etc.

share|improve this question
add comment

5 Answers

up vote 2 down vote accepted

I am in the process of setting up something similar to your with a different company. I cannot necessarily comment on how Linode works, but want to highlight some things that I've run across when looking into VPS:

  • I don't know the costs of linode's load balancers, but I found I like control over the configuration of a general VPS running HAProxy or NGINX for load balancing (I chose HAproxy).

  • Make sure you have enough RAM. Unfortunately the public offerings I've found so far do not let you adjust the amount of RAM and HDD (I need more RAM, but not nearly so much HDD. Oh well).

  • Make sure your VPS are persistent. You don't want to lose data (especially for MySQL) running on a VPS if it's shutdown or hardware failure. This is MySQL's documentation for EC2, but some of the same concepts apply).

  • Definitely make sure you're replicating MySQl to a different server, or even multiple slaves. Don't want to lose data.

  • Definitely make sure you retrieve the backups to an offsite server.

  • If you're deploying to multiple load-balanced servers (2 or more app servers), have a 'master' server that the backups rsync from to update the code. That simplifies your process of dev-pushing (take master out of load balance scheme, make sure slaves don't rsync from old master, push code to master, ensure everything is working as expected, turn rsyncing back on from master to slaves, then add master back to loadbalance scheme).

I'm sure there's more, but the setup outlined in the question seems fine.

share|improve this answer
    
We use HAproxy and Nginx for load balancing on different clusters and they both seem to handle load fine, I am just more used to using Nginx and it has proven to handle very high load. –  Chase Jul 22 '11 at 16:11
    
A basic Linode VPS costs the same as their load balancer. I don't think I'd need to tweak too much, but I'll look into using another VPS as a load balancer with HAProxy or Nginx. They don't come with enough ram out-of-the-box, but there is an option to add more. Linode VPS's are persistent. I use Linode a lot for my own personal projects, as well as other sites for work, and I'm not worried about data loss. I'll see if I can get them to approve funding for more MySQL servers. –  yrosen Jul 22 '11 at 17:06
add comment

This setup is very similar to our webserver clusters we use. We use Nginx upstreams on our "balance" server to pass requests to our web servers. I don't see why your above config wouldn't work, and it will allow you to expand horizontally as needed.

Do you currently have any backup/storage in place, or are you using RAID for redundancy? You may consider a NAS for snapshots/backups. Just a thought.

share|improve this answer
    
So far the concensus seems to be that I should roll my own balancers instead of Linode's LoadBalancer...I'm looking at this, might give it a test. I'll be setting up a local storage server to perform backups (probably rsync). As far as the live stuff, the host (so far, Linode) would handle redundancy/storage. –  yrosen Jul 22 '11 at 17:13
    
Rsync is great with BackupPC. It will allow for live restoration of a file (if a client edits/deletes something you can roll back to a previous date) and the GUI is very user-friendly. I don't have experience using Linode, but certainly Nginx and HAproxy are great for your senario, I can tell you from experience. We have serveral thousand clients, individually receiving thousands of hits a day and no issue (LB and webserver VMs are centos LAMP boxes/Nginx/HAproxy) –  Chase Jul 22 '11 at 17:20
add comment

Make sure you realize that building your own haproxy on 2 linode clusters will fail when you decide to take the keepalived route. Multicasts are NOT supported I'm sure. There is a haproxy fix to use unicasts, which are rumoured to work on those networks.

But realize that providers like rackspace/linode know that this kinda messes up their loadbalancer products otherwise.

That being said, you CAN get multicasts to work over a openvpn link (hint)

unicast support for haproxy limits itself to 2 nodes taking control of 1 VIP address.

share|improve this answer
add comment

I have Tow recommandations above all what has been said.

  • Make sure to use two LoadBalancers at least, using one LoadBalancer is a single point of failure
  • Use a clustered file system (gfs/OCFS2) to be actually sure that you have a single copy/version of code running on your cluster, otherwise, you will need to put your website into maintainance mode everytime you will need to push some new code or make some updates until changes are applied to all your cluster nodes (and yet you may forget to update one of them)
share|improve this answer
add comment

You're making me think deeper about this, I have several Linode machines (and Rackspace too) under my control.

My personal site runs like your plan:

  • W3TC plugin (with a nice CDN integration, I see where that Idea comes from)
  • PHP/fpm
  • NGINX (1.2)
  • MariaDB 5.2

I have more services on it too unrelated, tracd (edgewall) behind nginx as upstream and also a couchdb dev instance.

All that on a 768Mb linode (the second lowest). It handles up to 50 connections per second with memcached and disk cache on (so no APC or CDN). The cache plugin makes it go from 5/s to 50/s (biggest chunk being MariaDB).

As far as HAPROXY I have one of those running on a Linode as well. BTW, In find rackspace to be inferior to Linode since you cannot choose your geographic location from within the same account (you can have 2 accounts in us en outside). Also their interface doesn't look and feels as slick as Linode.

As far as scaling of MariaDB goes, I don't find that easy, running 2 masters is a big headache keeping them in sync. The classic route I'd take is setup a master server and several slaves (per node perhaps). Then in haproxy you cluster them together in TCP mode, but between your DB and HAPROXY I would put mysql-proxy in between to make sure write's are being sent to the correct master node, you can program that in LUA. Added benefit is that the creation of connections to mysql is greatly reduced. One big performance hog gone, every mysql connection takes up atleast 5Mb from what I experienced here.

That still gives you a single point of failure, being the mysql master server.

For that you can do the master-master approach with the second master not used anywhere directly unless the 1st goes down. I find Mysql DB's the biggest limiting factor in setting up large reduntant clusters. couchDB is so much easier in that aspect but it's a NOSQL solution...

The biggest point I would do different is , I would drop the Apache thing, you don't really need it since PHP/fpm + nginx is there. So I would not use nginx as a cache, but as the frontend server. Apache will eat up your Linode for breakfast compaired to what nginx can deliver on Linode...

Hope this contributed to your plan in the make. As always, make up your own mind and test this stuff first. Be dynamic in your solutions, drop something that doesn't work before you start using it in production, it's so much more work setting things straight later on.

share|improve this answer
    
This would probably make a better edit to your original answer, rather than a separate answer. –  jscott May 9 '12 at 12:21
    
I'll take it into account next time, yet I don't see the issue with it personally being separated. Also, I don't seem to be able to edit answers older than 5 minutes when I tried... –  Glenn Plas May 9 '12 at 13:13
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.