Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm looking into designing an architecture for a moodle based education site, it will serve several thousand users at first but needs to be able to grow to support hundreds of thousands to millions of users across several countries.

I was thinking of a load balancer to distribute requests to several web servers. The web servers can be split by some serving static and some serving dynamic content. Then it is to write to a mysql master node and read from slave nodes.

What kind of load balancer will work well with moodle, should I get a hardware load balancer solution from one of the vendors, or build one myself with open source solution like LVS or reverse proxy?

I was planning to use apache server to serve the web pages at first then as loads become higher, split into lighttpd webserver for static content and apache application server for dynamic content. Stuff like gzip compressing, squid cache, memcache will also be deployed if required.

For the web server hardware, should I use one-u single socket server or a blade solution? Which one will end up being cheaper to run and expand? Supermicro have an interesting product with twin servers in 1u chassis and 4 servers in 2U chassis with infiniband. Has anyone here tried these server before?

For the storage, should I use a SAN or storage server like Sun unified storage 7000 will be sufficient. For a mysql cluster setup should I have two different storage systems, on to use for master node writing access and another for slave to read? Or should all the nodes have separate storage?

Since this website will likely be more heavy on read operations, what consideration should be made for the mysql cluster and storage setup?

For the management part I am planning to use dsh, ganglia, nagios, splunk, kickstart.

For backup I am planning for an LTO tape autoloader. This site will be primarily used for the Asia region, so there will be several hours of low traffic at night. What is the best way to back up up a mysql cluster? Can I temporarily disable write and take out the master to perform the backup?

Please advise if you have experience with setting up this kind of scalable web site, most of my experience has been in working with large unix boxes, or smaller standalone unix/linux boxes. So this kind of scale out implementation is a first time for me.

Thanks

Robert.

share|improve this question

3 Answers 3

Robert, you're clearly a smart guy, but respectfully, get a consultant with prior domain knowledge, or begin building something small now and see where it takes you. There is no way to answer your post; it has too many abstract concepts and no hard numbers.

A few thoughts:

will serve several thousand users at first ... grow to support hundreds of thousands to millions of users

Prove that you need that level of scale first. Don't build a scale-out architecture in anticipation of users that never show up. Sorry if I sound harsh, but 99% of all websites don't grow to the large end of the scale. See Stack Overflow / Server Fault; they're serving a million users monthly from a handful of fairly conventional servers.

should I get a hardware load balancer solution from one of the vendors, or build one myself with open source solution

Depends on your skills and your situation regarding time vs money. Once built, the open source and commercial offerings work pretty much exactly the same. Commercial solutions tend to have better statistics and nicer management interfaces out of the box.

For the web server hardware, should I use one-u single socket server or a blade solution?

Ask your server vendor for prices. Ask your datacenter about power density, i.e. their preferred balance between size and power consumption -- often you'll be power limited, so a dense solution like blades may not win you anything.

For the storage, should I use a SAN or storage server like Sun unified storage 7000 will be sufficient.

Get SAN when you have a proven need for SAN; then you will also better understand what needs your SAN should solve for you.

Since this website will likely be more heavy on read operations, what consideration should be made for the mysql cluster and storage setup?

Create a really good caching solution. Either full page caching like Squid (Varnish), or application data caching like Memcached, or a combination of both. Consider cache invalidation, could you need to quickly purge content from your caches to avoid it being served again?

What is the best way to back up up a mysql cluster?

Opinions vary, but one common approach is to have a dedicated slave MySQL just for backups, and use something like InnoBackup or Maatkit for a self-scripted backup solution.

Edit: If you're really going to build this from scratch now, then please take a good look at cloud computing before committing. Cloud computing isn't just about scalability, even if scalability is a great strenght. Certain services that come as part of the package can really help in making day to day operations easier. Some examples:

  • Live snapshots of Amazon EBS volumes make for easy database backups.
  • Amazon has load balancing as a set and forget service (of course more feature limited than good self-hosted load balancer, but easy to get started with).
  • Rightscale has extensive server monitoring built into their images, which makes for easy capacity planning / application introspection.
share|improve this answer
    
Jesper, you are right about building something for user that never show up, that is my councern as well. what i am trying to do is to get some idea on way to the implement a scale-out architecture in anticipation of the best growth projection in the business plan. while at the sametime minimizing waste in hardware cost during the start of the project. thanks for all you tips by the way. –  Robert Sep 3 '09 at 7:09

While I don't know much about the specifics of Moodle, I can offer some tips for general scalability.

Blades and SANs are often mis-sold by vendors. I suspect that a cluster of commodity 1U servers would probably be the best for your needs. There's a number of datacentres that won't take blade systems because the power draw is very high, and the cooling requirements are quite needy too!

I'm a big fan of Gluster for distributed/replicated storage, you might find it interesting to investigate as an alternative to a SAN solution from a large vendor.

A whole stack of HP DL360s would do just as well (or cheaper commodity servers (I highly recommend DNUK)). I seriously doubt that you'd need Infiniband interconnections between your servers (the infrastructure is expensive, and largely unnecessary for web serving purposes, if you were doing HPC modelling of genome expression, my answer might be different!)

With regard to network infrastructure (if you've got to consider this too..), I recommend Cisco routers, with either Cisco Catalyst switches, or HP Procurves (pretty evenly matched, IMO, and cheaper)

As far as load-balancing is concerned, a dedicated linux server running LVS will easily handle traffic to multiple cluster nodes. If you had the money ($30k+), then a citrix netscaler might be the right caching/acceleration/load-balancing platform, but bear in mind that you'd need 2 (ideally 3) of them for redundancy.

You should probably try to include memcache from the outset, it's easy to add scalability, and vastly improves caching performance, especially when reading from a MySQL database cluster. There's other things you can do to tune your MySQL performance too, such as using InnoDB over MyISAM.

I suspect you would be better off with a reverse proxy cache, such as Varnish, as opposed to Squid, which works better as a client-side cache.
You could quite easily have a couple of dedicated Varnish cache nodes, or run Varnish on the same server as the Apache / lighttpd servers.

Try to avoid getting into a state where you get vendor lock-in, as this can prove very expensive when it comes to licensing issues. It's very possible to build a scalable site using free/open-source software, entirely. Of course, software load balancers aren't going to be as fast as hardware ones with dedicated ASICs, but with a good network infrastructure, it can come pretty close.

For the management part I am planning to use dsh, ganglia, nagios, splunk, kickstart.

Just need to add puppet to that list, and you're onto a winner. Watch out for splunk's expensive licensing (when you get into processing 10s of GB of logs per day, it might bite you).

Munin is a great free monitoring tool, and has advantages over apps like Zabbix, because it can automatically configure graphs from the plugin script (so you don't have to keep track in advance of what you're monitoring).

share|improve this answer
    
i was thinking of gigabit network for the server communication, 100mb for management network. and if mysql cluster grow larger implement 10gigabit network for the cluster communitcation. from my reseach on mysql performance seen that there are many that opinion that the current release does not scale well on server with large number of core. by next year mainstream server will be having dual 8 core processor and up to 200+gb of max memory. there is also design of 8 socket machice for nehalem, which will push the thread count to 128. will mysql benefit from these large number of core? –  Robert Sep 3 '09 at 8:05
    
WRT MySQL, it's very much a scale-out vs scale-up issue. I've usually seen architectures where you have multiple fairly low end slaves, and a powerful master. Might be best to look at some benchmarks each way (I'm pretty sure Sun published some on their tech blog). –  Tom O'Connor Sep 3 '09 at 8:56

While I have never administered a Moodle system that could be considered large (at most one having a few thousand active users) and I'm almost certain you have more Linux experience than me, I can offer some observations.

A Moodle installation with millions of users would be an order of magnitude larger than any others that I have heard of. Even the Open University, with students spread across the UK and the world, is only anticipating 200,000 users. Large US universities tend to have only tens of thousands of users. For a good idea of size have a look at http://docs.moodle.org/en/Large_installations Will you really get millions of people using the system? Are they all going to be appearing at once or gradually enrolling over years? You don't need a system capable of handling millions if you're only going to get 10,000 in the first year. In addition, a lot of institutions have theoretical numbers of students who will use Moodle, but only a small percentage of those actually use the system. In short, start small and scale up.

A Linux distro will make life easiest when administering Moodle. The online community help that's available isn't orientated to Windows at all!

Consider a recruiting a local Moodle Partner (sorry, I'm new and so can't post this second link) if you're serious about this. They can offer advice based upon personal experience in setting-up and administering Moodle systems.

Moodle is generally very easy on system resources. Just watch the databases, as the transaction rate can be very large. You haven't mentioned it, but consider separating db servers from the web server and concentrate resources on clustering the db. With caching (eaccelerator or memcached) web-access is negligible. File storage is also generally non-intensive and a link to a decent raid array, local or on a separate machine, is all that is needed. If you've got a SAN, use it. If not, just stick with simple stuff.

As always, backup, backup, backup!

Good luck!

share|improve this answer
    
the millions users senario may not happen in two or three year, if at all. i will start with a smaller web and database server, then throw in more web server and configure database cluster. the planning for the is the get a flexible setup in place if the website really become super popular. i believe that the currently available server have gotten so powerfull that one 2u web server and one 4u database server will be able to handle a few thousand request per second. the issue i can imagine for this site is the before exam time there is will be usage spike. –  Robert Sep 3 '09 at 6:56
    
one question with cacheing, if there is caching at application level, will cache at database level make improvement to the performance. –  Robert Sep 3 '09 at 7:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.