I'm looking into designing an architecture for a moodle based education site, it will serve several thousand users at first but needs to be able to grow to support hundreds of thousands to millions of users across several countries.
I was thinking of a load balancer to distribute requests to several web servers. The web servers can be split by some serving static and some serving dynamic content. Then it is to write to a mysql master node and read from slave nodes.
What kind of load balancer will work well with moodle, should I get a hardware load balancer solution from one of the vendors, or build one myself with open source solution like LVS or reverse proxy?
I was planning to use apache server to serve the web pages at first then as loads become higher, split into lighttpd webserver for static content and apache application server for dynamic content. Stuff like gzip compressing, squid cache, memcache will also be deployed if required.
For the web server hardware, should I use one-u single socket server or a blade solution? Which one will end up being cheaper to run and expand? Supermicro have an interesting product with twin servers in 1u chassis and 4 servers in 2U chassis with infiniband. Has anyone here tried these server before?
For the storage, should I use a SAN or storage server like Sun unified storage 7000 will be sufficient. For a mysql cluster setup should I have two different storage systems, on to use for master node writing access and another for slave to read? Or should all the nodes have separate storage?
Since this website will likely be more heavy on read operations, what consideration should be made for the mysql cluster and storage setup?
For the management part I am planning to use dsh, ganglia, nagios, splunk, kickstart.
For backup I am planning for an LTO tape autoloader. This site will be primarily used for the Asia region, so there will be several hours of low traffic at night. What is the best way to back up up a mysql cluster? Can I temporarily disable write and take out the master to perform the backup?
Please advise if you have experience with setting up this kind of scalable web site, most of my experience has been in working with large unix boxes, or smaller standalone unix/linux boxes. So this kind of scale out implementation is a first time for me.