2

I've been tasked with figuring out how to do a large scale deployment for a high availability, high traffic, dynamic, flash based web application. There is a good possibility that this application can grow to 2 million users or possibly a lot more.

Of course when the time comes to actually do it, I will likely bring in an expert to help me. For now, this is all theoretical, and I just wanted to get a good idea about how to do it.

Here is my current thinking:

  • Get a 1 or 2 gigabit internet connection.
  • 2x HA Proxies / load balancers.
  • 12x physical servers (2x quad cores, 32GB of ram) to run 25 Virtual Machine Apache web servers - 300 VMs total.
  • 2x memcached Servers (2x quad cores, 32GB of ram)
  • 2x Master Mysql DB servers, 3x read only slave mysql BD servers
  • 2x 6TB SAN RAIDs
  • All 10GbE or fiber connected...

My job is mostly to figure out the hardware, and how to deploy and maintain it.

As for maintaining this beast, I'm looking into Slack, Rightscale, Landscape, VMware.

I am also looking into just deploying it on Amazon EC2, but I would prefer to rack it myself as it looks like it would be much cheaper in the long run. I may start out with this rack and add EC2 instances if more is needed.

What do you think, is this overkill? Not enough? Have I got it figured out, and is there anything I'm missing?

Thanks to anyone who reads and responds!

Improve this question
3
  • 6
    1. Place app on web. 2. Attract 2M users. 3. ???? 4. PROFIT!
    – jscott
    Oct 1, 2010 at 15:57
  • 1
    1. Come up with a decent idea. 2. Make it work, even just very small scale. 3. If it's successful worry about scaling it. I think you've got the horse behind the carriage.
    – Chris S
    Oct 1, 2010 at 16:10
  • 1
    Don't forget Puppet for configuration management, because deploying and managing that many machines will be tough.
    – Martijn Heemels
    Oct 1, 2010 at 16:46

7 Answers 7

Reset to default
6

Develop it on EC2. If/when you want to move it in-house, set up your own Eucalyptus cluster and move it to there, which will be easy since Eucalyptus is an open source EC2 infrastructure clone.

Improve this answer
2
  • Thanks pjz. I should have worded it differently in my original question. I will probably put it on Amazon initially, then move it to my hardware as soon as I could. I did not know about Eucalyptus, and will definitely have a good look at it. Thanks!
    – lysdexic
    Oct 1, 2010 at 17:04
  • 1
    You should also keep an eye on the OpenStack project.
    – DerekB
    Oct 1, 2010 at 20:39
3

From what I've read on High Scalability, your best strategy is simply to work on the current bottlenecks and start to plan for the next bottlenecks. Unless you're using exactly the same software and hardware as someone has already used, and unless the usage patterns are identical as you scale, you can't plan your entire growth now. You need to plan for the current phase and the next one, not the 2-million-user mark.

Improve this answer
2
  • Good advice. I'd start with some base smaller configuration and make it scalable so that it can be adjusted and increased in size as time goes on. Plan for the 5,000 user mark first.
    – djangofan
    Oct 1, 2010 at 16:08
  • Thanks for the comments. However, I supposed I should not have used the word "grow" in my question. This is necessarily not an application for public consumption, and the expectation is that most of those users are already accounted for. So I need to actually be prepared for this many users, without having the metrics and data in place to see it grow to that scale. The application will start out at that scale...
    – lysdexic
    Oct 1, 2010 at 19:46
3

I think you've seriously overestimated your requirements. As a point of reference, a 2 ipvs primary/primary in front of a pair of dual quadcore xeons running nginx+tornado processes 286 million ~2400 byte iframe adblocks + full adstream logging for the 7 elements within the iframe. Utilization of each is <40% (allowing for a single node to fail with no 'overload')

The second issue you say is dynamic 'flash-based' which means that the majority of your bandwidth is going to be serving the initial flash application - a candidate for a CDN or cache.

Your dynamic flash application is likely to have small messages sent back and forth using gevent or some similar technology. Those messages are likely to be small. Your application intelligence will likely have to parse, log and respond to those, but, 2 million users a day, 80 interactions each is 160 million interactions. At peak times, you would probably have to handle 3600tps. If you need 300 servers to handle that, you need to rewrite your code.

Virtualizing your webservers as mentioned above is probably a waste of performance. Even the lightest weight virtualization won't run as quickly as bare iron.

However, without knowing your architecture or system design, any opinions given are guesswork given the loosely defined requirements.

Improve this answer
1
  • Excellent stuff here, this does help put things into perspective. This is all guesswork at this point in time, so any input is very much appreciated.
    – lysdexic
    Oct 5, 2010 at 17:47
2

Why would you put your web servers in a VM enviroment? It just causes overhead that you dont need, they should all serve data from the same storage and have identical configuration.

Improve this answer
2
  • Well, this is part of what I've been trying to figure out. I was told that to support 2 million users (hundreds of thousands of concurrent users) that I would need around 300 to 400 web servers. Of course they would all have the same storage and identical config. The web servers would only be there to process the incoming requests, and serve up the pages. Are you suggesting that I don't need that many web servers, or that they should all be physical? Thanks
    – lysdexic
    Oct 1, 2010 at 17:10
  • I'm hinting that you would get more performance per server if you deploy them as physical servers, as you're going to need really fast backends for this kind of traffic. Virtualization just gives you a extra, not needed overhead in this situation.
    – pauska
    Oct 1, 2010 at 19:01
1

I can reply for the sizing of the haproxy servers. Use a recent Linux kernel (>= 2.6.27) to benefit from the TCP splicing feature that will save you a lot of CPU on high bandwidth. Use the highest frequency you can find for the CPU. No need for many cores, better find a 3.6 GHz dual-core than a 2 GHz 8-core. For the RAM, count 1GB per 20k concurrent connections. Use two VIPs for the LBs (active/active) managed by the keepalived daemon and announce them both in your DNS. That way your two LBs will work at the same time and if either fails, the other one takes over its IP. One very large site I know runs more than 300k concurrent connections on only two properly tuned LBs with quite some margin.

Do not underestimate the network cards. The lower latency of some NICs such as Myricom's Myri10GE NICs compared to other solutions (1 or 10 GbE) can bring nice savings on the packet processing cost and the number of concurrent connections on the servers.

Concerning the apache servers, haproxy can concentrate incoming connections into fewer outgoing, thus reducing the number of servers and protecting them from traffic surges. However, that's usable for short connections. Don't rely on that for long-polling requests, chat or large file uploads/downloads.

Improve this answer
1
  • Awesome, thank you Willy, this is going into my documentation. I don't have much experience with haproxy servers, I plan to study it extensively. This helps a lot!
    – lysdexic
    Oct 1, 2010 at 19:34
0

Take a look at systemimager

http://wiki.systemimager.org/index.php/Main_Page

Improve this answer
0
0

As far as reading on this topic goes, I highly recommend Scalable Internet Architectures by Theo Schlossnagle. It covers HA deployments and horizontal scalability in rather exhaustive detail.

http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X

Also, High Performance Web Sites and Cloud Application Architectures might be worth taking a look at, however the latter might be slightly outdated.

Improve this answer
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .