So, few months ago my boss throws at me "what happens we needed to scale to X clients, in a reasonably short period of time." sort of scenarios, somewhat of an engineering challenge if anything else, but of course more clients means more income.
So, right now we have a setup similar to below. Redundant Load Balancers, serving data from two webservers (sessions shared via memcache), and two backend fully redundant File Servers. Each file server has the whole shabang, RAID6 Array across the drives, hot-space drive, dual raid controllers, dual nics, multipathing yada yada yada.
The design is one that I feel is solid, High Availability through no single-point-of failure. High Performance by splitting the load to multiple web servers, high scalability in the sense that we should be able to just keeping adding more and more machines horizontally to scale to more and more clients. The primary bottleneck being how much storage each file server could hold, and how much space is allotted to each client. That part of this would scale out faster then the rest of the system. FileServer/Client "Pools" route was chosen because it scales horizontally and is cheaper then saying "okay we need to buy an even BIGGER SAN."
So, this is all very easy, more and more online storage, means more and more NFS mounts. And that's where my second-guessing comes into play. I want to make sure I've ironed out all the potential issues in this design before putting anything in place. As long as each piece of this puzzle is properly monitored, I think its controllable, but I wanted to get some other opinions first, maybe from people who've been down this road before.
I know of a few key issues that will have to be watched for.
- Hotspots on the file servers, or particular machines working harder then the rest of the pool.
- Bandwidth & Backend Switching. being there will be a lot of talking between webservers to NFS devices, A high quality switch will have to be in place with a high switch fabric capacity.
- NFS Mounts on Webservers, Will there be any issues with having each webserver have 2... 5... 10... 100 NFS Mounts open at any given time? Is there any way to make this easier or more friendly? Maybe a NFS Proxy of some kind? (that'd create an internal bottle neck, which makes me frown). I thought about a Distributed file system, and have the webservers just mount that, However it seems that Non-Proprietary, POSIX Compliant, Expandable without downtime, Internally Redundant file systems are either too immature for production work, are outrageously expensive, or are really good at hiding from Google.
Let me know what you guys think, and if you see any suggestions & optimizations, it would be greatly appreciated.
((Due it it being an open-ended question where there is no one specific "right answer," question is a community Wiki))