We are setting up a number of Virtual Machines (around 250) for a customer who is basically using an application written in LAMP + Java with a bit of customizations in each of these 250 instances. Essentially each app can be considered as a web app, which can scale to around 200 -500 GB of data ( mostly files), and approximately 1-1.5 GB in size of MySQL database.
We have a standard 42 U rack at our disposal, and we are considering an architecture like this.
1) Run 250 Vms on around 14 2x Quad/6 core 1 U Servers with 32 GB RAM 250 GB SAS 15 K RPM x 2 (RAID 1). Basic App fits in comfortably within 2 GB inclusive of operating system
2) Have a dedicated set of 1 U MySQL database servers with 300 GB SAS x 3 (RAID 5 H/w) enabled. Add more as we need to scale up
3) Have a bunch of JBODs ( around 6) with 30 TB capacity for storage, with a 1:1 failover. That is each 30 TB JBOD has a mirrored 30 TB JBOD,. These would be used to store files that are primarily consisting of PDF, word, excel, JPG and some movie files.
Each app would be having around 2-10 concurrent users through the day who would be writing into the database as well as uploading files. The app is a business app where day-to-day transactions of each department is stored digitally. There could be a few hundred page views or attempts to download some of the docs that are uploaded for each app.
Roughly this would amount to on the raw NAS servers over a 10 hour business hour period, somewhere close to 250,000 reads and writes. Similarly loads are expected on the MySQL servers
We do not want to invest in storage boxes from EMC or NetAPP or any of the large storage vendors, because of lack of funds. We are in fact considering running RAID 0 based 30 TB arrays, with each array being mirrored to another array. So in case of any failure the redundant box takes over, giving us ample time to switchover. We have some spare 3 U chassis, with 15 hot-swap drive bays. We are considering throwing Dual Xeon processors, with 16 GB ECC RAM on each and also thinking of a Software RAID instead of H/W RAID on each box. Idea is that with powerful CPUs, software RAIDs should work better than H/W RAIDs.
We've received some criticism primarily from some vendors who want us to buy their propreitary storage boxes. What kind of performance issues are we likely to face. I have a friend who is a an ex-sys admin with Amazon who said that this kind of architecture is very similar to what an Amazon or google uses, and has pointed that since we are not actually setting up a public facing web site which can have potentially millions of hits, this architecture is good enough. He also told me that Software RAID on UNIX systems performs as well as H/w RAID. This is a reason in most servers at Google or other places are in fact very inexpensive computers.
I would like a second opinion on the same.....