We are setting up a number of Virtual Machines (around 250) for a customer who is basically using an application written in LAMP + Java with a bit of customizations in each of these 250 instances. Essentially each app can be considered as a web app, which can scale to around 200 -500 GB of data ( mostly files), and approximately 1-1.5 GB in size of MySQL database.

We have a standard 42 U rack at our disposal, and we are considering an architecture like this.

1) Run 250 Vms on around 14 2x Quad/6 core 1 U Servers with 32 GB RAM 250 GB SAS 15 K RPM x 2 (RAID 1). Basic App fits in comfortably within 2 GB inclusive of operating system

2) Have a dedicated set of 1 U MySQL database servers with 300 GB SAS x 3 (RAID 5 H/w) enabled. Add more as we need to scale up

3) Have a bunch of JBODs ( around 6) with 30 TB capacity for storage, with a 1:1 failover. That is each 30 TB JBOD has a mirrored 30 TB JBOD,. These would be used to store files that are primarily consisting of PDF, word, excel, JPG and some movie files.

Each app would be having around 2-10 concurrent users through the day who would be writing into the database as well as uploading files. The app is a business app where day-to-day transactions of each department is stored digitally. There could be a few hundred page views or attempts to download some of the docs that are uploaded for each app.

Roughly this would amount to on the raw NAS servers over a 10 hour business hour period, somewhere close to 250,000 reads and writes. Similarly loads are expected on the MySQL servers

We do not want to invest in storage boxes from EMC or NetAPP or any of the large storage vendors, because of lack of funds. We are in fact considering running RAID 0 based 30 TB arrays, with each array being mirrored to another array. So in case of any failure the redundant box takes over, giving us ample time to switchover. We have some spare 3 U chassis, with 15 hot-swap drive bays. We are considering throwing Dual Xeon processors, with 16 GB ECC RAM on each and also thinking of a Software RAID instead of H/W RAID on each box. Idea is that with powerful CPUs, software RAIDs should work better than H/W RAIDs.

We've received some criticism primarily from some vendors who want us to buy their propreitary storage boxes. What kind of performance issues are we likely to face. I have a friend who is a an ex-sys admin with Amazon who said that this kind of architecture is very similar to what an Amazon or google uses, and has pointed that since we are not actually setting up a public facing web site which can have potentially millions of hits, this architecture is good enough. He also told me that Software RAID on UNIX systems performs as well as H/w RAID. This is a reason in most servers at Google or other places are in fact very inexpensive computers.

I would like a second opinion on the same.....


We are in fact considering running RAID 0 based 30 TB arrays, with each array being mirrored to another array. So in case of any failure the redundant box takes over, giving us ample time to switchover.

Unless I'm missing something here this is a dangerous setup. A single disk failure in the second array will lose your entire data set. This is a very likely scenario when you're talking about arrays with at least 10 disks each.

Amazon and Google have their own dedicated storage technologies that's specifically written to work well with their hardware loadouts (lots of cheap disparate units). Specifically, their software detects failures in a storage block and continuously ensures that each item is stored in at least x additional locations. When a storage device fails, all its contents are immediately having a new duplicate added to some other storage pool. Unless you're rolling similar custom software for your storage tier, you can't use them as a basis for comparison.

Regarding vendors - It's true that you can potentially do without an array from one of the big enterprise players here - Netapp/EMC or similar. Their storage is designed for things like running lots of concurrent VMs directly from them. However you're talking about dumb NAS serving up flat files... much simpler use case and the overheads and randomness of your IO is way down. You're still going to want to consider RAID 6 at the very least, though.

What's your backup strategy?


For 3) I would use HW-Raid 5. That is a good comprimise between speed and availability. You don`t want to re-mirror 30 TB. During that re-mirror you might loose an additional disk.

I would also recommend using HW-Raid, because of its abilities to warn about disk failure und its ability to do an automatic rebuild.

It is also much easier to setup and maintain. With HW-Raid I am talking about real raid-controllers that present raids as disks to the OS.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.