Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote -10 down vote favorite

I'm building a iscsi storage system for hosting about ~500 Vmware vm running concurrently. And I have a disk array with 15 disks, I only need moderate write performance but at least raid1. so, that leaves me with RAID1 / RAID10 , I have couple choices:

1) 3x LUN 4disk RAID10 + 3 hot-swap

2) 1x LUN 14disk RAID10 + 1 hot-swap

3) 7x LUN 2disk RAID1 + 1 host-swap

Which way is better? Is there a real problem running 500 vms on single LUN? and would it be better to resort to 7 LUNs so each VM is better isolated with each other?

Edit:

To save you some mocking, right now I have a LUN with 300 vms running on it, and they are doing 3000-4000 IOPS on this 14disk radi10(in aggregate), and I think that's not too bad. And I want to know if anyone is doing similar things.

I think multiple datastore on single disk array will not work well because I also want SIOC to balance the IO from different VM and I don't think it will work well with some input not under its control.

More Edit: 250k for 500 vm, so you are targeting at 500$ per VM, I can just say wow. It would be cheaper for me to simply buy a single machines with 2 disk for each VM and that would still fit under that budget (they will use more power though). Like I said, I am only looking for moderate performance , it's not like I want to host 500 database vm and expect them to all run at full speed

Final Edit:

I ends up using about 200VMs per LUN , each VM has 2 disk , so 400 virtual disk per LUN (14D raid10) and that's working perfectly fine. a little tweak here and there, if you can afford a UPS you can use WB cache to further sky rock your performance.

And at this level, nothing is sequential, all random access, so queue depth is never a issue anyway.

share|improve this question
Why are you not considering RAID 5 or 6 with hot spares? – nearora Jun 20 '12 at 3:41
1  
Imagine how slow it'd be to boot ~500 machines from a single storage server? What if you need to shut down that storage server for maintenance reasons? you need to really think about your architecture a lot more and talk to some experts. – Matt Jun 20 '12 at 4:36
1  
RAID 5 and 6 are terrible for VM storage because of their write penalty. Not to mention the much longer rebuild time compared to RAID 10. – MDMarra Jun 20 '12 at 10:39
exactly! and I am not concerned with space, so there's no need to for more raid penalty – Yucong Sun Jun 20 '12 at 18:36
4  
Seriously, what you're saying is "I want to run a massive amount of VMs. I want it to be as cheap as possible. How should I do it? The answer is "you shouldn't." This type of thing costs a shitload of money. At my last job, I spent about 80k to virtualize 40 low-spec servers. I had 4 physical hosts, vSphere licensing, and a storage array as my cost. I did it a lot cheaper than most do, too. For 500 average load VMs, I'd be looking to invest at least $250k, with the bulk of that going into storage. If you try and cheap out, you're going to have a bad time. – MDMarra Jun 20 '12 at 19:33
show 3 more comments

4 Answers

up vote 16 down vote

I think your confusion (and anger) may be due to an error in your terminology.

First let's answer your question - 1 x 14-disk RAID 10 array with a single hotswap disk will be the best configuration.

Now onto the confusion. You're not presenting all that space as a single LUN are you? that would be silly, because of SCSI locks and queue-depths as others have said.

What you've probably done is create one array and presented it to the host/s as multiple smaller LUNs - that's exactly how everyone else does it and it works fine. I personally create lots of 500GB LUNs to stop my administrators from over filling these datastores with VMs but any size is fine.

Now obviously you could put up to 2048 powered-on VMs in a single datastore but that would be crazy, nobody would do that.

So anyway, back to my recommendation - one 14 disk R10 + single hotswap, then carve that up into a bunch of LUNs/datastores of whatever size you like and away you go - that'll work fine.

That said unless your VMs generate very low IO requests you may find queue and latency issues with such a small array, I know you're budget-constrained and doing all you can but I'd imagine you have at best a 14 x 200 IOPS IO budget (2,800) so that's less than 5 IOPs/VM - as someone said a non-planned/scheduled bootstorm could be problematic.

Hope this helps, oh and can I ask that you try to understand that your comments appear needlessly aggressive and that one of the rules of the site is to be pleasant where possible to other users, especially to those who have gone out of their way to try to help you.

share|improve this answer
1  
I just want to add this for future readers: SCSI locks are obsolete when you have a storage array capable of VAAI for ESXi and ODX for Hyper-V. You should still do as everyone in this thread says - think about queue depth and latency. – pauska Jun 20 '12 at 20:17
Yes, but as I state previously, my current configuration is 300vm on single LUN on 14 raid10 disks. It maybe silly but I'm getting about 3000-4000 iops (with 10G ram cache) on it. I know about the queuing issue and it will disrupt the sequential IO, but I'm more lean towards provision for random IO anyway. And, since this is a hosting platform, we are expecting to oversubscribe IO (not surprisingly right?) , and in reality I think it is a good trade-off for money/speed – Yucong Sun Jun 20 '12 at 20:51
@YucongSun - your 3-4k IOPS is reliant on caching, without caching (unless you're using SSDs or Hybrid disks - which I imagine you'd have mentioned) the best you're getting is about 2k random. It's generally accepted that you shouldn't plan on caching to get you out of a hole unless you thoroughly understand your RW exposure. That said given you're in the business of shoehorning as many customers into as cheap a box as you can I do understand your mindset on this, though I'd still urge you to carve the array into smaller datastores ok. – Chopper3 Jun 20 '12 at 21:01
2  
@YucongSun By the way would you let us all know what hosted service you're involved in, that would be kind. – Chopper3 Jun 20 '12 at 21:02
Please see my comments below about why I think multiple datastore on single array would not work, mostly because SIOC won't work in such enviroment and without SIOC there's no way to balance IO among vms – Yucong Sun Jun 20 '12 at 21:04
show 3 more comments
up vote 4 down vote

Two words: queue depth

Don't put 500 VMs on one LUN. 500 is a lot to share a single SCSI queue.

It sounds like you might be a bit over your head. Please consider going to some VMWare training classes or hiring a consultant.

share|improve this answer
This is not helpful, I know about the queue depth and scsi locking issue, the question is the trade off. If I can have dedicated 4 disk raid10 for each vm , I would not be here to ask questions. Most people just simply answer, oh that is too much, without explaining what will happen, and I seriously doubt they have any real world experiences on such matter. The fact that I need a cheap solution for hosting 500 vms, is a fact, and I want to balance the lun size and money. – Yucong Sun Jun 20 '12 at 18:37
4  
The iSCSI queue is per-target. If you have one massive LUN/volume, then you'll only have one target, thus a single iSCSI queue. If you break that down into multiple targets, then you have more queues. Not sure why you don't think that this is not helpful, but if any of those VMs are doing any kind of meaningful I/I, having 500 on a single target is suicide. Depending on I/O, somewhere between 10-15 VMs per volume is normally the sweet spot, but you should profile and test your environment. 500 is just a ridiculous number. – MDMarra Jun 20 '12 at 19:17
Yes I know too many vm compete iscsi queue will disrupt sequential IO since they will not be combined. But as such a big datastore, none-of the IO will be combinable when it passed down to the disk anyway. With 64k block size on disk, it's pretty much grauntee you will never have SEQUENTIAL io on disk, so that's why I simply don't think provision for sequential IO will buy much, it's only random IO that matters. – Yucong Sun Jun 20 '12 at 21:06
Ok. I'm done trying to help you. You seem to know it all already – MDMarra Jun 20 '12 at 23:48
up vote 2 down vote

As @MDMarra said, 500VMs are a lot on 1 single LUN. If you are consuming iSCSI on the ESX{,i} end, you are going to be using VMFS. You will consequently be affected by SCSI locks for operations that have to be exclusive when using VMFS. The following article has good explanation and a way to calculate your LUN sizing and VMs per LUN:

http://www.yellow-bricks.com/2009/07/07/max-amount-of-vms-per-vmfs-volume/

I'll edit this answer after you answer my RAID related comment.

share|improve this answer
I've saw that article, and it's useless, I found it hilarious to say that we should carve out 5 luns on a single 20disk raid10 and let them compete IO. I am having a hard time believe that would works better. – Yucong Sun Jun 20 '12 at 18:45
@YucongSun then I'm sorry man, you need to understand how SCSI works if you want any kind of help from us with this. Queue depths will KILL you with a single LUN. – pauska Jun 20 '12 at 20:36
2  
You should test it and get back to us. At this point, you're refusing to believe the accurate answers you've gotten, and I don't see that there's anything left to say here. – Basil Jun 20 '12 at 20:39
The problem with multiple LUN on same array is it doesn't plays with SIOC. with SIOC it will throttle IO base on latency. And If you have multiple SIOC datastore on single disk array, I would think it will result in burst/oscillation a lot. And, yet SIOC is very important to me for distributing IO to ensure at least some IOPS for each vm. – Yucong Sun Jun 20 '12 at 20:54
up vote -5 down vote

If you would like the VM's to be able to fail between cluster nodes (assuming they are in a cluster) it would be a good idea to split the VM's up as much as possible between the LUNs. The reason behind this is if one of your servers fails and all of your VM's are on one LUN then they would all have to fail over to another hypervisor together (I do beleive only one server can have control over an ISCSI node at the same time). Even if you aren't in a clustered environment would likely be a good idea to segregate as much as possible to reduce growing pains later on.

share|improve this answer
6  
This is wrong. You can have multiple VMWare vSphere hosts accessing and running VMs from the same volume. – MDMarra Jun 20 '12 at 11:54
Correct MDMarra, 32 hosts can access a single LUN concurrently. – Chopper3 Jun 20 '12 at 19:16
This was the way early versions of Hyper-V failover worked, but even Hyper-V doesn't have that issue any more – Mark Henderson♦ Jun 20 '12 at 21:44

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.