Enterprise class storage best practices

Question

One thing that has always perplexed me is storage best practices. Filesystems brag about how they can be petabytes or exabytes in size. Yet, I do not know many sysadmins who are willing to let a single volume grow over several terrabytes. I do know the primary reason behind this is how long it would take to rebuild the array should a drive fail. The more drives in a single LUN, the longer this takes and the greater your risk of losing another drive while the rebuild is taking place.

Then there's usage reasons. Admins will carve out a LUN based on how much space they think needs to be allocated to the project. It seems more practical to me for the LUN to be one large array and to use quotas. I understand this wouldn't satisfy every requirement (iSCSI), but I see a lot of NAS systems (NFS) managed this way. I also understand that the underlying volumes can be grown/shrunk as needed quite easily, but wouldn't it be less "risky" to use quotas rather than manipulating volumes and bringing possible data loss into the equation?

There may be some other reasons I'm missing, so please enlighten me. Can we not expect filesystems to ever be so large? Are we waiting for the hardware to get faster to cut down on rebuild times?

Answer 1

Spindle separation is a good reason not to have one large volume. If you are running Exchange, SQL Server, and other random things (ESXi guests maybe) off of a single NFS device you will certainly want spindle separation. Exchange data and longs should be on separate spindles from eachother as well as SQL databases and logs.

Answer 2

To rebuild a drive doesn't depend on the file system but on the drive itself. If you used 2TB drives, then you'll need a lot of time to rebuild.
Rebuilding is a problem of raid 5. For this now may controllers support raid 6 or much better you could make a raid 10 (if you want the best performance). Exporting a lun is a job for a SAN, exporting a file system is a job of NAS.
Both have their pros and cons (search on google, there're tons of paper about). Making a lot of independent luns (or file systems) has the advantage of snapshot backups.

Answer 3

Personally I wouldn't run true enterprise traffic (Oracle, MSSQL, Production ESX) over iSCSI or NFS at all - I know and trust FC, both for general performance and during failure situations. Oh course this means that I can't just create massive LUNs and subdivide and it does create quite a bit of complexity/work but our system availability and end-user-experience-figures have been better and more consistent than we'd hoped.

As for your actual answer - well your posts seems very NetApp/filer-oriented, hence my last paragraph - but you've laid out the reasons yourself. Disk interfaces have fallen significantly behind disk capacity - meaning that rebuild times can be ridiculous, especially if you want to see decent performance during a rebuild to your existing traffic. Disks die all the time and the idea of affecting multiple platforms performance during a rebuild to simply make life easier for the admin would be a short-lived method indeed. Also purely from a platform security perspective you might want to partition your systems and the 'big LUN' approach may well not work in that scenario (based on hardware of course).

Ultimately Enterprise SANs are inherently business or mission critical, demanding availability and consistency over cost, ease of admin or even ultimate performance.