I'm being tasked with defining SAN policies for an external organization. I'm not a sysadmin, these aren't systems we have any authority over. I'm also no SAN expert. Whoever said work had to make sense?

I've come up with a few bullet points based on the documentation provided by the vendors involved (which the external org that will is currently running the SAN has not bothered to look into). Things like "don't put high I/O test storage on the same slice as Prod data stores" which seem like they should be obvious but clearly weren't.

Any recommendations on general SAN conventions that should be in place to improve performance and reliability?

Specific to our setup (EMC hardware, DB2) these are the key items I have:

  • Ideally each Logical Unit (LUN) of the SAN will be spread across multiple physical devices to allow concurrent I/O thus improving performance.
  • Each LUN should be dedicated to single use (e.g. the DB2 store for a particular application)
  • For DB2 transaction logs should be located on a separate LUN on a physically separate spindle or set of spindles from the table data
  • Data LUNs should be RAID-5 as it provides the best performance though with reduced redundancy
  • Log LUNs should be RAID-10 to provide maximum redundancy
  • If LUNs are set up to use file systems instead of raw partitions (which is recommended) tablespaces should use the NO FILE SYSTEM
  • CACHING clause to improve performance
  • 2
    Just an off-the-wall, mostly irrelevant question: If you aren't a SysAdmin, aren't a SAN expert and don't have authority over the systems... how did you manage to get saddled with this task? =) – Wesley Dec 28 '09 at 19:42
  • I have a reputation for completing assigned tasks. Usually well. Yes, sometimes the bar here is set absurdly low. To be fair, it was assigned to the external group about a year ago and my manager has simply given up on them producing anything ever. I'm a jack-of-all-trades with some ability to write coherent explanations of tech topics, so go me. – user30226 Dec 28 '09 at 20:45

Before writing a policy, it'd be useful to know what you're trying to define a policy for. Is it for optimal performance? Data protection? Something specific in regards to company retention policies? Can we assume it's just a generic performance/reliability document based on your statement?

The reason I ask is that a SAN (like any network equipment) is usually customized to fit a role. It's hardware configuration can greatly affect the recommendations one might make for it. For example, SQL LUNs are generally best when they're made up of a numerous fast drives (spindle dependent) whereas something like user shares or archive data is better suited to larger, slower volumes (you seem to be aware of this). That said, I'd be hard pressed to definitively define a RAID level since different vendors have various views of it. For example, EMC might feel RAID10 is preferred whereas NetApp feels a 24 spindle RAID 6 is ideal.

Generically I'd say:

  • Isolate data LUNs from database/log LUNs
  • RAID levels and spindle counts should be determined by the application and vendor recommendations
  • Put low priority data (like user shares and archive data) on slowest disks
  • Ideally database/log LUNs have several small, fast drives to increase spindle count
  • I/O intensive apps should be split between controllers (if that is an option)
  • If you have a test environment that should be isolated to limit its impact on the production environment (separate controller/volume)

It's going to be difficult to give you more beyond those very generic options because you start getting into vendor, hardware, and application specific recommendations. You also get into security and company policy. You'd probably have better success in defining requirements for a specific application than creating a generic SAN guide.

  • I wish I had more specific directives. The external service org is very black box. What we don't know won't hurt us seems to be their philosophy. So my manager wants us to present them with SAN policies that we want them to follow unilaterally. Our particular interest is in supporting an online transaction processing and reporting system, but there are a plethora of other groups that will be using this SAN for who knows what. I accept that the policy recommendations I provide will likely be ignored, but I'd like them to be good recommendations regardless. Your comments are very helpful! – user30226 Dec 28 '09 at 19:41

Each LUN should be distributed across spindles on the backend of your storage targets, but returning to the frontend adapters, the possible demand (# exchanges * size of exchanges * number of servers) should be balanced. For example, if I don't alter my servers, they'll probably have a queue depth of 254. If my exchanges are 4 frames each (8k), then each of those servers can choke 2k of an FA. I would want to balance my SAN so that the total possible load, and the possible load at certain times (backup traffic hits when daily traffic doesn't) is balanced. Feel free to define QueueDepth limits, and catch those who exceed it. If you son't know how to catch QD violators, I can show you.

I would also try to enforce a policy: if you're not using the SAN, you'll not be zoned, and your port will be offline. Many environments provide SAN to all servers, but they don't immediately (ever?) come online on the SAN. They tend to thrash on/off/on/off/on/off though, and it's those devices that cause storms of updates across fabrics. Let's choke those before they become a problem.

Put devices into a default/chassis VSAN or VF: VSAN0001 on Cisco, or VFAB128 on Brocade. When a user decides where the port needs to be, move it to a VSAN or VF. VSAN0001 or VF128 do not pass ISLs/XISLs -- this reduces the broadcast storm risk.

New devices should have the requestor indicate whether they are single-path, or multipath, and if multi path, whether they are active passive, balanced multi path, or unbalanced multi path so that when they're not balanced, you can see if a config issue has occurred or the multi path tools have misbehaved.

Name everything. Aliases help. Have a naming scheme, so that Oracle14_HBA0 would make you expect an Oracle14_HBA1. That helps when there's a problem: you can decide whether Oracle14_HBA0 is worth getting out of bed right now, or wait until next workday.

Require requestors to request storage in terms of latency (ms) against demand (MB/sec or IOPS). They;ll want to say "Tier1! Tier1! My stuff rocks, I need Tier1" without knowing what that is. Push for an SLA such as "40ms for 200MByte/sec", which is a fairly easy latency at 2GB single-path link. If they don't know, then tell them "40ms @ 200mb/sec", and wait for them to restate. They'll eventually move to 9ms for database intent-log LUNs, but not immediately, and you'll have your crazy-expensive Flash-backed-by-SAS LUNs for only the places that need them.

VMAX rate-limiting is your friend: squelch the bursty demand before it triggers your array into write-pending. See above: "40ms @ 200MB/sec".

Those are some thoughts based on teaching FibreChannel to up to 50 people at a time and seeing the issues they have.

  • I can count the number of times I've got a good answer to what performance does your application/service need on the fingers of one hand. So tend to apply some extremely generous suggested metrics on my part. I'd tend to include IOPs as well as MB/sec though. – Sobrique Apr 17 '14 at 9:05

Others have offered advice already, I will add a few suggestions of my own:

  • Every server must have two physically distinct paths to storage. That means two HBAs per server, going through two completely separate sets of SAN switches on two separate fabrics, to two different back end controllers. Don't scrimp and buy a dual port HBA - that gives bandwidth, but not resilience. (If at all possible, have the fibers using different physical routes though the server room).

  • Multipath everything. Two paths at least, add more if you need better performance.

  • Use aliases for HBAs and controllers. Zone to these aliases. Stick with single initiator zoning. It's the least likely to cause problems if you don't fully understand the implications. Name your zones based on what they contain. Optionally include a logical grouping in the zone name. ('e.g' oracle_clus2_hostname_HBA0_array_port4)

  • Ask about performance, expect to get 'don't know' or 'lots!' type answers. It is rare to get a good answer on this sort of question, but it's important in a consolidated storage environment. The whole point of consolidation is to get better peak performance and lower average - this suits most workloads, because 'user facing' operations care more about a single transaction response (and don't care about it being idle).

  • Don't get hung up on RAID types - it can be a bit of a red herring. Caching makes more difference than RAID types, and caching affects different sorts of workload in different ways.

A read IO is under a hard time constraint - to complete the read to the host, the array has to fetch the block from disk. Caches prefetch and try to guess what'll be needed next - so predictable read IO is faster than random.

Write IO is under a soft time constraint - you can write cache, and acknowledge the write to the host - and destage to disk later. This means you can do some nice things like write coalescing and full stripe writes, and reduces the 'overhead' from the RAID type significantly. With a sequential workload such as logs/journals, RAID-5 can actually be faster than RAID 1+0 as a result.

The term often used is 'write penalty' - how many disk operations are required to 'complete' a write[1].

  • RAID 0, the write penalty is 1 - one write IO needed per incoming write.
  • RAID 1 - you write to both mirrors, so write penalty of 2.
  • RAID 5 - you need to write to a spindle, read/update parity. The write penalty is 4.
  • RAID 6 - like raid 5, but with two parity calculations, so attracts a write penalty of 6.

This means for a pure, random write workload - you need to divide your theoretical usable IOPs per spindle (~150 for FC, ~75 for SATA) by this write penalty.

For all raid types the 'read penalty' is basically the same - you need to complete a read from somewhere in your array. You get some slight advantage from RAID1, because it could be read from two different locations and see which responds faster, but there's not much in it - disks still have to rotate and seek.

With stripe coalescing - which most arrays can do with sequential writes - you get to cut down the write penalty. If you've got all of a parity stripe for your RAID 5 for example - you don't need to read back the parity, you can calculate it from what's in your cache. In that scenario, write penalty drops to 1+ 1/Raidgroup size so for a 4+1 RAID 5 group: 1.25. Which means it's better than RAID 1+0 for sustained serial write workloads. (e.g. database logs)

Resilience - you've got some different calculations to make, but I'd still say - don't get too hung up on RAID types here either - you'll get compound failures if you've a long enough time line regardless, so here's no mitigating needing a decent recovery solution. There are resilience differences between the RAID types, but the only one you definitely shouldn't use is RAID0 :).

(Other custom RAID types exist. E.g. NetApp use something they call RAID-DP for dual parity. It's basically RAID-4 with an extra parity spindle, but because of the way NetApps use their 'WAFL' filesystem, actually has a very low write penalty)