7.2K Near Line SAS with Large RAID Controller Cache vs 10/15K SAS

Question

I'm working on an application to capture a lot (10 million +) of really small blocks of data (16 bytes) every day. The data is not sequential (i.e lots of seeking about to write) and it's not a constant stream (there are periods of quiet).

The app has caching servers in front of it so reads are less of an issue and I anticpate only 1% of the data will be of interest on a given day and that 1% will sit in the cache. Only the 1st read should be slow.

I have a good but limited budget and I want RAID 1 which doubles my disk cost.

My choices are:

• Fast SAS Disks in RAID 1 - expensive not much storage but fast.
• Big Near-Line disks RAID 1 + 1gb NVCache on a controller (PERC H700)

What would you do? Or put another way, does a large cache on the controller compensate, in terms of writing, for a slower seek time?

We're a DELL shop and I'm looking at the R410/R510.

Answer 1

I'm not sure you're going to get a useful answer here. I'd be performing benchmarks with the application and the prospective hardware to get an idea of how it performed because I suspect there is enough complexity that attempting to model it "back of the envelope" is likely too simplistic.

In general the on-controller cache can buffer writes and allow the RAID volume to respond more quickly to the operating system. If your write rate exceeds the speed that the cache can be committed to disk for long enough to fill the cache then the controller will begin to block on writes (falling back to the speed of the physical disks).

It sounds like you're not using an off-the-shelf database management system but, rather, are managing the data storage yourself. You're going to have to evaluate how your application interacts with the OS cache manager and underlying filesystem (assuming you're not storing data on raw disk blocks) as well as the RAID controller. If you are using a database management system then, obviously, you'll have to see how that interacts, too.

When you say "working on" I wonder if you're involved in the development of the application. If so, I think it's worth looking at an application architecture that buffers incoming writes into a sequentially-written log and later lazy-writes that sequential log onto the random access storage structure. You'd be accomplishing the same thing, in effect, as the controller caching writes, but you'd have more granular control of the process (you could explicitly tier the storage for the sequential versus the random access log).

Answer 2

Or put another way, does a large cache on the controller compensate, in terms of writing, for a slower seek time?

To a certain extent. There are some factors to consider:

• the cache will only have the desired effect as long as it is not overrun - if your data is coming in bursts or at a sustained rate where the disks cannot cope with the load, caches will fill up, the worst case being an I/O block until the caches flush down to the low-watermark for further operation
• caching algorithms often ensure that data in cache can be no older than "X", initiating a flush for it even when there still would be room for more
• caching happens in "blocks", so even if your records are just 16 Bytes in size, it does not mean you could store 67 million records in 1 GB of cache RAM
• mixed random read/write load is tough even for a larger cache
• you might well run into filling command queues even with large caches, so if your storage requirements not only include IOPS and bandwidth requirements but low latency (low service times) as well, it would be hard to achieve with the given setup options

Some estimate math: assuming a typical service time for a single request to be 20 ms for nearline SATA disks, the I/O subsystem would take 200,000 seconds to write 10,000,000 to the disks - that's more than 55 hours of 100% disk utilization. If you are getting this magnitude of write requests per day, you are likely to overrun your I/O subsystem.

How hard you will be hit by one or the other boundary condition, heavily depends on the implementation of the controller and its caching mechanism. You would need to run thorough tests so there would be no unpleasant surprises.

Answer 3

If the RAID cache is a limiting factor (one of the previous answers indicates that it may be) I would consider adding some smarts to the caching in front to stripe writes over separate arrays - say, 4 mirrors of 2 disks each - and hash the destination so you spread the load evenly.

This will not improve cache usage per se, but it will provide you with 4 sets of independent spindles to write to, thus avoiding most of the latency incurred in having to write to all spindles at once.

As the first responder said, though - you need to test what works best.

Answer 4

Have you thought about the H700 with the 512 or 1GB cache, then trowing in an SSD or two to use as extra cache for the drives. Dell calls it their Cachecade technology.

See here: http://www.dell.com/downloads/global/products/pedge/en/perc-h700-cachecade.pdf