Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm working on an application to capture a lot (10 million +) of really small blocks of data (16 bytes) every day. The data is not sequential (i.e lots of seeking about to write) and it's not a constant stream (there are periods of quiet).

The app has caching servers in front of it so reads are less of an issue and I anticpate only 1% of the data will be of interest on a given day and that 1% will sit in the cache. Only the 1st read should be slow.

I have a good but limited budget and I want RAID 1 which doubles my disk cost.

My choices are:

  • Fast SAS Disks in RAID 1 - expensive not much storage but fast.
  • Big Near-Line disks RAID 1 + 1gb NVCache on a controller (PERC H700)

What would you do? Or put another way, does a large cache on the controller compensate, in terms of writing, for a slower seek time?

We're a DELL shop and I'm looking at the R410/R510.

share|improve this question

4 Answers 4

up vote 6 down vote accepted

I'm not sure you're going to get a useful answer here. I'd be performing benchmarks with the application and the prospective hardware to get an idea of how it performed because I suspect there is enough complexity that attempting to model it "back of the envelope" is likely too simplistic.

In general the on-controller cache can buffer writes and allow the RAID volume to respond more quickly to the operating system. If your write rate exceeds the speed that the cache can be committed to disk for long enough to fill the cache then the controller will begin to block on writes (falling back to the speed of the physical disks).

It sounds like you're not using an off-the-shelf database management system but, rather, are managing the data storage yourself. You're going to have to evaluate how your application interacts with the OS cache manager and underlying filesystem (assuming you're not storing data on raw disk blocks) as well as the RAID controller. If you are using a database management system then, obviously, you'll have to see how that interacts, too.

When you say "working on" I wonder if you're involved in the development of the application. If so, I think it's worth looking at an application architecture that buffers incoming writes into a sequentially-written log and later lazy-writes that sequential log onto the random access storage structure. You'd be accomplishing the same thing, in effect, as the controller caching writes, but you'd have more granular control of the process (you could explicitly tier the storage for the sequential versus the random access log).

share|improve this answer
1  
Thanks! Yes I am the developer (and also the finance) and we're managing the storage. I'm trying to get opinions/experiences here to help make an informed buying decision. I'm not in a position to buy/try. Ironically, I'm rewriting the storage engine from a 'lazy' update to avoid a problem W2k8R2 bit.ly/p9Nx2z –  The Diamond Z Feb 8 '12 at 16:48
    
@TheDiamondZ thanks, this makes an interesting read –  the-wabbit Feb 8 '12 at 17:08
    
@TheDiamondZ what exactly is the problem you are trying to avoid? The "problem" (in quotes because oddly enough I would expect a fileserver to cache filelocations) shouldn't occur if you are writing to a database or a single file. –  Jim B Feb 8 '12 at 17:28
    
We're appending individual update files (1 for each 16b data set, but each file could contain multuple days of updates) and coming along to do a lazy update/defrag type thing to a master data set on a schedule or when the data is 'read' by a user request. So 1-2% on demand 99% over a few days on a continuous cycle. This works really well on a 32 bit server (original development/deloyment) but our 8gb w2k8r2 host would regularly run out of memory and slow to a crawl. –  The Diamond Z Feb 8 '12 at 17:41
    
@TheDiamondZ: It's not clear to me why you'd be exhausting RAM, though. Are you saying that you're writing a large number of very small files (MFT-resident files)? The "lazy writer" architecture would be, to my mind, your best bet for future scalability. If you're running afoul of the OS cache manager I'd investigate how you can work with it better versus scrapping the whole idea and moving toward an infrastructure that will force you into buying more and more hardware to scale. –  Evan Anderson Feb 8 '12 at 21:08

Or put another way, does a large cache on the controller compensate, in terms of writing, for a slower seek time?

To a certain extent. There are some factors to consider:

  • the cache will only have the desired effect as long as it is not overrun - if your data is coming in bursts or at a sustained rate where the disks cannot cope with the load, caches will fill up, the worst case being an I/O block until the caches flush down to the low-watermark for further operation
  • caching algorithms often ensure that data in cache can be no older than "X", initiating a flush for it even when there still would be room for more
  • caching happens in "blocks", so even if your records are just 16 Bytes in size, it does not mean you could store 67 million records in 1 GB of cache RAM
  • mixed random read/write load is tough even for a larger cache
  • you might well run into filling command queues even with large caches, so if your storage requirements not only include IOPS and bandwidth requirements but low latency (low service times) as well, it would be hard to achieve with the given setup options

Some estimate math: assuming a typical service time for a single request to be 20 ms for nearline SATA disks, the I/O subsystem would take 200,000 seconds to write 10,000,000 to the disks - that's more than 55 hours of 100% disk utilization. If you are getting this magnitude of write requests per day, you are likely to overrun your I/O subsystem.

How hard you will be hit by one or the other boundary condition, heavily depends on the implementation of the controller and its caching mechanism. You would need to run thorough tests so there would be no unpleasant surprises.

share|improve this answer

If the RAID cache is a limiting factor (one of the previous answers indicates that it may be) I would consider adding some smarts to the caching in front to stripe writes over separate arrays - say, 4 mirrors of 2 disks each - and hash the destination so you spread the load evenly.

This will not improve cache usage per se, but it will provide you with 4 sets of independent spindles to write to, thus avoiding most of the latency incurred in having to write to all spindles at once.

As the first responder said, though - you need to test what works best.

share|improve this answer

Have you thought about the H700 with the 512 or 1GB cache, then trowing in an SSD or two to use as extra cache for the drives. Dell calls it their Cachecade technology.

See here: http://www.dell.com/downloads/global/products/pedge/en/perc-h700-cachecade.pdf

share|improve this answer
    
CacheCade not a Dell invention, it is LSI's brand - Dell just happens to use LSI controller hardware and re-brand them as Dell PERC (along with a custom firmware) –  the-wabbit Feb 9 '12 at 14:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.