Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have trouble getting the max throughput out of my setup. The hardware is as follow :

  • dual Quad-Core AMD Opteron(tm) Processor 2376
  • 16 GB DDR2 ECC RAM
  • dual Adaptec 52245 RAID controllers
  • 48 1 TB SATA drives set up as 2 RAID-6 arrays (256KB stripe) + spares.

Software :

  • Plain vanilla 2.6.32.25 kernel, compiled for AMD-64, optimized for NUMA; Debian Lenny userland.
  • benchmarks run : disktest, bonnie++, dd, etc. All give the same results. No discrepancy here.
  • io scheduler used : noop. Yeah, no trick here.

Up until now I basically assumed that striping (RAID 0) several physical devices should augment performance roughly linearly. However this is not the case here :

  • each RAID array achieves about 780 MB/s write, sustained, and 1 GB/s read, sustained.
  • writing to both RAID arrays simultaneously with two different processes gives 750 + 750 MB/s, and reading from both gives 1 + 1 GB/s.
  • however when I stripe both arrays together, using either mdadm or lvm, the performance is about 850 MB/s writing and 1.4 GB/s reading. at least 30% less than expected!
  • running two parallel writer or reader processes against the striped arrays doesn't enhance the figures, in fact it degrades performance even further.

So what's happening here? Basically I ruled out bus or memory contention, because when I run dd on both drives simultaneously, aggregate write speed actually reach 1.5 GB/s and reading speed tops 2 GB/s.

So it's not the PCIe bus. I suppose it's not the RAM. It's not the filesystem, because I get exactly the same numbers benchmarking against the raw device or using XFS. And I also get exactly the same performance using either LVM striping and md striping.

What's wrong? What's preventing a process from going up to the max possible throughput? Is Linux striping defective? What other tests could I run?

share|improve this question
    
I don't understand why you're going RAID 6+0 here, why would you do that when you can just use regular RAID 1+0? –  Chopper3 Dec 17 '10 at 15:28
    
That won't solve the problem. md/lvm share the same behaviour when striping RAID-10 arrays. I'm looking after the general lack of performance, not a particular setup. This is a test system, not a production machine. –  wazoox Dec 17 '10 at 15:31
5  
do your raid 0 stripes align with your raid 6 stripes? Generally layered RAID is a very precarious area and it is not ok to assume a given RAID will act as if it was a single harddrive of the same speed. –  JamesRyan Dec 17 '10 at 15:59
1  
have you tried configuring the Adaptec cards as JBOD and doing RAID-10 only in software (md)? it wouldn't surprise me if the RAID feature of the controller is in parte handled by the drivers, negating any performance advantage once you interleave two different RAIDs. md, on the other hand, will try to optimize access to each drive independently, aproaching theoretical times (if there's no other bottleneck, of course) –  Javier Dec 17 '10 at 16:55
1  
That doesn't sound like an easy investigation. At first glance I would favor the possibility that the striping program is not able to perform via a multi-threading algorithm. Since you are using the soft RAID from mdadm, I would suggest you to have a look at the source. –  ring0 Dec 23 '10 at 14:24
show 2 more comments

2 Answers 2

Have you tried to run latencytop while doing benchmarks? might be helpful to see which linux syscall is the culprit (if any).

share|improve this answer
    
Thank you for the idea, I'll try that. –  wazoox Dec 24 '10 at 14:12
add comment

That's an x8 PICe Gen 1 card as far as I can tell - the absolute maximum data rate it could support is 2GBytes/sec assuming zero overheads. Adaptec themselves only claim that the cards can sustain 1.2Gbytes/sec at best and you are exceeding that.

Equipped with industry-leading dual-core RAID on Chip (RoC), x8 PCI Express connectivity and 512MB of DDR cache, they provide over 250,000 IO per second and 1.2GB/s.

My guess is that since you are able to significantly exceed their claimed performance with two RAID 0 sets acting independently the additional load, small and all is it might be, that striping adds to that is overstressing the RAID CPU's, or possibly the RAM subsystem on the controller, at GByte/sec loads.

share|improve this answer
1  
I don't quite understand... As OP phrases it, it seems to me that he has 2 RAID controllers, each handling an RAID6 array. Then he RAID 0's the two RAID 6 arrays in software, ie. using Linux' built in kernel RAID. In this case, each controller should only handle half the load, so each controller just needs to do 780MB/s write and 1GB/s read. The controllers have already proven they can do this (before software RAID was added). So the PCI-Express bus / RAID controller itself should not be the limiting factor? –  Jesper Mortensen Dec 23 '10 at 14:35
    
Fair point - missed the dual controller comment (and the lvm\md part that emphasised that). His point about it not being a bus\IO limitation is pretty much proven then. –  Helvick Dec 23 '10 at 14:59
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.