We are currently testing the maximum sequential read throughput of a storage system (48 disks total behind two HP P2000 arrays) connected to HP DL580 G7 running RHEL 5 with 128 GB of memory.

Initial testing has been mainly done by running DD-commands like this:

dd if=/dev/mapper/mpath1 of=/dev/null bs=1M count=3000

In parallel for each disk.

However, we have been unable to scale the results from one array (maximum throughput of 1.3 GB/s) to two (almost the same total throughput). Each array is connected to a dedicated FC host bust adapter, so they should not be the bottleneck. The disks are currently in JBOD configuration, so each disk can be addressed directly.

I have two questions:

  1. Is running multiple DD commands in parallel really a good way to test maximum read throughput?

  2. How shoud we proceed in trying to find the reason for the scaling problem? Do you thing the server itself is the bottleneck here, or could there be some linux parameters that we have overlooked?

  • "Each array is connected to a dedicated host bust adapter"? Is that FC or sas? – Basil Nov 30 '11 at 16:10
  • FC, Brocade 825 – Unknown Dec 1 '11 at 13:50

I would recommend trying a dedicated benchmarking tool like bonnie++ or iozone (both available via yum from RPMForge) to get an accurate representation of the storage system's capabilities. Iozone will allow you to specify number of threads, block size and read/write patterns. Are IOPs not important in your use-case?

How are you connecting to this unit? FC? SAS? Can you describe your RAID configuration? You did not specify the exact P2000 model or generation (I'm assuming a G2 or G3 24SFF versus 12LFF disks per enclosure), but there's some level of oversubscription in the 24SFF units, so I'd expect that your sequential read speed is limited by that.

The HP QuickSpecs for the MSA2324 G3 LFF show 4Gb FC RAID 1+0 read speeds of 1.6GB/s. This is the same for RAID 5 and RAID 6, so the controller/interface appears to be the limiting factor.

  • 1
    Assuming it's FC Getting 1.3GB/sec on a controller that is rated for 1.6GB/sec is pretty good (81% of rated speed, and rated speed usually being "manufacturer-contrived benchmarking scenario of optimal performance") – voretaq7 Nov 30 '11 at 15:27
  • This will be used for a data warehouse where IOPS is not important. Connection is FC, one 2-channel Brocaden 825 card for each P2000. The model for P2000 is G3, 12LFF + encosure for another 12 disks. The results for single P2000 + enclosure are good, we are just wondering why it doesn't scale when running two P2000 in parallel (connected to different FC cards). – Unknown Dec 1 '11 at 13:27
  • @voretaq7: no, the "rated speed" is exactly that - the theoretical maximum throughput. As such, achieving 81% of it is an excellent result, especially considering there may be many concurrent requests (which bring down throughput.) The OP should, of course, consider more relevant metrics than just the kneejerk DD statistics. – adaptr Dec 1 '11 at 13:56
  • It is excellent result for single P2000+enclosure pair, but why can't we archive more total throughput by installing two pairs behind independent IO paths? There must be a bottleneck somewhere in the server. We have already confirmed that parallel DD results closely match actual database query throughput and the system is not yet CPU-bound. Of course we also have more complex queries, but it is common for a single query to scan whole table (in parallel) and return a few aggregate values. – Unknown Dec 2 '11 at 6:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.