3

I just got a HP DL180 G6 that has 25X 146GB 15K SAS drives, 36GB RAM, 2X 2.0GHz Xeon 1333Mhz FSB. For fun I configured them all in a single RAID 0 and installed Ubuntu on it to see how fast it could get with 25 drives on a HP Smart Array P410 Controller w/ 512MB RAM.

When I ran hdparm -tT /dev/mapper/concorde--vg-root I get

Timing cached reads:   5658MB in  1.99 seconds = 2834.13 MB/sec
Timing buffered disk reads: 1192 MB in  3.00 seconds = 397.13 MB/sec

When I run the same command on my other server (HP DL360 G5 - 32GB RAM - 2X 2.66GHz 667Mhz FSB) that only has 4X 15K drives I get:

Timing cached reads:   13268 MB in  1.99 seconds = 6665.18 MB/sec
Timing buffered disk reads: 712 MB in  3.00 seconds = 237.17 MB/sec

I would have expected this to run 5 times faster than the old one, not slower. The server is intended to deliver streaming media and so I need super fast access and transfer to keep up with 2 1Gb network ports I hope to max out at times along with performing it's other tasks.

I just put together a bunch of copies of a 400MB MP4 file to get 45GB to copy from one directory to another and it took 96 seconds, which just seems wrong for everything I have ever heard of the performance boost of RAID 0.

It is setup as a hardware raid, is there anything I need to do in Linux to take advantage of the extra speed that should be there? Does it matter which flavor of Linux I use? I am comfortable with CentOS and Ubuntu but could do others if needed.

Is there a different command I should use to measure performance? I tried using iotop and iostat yesterday to monitor the RAID usage and couldn't get it to report any usage while copying 2GB files over FTP, so kind of stuck trying to set a benchmark, comparing it's performance across servers, and monitoring it so I know when the hard drives are maxing out and need to be replaced with SSD.

  • The buffered disk read is better on the new array (as you'd expect.) You are attributing too much weight to the cached read. – Aaron Copley Dec 31 '14 at 20:29
  • Sorry, you are right, but with 25 drives in the RAID shouldn't it be 5X faster on the buffered disk read, not marginally better? – Alan Dec 31 '14 at 20:36
  • 1
    RAID 0 isn't going to be the final configuration, is it?! – Michael Hampton Dec 31 '14 at 20:40
  • 1
    No, I don't think so. What are you basing this 5x figure on? In my experience, throughput (read or write) does not increase linearly for RAID 0 despite popular lore. – Aaron Copley Dec 31 '14 at 20:49
  • 1
    @Johnny Since I answered his last question, I know exactly what his use case actually is. :) – Michael Hampton Jan 1 '15 at 2:18
8

Wow... there's a lot to address here.

  • Disk performance isn't just about throughput. There's the notion of IOPS and latency and service times to contend with. Most workloads are a bit random in nature, so 25 enterprise disks in an array will always trump 4 disks from an IOPS perspective.

  • hdparm is not the right tool to benchmark enterprise storage. Look into purpose-built programs like iozone and fio.

An example iozone command that could be helpful for you is (run from a large directory on the disk array you wish to test): iozone -t1 -i0 -i1 -i2 -r1m -s72g

  • The design of this server means that your disk backplane is oversubscribed. There's an expander chip on the server and those 25 disks are sharing a 4-lane 6Gbps connection to the RAID controller. That means that you have a theoretical maximum throughput of 24Gbps (or 3,000 Megabyte/second) to the array. That's a ceiling, and you won't see performance beyond that point.

  • Ubuntu is almost never the best choice when hardware drivers and support are concerned.It's not officially supported by the server. CentOS or RHEL would be a better fit for this hardware.

  • HP Smart Array controllers have the ability to carve a group of disks (an array) into multiple logical drives of varying capacity and RAID levels. The example below shows a 4-disk array carved into three logical drives. One of the logical drives is configured with a different RAID level than the others.

Smart Array P420i in Slot 0 (Embedded) (sn: 0014380296200A0)

  logicaldrive 1 (72.0 GB, RAID 1+0, OK)
  logicaldrive 2 (1024.0 GB, RAID 1+0, OK)
  logicaldrive 3 (869.1 GB, RAID 5, OK)

  physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SAS, 900.1 GB, OK)
  physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SAS, 900.1 GB, OK)
  physicaldrive 1I:2:3 (port 1I:box 2:bay 3, SAS, 900.1 GB, OK)
  physicaldrive 1I:2:4 (port 1I:box 2:bay 4, SAS, 900.1 GB, OK)
  • At no point should you use RAID 0 for a logical drive here. If you can spare the space, RAID 1+0 will perform very well with this hardware combination.

  • You have LVM in place. That's not the best approach when working with these HP Smart Array RAID controllers. It's an additional layer of abstraction, and you won't see the best performance (although it can be tuned to work well).

  • Firmware. You'll want to update the firmware of the server and related components. Functionality improves with each HP RAID controller firmware revision.

  • RAID cache configuration. Make sure the RAID battery is healthy and that the cache read/write balance is right for your workload.

  • Filesystem choice. XFS is a good option for streaming media. But the rates you're asking for are relatively low. Any modern Linux filesystem should be fine.

  • I disagree with a blanket statement saying that RAID-0 should not be used, there are lots of use cases for RAID0 where the increase in physical I/O (or storage capacity) outweighs the inconvenience of the loss of an array or an entire node. – Johnny Jan 1 '15 at 2:14
  • With this setup, a RAID 0 logical drive will have diminishing returns due to controller and backplane limitations. In addition, recovering a RAID 0 logical drive on an HP controller is a hassle. Not worth the perceived speed increase. – ewwhite Jan 1 '15 at 2:25
  • Whether the vendor is willing to provide contractual support for one distro or not has no bearing at all on how well it can be expected to perform. Same kernel, same drivers, same performance can be expected. – psusi Jan 1 '15 at 5:23
  • When I get back to the office I will install new firmware, divide up the RAID into multiple sub sets, install CentOS and check battery. I didn't have a voltmeter with me yesterday to check it. Raid 1+0 should be just fine, I was just doing it this first way as a fun test - which didn't work out like I thought. – Alan Jan 1 '15 at 16:30
  • @psusi Vendor support does matter in the form of drivers and monitoring tools. HP ProLiant servers have some value-added features that are often not available or intended for Ubuntu. Sometimes it's just a non-starter. – ewwhite Jan 1 '15 at 16:32
1

One thing that jumps out at me right away is the low cached througput on the new server, which indicates you have a ram bottleneck. You mention it has 36 GB of ram in it, which is a bit of an odd number. I am betting that you failed to install the ram in pairs ( or 3x if this cpu/board is capable of triple interleaving, as I suspect it is ), and so the ram is not interleaved properly. You might run memtest86+ to get a good measurement of the memory throughput.

As for the actual IO throughput, for an array this large you are probably not throwing enough requests at it. hdparm only reads 2 MiB at a time, so if the array is using a 512k stripe factor, that will only keep 4 of the drives busy at a time. For other commands like dd, it depends on the readahead size and the default readahead size is only 128 kb. Try using dd instead of hdparm ( without iflag=direct ) and pump up the readahead factor in /sys/block/dm-X/queue/read_ahead_kb to make sure the kernel sends large enough requests to keep all of the drives busy, or if you do use iflag=direct with dd, then make sure to use a very large value for bs=, like 32MiB, and then the kernel readahead setting doesn't matter.

  • Thanks psusi, I have more ram on order and will be filling all banks with 8GB sticks. I thought the ram arrangement seemed wrong, but where I bought it from was having trouble getting it all to show in other configurations. When I get back to the office I will try a few of those other commands like you suggest. One thing is you mentioned that hdparm only does 2MiB at a time, but I tried copying 45GB which took 96 seconds. Now this may have been a faulty test because I was reading and writing to the same raid, I can try setting up several other raids to try more stuff. – Alan Jan 1 '15 at 16:19
  • There is only one way to install/situate RAM on this model. Please refer to the DIMM Population Guidelines section in the DL180 G6 QuickSpecs document. – ewwhite Jan 1 '15 at 16:35
  • @Alan, best test is to dd to read from the drive and write to /dev/null. – psusi Jan 1 '15 at 16:51
  • Just a quick update @ewwhite, @psusi. Back in the office today. I reconfigured the RAM and added 32GB from another computer and got a little better performance on hdparm (3431MB/s and 419MB/s). I checked the battery and it was completely dead, trying to see if I can pick one up locally where I bought it but they may be closed today. Using the dd command I was able to copy a 439MB file in 2 seconds for a rate of 220 MB/s. I am in the process of downloading and installing CentOS and will configure the filesystem, stripe and stuff from your suggestions and try again. – Alan Jan 2 '15 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.