Limited I/O Performance in Dell T410

Question

We have a small number of Dell T410 and T710 systems. Each has a Dell SAS 6iR controller with 6 SATAII disks, 1x160GB OS and 5x3TB DATA disks. We have been having disk access speed issues all along, each disk giving little more than 100MB/s alone, but slows down quickly under load. We have been told it was our disk controllers. Tried replacing one of them with a H700 and saw little difference, maybe 5% speed increase. Convinced ourselves it must be something with the SAS -> SAS Backplane -> SATAII disks.

But now we are trying to setup ceph cluster between systems and to speed up the journal files we bought OCZ Vertex 4 SDD's with a Highpoint 620 PCI-e 1X card since SAS is all tied up with SAS Backplane. That is is all fine.

But in testing, I have tried 'dd if=/dev/zero bs=4k count=1024k of=/mnt/test1.img' where the SSD is mounted with a EXT4 filesystem. I have 'iostat -dmx 1' open in another window. I see the write finish, reporting 390MB/s write to disk, but this is with the disk caching. In iostat I see it flushing to the disk at the same 100-120MB/sec I see from our SATAII disks!

Tried the same test in the other direction, 'echo 3 > /proc/sys/vm/drop_caches; dd if=/mnt/test1.img bs=4k of=/dev/null', which reported 194MB/s and saw 189MB/sec in iostat.

I tried this same thing with Ubuntu 12.10 and CentOS Live CD's. Our Windows 2008 R2 systems have always seemed slow. All BIOS and Firmware are up to date. Tried noop, deadline and cfq schedulers, all the same results. Turned off any and all Memory and CPU options in BIOS, with no change. Tried with IOATDMA enabled, no change. Tried with taking out all network and SAS cards, leaving just the SATAIII card and SSD, no change. The SATAIII card is recognized correctly and identifies the SSD as 6G speed.

SATAIII card and SSD get 400MB/s+ in other, desktop, systems.

Any ideas what to try next?

We cannot afford to replace and start all over, again. It is acting as if there is some resource that is limiting the amount of I/O resources or timings so that it cannot reach full speed. Limited Dell BIOS does not have many options to do this and the card and SSD works fine in other systems, so it should not be the culprit. And it has always been slow with the SATAII disks all along, even if they perform better in other systems, as well.

I even see the slow perfomance on meta disks on the system. This is visible when I tried LVM with striping or mirroring in CentOS, the metadisks it sets up the background to implement the LVM volumes were limited to the same speed as the disks behind them. At the time I though it may have been something to do with the new code implementing it or the blocking IO aspects of the RAID operations. Now I think it is all related.

Answer 1

Well from my point of view, the issue might be in a faulty PCI-E bus or some issues with the Controller in general. On the controller side, have you tried to disable the Write cache for the SSD drives? Officially Dell suggests to disable the write cache when using SSD drives...

Another point, if I recall correctly any Dell controller disables the write cache if you don't have any Battery backup unit (BBU) installed in the server, do you have one? If not then, force the write cache in the PERC BIOS and check if the performance changes. Keep in mind that write cache without a battery exposes you to catastrophic data loss in a power failure event!

As a last resort, if it's a PCI-E bus issue, the fastest and simplest way should be to change the controller slot.

EDIT: forgot to mention

Have you upgraded system firmwares to latest versions? BIOS, PERC Firmwares and related stuff? If not then get latest from support.dell.com and try again. Maybe it will not change anything, but old Dell BIOS firmwares had a bug where they were throttling CPU in the post Nehalem architectures, try to disable C-States in the CPU section of the BIOS, NOTE it will increase power consumption!