Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a fresh mdadm RAID10 device which is healthy, built from 4 x 7200rpm SATA 3 drives:

# cat /proc/mdstat 
Personalities : [raid10] [raid1] 
md2 : active raid10 sdc3[5] sda3[4] sdb3[1] sdd3[3]
      3885267968 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

(There are 2 other mdadm devices, but md2 is the relevant one)

On top of md2 is an ext4 filesystem, created with:

mkfs.ext4 -m0 -L bups -b 4096 -E stride=8,stripe-width=32 /dev/md2

This has horrible performance when using it as a backup target over NFS. With a single client writing, I see figures like this, with ~30% to 50% I/O wait in top:

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda             197.00         0.00         1.07          0          1
sdb             189.00         0.00         1.09          0          1
sdc             300.00         0.00         1.78          0          1
sdd             290.00         0.00         1.78          0          1
md2             785.00         0.00         2.86          0          2

Load Average is high:

# uptime
 11:40:38 up 21:21,  1 user,  load average: 9.90, 9.18, 8.92

Using dd shows decent performance:

# dd if=/dev/zero of=/bup/test.dd bs=4096 count=1024000
1024000+0 records in
1024000+0 records out
4194304000 bytes (4.2 GB) copied, 18.7801 s, 223 MB/s

The connectivity is gigabit all the way. This is CentOS 6.2 x64 on a brand-new HP DL160 Gen8 with 16gb RAM, supposed to be a dedicated backup target, but not with this kind of performance.

EDIT

The controller is a HP B120i in SATA AHCI mode.

/etc/exports on the server:

/bup/phdv   172.31.42.30(rw,no_root_squash)

/etc/fstab extract on the client (also CentOS 6.4 x64):

svr-bup1.example.com:/bup/pronto    /mnt/bup    nfs soft    0 0

EDIT 2

Relevant information from /proc/mounts:

svr-bup1.example.com:/bup/pronto /mnt/bup2 nfs4 rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=2001:xxxx:xxxx:xxxx::240,minorversion=0,local_lock=none,addr=2001:xxxx:xxxx:xxxx::210 0 0
share|improve this question
    
You have no write cache... That's why. Is there a RAID controller in the server? –  ewwhite Jan 3 at 1:29
    
Also, can you share your NFS export and mount options? –  ewwhite Jan 3 at 1:59
    
@ewwhite Thanks for the comments - I'm writing nearly 100gb of data, so I don't think a write cache would make a difference with that much data? I'll edit the post with the export and mount details. FWIW, this server is replacing a Synology NAS performing the same task (backups, mounted via NFS) and it works fine, so it has to be something wrong on this host. –  fukawi2 Jan 3 at 2:41
    
@ewwhite You made me think about comparing the Synology exports to mine - the Synology has no_wdelay enabled for the share. Adding that to mine has reduced the load significantly down to between 1 and 2, the TPS in iostat has reduced to <4 for each block device, and write speeds seem to have increased (although iostat says 0.00MB/s for each block dev). I think that may be the issue, although I don't understand it fully. Would having a write cache have negated the impact from wdelay? –  fukawi2 Jan 3 at 2:50
    
Write cache on the RAID controller would benefit synchronous writes from an NFS client... Otherwise, your client will be waiting on write confirmation to the 7200 RPM disks rather than a low-latency cache. –  ewwhite Jan 3 at 3:13
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.