Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a fresh mdadm RAID10 device which is healthy, built from 4 x 7200rpm SATA 3 drives:

# cat /proc/mdstat 
Personalities : [raid10] [raid1] 
md2 : active raid10 sdc3[5] sda3[4] sdb3[1] sdd3[3]
      3885267968 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

(There are 2 other mdadm devices, but md2 is the relevant one)

On top of md2 is an ext4 filesystem, created with:

mkfs.ext4 -m0 -L bups -b 4096 -E stride=8,stripe-width=32 /dev/md2

This has horrible performance when using it as a backup target over NFS. With a single client writing, I see figures like this, with ~30% to 50% I/O wait in top:

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda             197.00         0.00         1.07          0          1
sdb             189.00         0.00         1.09          0          1
sdc             300.00         0.00         1.78          0          1
sdd             290.00         0.00         1.78          0          1
md2             785.00         0.00         2.86          0          2

Load Average is high:

# uptime
 11:40:38 up 21:21,  1 user,  load average: 9.90, 9.18, 8.92

Using dd shows decent performance:

# dd if=/dev/zero of=/bup/test.dd bs=4096 count=1024000
1024000+0 records in
1024000+0 records out
4194304000 bytes (4.2 GB) copied, 18.7801 s, 223 MB/s

The connectivity is gigabit all the way. This is CentOS 6.2 x64 on a brand-new HP DL160 Gen8 with 16gb RAM, supposed to be a dedicated backup target, but not with this kind of performance.

EDIT

The controller is a HP B120i in SATA AHCI mode.

/etc/exports on the server:

/bup/phdv   172.31.42.30(rw,no_root_squash)

/etc/fstab extract on the client (also CentOS 6.4 x64):

svr-bup1.example.com:/bup/pronto    /mnt/bup    nfs soft    0 0

EDIT 2

Relevant information from /proc/mounts:

svr-bup1.example.com:/bup/pronto /mnt/bup2 nfs4 rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=2001:xxxx:xxxx:xxxx::240,minorversion=0,local_lock=none,addr=2001:xxxx:xxxx:xxxx::210 0 0
share|improve this question
    
You have no write cache... That's why. Is there a RAID controller in the server? –  ewwhite Jan 3 at 1:29
    
Also, can you share your NFS export and mount options? –  ewwhite Jan 3 at 1:59
    
@ewwhite Thanks for the comments - I'm writing nearly 100gb of data, so I don't think a write cache would make a difference with that much data? I'll edit the post with the export and mount details. FWIW, this server is replacing a Synology NAS performing the same task (backups, mounted via NFS) and it works fine, so it has to be something wrong on this host. –  fukawi2 Jan 3 at 2:41
    
@ewwhite You made me think about comparing the Synology exports to mine - the Synology has no_wdelay enabled for the share. Adding that to mine has reduced the load significantly down to between 1 and 2, the TPS in iostat has reduced to <4 for each block device, and write speeds seem to have increased (although iostat says 0.00MB/s for each block dev). I think that may be the issue, although I don't understand it fully. Would having a write cache have negated the impact from wdelay? –  fukawi2 Jan 3 at 2:50
    
Write cache on the RAID controller would benefit synchronous writes from an NFS client... Otherwise, your client will be waiting on write confirmation to the 7200 RPM disks rather than a low-latency cache. –  ewwhite Jan 3 at 3:13

1 Answer 1

On server increasing RPCNFSDCOUNT number in /etc/default/nfs-kernel-server may help;
on client(s) it may be worth trying smaller rsize and wsize values...

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.