Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have three VMware ESXi 4 hosts serving VM's from an OpenFiler NFS share. Every host has a direct gigabit connection to the NAS. Although read performance has been great, writing files inside the VM guests are suffering.

The recommended configuration for data integrity is to export NFS shares with the sync option and mount ext3 with data=journal.

I'd like to compare the behaviour of the maximum integrity configuration with the maximum I/O performance configuration. To configure for performance I exported the NFS share as

/mnt/raided/main/vm 10.0.0.0/255.255.0.0(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,async)

while ext3 is mounted with

/dev/raided/main /mnt/raided/main ext3 defaults,usrquota,grpquota,acl,user_xattr,data=writeback,noatime

Will these configuration options give me optimal I/O performance? How about changing the file system? Will XFS improve performance significantly?

Other than the NAS crashing or power failures, what can cause data integrity issues with this configuration?

share|improve this question
    
You fail to mention the level of redundancy you are using to provide the storage. RAID-5 has poor write characteristics. –  Jan Jungnickel Jul 28 '09 at 6:51
    
I'm using software RAID 10 –  Hans Malherbe Jul 28 '09 at 11:23
1  
Our Openfiler+ESXi crude test showed ~30% improvement using XFS than ext3. Nevertheless, we decided to use iSCSI target feature. YMMV. –  Arie K Jul 29 '09 at 17:27
    
I'd like to know how you tested as I don't have good ones. –  Hans Malherbe Jul 29 '09 at 20:27
    
We used dd command from Linux VMs to write a large file. Hence the 'crude'. –  Arie K Jul 30 '09 at 3:22

3 Answers 3

I would think hard about using sync. I'm reminded about the Dirty Harry 'Do you feel lucky' speech.

NFS v2 and v3 is designed such that when the write is acked from the server to the client the data is on disk. This allows NFS to be stateless, and, therefore, the server COULD reboot between each request. One hopes not, but it could happen.

This means that if the client sees the ACK on the write then it doesn't have to care anymore about the data being on disk.

If you use async then this is not true anymore. It will be way faster though. Basically though if you use async, and, the server crashes, you probably should reboot the clients unless you know exactly what they do since if they are expecting to keep multiple files in sync the client may believe they are in sync, when, in fact, they are not.

share|improve this answer
    
I'm a big fan of data integrity. The above mentioned performance exercise gives me an idea of what the cost is. I am quite shocked actually. I'm considering striking a balance by splitting the important VM's from the easily replaceable ones. –  Hans Malherbe Jul 29 '09 at 15:53

My understanding is that OpenFiler is based on the CentOS/RedHat family of OS builds. Linux's nfs server isn't the best to begin with*, and CentOS/RedHat's is worse than average.

*= when compared to NetApp filers, *bsd, or Solaris servers.

(Wonders if I'm going to get down-modded for that.)

share|improve this answer
    
Have you got a reference for the *bsd comparison? –  Hans Malherbe Jul 29 '09 at 20:25
    
Sorry, no documentation, it just something my *bsd-using cohorts happily claimed while I was fighting with a linux nfs server. No one has claimed any different as far as I am aware. –  David Mackintosh Jul 30 '09 at 1:34

yup, 'sync' is definitely a huge write-performance killer. I wouldn't ever consider it unless the data integrity demand is high. If you don't have battery-backed RAID and UPS, don't even bother because it will have little or no effect anyway.

'ext3 data=journal' doesn't help neither. If you crave for write performance, ditch ext3 and go for xfs instead, it's so much faster it's not even funny.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.