Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

The server and client are both 64-bit Ubuntu 10.04 LTS.

The application in question is a custom app that uses mmap() for fast random file access. Its ideal state is when the entire file is cached in RAM.

The network connections are really fast 10Gb Ethernet. It is a virtual server blade setup.

It isn't the network connections slowing things down because everything performs superbly when using a virtual disk (iSCSI to the SAN). But when we run the application on a NFS home directory mount, performance goes to the dogs.

It appears that the Linux kernel isn't caching anything. So it is reading every single disk block needed by mmap() accesses over and over and over again.

The NFS mount is done through autofs, which has only default settings. /proc/mounts shows the NFS mount is done with the following options:

rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.11.52,mountvers=3,mountproto=tcp,addr=192.168.11.52

How can I make Ubuntu 10.04 cache the file instead of reloading it all the time?

share|improve this question
    
What MMAP flags are you passing? –  Matthew Ife Mar 1 at 15:00

1 Answer 1

Have you tuned your NFS exports or mounts at all? What are your present mount options? Which filesystem are you using on the NFS server?

I know you're using Ubuntu, but this NFS tuning guide from Red Hat might be a good start, especially if you have a good performance metric to test with different settings. Some of the hints in the document cover limiting metadata changes when accessing files over NFS and extending the attributes cache.

Is NFS caching (FS-Cache, CacheFS) an option in this case?

share|improve this answer
    
Sequential read/write runs at 80MB/s per second, so that part of the NFS tuning is okay. –  Zan Lynx Dec 13 '11 at 2:07
    
Right, but look at the attributes side of things. The application you describe may have an impact on access times and caching the attributes may make a difference, depending on how you're benchmarking. –  ewwhite Dec 13 '11 at 2:12
    
NFS caching with CacheFS will make things worse in his case. His local disks are likely slower than his network. –  David Schwartz Dec 13 '11 at 2:51
    
@DavidSchwartz: Right. Since these are virtual systems there aren't any real local disks. It's all virtual. The "local disk" is a VMware image backed by iSCSI. –  Zan Lynx Dec 13 '11 at 18:38
    
I added some info about proc mounts to the question. –  Zan Lynx Dec 13 '11 at 18:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.