Since a recent linux kernel upgrade (3.2.78 on Debian Wheezy) on both my NFS clients and NFS server, I'm seeing really high increase (10 to 20 folds) in the number of nfs_lookup requests from the clients to the server. This is making a huge performance degradation to the application running on my NFS clients (apache+php-fpm).
What can be the cause of such a high increase of nfs_lookup?
mountstats --nfs on one of the clients after about 1h of uptime is as follows:
Stats for ruchbah.nfs://membres mounted on /srv/membres: NFS mount options: rw,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.74.13,minorversion=0,local_lock=none NFS server capabilities: caps=0x7fff,wtmult=512,dtsize=32768,bsize=0,namlen=255 NFSv4 capability flags: bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3 NFS security flavor: 1 pseudoflavor: 0 Cache events: data cache invalidated 1231 times attribute cache invalidated 23693734 times inodes synced 1790 times VFS calls: VFS requested 46296214 inode revalidations VFS requested 35900771 dentry revalidations VFS called nfs_readdir() 3678 times VFS called nfs_lookup() 35200320 times VFS called nfs_permission() 11201 times VFS called nfs_file_open() 22547 times VFS called nfs_file_flush() 1400 times VFS called nfs_lock() 7611 times VFS called nfs_fsync() 65321 times VFS called nfs_file_release() 3731 times VM calls: VFS called nfs_readpage() 856 times VFS called nfs_readpages() 6945 times VFS called nfs_writepage() 10789 times VFS called nfs_writepages() 945872 times Generic NFS counters: File size changing operations: truncating SETATTRs: 65272 extending WRITEs: 0 291 silly renames short reads: 11201 short writes: 17 NFSERR_DELAYs from server: 0 NFS byte counts: applications read 162084951 bytes via read(2) applications wrote 19636930 bytes via write(2) applications read 0 bytes via O_DIRECT read(2) applications wrote 0 bytes via O_DIRECT write(2) client read 152501261 bytes via NFS READ client wrote 20783074 bytes via NFS WRITE
The number of VFS calls (nfs_lookup in particular) is very high. The cache attribute invalidation is also abnormally high. The clients mostly uses the NFS mount to read data.