I have a weird issue with one of the NFS clients (Ubuntu 16.04 LTS). I have been working for the past few days to try and debug this, but so far without success. After I mount the partition, everything works perfectly for a few days, transfer speed is 1 Gbps between the client and server. After a few days the speed drops to less than 10 mbps and even a simple directory listing takes a few seconds and I/O wait is at 100%

What I noticed is that the backlog wait, especially for the write operations is extremely high:

root@srv:~# mountstats /mnt/data
Stats for mounted on /mnt/data:
  NFS mount options: rw,vers=4.0,rsize=16384,wsize=16384,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=,local_lock=none
  NFS server capabilities: caps=0xffdf,wtmult=512,dtsize=16384,bsize=0,namlen=255
  NFSv4 capability flags: bm0=0xfdffbfff,bm1=0xf9be3e,bm2=0x0,acl=0x3,pnfs=notconfigured
  NFS security flavor: 1  pseudoflavor: 0

NFS byte counts:
  applications read 8168679407142 bytes via read(2)
  applications wrote 4833000353435 bytes via write(2)
  applications read 0 bytes via O_DIRECT read(2)
  applications wrote 0 bytes via O_DIRECT write(2)
  client read 4218977852758 bytes via NFS READ
  client wrote 4832098253207 bytes via NFS WRITE

RPC statistics:
  561421762 RPC requests sent, 561421608 RPC replies received (1 XIDs not found)
  average backlog queue length: 0

        263822474 ops (46%)     0 retrans (0%)  0 major timeouts
        avg bytes sent per op: 184      avg bytes received per op: 16051
        backlog wait: 8.772689  RTT: 27.972131  total execute time: 36.752241 (milliseconds)
        295296111 ops (52%)     0 retrans (0%)  0 major timeouts
        avg bytes sent per op: 16567    avg bytes received per op: 132
        backlog wait: 62468603019.791718        RTT: 78.030143  total execute time: 62468603097.830574 (milliseconds)

There are no errors, no warnings, I tried debugging with "echo 1 > /proc/sys/vm/block_dump" (which has done miracles for me in the past), but this time there's nothing visible related to NFS. Any idea how I can debug further and see what's causing the extremely high backlog wait?


Just in case anyone has the same problem, the only fix I could find was to force NFS3 instead NFS4. The problem has disappeared now.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.