System is running CentOS 7.3, serving files over both ethernet and infiniband, but the vast majority of traffic is over infiniband

I just stood up a new file server that is now generating large numbers of the following error:

[2007827.701033] rpc-srv/tcp: nfsd: sent only 618872 when sending 1048708 bytes - shutting down socket
[2007835.313884] rpc-srv/tcp: nfsd: sent only 75908 when sending 1048708 bytes - shutting down socket

I increased the tcp_rmem and wmem to a max of 32mb, but the problem persists

net.ipv4.tcp_rmem = 4096 87380 33554432 
net.ipv4.tcp_wmem = 4096 65536 33554432

Are there any other ways I could tune this to get around the issue?


This site reports some more suggestions, including increasing the NIC txqueuelen. These suggestions are mainly aimed at high latency networks, however I noticed that under Fedora24 Infiniband interfaces get by default a lower txqueuelen value (256) than other interfaces (1000), so the txquelen suggestion, besides the others, may also make sense in this case.

The modern way to do so is actually:

ip link set ib0 txqueuelen 10000

After following the suggestions on the indicated page, nfs-on-Infiniband errors of the type you reported have ceased in my nfs server.

| improve this answer | |
  • 1
    That's a nice first answer. Welcome to ServerFault. – Andrew Schulman Feb 7 '18 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.