We have an NFS mount and a PBS cluster in which all nodes mount that NFS mountpoint.

We are experiencing that a single process on one of the PBS cluster nodes can completely saturate and bog down the NFS mount on that node. This is a huge problem since PBS will usually schedule multiple jobs on the same node, the jobs only specify CPU and memory constraints but PBS doesn't seem to be concerned with IO constraints. This means that a single IO bound job that may not require much CPU and memory may render the whole node unusable, at least concerning the NFS mount.

I can reproduce this problem by simply running the following command on a node:

cat /nfsmount/verylargefile.txt > /dev/null

This will essentially prevent any other process on that node from accessing the NFS mount. Even doing a simple "ls /nfsmount" will take forever. Note that the performance of the same NFS mount point is not affected on other nodes so this seems to be a client side issue.

Does anybody have any experience with this type of behavior? Are there any pitfalls that I should be looking for? I would have thought there must be some logic that would try to distribute the "bandwidth" of an NFS mount if there are multiple processes accessing it to prevent this type of resource hogging by a single process.

Our PBS cluster is running RedHat 6.5 but I have reproduced this on my Ubuntu 14.04 workstation as well.

  • Linux's nfs client uses only a single connection to the server and doesn't send a new request as long as reply for last one not arrived. You can try to mount with smaller rsize/wsize and and shorter timeout. – kofemann Apr 26 '14 at 13:06
  • Thank you for the comment. I had not realized there is a single "global" connection but I guess it makes some sense though. It just strikes me as a bit odd that there isn't a better resource allocation mechanism that would make sure that processes don't starve. I tried messing with the *size parameters and timeo but it didn't have much effect. – StFS Apr 29 '14 at 14:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.