We have an NFS mount and a PBS cluster in which all nodes mount that NFS mountpoint.
We are experiencing that a single process on one of the PBS cluster nodes can completely saturate and bog down the NFS mount on that node. This is a huge problem since PBS will usually schedule multiple jobs on the same node, the jobs only specify CPU and memory constraints but PBS doesn't seem to be concerned with IO constraints. This means that a single IO bound job that may not require much CPU and memory may render the whole node unusable, at least concerning the NFS mount.
I can reproduce this problem by simply running the following command on a node:
cat /nfsmount/verylargefile.txt > /dev/null
This will essentially prevent any other process on that node from accessing the NFS mount. Even doing a simple "ls /nfsmount" will take forever. Note that the performance of the same NFS mount point is not affected on other nodes so this seems to be a client side issue.
Does anybody have any experience with this type of behavior? Are there any pitfalls that I should be looking for? I would have thought there must be some logic that would try to distribute the "bandwidth" of an NFS mount if there are multiple processes accessing it to prevent this type of resource hogging by a single process.
Our PBS cluster is running RedHat 6.5 but I have reproduced this on my Ubuntu 14.04 workstation as well.