We are using Lustre in a cluster with approximately 200TB of storage, 12 Object Storage Targets (that connect to a DDN storage system using QDR Infiniband), and roughly 160 quad and 8-core compute notes. Most of the users of this system have no problems at all, but my tasks are I/O intensive. When I run an array job that has 250-500 processes that are simultaneously pounding the file system typically between 10 and 20 of my processes will fail. The log files indicate that the load on the OSTs are going over 2 and that the Lustre client is returning either bad data or failed
read() function calls.
Currently the only way we have of resolving my problem is to run fewer simultaneous jobs. This is unsatisfactory, because there is no way to know in advance if my workload will be CPU-heavy or I/O heavy. Besides, just turning down the load isn't the way to run a supercomptuer: we would like it to run slower when running under load, not produce incorrect answers.
I'd like to know how to configure Lustre so that clients block when the load on the OSTs goes too high, rather than having the clients get bad data.
How do I configure Lustre to make the clients block?