I run a moderately sized cluster and I am having a queue problem. For example I have a node with 128GB of memory and a program that I set the amount of memory per core. So for example if I set memory at 8GB a core I would only be able to run 16 cores before I hit virtual memory, which must be avoided. Right now I am running Torque as a queueing method and if I submit two 10 core jobs to a single node at 8GB each it will try to use 160GB of memory. What I want to be able to do is limit the amount of jobs on a node so that their total memory usage is only the physical memory of the node. The amount of memory per core is highly dynamic and a job can easily use all of the physical memory but only a fraction of the available cores.
It is my understanding from reading and experiment that the pmem,vmem,mem commands for Torque only work if a job exceeded these limits after they are running on the node and does not regulate these before the jobs go on. As a side note many of these jobs have periods where they only use ~10% of the memory needed later in the process.
So is there a way to do this with Torque. If not is there another queuing program that you would suggest?
Thank you for your time.