We have an application that takes large amount of RAM for execution. To execute in parallel, this application also forks out copies of itself that perform multiple dedicated tasks.
During fork, we specify properties, so that the child processes share the virtual memory space and all run on the same host. DUe to usage of shared memory, we have seen that the overhead of memory for running upto 10 child processes is just about 50%. (i.e., if sequential run of the application took X amount of memory, the peak usage with a parallel run of the application with up to 10 child processes is just 1.5 X).
Now we are trying to build "parallel environment" with sun grid to submit/dispatch jobs of this application. For jobs submission, we use "mem_free" as a consumable resource and thus the job starts only on a host with sufficient memory and also specified memory is reserved on the host while the job is running. We also specify h_vmem parameter to stop the process if it oversubscribes the memory. To account for overhead of child processes, we submit sun grid submission with 1.5X of desired memory.
However, it seems that grid mis-computes the memory consumption by the application and computes the memory by adding up the memory consumption by the child processes (thus dis-regarding the shared memory used by the children processes) and thus at some point, considers the job to have oversubscribe the memory usage and kills it.
Has anybody run into similar isue and have any suggestion as to how to modify the parallel environment settings to avoid this kind of the issue?