parallel environment, grid mis-computing memory consumption by dis regarding shared memory of children processes?

Question

We have an application that takes large amount of RAM for execution. To execute in parallel, this application also forks out copies of itself that perform multiple dedicated tasks.

During fork, we specify properties, so that the child processes share the virtual memory space and all run on the same host. DUe to usage of shared memory, we have seen that the overhead of memory for running upto 10 child processes is just about 50%. (i.e., if sequential run of the application took X amount of memory, the peak usage with a parallel run of the application with up to 10 child processes is just 1.5 X).

Now we are trying to build "parallel environment" with sun grid to submit/dispatch jobs of this application. For jobs submission, we use "mem_free" as a consumable resource and thus the job starts only on a host with sufficient memory and also specified memory is reserved on the host while the job is running. We also specify h_vmem parameter to stop the process if it oversubscribes the memory. To account for overhead of child processes, we submit sun grid submission with 1.5X of desired memory.

However, it seems that grid mis-computes the memory consumption by the application and computes the memory by adding up the memory consumption by the child processes (thus dis-regarding the shared memory used by the children processes) and thus at some point, considers the job to have oversubscribe the memory usage and kills it.

Has anybody run into similar isue and have any suggestion as to how to modify the parallel environment settings to avoid this kind of the issue?

Answer 1

GridEngine measures the virtual memory usage by checking the memory statistics given by the OS. If you are on Linux then you can see a process's virtual memory usage in /proc/<pid>/status in the row that says VmSize.

Unfortunately this measure includes the total virtual size of the process, including and shared pages with a fork parent and any shared libraries and so will generally tend to be much higher than the physical memory usage, VmRSS.

What this means is that under grid engine if you are submitting a process that uses 1GB of virtual memory which is then going to fork 3 child processes, you need to request at least 4GB of the h_vmem resource.