Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I want to be able to set up memory requirements per job.

For instance: to run 5 jobs, for which I know each will need 4 GB memory. I have 16 GB RAM on the Ubuntu server and 16 GB swap. I want to avoid using the swap. Can I do something like: qsub -l mem_required_for_my_job=4G job1; qsub -l mem_required_for_my_job=4G job2; qsub -l mem_required_for_my_job=4G job3; qsub -l mem_required_for_my_job=4G job4; qsub -l mem_required_for_my_job=4G job5? The jobs will require 4 GB at some moment, but not in the beginning.

How to tell SGE what my requirements are? How to avoid scheduling 5 x 4 GB when only 16 GB available?

I read the user guide and tried s_vmem, h_vmem, mem_free, mem_used. None of them is what I want. I do not want my jobs to get killed in the middle of the processing. I want them not to be scheduled, unless the maximum resources needed are available.

Can I do this?

Thank you all!

share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

On our cluster we use h_vmem to enforce job memory allocation.

The thing you appear to be missing is setting the available amount as a consumable resource. In qconf -mc or in the qmon complexes dialog you need to set the resource as requestable and consumable.

Then, on each host with qconf -me you need to set the amount of available memory in complex_values.

For example we have host definitions that look like:

hostname              node004
load_scaling          NONE
complex_values        h_vmem=30.9G,exclusive=1,version=3.3.0
share|improve this answer
    
I tried with h_vmem, virtual_free and quotas. None of them carry the task in the way I want it. h_vmem protects the system, killing the jobs if they overcome their requirements. Both quota and virtual_free didn't do anything to prevent the memory exhausting. The OS killed the job in this case. I want to avoid killing the jobs. I would like to make them stay in the queue, waiting for their requested resources to be available. Is this possible? –  grs Mar 2 '11 at 22:28
    
I'm not sure what you mean. With the h_vmem limit, GE will only run the job on the node if it has enough of the h_vmem resource available to run it. It also sets the ulimit on virtual memory. If the job tries to allocate more than the limit, it will be killed by the OS. –  Kamil Kisiel Mar 3 '11 at 1:06
1  
GridEngine will run exactly as many jobs as match the available complex resources. So if you define a node to have 16GB of h_vmem available, and you submit 5 jobs requesting 4GB, GridEngine will only place 4 of them on the node at once. Your job should always request the peak amount it expects to use. If you have swap space on your nodes, you can include that amount in the h_vmem complex value. –  Kamil Kisiel Mar 3 '11 at 2:22
1  
The arguments are the same as the name of the complex. So if you have h_vmem configured as a consumable resource and under a host's complex_values, your command would look like: qsub -l h_vmem=4G job1. –  Kamil Kisiel Mar 3 '11 at 6:21
2  
Also here's a quick blog post from gridengine.info regarding memory limits: gridengine.info/2009/12/01/… –  Kamil Kisiel Mar 3 '11 at 6:22
show 3 more comments

I think the what you are looking for is virtual_free

"If a job request for virtual_free exceeds the available amount, the job is not dispatched to a queue on that host." from http://wikis.sun.com/display/GridEngine/Configuring+Complex+Resource+Attributes

See also http://markmail.org/message/2iw4esthqvs5rc2a

share|improve this answer
    
Please check the comment on the other answer. –  grs Mar 2 '11 at 22:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.