I have Sun Grid Engine set up (version 6.2u5) on a Ubuntu 10.10 server with 8 cores. In order to be able to reserve multiple slots, I have a parallel environment (PE) set up like this:

pe_name            serial
slots              999
user_lists         NONE
xuser_lists        NONE
start_proc_args    /bin/true
stop_proc_args     /bin/true
allocation_rule    $pe_slots
control_slaves     FALSE
job_is_first_task  TRUE
urgency_slots      min
accounting_summary FALSE

This is associated with the all.q on the server in question (let's call the server A). However, when I submit a job that uses 4 threads with e.g. qsub -q all.q@A -pe serial 4 mycmd.sh, it never gets scheduled, and I get the following reasoning from qstat:

cannot run in PE "serial" because it only offers 0 slots

Why is SGE saying "serial" only offers 0 slots, since there are 8 slots available on the server I specified (server A)?

The queue in question is configured thus (server names changed):

qname                 all.q
hostlist              @allhosts
seq_no                0
load_thresholds       np_load_avg=1.75
suspend_thresholds    NONE
nsuspend              1
suspend_interval      00:05:00
priority              0
min_cpu_interval      00:05:00
processors            UNDEFINED
qtype                 BATCH INTERACTIVE
ckpt_list             NONE
pe_list               make orte serial
rerun                 FALSE
slots                 1,[D=32],[C=8], \
tmpdir                /tmp
shell                 /bin/sh
prolog                NONE
epilog                NONE
shell_start_mode      posix_compliant
starter_method        NONE
suspend_method        NONE
resume_method         NONE
terminate_method      NONE
notify                00:00:60
owner_list            NONE
user_lists            NONE
xuser_lists           NONE
subordinate_list      NONE
complex_values        NONE
projects              NONE
xprojects             NONE
calendar              NONE
initial_state         default
s_rt                  INFINITY
h_rt                  08:00:00
s_cpu                 INFINITY
h_cpu                 INFINITY
s_fsize               INFINITY
h_fsize               INFINITY
s_data                INFINITY
h_data                INFINITY
s_stack               INFINITY
h_stack               INFINITY
s_core                INFINITY
h_core                INFINITY
s_rss                 INFINITY
h_rss                 INFINITY
s_vmem                INFINITY
h_vmem                INFINITY,[A=30g], \
  • is your "A" server also defined as execution host? – luxifer Oct 17 '11 at 7:12

The allocation rule $pe_slots requires you to submit jobs that are of the exact same size as the number of cores on any of your machines. Either change allocation rule to $fill_up or try submitting a job which requires 8, 30 or 32 cores.

  • The above statement is untrue. All an allocation_rule of $pe_slots does is force all requested slots to be allocated on a single host. I have a lot of jobs running on the cluster I admin that demonstrate this. – William Hay Oct 29 '13 at 8:54
  • It could be one or many hosts, but your job must fill up all cores on all machines. – Jimmy Hedman Oct 15 '14 at 9:08

I don't know the answer but you can get a lot more information on why grid engine isn't scheduling a job by running qalter -w p and qalter -w v. The only caveat is that,at least in some versions, qalter -w v doesn't ignore exclusive resources as it should.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.