Questions tagged [slurm]

Slurm Workload Manager (formerly known as Simple Linux Utility for Resource Management or SLURM), or Slurm, is a free and open-source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and computer clusters.

Filter by
Sorted by
Tagged with
0
votes
0answers
15 views

Can lower priority slurm jobs push back higher priority job start times if submitted to a different but 100% overlapping partition?

TL;DR: If all jobs in front of mine on Wednesday were submitted before mine, if my start time was pushed back roughly another 24 hours when I looked on Thursday, and if all jobs in front of mine are ...
1
vote
0answers
16 views

Slurm Error: “If using PrologFlag=Contain for pam_slurm_adopt, either proctrack/cgroup or proctrack/crau_aries is required.”

I'm using the flag x11 (PrologFlags=x11) in my slurm.conf file and jobs with x11 works perfectly, but I am getting this error every time I run a slurm command (e.g. sbatch, srun, sacctmgr): scontrol: ...
0
votes
0answers
14 views

The Grafana graph stops updating

Apr 1 13:10:01 grid CRON[9839]: (root) CMD (/usr/sbin/icsisnap /var/lib/icsisnap) Apr 1 13:12:00 grid telegraf[1172]: 2020-04-01T20:12:00Z E! [inputs.exec]: Error in plugin: exec: exit status 1 for ...
0
votes
0answers
28 views

Single-node SLURM server: restrict interactive CPU usage

I have SLURM setup on a single node, which is also a 'login node'. I would like to restrict interactive CPU usage, e.g. outside the scheduling system. I found the following article which suggests to ...
1
vote
2answers
46 views

Query peak GPU memory used by finished job

I have a SLURM job I submit with sbatch, such as sbatch --gres gpu:Tesla-V100:1 job.sh job.sh trains a model on a V100 GPU. The code itself does not log GPU memory usage. Is there a SLURM command ...
1
vote
0answers
131 views

Why does Slurm fail to start with systemd but work when starting manually?

I've just set up slurm where one physical machine will be the only system in the cluster (so far). This is on Ubuntu 18.04. I have slurmdbd running, but when I attempt to start up slurmd and ...
0
votes
0answers
35 views

How store the result of the operation in each node

I used to SLURM My SLURM configuration is One Master Node , Two Compute Node I want to Store the result of the operation in each node. So I try to this Script #! /bin/bash #SBATCH -J testing #...
1
vote
0answers
115 views

i try to srun /bin/hostname. slurmctld not respones

I have Master Node (Ubuntu 18.04) and Two Compute Node (Ubuntu 18.04) There is no problem with the connection Munge is good, I try to command (sinfo, scontrol show nodes).. that is no problem I try ...
1
vote
0answers
547 views

slurm service running failed again. i don't know why

I have one master node and two slave nodes. One slave node connects successfully but one node connection failed. Each node has 18.04 Ubuntu and 17.11 Slurm If running to systemctl status slurmd....
0
votes
0answers
15 views

Is there a way to make Slurm pull nodes from multiple partitions at once?

I'm trying to make a Slurm job utilize compute nodes from two partitions at once. One partition has less nodes than the other, and can only be used by certain users. I'd like the job to use the nodes ...
0
votes
0answers
100 views

Installing openmpi with slurm - ofi error

I am having trouble understanding why these errors appears. I do not understand what ofi is doing, something with the network communication of openmpi I guess? Error 1 (from stderr): ----------------...
0
votes
1answer
90 views

slurm salloc and how it get user login

Some theoretical question. I understand that the better way to know is to look at the code, but maybe I can do some cheat and just ask about it? I wonder that after salloc user can log in to the node....
2
votes
0answers
16 views

Slurm - Does it maintain ccNUMA?

Does a SLURM cluster control, maintain or enforce Cache Coherence across the Nodes? Is it a configuration property, or does something like this not exist? I can't find anything inside the docs.
1
vote
0answers
1k views

Unable to contact slurm controller

I followed the steps to troubleshoot here: https://slurm.schedmd.com/troubleshoot.html. When running scontrol show slurmd, I get: Active Steps = NONE Actual CPUs = 1 Actual ...
3
votes
1answer
3k views

Slurm node daemon error: Can't open PID file

I run systemctl start slurmd.service, and it times out: Job for slurmd.service failed because a timeout was exceeded. The relevant lines from running systemctl status slurmd.service: Mar 23 17:13:...
0
votes
1answer
123 views

Trouble Installing slurm on Fedora 29

When I run slurmd, it gives a -bash: slurmd: command not found. I ran sudo yum install slurm to install slurm. I don't know why it isn't working, or if I installed all the required packages for slurm....
0
votes
1answer
165 views

What does “CPU Minutes” mean exactly?

I'm actually trying to report cluster utilization in Slurm but i don't understand the metric CPU Minutes. [root@XXXX]# sreport cluster Utilization Start=2018-12-01 End=2018-12-31 ---------------------...
0
votes
1answer
56 views

Allow other users to cancel jobs

I have a test cluster with Slurm in which I would like that other users where able to cancel other users' jobs. By default, the users are able to cancel their own jobs. How can I define several ...
2
votes
1answer
1k views

slurmdbd fails to start (initial installation)

I tried to install slurmdbd for accounting on a Ubuntu 16.04 from the standard repositories (version: 15.08.7-1build1). Here are the commands: $ sudo apt-get install mysql-server $ sudo mysql > ...
0
votes
0answers
373 views

SLURM, SSH, adn NOHUP Behaviour

I am the administrator of a cluster running on CentOS and using SLURM to send jobs from a login node to compute nodes. Recently, a user complained about some unexpected behaviour with their jobs. If a ...
2
votes
3answers
386 views

Randomize Slurm Node Allocation

Has anyone had luck randomizing Slurm node allocations? We have a small cluster of 12 nodes that could be used by anywhere from 1-8 people at a time with jobs of various size/length. When testing our ...
1
vote
1answer
536 views

How do I prevent additional jobs from a given user from starting?

With the Slurm workload manager how can I prevent more jobs from user bob from starting? Existing jobs should continue to run. The user should be able to submit more jobs but they shouldn't be able to ...
1
vote
1answer
574 views

How to upgrade Slurm?

I've been asked to upgrade our Slurm Workload Manager installation. I have a slurm 2.3.4 on a Debian 7.0 wheezy cluster (1 master + 8 nodes). I've not installed it so I'm a bit confused about how to ...
1
vote
0answers
46 views

program on cluster exceeds RSS memory limit

I have been trying to run a python script on a computer cluster but keep running into a error saying that RSS memory limit exceeded. I am using this program to analyse a data set consisting of around ...
1
vote
0answers
53 views

Slurm not filtering sacct results by date

We're using Slurm as a resource manager on our Beowulf cluster, so I installed Slurm on my workstation to test out my scripts before I submit them to the cluster. When I try to list old jobs on my ...