Questions tagged [slurm]

Slurm Workload Manager (formerly known as Simple Linux Utility for Resource Management or SLURM), or Slurm, is a free and open-source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and computer clusters.

Filter by
Sorted by
Tagged with
0
votes
0answers
55 views

slurm nvidia-docker ignores CUDA_VISIBLE_DEVICES

I have a problem running nvidia-docker containers on a slurm cluster. When inside the container all gpus are visible so basically it ignores the CUDA_VISIBLE_DEVICES set env by slurm. Outside the ...
0
votes
0answers
7 views

Does deleted Slurm users' recorded usage disappear from usage statistics?

I am part of a team managing a Slurm cluster. We have a number of users in Slurm organised in a number of accounts and sub-accounts. Some of these Slurm users are "dead" users, who do not ...
1
vote
0answers
21 views

SLURM with “partial” head node

I am trying to install SLURM with NFS on a small ubuntu 18.04 HPC cluster, in a typical fashion, e.g. configure controller (slurmctld) and clients (slurmd) and shared directory, etc. What I am curious ...
0
votes
1answer
59 views

Update SLURM node state prior/after playbook execution

I would like to automatically set the state of a node in a SLURM cluster before/after running my Ansible playbook (from idle to drained and after applying the playbook back to idle). The scontrol ...
0
votes
1answer
25 views

Slurm cluster in Google Cloud - how do I attach a Filestore instance

If one clones the slurm-gcp project and deploys the stock cluster defined in there, things work well. What I would like to do, is to use a GCP Filestore instance to provide (more) persistent storage ...
0
votes
1answer
359 views

Wrong LDAP user ID is mapped into Slurm account management service

I configured a Slurm head node as follows: sssd to contact openLDAP slurmctld/slurmdbd/slurmd/munged to act as the Slurm controller and compute node ...where ray.williams is an LDAP user. Its UID ...
1
vote
0answers
53 views

View/request instruction sets available on SGE host

How can I view or request hosts that can handle a particular instruction set in SGE? With Slurm, to view available instruction sets on each host I can use sinfo --Node -o '%n %f', and to submit a ...
-1
votes
1answer
83 views

Using HPC managers like Slurm on multiple servers in LAN [closed]

I have access to a group of servers connected with a 1Gb LAN, and each of them has 40+ cores and Ubuntu OS. They all have a common NAS. I installed SLURM on a few of them and configured it so that ...
1
vote
0answers
239 views

Slurmd remains inactive/failed on start

I currently have a cluster of 10 worker nodes managed by Slurm with 1 master node. I have previously successfully set up the cluster, after some teething problems, but managed to get it working. I put ...
2
votes
1answer
602 views

Slurm: “Connection refused” for certain sacctmgr commands

I have an existing slurm cluster up and running but as of today without a configuration change I get an error when I run certain sacctmgr commands and slurmdbd crashes: $ sacctmgr list associations ...
1
vote
0answers
84 views

Slurm Error: “If using PrologFlag=Contain for pam_slurm_adopt, either proctrack/cgroup or proctrack/crau_aries is required.”

I'm using the flag x11 (PrologFlags=x11) in my slurm.conf file and jobs with x11 works perfectly, but I am getting this error every time I run a slurm command (e.g. sbatch, srun, sacctmgr): scontrol: ...
1
vote
1answer
199 views

Single-node SLURM server: restrict interactive CPU usage

I have SLURM setup on a single node, which is also a 'login node'. I would like to restrict interactive CPU usage, e.g. outside the scheduling system. I found the following article which suggests to ...
1
vote
2answers
340 views

Query peak GPU memory used by finished job

I have a SLURM job I submit with sbatch, such as sbatch --gres gpu:Tesla-V100:1 job.sh job.sh trains a model on a V100 GPU. The code itself does not log GPU memory usage. Is there a SLURM command ...
3
votes
1answer
2k views

Why does Slurm fail to start with systemd but work when starting manually?

I've just set up slurm where one physical machine will be the only system in the cluster (so far). This is on Ubuntu 18.04. I have slurmdbd running, but when I attempt to start up slurmd and ...
1
vote
0answers
208 views

i try to srun /bin/hostname. slurmctld not respones

I have Master Node (Ubuntu 18.04) and Two Compute Node (Ubuntu 18.04) There is no problem with the connection Munge is good, I try to command (sinfo, scontrol show nodes).. that is no problem I try ...
1
vote
0answers
2k views

slurm service running failed again. i don't know why

I have one master node and two slave nodes. One slave node connects successfully but one node connection failed. Each node has 18.04 Ubuntu and 17.11 Slurm If running to systemctl status slurmd....
0
votes
1answer
105 views

slurm salloc and how it get user login

Some theoretical question. I understand that the better way to know is to look at the code, but maybe I can do some cheat and just ask about it? I wonder that after salloc user can log in to the node....
2
votes
0answers
18 views

Slurm - Does it maintain ccNUMA?

Does a SLURM cluster control, maintain or enforce Cache Coherence across the Nodes? Is it a configuration property, or does something like this not exist? I can't find anything inside the docs.
2
votes
1answer
3k views

Unable to contact slurm controller

I followed the steps to troubleshoot here: https://slurm.schedmd.com/troubleshoot.html. When running scontrol show slurmd, I get: Active Steps = NONE Actual CPUs = 1 Actual ...
6
votes
1answer
7k views

Slurm node daemon error: Can't open PID file

I run systemctl start slurmd.service, and it times out: Job for slurmd.service failed because a timeout was exceeded. The relevant lines from running systemctl status slurmd.service: Mar 23 17:13:...
0
votes
1answer
188 views

Trouble Installing slurm on Fedora 29

When I run slurmd, it gives a -bash: slurmd: command not found. I ran sudo yum install slurm to install slurm. I don't know why it isn't working, or if I installed all the required packages for slurm....
0
votes
1answer
323 views

What does “CPU Minutes” mean exactly?

I'm actually trying to report cluster utilization in Slurm but i don't understand the metric CPU Minutes. [root@XXXX]# sreport cluster Utilization Start=2018-12-01 End=2018-12-31 ---------------------...
0
votes
1answer
97 views

Allow other users to cancel jobs

I have a test cluster with Slurm in which I would like that other users where able to cancel other users' jobs. By default, the users are able to cancel their own jobs. How can I define several ...
2
votes
1answer
2k views

slurmdbd fails to start (initial installation)

I tried to install slurmdbd for accounting on a Ubuntu 16.04 from the standard repositories (version: 15.08.7-1build1). Here are the commands: $ sudo apt-get install mysql-server $ sudo mysql > ...
0
votes
0answers
498 views

SLURM, SSH, adn NOHUP Behaviour

I am the administrator of a cluster running on CentOS and using SLURM to send jobs from a login node to compute nodes. Recently, a user complained about some unexpected behaviour with their jobs. If a ...
3
votes
3answers
642 views

Randomize Slurm Node Allocation

Has anyone had luck randomizing Slurm node allocations? We have a small cluster of 12 nodes that could be used by anywhere from 1-8 people at a time with jobs of various size/length. When testing our ...
1
vote
1answer
817 views

How do I prevent additional jobs from a given user from starting?

With the Slurm workload manager how can I prevent more jobs from user bob from starting? Existing jobs should continue to run. The user should be able to submit more jobs but they shouldn't be able to ...
1
vote
1answer
777 views

How to upgrade Slurm?

I've been asked to upgrade our Slurm Workload Manager installation. I have a slurm 2.3.4 on a Debian 7.0 wheezy cluster (1 master + 8 nodes). I've not installed it so I'm a bit confused about how to ...
1
vote
0answers
53 views

program on cluster exceeds RSS memory limit

I have been trying to run a python script on a computer cluster but keep running into a error saying that RSS memory limit exceeded. I am using this program to analyse a data set consisting of around ...
1
vote
0answers
70 views

Slurm not filtering sacct results by date

We're using Slurm as a resource manager on our Beowulf cluster, so I installed Slurm on my workstation to test out my scripts before I submit them to the cluster. When I try to list old jobs on my ...