Questions tagged [slurm]

Slurm Workload Manager (formerly known as Simple Linux Utility for Resource Management or SLURM), or Slurm, is a free and open-source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and computer clusters.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Slurm Config for limit the endtime for all new jobs on the cluster to a certain date

Our cluster has to be shutdown for an update in two weeks. We would like to let users use the cluster until the last day, but we want to make sure, no job can be started, which would end after the ...
user avatar
0 votes
0 answers
186 views

slurm_load_partitions:Unable to contact slurm controller

I had this problem using the slurm command.Unable to contact slurm controller. this is part of my slurm.conf: ######################################## # YHPC NAME ######################################...
user avatar
  • 1
1 vote
1 answer
19 views

Computer cluster admin: how to limit users running program but permit file transferring

I am managing a small computer cluster with slurm on CentOS 7. I want to discourage users to run programs on login node. This can be achieved by adding user hard cpu 1 to file /etc/security/...
user avatar
  • 143
2 votes
1 answer
206 views

How can I set up interactive-job-only or batch-job-only partition on a SLURM cluster?

I'm managing a PBS/torque HPC cluster, and now I'm setting up another cluster with SLURM. On the PBS cluster, I can set a queue to accept only interactive jobs by qmgr -c "set queue interactive_q ...
user avatar
  • 143
0 votes
0 answers
137 views

Setting up slurm on a cluster

My IT admin has setup a cluster with 3 nodes, which is administered via Windows server. VMs are hosted via Hyper-V, including an Ubuntu VM to which a substantial portion of the cluster's resources ...
user avatar
1 vote
0 answers
629 views

Slurm srun cannot allocate ressources for GPUs - Invalid generic resource specification

I am able to launch a job on a GPU server the traditional way (using CPU and MEM as consumables): ~ srun -c 1 --mem 1M -w serverGpu1 hostname serverGpu1 but trying to use the GPUs will give an error: ...
user avatar
2 votes
0 answers
347 views

Managing SLURM memory on single node installation (issues)

I have SLURM setup on a single CentOS 7 node with 64 cores (128 CPU's). I have been using SLURM to submit jobs successfully using both srun and sbatch. However, it is with the caveat that I don't ...
user avatar
  • 71
0 votes
0 answers
69 views

Error with docker in a Slurm environment that uses LDAP as user management

I have the next environment: a cluster with a head node, which is used to access all the compute nodes and the storage nodes. Slurm is installed in the head node and all the compute nodes, where the ...
user avatar
0 votes
0 answers
21 views

Is there a way of killing user processes on machines they have no jobs on SLURM?

I'm starting to configure a small cluster using SLURM. One thing I've noticed is that users can still have running jobs on any machine if he follows these steps: Get a machine with salloc SSH into ...
user avatar
0 votes
0 answers
84 views

Some process are in unkillable sleeping state while i/o is low

I am the system administrator of an Arch Linux-based workstation. Our workstation uses Slurm as the load manager and consists of one master machine and 4 other computation nodes. In the past few ...
user avatar
0 votes
0 answers
17 views

no libslurmfull.so error after a newstart for all slurm commands

after having installed/configured slurm/nodes (in my case it is the same server) I can run jobs but when I restart the server I get: squeue: error while loading shared libraries: libslurmfull.so: ...
user avatar
0 votes
0 answers
56 views

Installing latest SLURM bundle (package) on Clear Linux fails on service start with symbol error

I would like to install SLURM on Clear Linux because of its good benchmarks. I have followed the tutorial at https://docs.01.org/clearlinux/latest/tutorials/hpc.html. When I got to the step of the ...
user avatar
1 vote
1 answer
239 views

slurm nvidia-docker ignores CUDA_VISIBLE_DEVICES

I have a problem running nvidia-docker containers on a slurm cluster. When inside the container all gpus are visible so basically it ignores the CUDA_VISIBLE_DEVICES set env by slurm. Outside the ...
user avatar
2 votes
1 answer
94 views

SLURM with "partial" head node

I am trying to install SLURM with NFS on a small ubuntu 18.04 HPC cluster, in a typical fashion, e.g. configure controller (slurmctld) and clients (slurmd) and shared directory, etc. What I am curious ...
user avatar
  • 123
0 votes
1 answer
128 views

Update SLURM node state prior/after playbook execution

I would like to automatically set the state of a node in a SLURM cluster before/after running my Ansible playbook (from idle to drained and after applying the playbook back to idle). The scontrol ...
user avatar
  • 121
0 votes
1 answer
51 views

Slurm cluster in Google Cloud - how do I attach a Filestore instance

If one clones the slurm-gcp project and deploys the stock cluster defined in there, things work well. What I would like to do, is to use a GCP Filestore instance to provide (more) persistent storage ...
user avatar
  • 111
1 vote
1 answer
979 views

Wrong LDAP user ID is mapped into Slurm account management service

I configured a Slurm head node as follows: sssd to contact openLDAP slurmctld/slurmdbd/slurmd/munged to act as the Slurm controller and compute node ...where ray.williams is an LDAP user. Its UID ...
user avatar
1 vote
0 answers
55 views

View/request instruction sets available on SGE host

How can I view or request hosts that can handle a particular instruction set in SGE? With Slurm, to view available instruction sets on each host I can use sinfo --Node -o '%n %f', and to submit a ...
user avatar
-1 votes
1 answer
166 views

Using HPC managers like Slurm on multiple servers in LAN [closed]

I have access to a group of servers connected with a 1Gb LAN, and each of them has 40+ cores and Ubuntu OS. They all have a common NAS. I installed SLURM on a few of them and configured it so that ...
user avatar
1 vote
0 answers
453 views

Slurmd remains inactive/failed on start

I currently have a cluster of 10 worker nodes managed by Slurm with 1 master node. I have previously successfully set up the cluster, after some teething problems, but managed to get it working. I put ...
user avatar
2 votes
1 answer
1k views

Slurm: "Connection refused" for certain sacctmgr commands

I have an existing slurm cluster up and running but as of today without a configuration change I get an error when I run certain sacctmgr commands and slurmdbd crashes: $ sacctmgr list associations ...
user avatar
  • 497
1 vote
0 answers
193 views

Slurm Error: “If using PrologFlag=Contain for pam_slurm_adopt, either proctrack/cgroup or proctrack/crau_aries is required.”

I'm using the flag x11 (PrologFlags=x11) in my slurm.conf file and jobs with x11 works perfectly, but I am getting this error every time I run a slurm command (e.g. sbatch, srun, sacctmgr): scontrol: ...
user avatar
1 vote
1 answer
379 views

Single-node SLURM server: restrict interactive CPU usage

I have SLURM setup on a single node, which is also a 'login node'. I would like to restrict interactive CPU usage, e.g. outside the scheduling system. I found the following article which suggests to ...
user avatar
  • 375
2 votes
2 answers
723 views

Query peak GPU memory used by finished job

I have a SLURM job I submit with sbatch, such as sbatch --gres gpu:Tesla-V100:1 job.sh job.sh trains a model on a V100 GPU. The code itself does not log GPU memory usage. Is there a SLURM command ...
user avatar
4 votes
1 answer
6k views

Why does Slurm fail to start with systemd but work when starting manually?

I've just set up slurm where one physical machine will be the only system in the cluster (so far). This is on Ubuntu 18.04. I have slurmdbd running, but when I attempt to start up slurmd and ...
user avatar
1 vote
0 answers
335 views

i try to srun /bin/hostname. slurmctld not respones

I have Master Node (Ubuntu 18.04) and Two Compute Node (Ubuntu 18.04) There is no problem with the connection Munge is good, I try to command (sinfo, scontrol show nodes).. that is no problem I try ...
user avatar
1 vote
0 answers
4k views

slurm service running failed again. i don't know why

I have one master node and two slave nodes. One slave node connects successfully but one node connection failed. Each node has 18.04 Ubuntu and 17.11 Slurm If running to systemctl status slurmd....
user avatar
0 votes
1 answer
158 views

slurm salloc and how it get user login

Some theoretical question. I understand that the better way to know is to look at the code, but maybe I can do some cheat and just ask about it? I wonder that after salloc user can log in to the node....
user avatar
2 votes
0 answers
20 views

Slurm - Does it maintain ccNUMA?

Does a SLURM cluster control, maintain or enforce Cache Coherence across the Nodes? Is it a configuration property, or does something like this not exist? I can't find anything inside the docs.
user avatar
  • 271
3 votes
1 answer
8k views

Unable to contact slurm controller

I followed the steps to troubleshoot here: https://slurm.schedmd.com/troubleshoot.html. When running scontrol show slurmd, I get: Active Steps = NONE Actual CPUs = 1 Actual ...
user avatar
8 votes
1 answer
9k views

Slurm node daemon error: Can't open PID file

I run systemctl start slurmd.service, and it times out: Job for slurmd.service failed because a timeout was exceeded. The relevant lines from running systemctl status slurmd.service: Mar 23 17:13:...
user avatar
0 votes
1 answer
243 views

Trouble Installing slurm on Fedora 29

When I run slurmd, it gives a -bash: slurmd: command not found. I ran sudo yum install slurm to install slurm. I don't know why it isn't working, or if I installed all the required packages for slurm....
user avatar
0 votes
1 answer
585 views

What does "CPU Minutes" mean exactly?

I'm actually trying to report cluster utilization in Slurm but i don't understand the metric CPU Minutes. [root@XXXX]# sreport cluster Utilization Start=2018-12-01 End=2018-12-31 ---------------------...
user avatar
  • 3
0 votes
1 answer
199 views

Allow other users to cancel jobs

I have a test cluster with Slurm in which I would like that other users where able to cancel other users' jobs. By default, the users are able to cancel their own jobs. How can I define several ...
user avatar
2 votes
1 answer
2k views

slurmdbd fails to start (initial installation)

I tried to install slurmdbd for accounting on a Ubuntu 16.04 from the standard repositories (version: 15.08.7-1build1). Here are the commands: $ sudo apt-get install mysql-server $ sudo mysql > ...
user avatar
  • 497
0 votes
0 answers
670 views

SLURM, SSH, adn NOHUP Behaviour

I am the administrator of a cluster running on CentOS and using SLURM to send jobs from a login node to compute nodes. Recently, a user complained about some unexpected behaviour with their jobs. If a ...
user avatar
3 votes
3 answers
795 views

Randomize Slurm Node Allocation

Has anyone had luck randomizing Slurm node allocations? We have a small cluster of 12 nodes that could be used by anywhere from 1-8 people at a time with jobs of various size/length. When testing our ...
user avatar
  • 31
1 vote
1 answer
1k views

How do I prevent additional jobs from a given user from starting?

With the Slurm workload manager how can I prevent more jobs from user bob from starting? Existing jobs should continue to run. The user should be able to submit more jobs but they shouldn't be able to ...
user avatar
1 vote
1 answer
1k views

How to upgrade Slurm?

I've been asked to upgrade our Slurm Workload Manager installation. I have a slurm 2.3.4 on a Debian 7.0 wheezy cluster (1 master + 8 nodes). I've not installed it so I'm a bit confused about how to ...
user avatar
1 vote
0 answers
92 views

program on cluster exceeds RSS memory limit

I have been trying to run a python script on a computer cluster but keep running into a error saying that RSS memory limit exceeded. I am using this program to analyse a data set consisting of around ...
user avatar
  • 111
1 vote
0 answers
94 views

Slurm not filtering sacct results by date

We're using Slurm as a resource manager on our Beowulf cluster, so I installed Slurm on my workstation to test out my scripts before I submit them to the cluster. When I try to list old jobs on my ...
user avatar
  • 1,124