Questions tagged [slurm]
Slurm Workload Manager (formerly known as Simple Linux Utility for Resource Management or SLURM), or Slurm, is a free and open-source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and computer clusters.
48
questions
0
votes
0
answers
16
views
End user reports slurm jobs suddenly died with little in logs to find out why
I got a bunch of messages in slurm with the following:
[2023-11-16T10:03:53.952] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=50461580 uid 1900007651
[2023-11-16T10:03:53.958] _slurm_rpc_kill_job: ...
0
votes
1
answer
205
views
SLURM able to send CPU, but not GPU jobs
I've built out a slurm "cluster" (just a job server and single compute server at the moment) and am trying to run jobs on it. I can run CPU jobs just fine, it sends them to the machine and ...
- 1
1
vote
1
answer
32
views
Can I release stale allocated GRES on a Slurm node?
Is there any way to clear stale allocated GRES in Slurm?
I have one node where 4 GPUs are allocated while no jobs are running on the node. Rebooting the node does not release the GPUs.
user@control1:~$...
- 24.3k
0
votes
1
answer
67
views
Why does the login node connect to external networks but allocated compute node fail in Slurm-GCP?
I've noticed that connecting to the internet from the allocated compute node via Slurm-GCP keeps failing. For example, using wget from the login node works successfully:
[me@gcp-login0 ~]$ wget https:/...
google-cloud-platform
0
votes
0
answers
152
views
SLURM / NFS based computing cluster with disk uniterruptible sleep issues (state : D)
Context :
We have a computing cluster based on 7 servers, running Debian 11:
a storage (HDD NAS, ~500TB, RAID5, LVM)
a frontal server, running SLURM, nfs-common
5 nodes on which the storage is ...
0
votes
1
answer
143
views
Setting up slurm with 2 different nodes and 2 different partation on 1 physical server
I have a requirement for setting up slurm on one physical server, with 2 different partation and 2 main node so, need to have,
partation1 wihich need to have node1 and need to be used by group1 user
...
- 5
0
votes
1
answer
456
views
Docker Compose + Ubuntu:22.04-Unable To Create cgroup...Read-only file system
I'm playing around with Docker Desktop(4.16.3) and Slurm. When I run slurmd, I get an error with the following complaint:
common_cgroup_instantiate: unable to create cgroup '/sys/fs/cgroup/freezer/...
- 533
0
votes
0
answers
536
views
Use more CPU cores using code in PyTorch that also uses GPU
I'm trying to successfully run code in PyTorch that uses DataLoader. It is possible to configure the DataLoader to load data using several processes (which speeds up data loading a lot), via the use ...
- 1
0
votes
0
answers
268
views
Troubleshooting slurm e-mail settings
I am trying to setup a slurm installation and I have advanced towards the e-mail stage. So far I do not receive any mails. I have a working setup using msmtp-mta and msmtp. When I batch a script the ...
- 101
0
votes
0
answers
131
views
Slurm jobs undesirably get access to all threads
I have one Ryzen R9 5950x CPU (16 cores/32 threads), one Xeon Phi 7120p card and partition/node in slurm.conf defined as:
NodeName=mic0 RealMemory=15000 Sockets=1 CoresPerSocket=61 ThreadsPerCore=4 ...
- 101
1
vote
1
answer
1k
views
ssh directly into a specific node on a cluster, without first ssh into login node?
I usually log on to a cluster, start a slurm interactive job, then I am able to ssh into specific running nodes.
My questions is, is it generally possible to ssh into a specific node from my local ...
- 111
2
votes
1
answer
56
views
Slurm Config for limit the endtime for all new jobs on the cluster to a certain date
Our cluster has to be shutdown for an update in two weeks. We would like to let users use the cluster until the last day, but we want to make sure, no job can be started, which would end after the ...
- 123
0
votes
0
answers
1k
views
slurm_load_partitions:Unable to contact slurm controller
I had this problem using the slurm command.Unable to contact slurm controller.
this is part of my slurm.conf:
########################################
# YHPC NAME
######################################...
- 1
1
vote
1
answer
36
views
Computer cluster admin: how to limit users running program but permit file transferring
I am managing a small computer cluster with slurm on CentOS 7. I want to discourage users to run programs on login node. This can be achieved by adding user hard cpu 1 to file /etc/security/...
- 153
2
votes
1
answer
538
views
How can I set up interactive-job-only or batch-job-only partition on a SLURM cluster?
I'm managing a PBS/torque HPC cluster, and now I'm setting up another cluster with SLURM. On the PBS cluster, I can set a queue to accept only interactive jobs by qmgr -c "set queue interactive_q ...
- 153
0
votes
0
answers
715
views
Setting up slurm on a cluster
My IT admin has setup a cluster with 3 nodes, which is administered via Windows server. VMs are hosted via Hyper-V, including an Ubuntu VM to which a substantial portion of the cluster's resources ...
2
votes
0
answers
4k
views
Slurm srun cannot allocate ressources for GPUs - Invalid generic resource specification
I am able to launch a job on a GPU server the traditional way (using CPU and MEM as consumables):
~ srun -c 1 --mem 1M -w serverGpu1 hostname
serverGpu1
but trying to use the GPUs will give an error:
...
- 121
3
votes
0
answers
1k
views
Managing SLURM memory on single node installation (issues)
I have SLURM setup on a single CentOS 7 node with 64 cores (128 CPU's). I have been using SLURM to submit jobs successfully using both srun and sbatch. However, it is with the caveat that I don't ...
- 81
0
votes
0
answers
282
views
Some process are in unkillable sleeping state while i/o is low
I am the system administrator of an Arch Linux-based workstation. Our workstation uses Slurm as the load manager and consists of one master machine and 4 other computation nodes. In the past few ...
1
vote
1
answer
546
views
slurm nvidia-docker ignores CUDA_VISIBLE_DEVICES
I have a problem running nvidia-docker containers on a slurm cluster. When inside the container all gpus are visible so basically it ignores the CUDA_VISIBLE_DEVICES set env by slurm. Outside the ...
2
votes
1
answer
465
views
SLURM with "partial" head node
I am trying to install SLURM with NFS on a small ubuntu 18.04 HPC cluster, in a typical fashion, e.g. configure controller (slurmctld) and clients (slurmd) and shared directory, etc. What I am curious ...
- 123
0
votes
1
answer
304
views
Update SLURM node state prior/after playbook execution
I would like to automatically set the state of a node in a SLURM cluster before/after running my Ansible playbook (from idle to drained and after applying the playbook back to idle). The scontrol ...
- 121
0
votes
1
answer
108
views
Slurm cluster in Google Cloud - how do I attach a Filestore instance
If one clones the slurm-gcp project and deploys the stock cluster defined in there, things work well.
What I would like to do, is to use a GCP Filestore instance to provide (more) persistent storage ...
1
vote
1
answer
2k
views
Wrong LDAP user ID is mapped into Slurm account management service
I configured a Slurm head node as follows:
sssd to contact openLDAP
slurmctld/slurmdbd/slurmd/munged to act as the Slurm controller and compute node
...where ray.williams is an LDAP user. Its UID ...
- 209
1
vote
0
answers
59
views
View/request instruction sets available on SGE host
How can I view or request hosts that can handle a particular instruction set in SGE?
With Slurm, to view available instruction sets on each host I can use
sinfo --Node -o '%n %f',
and to submit a ...
- 111
-1
votes
1
answer
329
views
Using HPC managers like Slurm on multiple servers in LAN [closed]
I have access to a group of servers connected with a 1Gb LAN, and each of them has 40+ cores and Ubuntu OS. They all have a common NAS. I installed SLURM on a few of them and configured it so that ...
- 123
1
vote
0
answers
697
views
Slurmd remains inactive/failed on start
I currently have a cluster of 10 worker nodes managed by Slurm with 1 master node. I have previously successfully set up the cluster, after some teething problems, but managed to get it working. I put ...
2
votes
1
answer
2k
views
Slurm: "Connection refused" for certain sacctmgr commands
I have an existing slurm cluster up and running but as of today without a configuration change I get an error when I run certain sacctmgr commands and slurmdbd crashes:
$ sacctmgr list associations
...
- 507
1
vote
0
answers
326
views
Slurm Error: “If using PrologFlag=Contain for pam_slurm_adopt, either proctrack/cgroup or proctrack/crau_aries is required.”
I'm using the flag x11 (PrologFlags=x11) in my slurm.conf file and jobs with x11 works perfectly, but I am getting this error every time I run a slurm command (e.g. sbatch, srun, sacctmgr):
scontrol: ...
1
vote
1
answer
641
views
Single-node SLURM server: restrict interactive CPU usage
I have SLURM setup on a single node, which is also a 'login node'. I would like to restrict interactive CPU usage, e.g. outside the scheduling system.
I found the following article which suggests to ...
- 384
3
votes
2
answers
1k
views
Query peak GPU memory used by finished job
I have a SLURM job I submit with sbatch, such as
sbatch --gres gpu:Tesla-V100:1 job.sh
job.sh trains a model on a V100 GPU. The code itself does not log GPU memory usage.
Is there a SLURM command ...
- 173
4
votes
1
answer
9k
views
Why does Slurm fail to start with systemd but work when starting manually?
I've just set up slurm where one physical machine will be the only system in the cluster (so far). This is on Ubuntu 18.04.
I have slurmdbd running, but when I attempt to start up slurmd and ...
- 41
1
vote
0
answers
474
views
i try to srun /bin/hostname. slurmctld not respones
I have Master Node (Ubuntu 18.04) and Two Compute Node (Ubuntu 18.04)
There is no problem with the connection
Munge is good, I try to command (sinfo, scontrol show nodes).. that is no problem
I try ...
1
vote
0
answers
6k
views
slurm service running failed again. i don't know why
I have one master node and two slave nodes.
One slave node connects successfully but one node connection failed.
Each node has 18.04 Ubuntu and 17.11 Slurm
If running to systemctl status slurmd....
0
votes
1
answer
397
views
slurm salloc and how it get user login
Some theoretical question. I understand that the better way to know is to look at the code, but maybe I can do some cheat and just ask about it?
I wonder that after salloc user can log in to the node....
- 35
2
votes
0
answers
40
views
Slurm - Does it maintain ccNUMA?
Does a SLURM cluster control, maintain or enforce Cache Coherence across the Nodes? Is it a configuration property, or does something like this not exist? I can't find anything inside the docs.
- 271
3
votes
1
answer
14k
views
Unable to contact slurm controller
I followed the steps to troubleshoot here: https://slurm.schedmd.com/troubleshoot.html.
When running scontrol show slurmd, I get:
Active Steps = NONE
Actual CPUs = 1
Actual ...
- 213
8
votes
1
answer
11k
views
Slurm node daemon error: Can't open PID file
I run systemctl start slurmd.service, and it times out:
Job for slurmd.service failed because a timeout was exceeded.
The relevant lines from running systemctl status slurmd.service:
Mar 23 17:13:...
- 213
0
votes
1
answer
379
views
Trouble Installing slurm on Fedora 29
When I run slurmd, it gives a -bash: slurmd: command not found.
I ran sudo yum install slurm to install slurm. I don't know why it isn't working, or if I installed all the required packages for slurm....
- 213
0
votes
1
answer
1k
views
What does "CPU Minutes" mean exactly?
I'm actually trying to report cluster utilization in Slurm but i don't understand the metric CPU Minutes.
[root@XXXX]# sreport cluster Utilization Start=2018-12-01 End=2018-12-31
---------------------...
- 3
1
vote
1
answer
532
views
Allow other users to cancel jobs
I have a test cluster with Slurm in which I would like that other users where able to cancel other users' jobs.
By default, the users are able to cancel their own jobs.
How can I define several ...
- 111
2
votes
1
answer
3k
views
slurmdbd fails to start (initial installation)
I tried to install slurmdbd for accounting on a Ubuntu 16.04 from the standard repositories (version: 15.08.7-1build1).
Here are the commands:
$ sudo apt-get install mysql-server
$ sudo mysql
> ...
- 507
0
votes
0
answers
1k
views
SLURM, SSH, adn NOHUP Behaviour
I am the administrator of a cluster running on CentOS and using SLURM to send jobs from a login node to compute nodes. Recently, a user complained about some unexpected behaviour with their jobs. If a ...
3
votes
3
answers
1k
views
Randomize Slurm Node Allocation
Has anyone had luck randomizing Slurm node allocations? We have a small cluster of 12 nodes that could be used by anywhere from 1-8 people at a time with jobs of various size/length. When testing our ...
- 31
1
vote
1
answer
1k
views
How do I prevent additional jobs from a given user from starting?
With the Slurm workload manager how can I prevent more jobs from user bob from starting? Existing jobs should continue to run. The user should be able to submit more jobs but they shouldn't be able to ...
- 111
1
vote
1
answer
1k
views
How to upgrade Slurm?
I've been asked to upgrade our Slurm Workload Manager installation. I have a slurm 2.3.4 on a Debian 7.0 wheezy cluster (1 master + 8 nodes). I've not installed it so I'm a bit confused about how to ...
- 223
1
vote
0
answers
201
views
program on cluster exceeds RSS memory limit
I have been trying to run a python script on a computer cluster but keep running into a error saying that RSS memory limit exceeded.
I am using this program to analyse a data set consisting of around ...
- 111
1
vote
0
answers
139
views
Slurm not filtering sacct results by date
We're using Slurm as a resource manager on our Beowulf cluster, so I installed Slurm on my workstation to test out my scripts before I submit them to the cluster.
When I try to list old jobs on my ...
- 1,404
Related Tags
linux × 10
hpc × 9
cluster × 9
ubuntu × 3
ssh × 3
cgroup × 2
fedora × 2
docker × 2
cpu-usage × 2
job-scheduler × 2
nfs × 2
memory-usage × 2
ubuntu-18.04 × 2
x11 × 1
virtual-machines × 1
ubuntu-16.04 × 1
torque × 1
sysadmin × 1
sssd × 1
ssh-tunnel × 1
python × 1
performance × 1
pbs × 1
more related tags
Hot Network Questions
- Can I use p-values without hypothesis?
- Find a simple proof that π is irrational
- Numerically solving radial Schrödinger equation with Yukawa potential
- Indonesia visa misprint by consulate
- RSA perfect square phi
- Correctness of mixed signed/unsigned arithmetic
- How do I reset file ownership at the command line when the files are owned by users from a dead domain?
- What would happen if a small black hole fell into a star?
- This is on the bottom of a bronze vase (Characters identified: 萬歷秊製)
- Did Huawei modem just tried to do a Man-In-The-Middle attack on me?
- Is "stife" a name for smoking cooking oil?
- Diploria labyrinthiformis - can we reproduce Ernst Haeckel's drawing of a brain coral?
- Why does the US still maintain so many military facilities in Western Europe?
- Does China have no homeless people?
- Can I decline a professor's request for video conferencing?
- What happens when an anti-electron collides with a neutrino?
- Sub numerotation for sub-definition
- bullet resistant space armor material?
- Identifying an Actor in Star Wars: A New Hope's Tantive IV Boarding Scene?
- Can I lose the game by flickering Platinum Angel
- How to let a recruiter know you're not seriously considering another job but you are exploring your options
- Negotiated conditions not met after taking back resignation
- How much time should I spend on illustrations?
- Which is More Positively Effective, Math-Wise: Modifier Bonuses, or Expertise?