Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [hpc]

High Performance Computing encompasses using "supercomputers" with high numbers of CPUs, large parallel storage systems and advanced networks to perform time-consuming calculations. Parallel algorithms and parallelization of storage are essential to this field, as well as issues with complex, fast networking fabrics such as Infiniband.

Filter by
Sorted by
Tagged with
0 votes
1 answer
31 views

Single SSH login for multilpe machines?

I have a number of physical (desktop) machines running at the office as part of a new network to handle processing & serving Open Source data; some of these machines also house VMs. At the moment, ...
user avatar
0 votes
0 answers
8 views

accounting GPU compute time on HPC clusters

How do you account for GPU compute time on your HPC clusters ? I have a growing, and quite heterogeneous (SXM4 A100s, PCIe A100s, NVlinked V100s, PCIe V100s, T4s, AMD cards arriving soon etc...), GPU ...
user avatar
0 votes
0 answers
9 views

Excluding specific processes from ulimit

I manage a shared CentOS 8.1 server. I don't want users running computationally intensive processes on this server because it is the login node to a computer cluster. So we use ulimit to kill ...
user avatar
2 votes
0 answers
63 views

Infiniband fabric with 3 nodes - newbie

I am trying to connect 3 HP z840 workstations using: Mellanox ConnectX-3 VPI 40 / 56GbE Dual-Port QSFP Adapter MCX354A-FCBT Mellanox SX6005 12-port Non-blocking Unmanaged 56Gb/s Description of ...
user avatar
  • 121
2 votes
1 answer
190 views

How can I set up interactive-job-only or batch-job-only partition on a SLURM cluster?

I'm managing a PBS/torque HPC cluster, and now I'm setting up another cluster with SLURM. On the PBS cluster, I can set a queue to accept only interactive jobs by qmgr -c "set queue interactive_q ...
user avatar
  • 143
0 votes
0 answers
102 views

Setting up slurm on a cluster

My IT admin has setup a cluster with 3 nodes, which is administered via Windows server. VMs are hosted via Hyper-V, including an Ubuntu VM to which a substantial portion of the cluster's resources ...
user avatar
1 vote
1 answer
82 views

Why can't the GPUs communicate in a multi-GPU server?

This is a Dell PowerEdge r750xa server with 4 Nvidia A40 GPUs, intended for AI applications. While the GPUs work well individually, multi-GPU training jobs or indeed any multi-GPU computational ...
user avatar
  • 321
1 vote
1 answer
53 views

Infiniband OpenSM N-to-N port routing configuration

I have 10 servers with two CPUs each and one Mellanox 100G Infiniband NIC per CPU. Each NIC is connected to a single Mellanox 36 port 100G IB switch. My RDMA application runs as one process per NUMA ...
user avatar
0 votes
0 answers
20 views

Is there a way of killing user processes on machines they have no jobs on SLURM?

I'm starting to configure a small cluster using SLURM. One thing I've noticed is that users can still have running jobs on any machine if he follows these steps: Get a machine with salloc SSH into ...
user avatar
2 votes
0 answers
26 views

Lustre glitch: latency of minutes

Using a HPC lustre filesystem, we occasionally experience glitchiness where even simply opening a terminal and typing "ls" can take minutes to return. That is, any process that involves the ...
user avatar
  • 121
1 vote
0 answers
27 views

Why a non-root installation can work across the whole cluster?

I recently installed anaconda (which includes a new python3) locally in my account folder on a cluster with a dozen of nodes (each node with several cores). I use it to install some package P that is ...
user avatar
0 votes
0 answers
21 views

User Statistics from PBS

Is there a way to check on the usage-statistics of the Portable Batch System (PBS), i.e. check how many real-time or CPU hours were consumed by one particular user or the users themselves? qstat --...
user avatar
0 votes
0 answers
113 views

BeegFS cluster - One node dead, need to move on without it

As the title suggests, a cluster consisting of four machines has one dead machine in the rack. The cluster is set up with a buddy mirror system for redundancy, so the data should still be intact. The ...
user avatar
  • 485
0 votes
1 answer
38 views

Changing the subnet on which my beegfs cluster operates

I've added some fiber channels between the machines that constitutes my BeegFS cluster in an effort to increase throughput. However, I have to leave the old coppy ethernet in place with its addressing ...
user avatar
  • 485
0 votes
1 answer
145 views

Latency of memory accesses via interconnectors

I'm trying to compare latencies of different node interconnects for a cluster. The goal is to minimize the memory access latency. I have obtained some benchmarks regarding one of the hardware ...
user avatar
  • 23
2 votes
0 answers
175 views

Current single system image solutions

I'm designing a cluster for a small research institute. Since our computations require a large amount of memory, I'm looking for a solution that will allow our applications access to the whole memory ...
user avatar
  • 23
1 vote
1 answer
199 views

ssh port forwarding (tunneling in HPC)

I have an application server that runs on a compute node. The server opens a port (9000) and I then run a command for tunneling between my local machine and the server: ssh -N -f -L 9000:compute-node:...
user avatar
  • 111
1 vote
1 answer
139 views

HPC cluster master node as virtual machine

For a given small HPC cluster (~16 nodes) a master node is used as a front-end for users to login and interact with SLURM, and not as a computing node. The master node is currently a bare-metal server....
user avatar
0 votes
1 answer
309 views

OpenLDAP implementation allows only root user to set passwords of accounts

I'm working application that requires the use of AWS ParallelCluster assets for some high performance processing. After the initial setup, we need to be able to add/remove user accounts and I am ...
user avatar
1 vote
1 answer
705 views

Considerations using consumer class (high-end) GPU in server?

Motivation: First of all, even if I have some knowledge of computer science, software development and server Linux administration, I never looked into a server hardware and I am a total "newbie&...
user avatar
1 vote
2 answers
272 views

Infiniband drivers : OFED or distro included?

I'm setting up a Linux cluster with infiniband network, and I'm quite a newby in infiniband wolrd, any advice is more than welcome ! We are currently using Mellanox OFED drivers, but our infiniband ...
user avatar
2 votes
1 answer
89 views

SLURM with "partial" head node

I am trying to install SLURM with NFS on a small ubuntu 18.04 HPC cluster, in a typical fashion, e.g. configure controller (slurmctld) and clients (slurmd) and shared directory, etc. What I am curious ...
user avatar
  • 123
3 votes
0 answers
39 views

Bad multicore performance on DL360 Gen 10 with 2xXeon 6154

I have some issue on the multi-core performance of some server. The server are HPE DL360 Gen10, mounting 2x Xeon Gold 6154 (18 cores). When i refer to the performances they are slower then some older ...
user avatar
  • 31
3 votes
2 answers
383 views

What is the overhead of ZFS RAIDz1/2 in HPC SSD Environment?

Example hardware / host: Modern 64 Core CPU, 128GB Memory 8 x Micron Pro 15.36TB u.2 SSDs SSDs connected by dedicated Oculink per device (no backplane or PCIe sharing) Ubuntu 20.04 Use case: A ...
user avatar
  • 376
1 vote
1 answer
934 views

Wrong LDAP user ID is mapped into Slurm account management service

I configured a Slurm head node as follows: sssd to contact openLDAP slurmctld/slurmdbd/slurmd/munged to act as the Slurm controller and compute node ...where ray.williams is an LDAP user. Its UID ...
user avatar
2 votes
1 answer
276 views

HTCondor high availability

I am currently trying to make the job queue and submission mechanism of a local, isolated HTCondor cluster highly available. The cluster consists of 2 master servers (previously 1) and several compute ...
user avatar
-1 votes
1 answer
162 views

Using HPC managers like Slurm on multiple servers in LAN [closed]

I have access to a group of servers connected with a 1Gb LAN, and each of them has 40+ cores and Ubuntu OS. They all have a common NAS. I installed SLURM on a few of them and configured it so that ...
user avatar
0 votes
0 answers
42 views

What can be a reason for different clock speeds between sockets on 2 x Xeon Scalable 6148?

I have server with dual Xeon Scalable 6148 CPUs running HPC application. Base clock: 2.4GHz All core Turbo: 3.1 GHz Some processing threads are not scaling well and are sensitive to cpu clock. I ...
user avatar
  • 1
1 vote
1 answer
369 views

Single-node SLURM server: restrict interactive CPU usage

I have SLURM setup on a single node, which is also a 'login node'. I would like to restrict interactive CPU usage, e.g. outside the scheduling system. I found the following article which suggests to ...
user avatar
  • 375
0 votes
1 answer
391 views

Ubuntu server vs Ubuntu desktop for Beowulf cluster

I want to create a beowulf cluster using Ubuntu 18. Looking at some guides they all seem to use ubuntu server for this an my question is why? Is it not possible to use ubuntu desktop for the client ...
user avatar
-1 votes
1 answer
75 views

How is it that Summit at Oak Ridge National Lab has 2,414,592 cores? [closed]

Top500 says that Summit has 2,414,592 cores: https://www.top500.org/system/179397. But they have 4608 nodes, 9216 chips (each node has 2 chips), and 22 cores per chip. This is 202,752 cores. Where ...
user avatar
0 votes
1 answer
553 views

What does "CPU Minutes" mean exactly?

I'm actually trying to report cluster utilization in Slurm but i don't understand the metric CPU Minutes. [root@XXXX]# sreport cluster Utilization Start=2018-12-01 End=2018-12-31 ---------------------...
user avatar
  • 3
0 votes
1 answer
79 views

Exascale Power Consumption

I have read a lot of articles about exascale and found out that it may consumes approximately 20MW power envelope. Is it a daily basis or a yearly basis or every second? Please enlighten me. Here are ...
user avatar
0 votes
1 answer
816 views

Configure Singularity to do headless rendering / use OpenGL / glxgears / glxinfo

I want to do headless rendering on a server where I do not have root permissions. Therefore, I created a Singularity container like this: Bootstrap: docker From: nvidia/cuda:9.0-runtime-ubuntu16.04 %...
user avatar
  • 101
2 votes
1 answer
2k views

Intel Xeon 6134 + One DIMM per channel or two DIMMs per channel for maximum memory bandwidth?

I'm unable to find this critical piece of information in spec sheets. Appreciate any insight. We're purchasing servers for HPC work with intel Xeon Gold 6134 (Skylake) cpus I want maximum memory ...
user avatar
3 votes
1 answer
109 views

Micosoft HPC Pack 2012 R2 does not run with Network Direct after joining new domain

I am working with a 13 computer cluster, running on Windows Server 2012 R2, using MS HPC Pack 2012 R2. The headnode is working properly. The servers are connected to the corporate network via IPv4 on ...
user avatar
2 votes
1 answer
37 views

Are processors more efficient at lower temperatures

I couldn't find a stackexchange site that's more suited for this question, I apologize. This doesn't necessarily have a lot to do with servers and stability.... I do not have problems with stability ...
user avatar
  • 343
0 votes
0 answers
194 views

How to handle mpi head node failure?

There is app which starting with mpirun. If compute node fail then all processes crush, but if only head node fail(for example reboot) then processes will stuck on compute nodes. How to get rid of ...
user avatar
  • 163
3 votes
3 answers
783 views

Randomize Slurm Node Allocation

Has anyone had luck randomizing Slurm node allocations? We have a small cluster of 12 nodes that could be used by anywhere from 1-8 people at a time with jobs of various size/length. When testing our ...
user avatar
  • 31
3 votes
1 answer
437 views

MAAS for diskless computational hpc cluster

I'm considering to use MAAS to deploy OS for a computational cluster. All nodes are diskless. Only head node and (probably) MAAS rack controller will have hard drives. It seems MAAS have to finish a ...
user avatar
  • 135
0 votes
0 answers
22 views

Ideal configuration for a head node? [duplicate]

Which hardware should I concentrate on, when assembling a head node for an HPC cluster? The main task for the head node is to relay instructions to the compute nodes which will be running artificial ...
user avatar
1 vote
1 answer
490 views

ifconfig apparently showing wrong RX/TX values for InfiniBand HCA

Recently, I executed a watch -n 1 ipconfig on one of our Linux cluster computing nodes while it was running a 48-process MPI run, disributed over several nodes. Oddly, while Ethernet packets seem to ...
user avatar
  • 133
1 vote
2 answers
290 views

Containers for HPC batch processing

We are facing the problem that a lot of people want to run different scientific software on our high performance computing cluster. Every user requires a different set of libraries and library ...
user avatar
  • 13
-3 votes
1 answer
236 views

Solutions for monetizing excess CPU cycles [closed]

My company has a big (relatively) computer farm, say, 100 physical servers (dual CPU hexacore e5 xeons with 160 Gb RAM) leased from some hardware provider (say Leaseweb or OVM) on monthly basis, means,...
user avatar
  • 185
2 votes
1 answer
103 views

How many infiniband adapters should be used in multi socket servers?

Should dual socket motherboards have an infinity band adapter for each CPU? That is, should there be two infiniband band adapters, one in each CPU's PCIe slot. Would this eliminate the signal going ...
user avatar
0 votes
1 answer
141 views

SSH vs qlogin to use all processors of a computing node

I have a SGE cluster consisting of four computing nodes, each with 20 processors. I do not mind to give one particular user the full capabilities of one specific node, i.e. I do not mind he/she uses ...
user avatar
-2 votes
2 answers
175 views

memory usage per user in SGE cluster

I would like to automate the estimation of monthly memory usage of all jobs performed by a given user in my cluster (SGE, ubuntu). I have seen there are many tools to compute the current memory usage ...
user avatar
1 vote
1 answer
1k views

Slurm: Have two separate queues for GPU and CPU-only jobs

At the moment, we have set up Slurm to manage a small cluster of six nodes with four GPUs each. That has been working great so far, but now we want to utilize the Intel Core i7-5820K CPUs for jobs ...
user avatar
  • 121
-5 votes
1 answer
52 views

why all hard drives in a HPC cluster must be of the same size and part number? [closed]

One of the hard drives of my HPCC is broken and I have to buy another one. I've heard it must be of the exactly same size and part number. Could anyone explain me why? Thanks
user avatar
1 vote
0 answers
108 views

Torque queue issue

I am having troubles with Torque + Maui. The problem is the following: I have 2 queues, each queue has 10 associated nodes. If i submit 10k jobs to the first queue and i submit 1 job to the second ...
user avatar
  • 11