3
votes
Accepted
How can I set up interactive-job-only or batch-job-only partition on a SLURM cluster?
As of Slurm 20, there isn't any direct way to separate interactive jobs from batch jobs in a partition.
However, it is possible to use a job submit plugin which can distinguish the two in several ways;...
- 594
3
votes
Accepted
Latency of memory accesses via interconnectors
Your single node numbers of 90 ns local vs 370 ns other socket seems reasonable. However, I think the 600 ns of Infiniband is supposed to be end to end, through a switch to a different frame.
600 ns ...
- 32.1k
3
votes
Accepted
What is the overhead of ZFS RAIDz1/2 in HPC SSD Environment?
Use ZFS. Use LZ4 compression. Tune your ZFS appropriately, as defaults won't be ideal for that many NVMe drives.
Test and benchmark with your actual workloads. We can't tell you how it will perform.
...
- 197k
3
votes
Accepted
What does "CPU Minutes" mean exactly?
The man page for sreport explains the different columns and the default format (-t) is Minutes.
cluster Utilization
This report will display total usage divided by Allocated, Down, Idle,
and ...
- 47.5k
3
votes
Accepted
Containers for HPC batch processing
We use docker containers extensively for ephemeral batch type jobs. In our case, it's intensive 3D imaging processing, but each container processes a "batch" of thousands of related images. We've ...
- 109k
3
votes
Solutions for monetizing excess CPU cycles
This sort of thing is possible but logistically speaking it would likely involve several much more specific questions about setting up the infrastructure, which would be instance specific.
This sort ...
- 2,751
2
votes
memory usage per user in SGE cluster
I have just realised you can check how much memory a user has used with
qacct -j -o <username>
- 199
2
votes
Micosoft HPC Pack 2012 R2 does not run with Network Direct after joining new domain
Flash Mellanox NICs into Etherent to have everything working reliably. InfiBand with Windows is such a PITA!
http://www.mellanox.com/page/firmware_table_ConnectX3IB
- 896
2
votes
Randomize Slurm Node Allocation
Look at the Weight setting in slurm.conf
The priority of the node for scheduling purposes. All things being
equal, jobs will be allocated the nodes with the lowest weight which
satisfies their ...
- 1,093
2
votes
Accepted
ifconfig apparently showing wrong RX/TX values for InfiniBand HCA
You are correct with your assumptions. When running MPI over Infiniband, you normally want to bypass the network stack and use RDMA/Verbs interface to have full performance. All communication sent ...
- 4,235
2
votes
How is it that Summit at Oak Ridge National Lab has 2,414,592 cores?
Validate the number with a rough order of magnitude estimate. 20 two socket servers in a rack, 20 CPU cores per socket. 300 racks is enormous for one computer, that's many MW of power draw. So, in ...
- 32.1k
2
votes
why all hard drives in a HPC cluster must be of the same size and part number?
Depending on the exact hardware (HPC cluster can mean a lot of things), it might be that the hardware used require disks with a vendor specific firmware and can't/won't operate with generic disks.
...
- 98.8k
2
votes
Accepted
Using HPC managers like Slurm on multiple servers in LAN
Yes, that is a compute cluster. I have a broad definition, as multiple compute nodes tasked with the same workload. Large build farm with multiple hosts running compute workers qualifies. So do the ...
- 32.1k
2
votes
What is the overhead of ZFS RAIDz1/2 in HPC SSD Environment?
As BTRFS RAID5 is not considered 100% stable, I do not suggest it even for a backup machine. Moreover, with these large SSDs I would use RAID6 rather than RAID5.
So I suggest using MD RAID6 with XFS ...
- 48.2k
2
votes
Infiniband drivers : OFED or distro included?
The "inbox" drivers have gone through the Linux Q&A process and through the distro Q&A. The MOFED drivers have not.
There are severe bugs in MOFED that prevent our code from running ...
2
votes
Accepted
Infiniband drivers : OFED or distro included?
By not using the vendor OFED distribution, in this case Mellanox OFED you should expect not only a performance penalty but lack of features and a lot of stability issues.
Infiniband is not rock solid ...
- 5,530
1
vote
Why can't the GPUs communicate in a multi-GPU server?
While there is no explicit "IOMMU off" setting in this BIOS flavour, the problem is still with the BIOS configuration.
In the BIOS, go to "Integrated Devices" and change the "...
- 341
1
vote
Infiniband OpenSM N-to-N port routing configuration
https://community.mellanox.com/s/question/0D51T00006RVtlU/rdmacm-connection-setup-issues
For the record, the issue has been largely resolved by ensuring the ibacm service (Infiniband Assistant ...
- 121
1
vote
ssh port forwarding (tunneling in HPC)
The statement is incorrect. That's just a local port redirect through your SSH connection that is authenticated with your SSH keypair.
There is nothing insecure in doing that, it has the same degree ...
- 5,530
1
vote
Accepted
Considerations using consumer class (high-end) GPU in server?
We use lots of GPUs in our servers - but there's a single rule to follow;
Only use parts explicitly supported for your exact server model by the manufacturer.
Do not break this rule.
- 101k
1
vote
Accepted
SLURM with "partial" head node
You're trying to consume the headnode as a compute node. This is perfectly normal on small clusters and even on workstations that have SLURM as a queue system to consume it as an easier way to enqueue ...
- 5,530
1
vote
Accepted
Wrong LDAP user ID is mapped into Slurm account management service
The solution turned out to be in the getent passwd.
I changed the following configuration in sssd:
$ cat /etc/sssd/sssd.conf
[domain/local.lan]
enumerate = true
I removed the users and added them ...
- 209
1
vote
Accepted
HTCondor high availability
With the help of the mailing list and some experimenting I managed to resolve the issue.
Since there are two master servers now, they need not only a shared filesystem for spooling (as described in ...
1
vote
Single-node SLURM server: restrict interactive CPU usage
I have been beating my head against exactly this problem for a while now and I finally came across this solution on stackexchange: https://unix.stackexchange.com/questions/526994/limit-resources-cpu-...
- 111
1
vote
Accepted
Ubuntu server vs Ubuntu desktop for Beowulf cluster
I don't see, why you couldn't use any of the two to build the cluster. Though having no GUI can increase performance, since there is no computation time used for creating the working GUI. Note, that ...
- 157
1
vote
Exascale Power Consumption
20MW is the total power. Power on the computer for an hour and you've used 20MWh. To calculate the cost, multiply with the cost of electricity, e.g. $0.10/kWh.
Example from Wikipedia:
Tianhe-1A ...
- 5,637
1
vote
Intel Xeon 6134 + One DIMM per channel or two DIMMs per channel for maximum memory bandwidth?
Adding answer to my own question.
Source: http://frankdenneman.nl/2015/02/25/memory-deep-dive-ddr4/
It appears that with DDR4 and Haswell cores, Image of performance comparison :
1 DIMM per channel ...
- 167
1
vote
Accepted
Are processors more efficient at lower temperatures
The resistance of semiconductor devices decreases with temperature and thus, load currents increase. Current - or more exactly, moving charge carriers -
wear out a chips traces, which is called ...
1
vote
Randomize Slurm Node Allocation
You can add "LLN=YES" to the partition.
LLN
Schedule resources to jobs on the least loaded nodes (based upon the number of idle CPUs).
- 11
1
vote
How many infiniband adapters should be used in multi socket servers?
That would be the optimal configuration, but it also depends on the applications that make use of the Infiniband connections and you would have to take care that those applications and their processes ...
- 4,235
Only top scored, non community-wiki answers of a minimum length are eligible