I am managing a small computer cluster with slurm on CentOS 7. I want to discourage users to run programs on login node. This can be achieved by adding user hard cpu 1 to file /etc/security/limits.conf. However, I do not want file transferring from another cluster through sshd to be disrupted. Is there a way to reconcile these seemingly conflicting goals?
Add a comment
|
1 Answer
CPU limits seem like the obvious answer, but an effective answer is to use memory limits instead. Tools like ssh tend to have a constant memory profile, so a suitable ulimit for memory can be found that will not affect them, where most real computing applications have large memory requirements that will quickly go over the limit.
Also, the operating system tends to be pretty good at balancing cpu use between processes under heavy cpu use. A per user total processes ulimit may also be necessary.
If you are dead set on limiting cpu use, instead of a hard cpu limit, it is possible to renice heavy cpu processes, or use cgroups to throttle them.
If you want to be draconian, use a script in cron that uses ps -o and filter by pcpu, total cpu use, and whitelist some commands like ssh, screen, tmux, and system users, and kill remaining suspect processes automatically. (I recommend printing candidates and manually reviewing them for a few months before flipping the kill switch.)
However, the key is to educate users to use slurm. Things like the above may be a form of hard persuasion to accomplish that.
