Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a 24 core server, which users connect to through SSH.

It is not meant for them to run CPU heavy programs like MATLAB, R or their own scripts that perform simulations or things like that.

Are there ways to detect and kill -9 CPU heavy processes automatically?

share|improve this question
6  
Why kill them, when you could nice them instead? –  MSalters Nov 4 '11 at 15:16
1  
That could be a good option too. How is that done? –  Sandra Nov 4 '11 at 15:24
4  
nice -n 19 matlab or, if it is started: renice -n 19 $(pgrep matlab) –  Mircea Vutcovici Nov 4 '11 at 20:31
1  
A bit more effective would be a small script that selects the top 5 CPU-consuming processes (ps -eo pcpu,pid,user | sort -k 1 -r | head -5) and renice those. You could do so from a cronjob. –  MSalters Nov 7 '11 at 8:45
    
@MSalters Won't that over time make all processes slow, as the the next 5 CPU heavy processes might just be legit processes? –  Sandra Nov 10 '11 at 11:16

2 Answers 2

up vote 3 down vote accepted

You should use cgroups to do this.

See "man cgrules.conf" and "man cgconfig.conf".

Later versions of systemctl on fedora should support sticking users directly into a cgroup so you can do it better that way.

This wont limit CPU in the sense that if there is available CPU resources (which nobody is using) it will use all the CPU however it something else is also demanding CPU it will allocate a share of the CPU based off of the configured "cpu.shares" value.

Also as suggested sticking a ulimit on CPU time will ensure a running process is given a cumalative number of jiffies before being killed for using too much CPU. This might negatively impact long running processes a user is using which over a long period of time have accumulated a certain number of jiffies naturally.

You could also use cgroups to enforce that all a users processes live on one of your cores only, so that you can at least guarantee if one CPU is being overwhelmed it has no negative impact on the rest of the operating systems processes.

CGroups is also a awesome way to limit memory usage. You can combine it with pam_limits to prevent fork bombing.

Edit: I should also point out what I think your asking for is not necessarily relevant. Having 1 process use up 100% of the CPU is not necessarily bad, providing time is given for other processes to run. The completely fair scheduler on linux guarantees this behaviour anyway.

If the CPU is just idling theres nothing wrong with one process using up all the CPU. Your problem only comes where multiple processes are demanding CPU time and one of the processes is hogging the CPU. This is where cgroups should be of benefit as it permits you control how much cpu time you'll allocate different process in the event of CPU contention.

share|improve this answer

You can limit users resources usage with PAM.

I never tried it so I have no clue if it works properly or not...

share|improve this answer
    
Very interesting! What does "Maximum CPU time" mean? I mean, what if the server is 50% utilized and a user starts a script that could max out one cpu core. Would this script then be detected? –  Sandra Nov 4 '11 at 14:30
    
I don't want to claim that I'm an expert with this type of question but...my understanding is that it will prevent users from "hugging" the CPU. So after a total of x seconds of CPU usage the process will take the back seat. Therefore preventing 100% CPU usage from a single user. –  Alex Nov 4 '11 at 14:40
4  
Considering it's referencing ulimit, seems this is implemented as RLIMIT_CPU : "CPU time limit in seconds. When the process reaches the soft limit, it is sent a SIGXCPU signal. The default action for this signal is to terminate the process. However, the signal can be caught, and the handler can return control to the main program. If the process continues to consume CPU time, it will be sent SIGXCPU once per second until the hard limit is reached, at which time it is sent SIGKILL" –  MSalters Nov 4 '11 at 15:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.