10

I'm encountering a weird problem where a Fedora Linux VPS server reports 100% CPU, and effectively becomes unusable, but I don't know why because the high load prevents me from SSHing into it to see what's wrong.

How do I prioritize or configure SSH so that I'm still able to connect even if some process is consuming all other CPU?

6
  • I don't think this is really possible, why not attack the problem at it's source? Work out what's causing the cpu to max out and fix THAT problem. – Robbie Mckennie May 24 '13 at 14:32
  • 9
    Maybe because he can't SSH in to attack the problem. Chicken meets egg. :) – kenchilada May 24 '13 at 14:35
  • @RobbieMckennie, And how do you propose I do that? – Cerin May 24 '13 at 14:36
  • It sounds like it is a little late to force ssh to have priority. If you have 'physical' access, you might be able to get in. You might be about to run ssh user@host sudo reboot to try to make it restart, with less overhead then getting a full shell. – demure May 24 '13 at 14:49
  • @demure, I don't have physical access. I can reboot it, but that destroys any evidence as to what the problem was. – Cerin May 24 '13 at 15:07
3

If this is for real a CPU / Load related problem you can (re)nice the sshd process on your server to give it a higher priority. To make that persistent you can add the nice inside the sshd init script.

3

Create a script which logs processes:

#!/bin/bash
top -b -c -n 1 -d 1 > /tmp/top-$(date "+%Y-%m-%d_%H-%M-%S")

Setup a cron job which runs the script every minute:

*/1 * * * * /path/to/script.sh

Next time your system goes unresponsive, you'll have logging of what was doing it.

1
  • For additional points, write another cron job which clears out old logging every few days so you don't fill up your filesystem. (hint: find and -mtime) – suprjami May 26 '13 at 12:25
2

I was running the code that was hogging all the resources through Docker, so I just limited the amount of CPU that could be consumed by the container by starting it with

--cpus=7.5

Because I have 8 cores, this means I should always have half a core for processing SSH sessions into the actual server running the container (unless some other process takes up those resources too).

For good measure I also limited the container's memory usage with

--memory=4g

https://docs.docker.com/config/containers/resource_constraints/#cpu

1

If you don't mind my saying, I think you're asking the wrong question. You really want to find out what is using so may resources to the point of your not being able to log in. I have come across this problem numerous times and the solution is to track what is going on the server constantly. I came up with this solution which is as light as possible, reducing its chances of being stopped by a resource hog:

http://linuxtech.ie/wordpress/2012/09/05/finding-a-severe-resource-hog-on-your-server/

I hope that helps

1
  • This is true. Treating the symptom, not the cause. I've NEVER had to make changes to the priority of the SSH daemon on a production system. I think this lies more with the deficiencies of the VPS provider or another resource issue. – ewwhite May 25 '13 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.