11

I'm encountering a weird problem where a Fedora Linux VPS server reports 100% CPU, and effectively becomes unusable, but I don't know why because the high load prevents me from SSHing into it to see what's wrong.

How do I prioritize or configure SSH so that I'm still able to connect even if some process is consuming all other CPU?

6
  • I don't think this is really possible, why not attack the problem at it's source? Work out what's causing the cpu to max out and fix THAT problem.
    – Rob M
    May 24, 2013 at 14:32
  • 10
    Maybe because he can't SSH in to attack the problem. Chicken meets egg. :)
    – kenchilada
    May 24, 2013 at 14:35
  • @RobbieMckennie, And how do you propose I do that?
    – Cerin
    May 24, 2013 at 14:36
  • It sounds like it is a little late to force ssh to have priority. If you have 'physical' access, you might be able to get in. You might be about to run ssh user@host sudo reboot to try to make it restart, with less overhead then getting a full shell.
    – demure
    May 24, 2013 at 14:49
  • @demure, I don't have physical access. I can reboot it, but that destroys any evidence as to what the problem was.
    – Cerin
    May 24, 2013 at 15:07

4 Answers 4

3

If this is for real a CPU / Load related problem you can (re)nice the sshd process on your server to give it a higher priority. To make that persistent you can add the nice inside the sshd init script.

3

Create a script which logs processes:

#!/bin/bash
top -b -c -n 1 -d 1 > /tmp/top-$(date "+%Y-%m-%d_%H-%M-%S")

Setup a cron job which runs the script every minute:

*/1 * * * * /path/to/script.sh

Next time your system goes unresponsive, you'll have logging of what was doing it.

1
  • For additional points, write another cron job which clears out old logging every few days so you don't fill up your filesystem. (hint: find and -mtime)
    – suprjami
    May 26, 2013 at 12:25
2

I was running the code that was hogging all the resources through Docker, so I just limited the amount of CPU that could be consumed by the container by starting it with

--cpus=7.5

Because I have 8 cores, this means I should always have half a core for processing SSH sessions into the actual server running the container (unless some other process takes up those resources too).

For good measure I also limited the container's memory usage with

--memory=4g

https://docs.docker.com/config/containers/resource_constraints/#cpu

1
  • What ensures the cores you do use aren't where sshd runs? Is sshd smart enough to use any available core?
    – Steven
    Jan 17, 2023 at 9:58
0

If you don't mind my saying, I think you're asking the wrong question. You really want to find out what is using so may resources to the point of your not being able to log in. I have come across this problem numerous times and the solution is to track what is going on the server constantly. I came up with this solution which is as light as possible, reducing its chances of being stopped by a resource hog:

http://linuxtech.ie/wordpress/2012/09/05/finding-a-severe-resource-hog-on-your-server/

I hope that helps

1
  • This is true. Treating the symptom, not the cause. I've NEVER had to make changes to the priority of the SSH daemon on a production system. I think this lies more with the deficiencies of the VPS provider or another resource issue.
    – ewwhite
    May 25, 2013 at 12:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .