Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We've had it a few times now. Suddenly our production server won't respond because a process is in an infinite loop, or the MySQL server stops serving new requests because one query is blocking everything...

We SSH to the server and use ps aux or top to find the culprit, or mytop or SHOW FULL PROCESSLIST in MySQL to find the offending process ID and kill it. Then offcourse we try to recreate the situation on the testserver and fix the bug.

But sometimes the server is so well hung your ps aux / top / mytop / SHOW FULL PROCESSLIST won't go through - even the admins are blocked.

What is the best way to ensure an admin can always access the server and kill offending processes or queries (both on Linux and MySQL)?

  • Can we allocate priorities to different users?
  • Reserve a part of the resources for root?

I've checked nice(1), but constantly having an open connection with nice -20 seems a bit excessive and difficult to work with (let alone dangerous as root).

share|improve this question

migrated from stackoverflow.com Jun 29 '10 at 13:51

This question came from our site for professional and enthusiast programmers.

5 Answers 5

up vote 2 down vote accepted

The pam_limits.so module it's a nifty tool to limit memory, open files, ... and to set nice priority for users and groups.

rpm -ql pam | grep limits
man limits.conf
less /etc/security/limits.conf
share|improve this answer

http://en.wikipedia.org/wiki/Magic_SysRq_key

share|improve this answer
    
Interesting, but getting physical access is not always possible (and getting into the serverroom is quite a process...) - does anyone have any experience with the remote sysrqd mentioned on the wiki page? –  Konerak Jun 29 '10 at 17:24
1  
You can install a KVM over IP and then use Magic SysRq Keys. –  Maxfer Jul 8 '10 at 15:11

We use Dell servers that have a remote access network card (DRAC) installed that allows us to access the server out of band via ssh or a web browser. We can get to a console screen, or power cycle the server. Most major server vendors support some similar device.

This doesn't help you if you want to log into a server that has 0 resources available to allow a login. Short of reserving resources for a log in, this is the next best thing to physical access to the server.

It sounds like you have issues surrounding problem applications. Why do you have apps that are going into infinite loops and MySQL queries that are exhausting your server resources?

share|improve this answer

I guess my answer wasn't good enough and someone felt the need to delete it.

monit or god, preempt things taking over by setting limits. If you're running virtual iron rather than bare metal, assign all but one core to your process and save one core for console access. KVM over IP can sometimes allow you to enter a key combination on the console. If network activity is doing it, shut off eth0 until things calm down, connect on eth1.

share|improve this answer
    
@karmawhore: answers won't be deleted because someone thinks they are wrong - they'll get downvoted. You can delete them yourself. Thank you for your suggestions :) –  Konerak Jun 29 '10 at 17:23
    
I answered the question earlier in a bit more detail, I didn't delete the answer. I came back to see someone else's potential solutions and noticed my original answer was gone. While you may say that answers aren't deleted, I have had several answers deleted where a higher karma user disagrees with my answer and has gotten upvoted on his incorrect answer, so, thanks for playing. –  karmawhore Jun 29 '10 at 18:24
    
There are less than 10 users on the system who could have deleted your answer, only three of which I know of who actively moderate, and I can't imagine any of them deleting your answer without cause. But really, you should go to meta.stackoverflow.com - If your answers are getting deleted, there's a reason (and it may be a bug in the code, not malicious intent). It's important to find out why. –  Matt Simmons Jun 29 '10 at 19:44
2  
Karmawhore, somehow you appear to have created about 7 different identically named accounts, perhaps this unnecessary duplication has caused you problems posting but nothing was deleted as no post ever truly is, they're just hidden from most users, and none were deleted for this question. –  Chopper3 Jul 7 '10 at 19:55
3  
@karmawhore : Don't pretend to be surprised. You're trolling. You've created a many different accounts all named 'karmawhore', and you use your other accounts to insult people, provide inappropriate answers, and have probably been been flagged over and over again by the community. And then with this account you provide great answers. So don't be surprised when others do something about your behavior. –  Stefan Lasiewski Jul 7 '10 at 21:33

Maybe SLURM could be the answer. It's a QoS Resource Manager for linux based cluster systems.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.