We've had it a few times now. Suddenly our production server won't respond because a process is in an infinite loop, or the MySQL server stops serving new requests because one query is blocking everything...
We SSH to the server and use
ps aux or
top to find the culprit, or
SHOW FULL PROCESSLIST in MySQL to find the offending process ID and
kill it. Then offcourse we try to recreate the situation on the testserver and fix the bug.
But sometimes the server is so well hung your
ps aux /
SHOW FULL PROCESSLIST won't go through - even the admins are blocked.
What is the best way to ensure an admin can always access the server and kill offending processes or queries (both on Linux and MySQL)?
- Can we allocate priorities to different users?
- Reserve a part of the resources for root?
I've checked nice(1), but constantly having an open connection with nice -20 seems a bit excessive and difficult to work with (let alone dangerous as root).