Locked out of server by not-sure-what, remote reboots don't help. What to do?

Question

I'm in charge of a dedicated web server for a client. He's got some pretty good traffic and it's been going down more and more often in the last 2 months. I was just in the process of finalizing a big improvement in terms of performance (set up nginx, varnish, cleaned up the lingering processes, etc) and the server did what it's been doing lately.

Now, as I said I was in the process of making modifications, so stuff is on there half-assed right now. All that happened is my shell stopped reacting to keystrokes. And my client called. Bottom line is, I need to figure out what is happening and I don't have a lot of resources.

On to the fun part:

The server is running CentOS 5.5 with cPanel and WHM.

It's also got: Monit & Nagios who both spat out email alerts; Apache 2 backed up by a nginx 0.8.54 front-end; PHP 5 with APC and memcached;

Client side, we're talking about a few very popular WordPress blogs with a MySQL5 backend.

I think that the problem is MySQL beaause it's the only data-intensive resource I haven't spent 20 straight hours optimizing.

However, I cannot connect through SSH. access Webmin, Nagios, WHM, cPanel, nginx, HTTP or HTTP-SSL - nothing.

Any suggestions?

SSH, nginx, Apache 2, MySQL 5, PHP 5, BIND

Answer 1

From your description it sounds as though the entire operating system is crashing for whatever reason, and it likely has little to do with individual services (mysql, bind, etc). The best place to start is to look at the systems logs when it comes back up (look under the /var/log directory) and try to identify events that are occurring at the time of the crash. Honestly, it could be a multitude of different things, and you'll need that base level of information to begin with. It's also possible that hardware is failing and many of the tools to help diagnose this are specific to the server vendor.

Answer 2

If the sites come up but you are locked out of the server (i.e. other hosts can still connect) it could be an issue with cPanel's cphulk security daemon - if you have access to a proxy or can otherwise SSH to the box from a different IP address, try running the following as root:

root@server # mysql cphulkd
> SELECT * FROM `brutes` WHERE `IP`='1.2.3.4';

(where 1.2.3.4 is the failing IP address)

If cphulk has identified your IP address as a potential troublemaker, you can delete from cphulkd.brutes accordingly.

If you don't have root access on the box, (and you suspect an issue with cphulk, as the server still appears to be serving requests) get someone who does have root access to disable cphulk by visiting the following WHM URI:

/scripts2/doautofixer?autofix=disable_cphulkd

Answer 3

If you have console access, you could try bringing the system up in single user mode. Then disable all the web and database services from starting at boot, and reboot. If the system is then able to boot normally, you can start enabling services in a controlled manner, watching to see which one(s) is/are the problem child/children. If the box doesn't boot normally, then it might be time to start looking at hardware issues.