I'm in charge of a dedicated web server for a client. He's got some pretty good traffic and it's been going down more and more often in the last 2 months. I was just in the process of finalizing a big improvement in terms of performance (set up nginx, varnish, cleaned up the lingering processes, etc) and the server did what it's been doing lately.
Now, as I said I was in the process of making modifications, so stuff is on there half-assed right now. All that happened is my shell stopped reacting to keystrokes. And my client called. Bottom line is, I need to figure out what is happening and I don't have a lot of resources.
On to the fun part:
The server is running CentOS 5.5 with cPanel and WHM.
It's also got: Monit & Nagios who both spat out email alerts; Apache 2 backed up by a nginx 0.8.54 front-end; PHP 5 with APC and memcached;
Client side, we're talking about a few very popular WordPress blogs with a MySQL5 backend.
I think that the problem is MySQL beaause it's the only data-intensive resource I haven't spent 20 straight hours optimizing.
However, I cannot connect through SSH. access Webmin, Nagios, WHM, cPanel, nginx, HTTP or HTTP-SSL - nothing.
SSH, nginx, Apache 2, MySQL 5, PHP 5, BIND