Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm in charge of a dedicated web server for a client. He's got some pretty good traffic and it's been going down more and more often in the last 2 months. I was just in the process of finalizing a big improvement in terms of performance (set up nginx, varnish, cleaned up the lingering processes, etc) and the server did what it's been doing lately.

Now, as I said I was in the process of making modifications, so stuff is on there half-assed right now. All that happened is my shell stopped reacting to keystrokes. And my client called. Bottom line is, I need to figure out what is happening and I don't have a lot of resources.

On to the fun part:

The server is running CentOS 5.5 with cPanel and WHM.

It's also got: Monit & Nagios who both spat out email alerts; Apache 2 backed up by a nginx 0.8.54 front-end; PHP 5 with APC and memcached;

Client side, we're talking about a few very popular WordPress blogs with a MySQL5 backend.

I think that the problem is MySQL beaause it's the only data-intensive resource I haven't spent 20 straight hours optimizing.

However, I cannot connect through SSH. access Webmin, Nagios, WHM, cPanel, nginx, HTTP or HTTP-SSL - nothing.

Any suggestions?

SSH, nginx, Apache 2, MySQL 5, PHP 5, BIND

share|improve this question
    
Home built hardware or branded server? Does it have remote management card? –  SpacemanSpiff Jan 18 '11 at 19:42
    
walk up to the server and make sure nothing but ssh is started. If you have to, have your datacenter do this. Then once you're logged in remotely, you can start services and see what happens. –  xenoterracide Jan 19 '11 at 2:13

3 Answers 3

From your description it sounds as though the entire operating system is crashing for whatever reason, and it likely has little to do with individual services (mysql, bind, etc). The best place to start is to look at the systems logs when it comes back up (look under the /var/log directory) and try to identify events that are occurring at the time of the crash. Honestly, it could be a multitude of different things, and you'll need that base level of information to begin with. It's also possible that hardware is failing and many of the tools to help diagnose this are specific to the server vendor.

share|improve this answer
    
When I read "it's been going down more and more often in the last 2 months", I was thinking: hardware failure. If there's no obvious software cause or hardware failure: just replace the PSU. –  DutchUncle Jan 19 '11 at 19:38

If the sites come up but you are locked out of the server (i.e. other hosts can still connect) it could be an issue with cPanel's cphulk security daemon - if you have access to a proxy or can otherwise SSH to the box from a different IP address, try running the following as root:

root@server # mysql cphulkd
> SELECT * FROM `brutes` WHERE `IP`='1.2.3.4';

(where 1.2.3.4 is the failing IP address)

If cphulk has identified your IP address as a potential troublemaker, you can delete from cphulkd.brutes accordingly.

If you don't have root access on the box, (and you suspect an issue with cphulk, as the server still appears to be serving requests) get someone who does have root access to disable cphulk by visiting the following WHM URI:

/scripts2/doautofixer?autofix=disable_cphulkd

share|improve this answer

If you have console access, you could try bringing the system up in single user mode. Then disable all the web and database services from starting at boot, and reboot. If the system is then able to boot normally, you can start enabling services in a controlled manner, watching to see which one(s) is/are the problem child/children. If the box doesn't boot normally, then it might be time to start looking at hardware issues.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.