Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Don't know if anyone may have any brainwaves with this one. We have a website that is quiet heavily used (delivers 4+ million pages a month). Runs on a Dell Poweredge server, and has run fine for the past two years.

The past few weeks, we've had issues with too many connections to the database. This has the problem that it takes the database down until mysqld is restart, at which point the problem clears and all returns to normal. It could be a week or so until another problem.

Today however, we've had an outage, which appears to the same thing, the lack of database access both through code and through PhpMyAdmin, brought me to look at the same thing - restart mysql ... only this time it failed. Two further attempts brought the same response. We've also had three crashed tables in the past two days, which may be adding to the trouble.

Nothing would clear the MySQL problem - the only recourse was a full reboot of the server. Because the SSH window locks up as soon as you think MySQL, its difficult to interrogate whats gone wrong. Does anyone have any ideas or suggestions? The service is running CentOS 5 and MySQL is 5.2 i think, one behind the latest version.

I'm rebooting the server as I type .... the previous reboot took the entire machine off line, as even httpd did not restart.

Any thoughts anyone???

Ta Rich

share|improve this question
    
Do you have a DRAC card installed so you can connect to the servers console even with the machine locked up? –  mrdenny Dec 22 '11 at 19:39
    
@mrdenny. Yes the DRAC card is installed in the machine. –  TIW Dec 22 '11 at 19:50
    
Ah! Secret to good server management - always keep your eye on how full your hard drives are! A backup process had filled the HDD to 100%, hence nothing would restart on the reboot! Durr moment! lol –  TIW Dec 22 '11 at 20:26
2  
Feel free to post the resolution to this as an answer. After the waiting period, you can mark is resolved and the system will know this issue is handled. –  sysadmin1138 Dec 22 '11 at 21:41
    
So, the key problem was that 'too many connections' thing, right? Do your clients connect with 'super' permissions? If not, there must be some connections reserved for the superuser. If yes, how did you restart MySQL, then? Did you just kill the process? –  minaev Dec 23 '11 at 8:14

2 Answers 2

Honestly i would suggest you upgrading the Broadcom NIC firmware, i had some trouble myself with that on very busy machines with old broadcom firmwares (netextreme2 - bcm5709C). Seems that the internal counters were somehow triggering a bug when you hit an N connections per time unit (until the counters reset) within the driver.

share|improve this answer

Make sure that you have sufficient free disk space to allow your services to run.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.