Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This website has been running fine without issues, Recently it went down.

After some investigation it looks like the combo of MYSQL and Apache bring the box to its knees. Apache can run find serving static web pages and MYSQL can run fine when the website isn't working.

As soon as the website is enabled with SQL running the CPU on the box remains at 100%.

Picture of the usage: http://i.stack.imgur.com/GG2NC.png

I've checked the sql database for errors, tried tuning nearly every parameter in apache/sql's conf file for performance. The server is a redhat based box running the latest software packages.

Any help/suggestions are welcome.

Doing an strace on a high cpu apache process I see the following:

read(14, "", 8192) = 0 close(14) = 0 socket(PF_FILE, SOCK_STREAM, 0) = 14 fcntl64(14, F_SETFL, O_RDONLY) = 0 fcntl64(14, F_GETFL) = 0x2 (flags O_RDWR) connect(14, {sa_family=AF_FILE, path="/var/lib/mysql/mysql.sock"...}, 110) = 0 setsockopt(14, SOL_SOCKET, SO_RCVTIMEO, "\2003\341\1\0\0\0\0", 8) = 0 setsockopt(14, SOL_SOCKET, SO_SNDTIMEO, "\2003\341\1\0\0\0\0", 8) = 0 setsockopt(14, SOL_IP, IP_TOS, [8], 4) = -1 EOPNOTSUPP (Operation not supported) setsockopt(14, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0

Here is what I see from a mysql process:

futex(0x86fc9a4, FUTEX_WAIT_PRIVATE, 39, NULL) = 0 futex(0x86fc734, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 futex(0x86fc734, FUTEX_WAKE_PRIVATE, 1) = 0 gettimeofday({1301465020, 141613}, NULL) = 0 clock_gettime(CLOCK_REALTIME, {1301465020, 141699633}) = 0 futex(0x8707a64, FUTEX_WAIT_PRIVATE, 1, {4, 999913367}) = 0 futex(0x8707a40, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 futex(0x8707a40, FUTEX_WAKE_PRIVATE, 1) = 0 exit_group(0) = ?

share|improve this question
    
Is this server a VPS or dedicated system? The top stats you posted don't make much sense. The CPU is at 100% but the load near zero (0.66 0.20 is very low). Similarly, the CPU usage of the MySQL/httpd processes are 25% but the TIME+ column shows almost no net usage (unless you restarted both apps just seconds prior to this). –  uesp Mar 31 '11 at 15:30

1 Answer 1

I know you said that you've tweaked Apache but is this a low traffic site ? Have you tried keeping the child instances of Apache right down: MaxSpareServers.

You could check ps -ef to look closer at the %CPU column per process. If it's a simple site on older hardware you could test lighthttpd as a fix.

Sorry these suggestions are a bit finger-in-the-air.

share|improve this answer
    
It's a relatively low traffic site. In my original post there was a break down of the CPU hogs. Typically once apaache is started httpd starts eatting CPU, then MYSQL quickly creeps up until they are both running so hot the system barely is able to do anything else (including respond to SSH, commands, etc). –  Tim Mar 30 '11 at 13:10
    
Updated some strace info to the original post. Hopefully that helps a little.. –  Tim Mar 30 '11 at 13:24
    
I'm not the man for this job, Tim :-( Other than suggesting that you upgrade to the latest versions of Apache and MySQL without getting my hands dirty, by logging in, nothing springs to mind I'm afraid. I'd look at an intermediate fix with another Web Server if all else fails: http://www.wikivs.com/wiki/Lighttpd_vs_nginx –  Jonathan Ross Mar 30 '11 at 15:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.